Bus dynamic multi-objective optimization scheduling method based on reinforcement learning

By dividing the full-day operation time of the bus into multiple rescheduling stages, and independently using reinforcement learning models for scheduling in each stage, combined with multi-objective deep reinforcement learning, the conflicts between multiple goals in dynamic bus scheduling are solved, and the operational efficiency and service quality of the bus system are improved.

CN120450357APending Publication Date: 2025-08-08武汉禾青优化科技有限公司
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202510599407.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-11
Publication Date
2025-08-08

AI Technical Summary

Technical Problem

The existing dynamic bus scheduling method based on reinforcement learning is difficult to effectively solve the conflict problem between multiple goals.

Method used

The full-day operation time of the bus is divided into multiple rescheduling stages. Each stage is independently scheduled using a reinforcement learning model, and a knowledge transfer strategy is designed to make decisions through multi-objective deep reinforcement learning (M-DRL), and multi-objective decisions are made in combination with punishment costs, vehicle operation costs and passenger travel costs.

Benefits of technology

The bus scheduling plan is achieved to be more adaptable to the actual operation environment, reduce passenger waiting time, improve operational efficiency and service quality, and solve the conflict problems between multiple goals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120450357A_ABST
    Figure CN120450357A_ABST
Patent Text Reader

Abstract

The invention provides a bus dynamic multi-objective optimization scheduling method based on reinforcement learning. The method comprises the following steps: firstly, initializing a motorcade scale and a one-way set, and segmenting an operation time period into a plurality of rescheduling stages; constructing a reinforcement learning model of the current rescheduling stage; sequentially carrying out decision making on each decision point in the current rescheduling stage by utilizing a reinforcement learning model to form an optimal solution of the current rescheduling stage; repeating the decision-making process of the current rescheduling stage for multiple times to form an optimal solution set of the current stage; and sequentially processing each rescheduling stage until all rescheduling stages are processed. Through the scheme provided by the invention, the vehicle scheduling scheme can be more suitable for a real operation environment, conflicts between targets are fully considered, and the operation level and the service quality of buses are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of public transportation scheduling optimization, and in particular relates to a public transportation dynamic multi-objective optimization scheduling method based on reinforcement learning. Background Art

[0002] Public transportation is the lifeline of urban transportation, fulfilling a vital role in citizens' daily commutes. However, in actual operations, public transportation systems often face unpredictable emergencies such as road congestion, vehicle breakdowns, and sudden changes in passenger flow. These factors can affect public transportation punctuality and operational efficiency. Therefore, implementing dynamic public transportation scheduling has become a key means of ensuring the stable operation of public transportation systems. By real-time monitoring of vehicle locations, passenger flow data, and road conditions, dynamic scheduling can intelligently adjust departure intervals and flexibly allocate vehicle resources to effectively respond to various emergencies. This not only reduces passenger wait times and improves the travel experience, but also ensures the efficiency and reliability of public transportation services, allowing urban public transportation systems to always maintain optimal performance in a complex and changing operating environment, providing citizens with better travel services.

[0003] Reinforcement learning algorithms, which have rapidly developed in recent years, have proven to have unique advantages in handling dynamic problems, particularly complex and dynamic system optimization problems like bus scheduling. Rather than relying on fixed rules or expert experience, they autonomously discover optimal strategies through continuous interaction with the environment, adapting to the personalized needs of different time periods and routes.

[0004] However, existing reinforcement learning-based dynamic bus scheduling methods weight all objectives, making it difficult to resolve conflicts between multiple objectives. Therefore, this patent proposes a reinforcement learning-based dynamic multi-objective optimization scheduling method for buses. On the one hand, the whole day is divided into multiple scheduling periods, and each period is independently scheduled using a reinforcement learning model. On the other hand, a knowledge transfer strategy is designed to accelerate the convergence of the reinforcement learning algorithm. At the same time, multi-objective decision-making is made based on three aspects: penalty cost, vehicle operating cost, and passenger travel cost, effectively resolving conflicts between objectives. Summary of the Invention

[0005] The existing bus dynamic scheduling method based on reinforcement learning only weights all objectives, making it difficult to resolve conflicts between multiple objectives. Therefore, this patent proposes a bus dynamic multi-objective optimization scheduling method based on reinforcement learning. The specific steps are as follows: Step 1: Initialize the bus fleet size and initial one-way trip set. For example, if the bus fleet size V = 50, it means that the fleet has 50 buses. The initial one-way trip set T contains n (for example, 5000) one-way trips. The one-way trips in T are sorted from smallest to largest in terms of departure time and numbered from 1 to n. The bus operating time throughout the day is divided into (for example, R = 4) rescheduling stages. Step 2: Build a model for the current rescheduling phase and construct the optimization process of each rescheduling phase as a Markov decision process, where each unexecuted single trip is considered a decision point. Step 3: Use the reinforcement learning model to make decisions for each decision point in the current rescheduling phase in turn to form the optimal solution for the current rescheduling phase; Step 4: Repeat the decision-making process of the current rescheduling stage P times (for example, P = 100) to form the solution set of the current rescheduling stage. Use the fast non-dominated sorting method to screen out a non-dominated solution set of size H (for example, H = 100) from the solution set as the optimal solution set output for the current rescheduling stage r; Step 5: Process each rescheduling stage in turn until all R rescheduling stages are processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, it is possible for a person skilled in the art to derive other drawings based on these drawings without inventive effort: Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention; Figure 2 1 is a schematic diagram of the coordination of multiple scheduling stages of a method according to an embodiment of the present invention; Figure 3 FIG. 1 is a schematic diagram of an optimization process of a method according to an embodiment of the invention in the rescheduling stage r. DETAILED DESCRIPTION

[0007] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0008] like Figure 1 As shown, the present invention provides a schematic flow chart of a method of an embodiment, including: Step 1: Initialize the bus fleet size and initial one-way trip set. For example, if the bus fleet size V = 50, it means that the fleet has 50 buses. The initial one-way trip set T contains n (for example, 5000) one-way trips. The one-way trips in T are sorted from smallest to largest in terms of departure time and numbered from 1 to n. The bus operating time throughout the day is divided into R (for example, R = 4) rescheduling stages. Step 2: Construct the current rescheduling stage model and construct the optimization process of each rescheduling stage as a Markov decision process, where each unexecuted single trip is regarded as a decision point; Specifically: Step 21: In any rescheduling stage r , there are V (e.g. 50) buses, and an initial one-way set T r , its scale is n express, n ≤5000; in this rescheduling phase r Before starting, T r None of the single passes in have started execution, so they are all considered as a decision point, that is, the Markov decision process contains n stages; Step 22: Define the rescheduling phase r The state space of any decision point in is represented as a vector of dimension V+2 s r ,Right now s r = ( v 1 r , ..., v k r , ..., v V r , p trip , c trip );vector s r middle, v k r = j Indicates the k The latest one-way mission performed by the vehicle is one-way j ,and v k r = 0 means the k The vehicle has not yet performed any single trip; p trip represents the last single pass designated for execution, and c trip It represents the one-way trip corresponding to the current decision point; Step 23: Define the action space based on vehicle allocation, that is, decide the current one-way c trip To which vehicle to assign; specifically, the decision point is at state s r The corresponding action space can be defined as the set A ( s r ) = { a 0, a 1, ..., a k ,..., a V}; Among them, action a = a 0 means the current one-way c trip Not executed by any vehicle, that is, the single trip is selected as not to be executed; and the action a = a k It means the current one-way c trip Will be by k The vehicle is executed, at this time, the one-way task is k The last one-way combination executed by the vehicle forms a new one-way connecting arc. The action space defined above can ensure that no matter which rescheduling stage it is in or how many decisions it makes, the action space always has a fixed (V+1)-dimensional structure, which greatly reduces the complexity of policy learning and helps improve the convergence speed of the algorithm and the diversity of the solution set.

[0009] Step 3: Use the reinforcement learning model to make decisions for each decision point in the current rescheduling phase in turn to form the optimal solution for the current rescheduling phase; Specifically: Step 31: Construct a multi-objective depth Q Learning network, referred to as M-DQN; computing action set A ( s r ) = { a 0, a 1, ..., a k , ..., a V The original source of all actions in Q value; Q The value represents the given state s r Next, the agent performs the action a k When , it corresponds to the first p The expected reward value that can be obtained by the optimization goal is recorded as Q p (a k ); larger Q The value indicates that the action will obtain higher rewards under the corresponding objective. The optimization problem contains three objectives: the penalty for passengers who fail to get on the bus due to vehicle capacity, the operating cost of the vehicle, and the travel cost of the passengers. Step 32: Design Q Value Output Adjustment Layer; First, Q Enter the value into the Q Value adjustment layer; then, the action space A ( s r ) = { a 0, a 1, ..., a k , ..., a n Each action in} is checked, which includes two steps. First, calculate the current action one by one a k The departure interval between the previous one-way trip of the corresponding station. If the departure interval does not meet the specified constraints, such as less than 3 minutes, the action is considered invalid; second, each bus's most recently executed one-way trip is connected to the selected current decision point one-way trip; if it is a feasible connection, it constitutes a candidate connection arc; otherwise, the action is considered invalid; the Q value adjustment layer updates each action based on the inspection results. a k of Q The Q value is used as the evaluation value of the action; the Q value of the invalid action will be assigned a large negative value (for example, -500) to indicate that the action is not feasible, thereby preventing it from being adopted in the subsequent selection process; finally, the optimal action is selected and executed based on the evaluation value, thus completing the one-way decision stage. The reinforcement learning network is trained online based on the decision result, and the next new one-way decision stage is entered; Step 33: Action decision, based on the updated Q Value, select a decision action a k ; Step 34: Knowledge recording, recording the decision status of the decision process at the current decision point s r , the decision action selected a k , and the evaluation result of the decision action Q p ( a k ) is recorded as a new decision-making knowledge and stored in the experience replay pool; Step 35: Update state and parameters, record decision actions a k The decision point status after execution is s r new , update the decision point status to s r = s r new Based on the accumulated decision knowledge, the agent parameters θ r Perform online updates; Step 36: Accordingly, each decision point in the one-way set of the current rescheduling phase is processed until all decision points have been decided; a vehicle scheduling plan, i.e., a solution to the problem, is output.

[0010] Step 4: Repeat the decision-making process of the current rescheduling stage P times (for example, P = 100) to form the solution set of the current rescheduling stage. Use the fast non-dominated sorting method to screen out a non-dominated solution set of size H (for example, H = 20) from the solution set as the optimal solution set output for the current rescheduling stage r. Step 5: Process each rescheduling stage in turn until all R rescheduling stages are processed.

[0011] like Figure 2 As shown, a schematic diagram of the coordination of multiple scheduling stages of the method of an embodiment of the present invention is shown. Specifically, each rescheduling stage is decided by an independent reinforcement learning model. Since the environment and decision-making process between multiple scheduling stages have certain similarities, the parameters of the reinforcement learning model of each rescheduling stage can also refer to the parameters of the model of its previous rescheduling stage. Therefore, we use the parameters of the reinforcement learning of the previous rescheduling stage as decision knowledge and set them as the initial parameters of the reinforcement learning model of the current rescheduling stage. The reinforcement learning model of the current rescheduling stage is trained based on the initial parameters, which can greatly reduce the convergence time of the model.

[0012] like Figure 3As shown, a schematic diagram of the optimization process of the method of an embodiment of the present invention in the rescheduling stage r, specifically, the optimization process of the rescheduling stage r includes a multi-objective deep reinforcement learning (M-DRL) agent, the external environment of stage r, the experience revisit pool, and the optimal scheduling solution set of this rescheduling stage; wherein, when the M-DRL agent makes a decision at each decision point, it first calculates the original Q values of all actions in the action set using the multi-objective deep Q network (M-DQN) according to the state of the current rescheduling stage, and then, in order to ensure that the selected action meets the scheduling constraints of the problem, the agent inputs the original Q value into the Q value adjustment layer, gives a corrected final Q value to each action as the evaluation value of the action, and then selects the optimal action according to the evaluation value and executes the action , thus completing the one-way decision stage and entering the next new one-way decision stage; during the entire rescheduling stage, the decision knowledge generated by the M-DRL agent in each one-way decision stage will be encoded as a "state-action-reward" triple data recorded in the experience replay pool, and used as training data to optimize the agent's decision-making ability online, so that it can learn the optimal decision-making strategy; among them, M-DQN is trained through the external environment of the rescheduling stage r, using environmental simulation to calculate the reward of each action, constructing a training data set, and then using the training data set to train M-DQN; after obtaining a new complete scheduling plan, the reward value of each action executed in the scheduling plan will be adjusted according to its non-dominated relationship in the optimal plan set.

[0013] Multi-objective deep reinforcement learning is trained using an online training method. Specifically, at each decision point, when the agent is in state s r Next select action a k After that, the state-action pair is optimized in each p The reward value on will be recorded in the experience replay pool for subsequent training of the agent. These data are in the form of six tuples ( s r , a k , w 1 rew ( a k ), w 2 rew ( a k ), w 3 rew ( a k ), s r new ), where s r newRepresents the new state the environment will transition to after an action is executed. To improve the stability of the training process, M-DQN uses a combination of a master network and a target network. These two networks share the same network structure. During training, the agent uses the Q-values output by the master network for decision-making, while the Q-values output by the target network serve as target values for parameter training. The target network's parameters are not trained directly, but are periodically updated by copying parameters from the master network at fixed intervals. This effectively prevents instability in the target Q-values during training caused by rapid changes in the master network's parameters. Furthermore, during each training step, a batch size of 80% of the data is randomly drawn from the experience replay pool for agent training, rather than using only the most recently acquired state-action pairs. This helps reduce data coupling and stabilizes the training process.

[0014] Although the present invention has been described in detail with reference to the accompanying drawings and in conjunction with preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and such modifications or substitutions shall be within the scope of the present invention or any changes or substitutions that a person of ordinary skill in the art can easily conceive within the technical scope disclosed by the present invention shall be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A dynamic multi-objective optimization scheduling method for public transportation based on reinforcement learning, characterized by the following steps: Step 1: Initialize the bus fleet size and initial one-way trip set. For example, if the bus fleet size V = 50, it means that the fleet has 50 buses. The initial one-way trip set T contains n (for example, 5000) one-way trips. The one-way trips in T are sorted from smallest to largest in terms of departure time and numbered from 1 to n. The bus operating time throughout the day is divided into R (for example, R = 4) rescheduling stages. Step 2: Build a model for the current rescheduling phase and construct the optimization process of each rescheduling phase as a Markov decision process, where each unexecuted single trip is considered a decision point. Step 3: Use the reinforcement learning model to make decisions for each decision point in the current rescheduling phase in turn to form the optimal solution for the current rescheduling phase; Step 4: Repeat the decision-making process of the current rescheduling stage P times (for example, P = 100) to form the solution set of the current rescheduling stage. Use the fast non-dominated sorting method to screen out a non-dominated solution set of size H (for example, H = 20) from the solution set as the optimal solution set output for the current rescheduling stage r. Step 5: Process each rescheduling stage in turn until all R rescheduling stages are processed.

2. The method according to claim 1, characterized in that Construct a model for the current rescheduling phase and construct the optimization process of each rescheduling phase as a Markov decision process, where each unexecuted single trip is considered a decision point. This process includes the following steps: Step 21: At any rescheduling stage r , there are V (e.g. 50) buses, and an initial one-way set T r , its scale is n express, n ≤5000; in this rescheduling phase r Before starting, T r None of the single passes in have started execution, so they are all considered as a decision point, that is, the Markov decision process contains n stages; Step 22: Define the rescheduling phase r The state space of any decision point in is represented as a vector of dimension V+2 s r ,Right now s r = ( v 1 r , ..., v k r , ..., v V r , p trip , c trip );vector s r middle, v k r = j Indicates the k The latest one-way mission performed by the vehicle is one-way j ,and v k r = 0 means the k The vehicle has not yet performed any single trip; p trip represents the last single pass designated for execution, and c trip It represents the one-way trip corresponding to the current decision point; Step 23: Define the action space based on vehicle allocation, that is, decide the current one-way c trip To which vehicle it is assigned; Specifically, the decision point is in the state s r The corresponding action space can be defined as the set A ( s r ) = { a 0, a 1, ..., a k , ..., a V }; Among them, action a = a 0 means the current one-way c trip Not executed by any vehicle, that is, the single trip is selected as not to be executed; and the action a = a k It means the current one-way c trip Will be by k The vehicle is executed, at this time, the one-way task is k The last one-way combination executed by the vehicle forms a new one-way connecting arc. The action space defined above can ensure that no matter which rescheduling stage it is in or how many decisions it makes, the action space always has a fixed (V+1)-dimensional structure, which greatly reduces the complexity of policy learning and helps improve the convergence speed of the algorithm and the diversity of the solution set.

3. The method according to claim 1, characterized in that Using the reinforcement learning model, we make decisions for each decision point in the current rescheduling phase in turn to form the optimal solution for the current rescheduling phase, including the following steps: Step 31: Construct a multi-object depth Q Learning network, referred to as M-DQN; computing action set A ( s r ) = { a 0, a 1,..., a k , ..., a V The original source of all actions in Q value; Q The value represents the given state s r Next, the agent performs the action a k When , it corresponds to the first p The expected reward value that can be obtained by the optimization goal is recorded as Q p ( a k ); larger Q The value indicates that the action will obtain higher rewards under the corresponding objective. The optimization problem contains three objectives: the penalty for passengers who fail to get on the bus due to vehicle capacity, the operating cost of the vehicle, and the travel cost of the passengers. Step 32: Design Q Value Output Adjustment Layer; First, Q Enter the value into the Q Value adjustment layer; then, the action space A ( s r ) = { a 0, a 1, ..., a k , ..., a n Each action in} is checked, which includes two steps. First, calculate the current action one by one a k The departure interval between the previous one-way trip of the corresponding station. If the departure interval does not meet the specified constraint conditions, such as less than 3 minutes, the action is considered invalid; second, each bus's most recently executed one-way trip is attempted to connect with the selected one-way trip of the current decision point; If it is a feasible connection, it constitutes a candidate connection arc; Otherwise, the action is considered invalid; the Q value adjustment layer updates each action based on the inspection results. a k of Q The Q value of an invalid action will be assigned a large negative value (e.g. -500) to indicate that the action is not feasible, thus preventing it from being adopted in the subsequent selection process. Finally, the optimal action is selected based on the evaluation value and executed, thus completing the one-way decision stage. The reinforcement learning network is trained online based on the decision result, and the next new one-way decision stage is entered. Step 33: Action decision, based on the updated Q Value, select a decision action a k ; Step 34: Knowledge recording, recording the decision status of the decision process at the current decision point s r , the decision action selected a k , and the evaluation result of the decision action Q p ( a k ) is recorded as a new decision-making knowledge and stored in the experience replay pool; Step 35: Update state and parameters, record decision actions a k The decision point status after execution is s r new , update the decision point status to s r = s r new Based on the accumulated decision knowledge, the agent parameters θ r Perform online updates; Step 36: Accordingly, each decision point in the one-way set of the current rescheduling phase is processed until all decision points have been decided; a vehicle scheduling plan, i.e., a solution to the problem, is output.

Citation Information

Cited By

  • Dynamic multi-objective optimization based on improved Wasserstein distance and adaptive transfer learning

    CN121542543A