Multi-line vehicle intelligent scheduling method and system, device, and storage medium

Abstract

The present disclosure provides a multi-line vehicle intelligent scheduling method and system, device, storage medium, belonging to the technical field of intelligent vehicle scheduling, which comprises the following steps: constructing a feature extraction network, the feature extraction network being used for extracting a feature vector of first information; constructing a vehicle selection network, the vehicle selection network being used for selecting a vehicle for scheduling; constructing a simulation environment, the simulation environment being used for simulating running information of the vehicle; constructing an agent based on the feature extraction network and the vehicle selection network; generating first information based on the simulation environment, and training the agent based on the first information and a reward function to obtain a reinforcement learning agent, wherein the first information is historical running information of the simulated vehicle, the reward function is a reward value calculation function corresponding to the first information, inputting second information into the reinforcement learning agent to obtain a target scheduling scheme, and the second information is multi-line vehicle information. The multi-line vehicle intelligent scheduling method and system, device, storage medium provided by the present disclosure can improve the response capability to complex vehicle scheduling problems.

Description

Technical Field

[0001] This disclosure belongs to the field of intelligent vehicle dispatching technology, and more specifically, relates to intelligent scheduling methods, systems, equipment, and storage media for vehicles on multiple routes. Background Technology

[0002] The key to improving service quality lies in optimizing vehicle scheduling.

[0003] Existing methods for solving vehicle scheduling problems are mainly categorized into exact algorithms, heuristic methods, and reinforcement learning methods. While exact algorithms and heuristic methods can find optimal or near-optimal solutions for a specific instance, these solutions become invalid if the instance changes, requiring both methods to be re-solved. Reinforcement learning methods, because they ultimately yield a policy, can better handle changes in instances; however, their generalization performance is limited when the instance changes significantly.

[0004] Therefore, designing a vehicle scheduling method capable of handling various complex scenarios is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method, system, device, and storage medium for intelligent scheduling of vehicles on multiple routes, so as to improve the ability to deal with complex vehicle scheduling problems.

[0006] A first aspect of this disclosure provides a method for intelligent scheduling of vehicles on multiple routes, comprising:

[0007] A feature extraction network is constructed to extract feature vectors of first information; a vehicle selection network is constructed to select vehicles for scheduling; a simulation environment is constructed to simulate vehicle operation information; and an intelligent agent is constructed based on the feature extraction network and the vehicle selection network.

[0008] First information is generated based on the simulated environment, and the agent is trained based on the first information and the reward function to obtain a reinforcement learning agent; wherein, the first information is the historical operation information of the simulated vehicle, and the reward function is the reward value calculation function corresponding to the first information;

[0009] The second information, which is multi-route vehicle information, is input into the reinforcement learning agent to obtain the target scheduling scheme.

[0010] A second aspect of this disclosure provides a multi-route vehicle intelligent scheduling system, comprising:

[0011] The intelligent agent construction module is used to construct a feature extraction network for extracting feature vectors of first information; construct a vehicle selection network for selecting vehicles for scheduling; construct a simulation environment for simulating vehicle operation information; and construct an intelligent agent based on the feature extraction network and the vehicle selection network.

[0012] The agent training module is used to generate first information based on the simulated environment, and to train the agent based on the first information and a reward function to obtain a reinforcement learning agent; wherein, the first information is the historical operation information of the simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information;

[0013] The intelligent scheduling module is used to input the second information into the reinforcement learning agent to obtain the target scheduling scheme, wherein the second information is multi-route vehicle information.

[0014] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described intelligent scheduling method for multi-line vehicles.

[0015] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described intelligent scheduling method for multi-line vehicles.

[0016] The beneficial effects of the intelligent scheduling method, system, device, and storage medium for multi-route vehicles provided in this disclosure are as follows: By constructing a simulated environment to simulate vehicle operation, the high cost of testing and training in real-world scenarios is effectively reduced, enabling research and development to be carried out efficiently and allowing for rapid iterative optimization of scheduling strategies in a virtual environment. The decision-making process is simplified, reducing decision complexity and allowing the agent to focus more intently and efficiently on vehicle allocation when handling multi-route vehicle scheduling. A special reinforcement learning network designed to address the challenges of variable action space and the need for additional learning of vehicle correspondence in reinforcement learning enhances the system's adaptability and processing capabilities to complex and changing situations, improving the accuracy and reliability of the scheduling scheme. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a multi-route vehicle intelligent scheduling method provided in an embodiment of this disclosure;

[0019] Figure 2 A flowchart illustrating a multi-route vehicle intelligent scheduling method provided in another embodiment of this disclosure;

[0020] Figure 3 A schematic diagram of the network structure of a reinforcement learning agent provided in an embodiment of this disclosure;

[0021] Figure 4 This is a structural block diagram of a multi-route vehicle intelligent scheduling system provided in an embodiment of the present disclosure;

[0022] Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0024] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0025] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a multi-route vehicle intelligent scheduling method provided in an embodiment of the present disclosure. The method may include steps S101 to S103.

[0026] S101: Construct a feature extraction network to extract feature vectors from the primary information. Construct a vehicle selection network to select vehicles for scheduling. Construct a simulation environment to simulate vehicle operation information. Construct an intelligent agent based on the feature extraction network and the vehicle selection network.

[0027] like Figure 3As shown, in this embodiment, the first information may include multiple vehicle information, multiple control point information, multiple route information, and timetable information. Vehicle information may include whether the vehicle has departed, whether it is currently in operation, rest time, current control point, and crossing time. Control point information may include its number, short-term vehicle demand, long-term vehicle demand, total number of vehicles, and total number of old vehicles. Route information may include the departure control point, arrival control point, and travel time. Timetable information may include the departure control point, arrival control point, departure time, and departure timetable.

[0028] The feature extraction network can include a vehicle feature extraction network, a control point feature extraction network, a route feature extraction network, and a timetable feature extraction network. The control point and route feature extraction networks are individual-level feature extraction networks, while the vehicle and timetable feature extraction networks are global-level feature extraction networks. The vehicle selection network is constructed based on an attention mechanism.

[0029] For example, the multi-route vehicle scheduling problem can be modeled as a sequential decision process. This problem involves multiple routes, each with its own departure timetable. The departure timetables for all routes are merged into a single overall departure timetable. At each departure time in this overall timetable, two types of decisions are executed: vehicle selection and route crossing. Vehicle selection involves choosing a vehicle to depart at that specific time to complete the corresponding trip. To reduce operating costs, vehicles from routes with a larger number of vehicles (more than a first-order number) need to be scheduled onto routes with fewer vehicles (less than a second-order number). This vehicle scheduling decision is the route crossing decision. To reduce the learning difficulty for the agent, the two heterogeneous decisions—vehicle selection and route crossing—can be merged into a single vehicle selection decision.

[0030] For example, a Markov decision process model can be constructed. A Markov decision process can contain five elements: a state space, an action space, state transition probabilities, a reward function, and a discount factor. The state transition probability is the probability of transitioning from one state to another, and the discount factor is a hyperparameter.

[0031] The state space comprises four parts: vehicles, control points, routes, and timetables. For vehicle information, vehicles for which information needs to be extracted are selected based on a vehicle importance ranking mechanism, and a 5-dimensional vehicle vector is designed for each vehicle, which may include: whether it has started working, whether it is currently working, its rest time, its current control point, and its crossing time. Control point information includes all control points, and a 5-dimensional control point vector is designed for each control point, which may include: number, short-term vehicle demand, long-term vehicle demand, total number of vehicles at that control point, and total number of old vehicles at that control point. Route information includes all routes, and a 3-dimensional route vector is designed for each route, which may include: departure control point, arrival control point, and travel time. For timetable information, for a fixed number of future time points (e.g., preset to 15), a 3-dimensional time point vector is designed for each time point, which may include: departure control point, arrival control point, and departure time.

[0032] The specific steps for the vehicle importance ranking mechanism are as follows: Vehicles are divided into three types: used vehicles that do not need to cross lines, used vehicles that need to cross lines, and new vehicles. Each type of vehicle is then ranked according to preset rules (used vehicles that do not need to cross lines are ranked from longest to shortest rest time; used vehicles that need to cross lines are ranked from shortest to longest crossing time; new vehicles are not ranked). Vehicles are then selected sequentially from each ranked vehicle set in the order of used vehicles that do not need to cross lines, used vehicles that need to cross lines, and new vehicles, until the number of selected vehicles reaches a preset value.

[0033] Action Space: An action space is constructed for vehicle selection decisions. The action space includes all available vehicles at the current departure time, which are divided into four categories: old vehicles that do not need to cross lines, old vehicles that need to cross lines, new vehicles that do not need to cross lines, and new vehicles that need to cross lines. Since the effect of selecting all new vehicles is similar, one vehicle type (new vehicle) is used to represent the two categories of new vehicles: new vehicles that do not need to cross lines and new vehicles that need to cross lines.

[0034] In this embodiment, the action space is often variable. Furthermore, the vehicles in the state space and the vehicles in the action space often correspond. To select the appropriate vehicle based on the information of each vehicle, it is necessary to learn the correspondence between vehicles in the state space and vehicles in the action space using a reinforcement learning network. Therefore, this disclosure designs a reinforcement learning network that can process and fuse multiple types of input information and perform vehicle selection based on an attention mechanism. This reinforcement learning network includes a feature extraction network and a vehicle selection network.

[0035] S102: Generate first information based on the simulated environment, and train the agent based on the first information and the reward function to obtain a reinforcement learning agent. Here, the first information is the historical operating information of the simulated vehicle, and the reward function is the reward value calculation function corresponding to the first information.

[0036] like Figure 2 As shown, in this embodiment, the simulation environment is used for:

[0037] The preset status information is updated based on the simulated time period.

[0038] The preset status information includes the vehicle's operating status.

[0039] If the vehicle is in the first operating state, the vehicle information is updated based on the corresponding route information, vehicle operating time, and simulation time period to obtain the target vehicle information.

[0040] If the vehicle is in the second operating state, the vehicle information is updated based on the vehicle's rest time and the simulated time period to obtain the target vehicle information.

[0041] The preset status information includes the initial number of vehicles.

[0042] The initial number of vehicles is updated based on the target vehicle information to obtain the target vehicle number information.

[0043] For example, an environment simulator is designed to build a simulation environment for the multi-line vehicle scheduling problem. The simulation environment is used to simulate the operation process of the multi-line vehicle scheduling problem and realize real-time interaction with the intelligent agent.

[0044] The simulation process in the simulation environment may include: dividing the simulation process into a sequence of simulation steps based on the departure time points in the timetable information. For each step in the simulation step sequence, first state information is constructed based on first information obtained from the simulation environment, and first action information is selected based on the first state information. The first action information is input into the simulation environment to obtain the execution result of the first action information, and the first state information in the simulation environment is updated based on the execution result of the first action information to obtain second state information.

[0045] For each step, the simulation process only simulates the process from the current time point. To the next time point The process between (i.e., the simulated time period). The simulation environment can manage four types of information: vehicle information, control point information, route information, and timetable information.

[0046] For example, regarding vehicle information, the simulation environment can manage specific operational information of vehicles, including whether the vehicle has started working, whether it is currently working, rest time, location at a control point, crossing time, route, and accumulated running time on that route. Vehicles can be divided into two categories: running vehicles and inactive vehicles. For running vehicles, the simulation environment focuses on the route the vehicle is on and the accumulated running time on that route, and adjusts accordingly. Update the corresponding information, if in Once the vehicle has completed its route, it will become inactive in the next step. Therefore, it's necessary to set information such as whether the vehicle is currently operating, its rest time, and its location at a control point. For inactive vehicles, the environment simulator primarily focuses on the vehicle's rest time and, based on... Update the corresponding information. If the vehicle is selected by the intelligent agent, the vehicle will start executing the corresponding trip. Therefore, you can also set information such as whether the vehicle is working, the route it is on, and the running time on the route.

[0047] Regarding control point information, the simulation environment can manage the number of vehicles parked at each control point and the relationship between the control point and the route. For the number of parked vehicles, if a vehicle arrives at the control point during the simulation period, the corresponding number of vehicles increases; if a vehicle departs from the control point during the simulation period, the corresponding number of vehicles decreases.

[0048] For route information, the simulation environment can manage information such as the departure control point, arrival control point, and travel time. Among them, the travel time information can be freely set, and the simulation environment can support hourly adjustments to the travel time of each route, simulating special events such as traffic jams and morning rush hours.

[0049] Regarding timetable information, the simulation environment can freely set different timetables to construct different instances. A timetable contains multiple departure time points, and each departure time point stores two parts of information: the corresponding route and the specific departure time.

[0050] For example, such as Figure 2 As shown, Proximal Policy Optimization (PPO) is selected as the reinforcement learning algorithm, as detailed below:

[0051] First, data collection is performed. The agent executes a series of actions in a simulated environment, collecting multi-step sample data (state, action, reward, next state). This data is used to calculate the advantage function, which measures the value of each action relative to the average level. Additionally, to construct the objective function, the probabilities corresponding to the agent's selected actions are stored. Next, policy updates are performed by optimizing the objective function, which includes a truncated policy ratio term to limit the magnitude of each update. Specifically, the PPO's objective function applies a truncated policy ratio to ensure that the change between the old and new policies does not exceed a preset threshold, thus balancing exploration and stability. By repeatedly executing these steps, PPO continuously optimizes the policy, ultimately achieving good task performance.

[0052] S103: Input the second information into the reinforcement learning agent to obtain the target scheduling plan. The second information is the multi-route vehicle information.

[0053] In this embodiment, the second piece of information is multi-route vehicle information. Analyzing this information yields corresponding vehicle information, control point information, timetable information, and route information. The reinforcement learning agent has been pre-trained using a large amount of data. After receiving multi-route vehicle information, the agent comprehensively evaluates the distribution, demand, and operational status of vehicles on each route. For example, it predicts vehicle dispatching demand based on vehicle locations and peak / off-peak passenger flow, thereby planning a target scheduling scheme that includes vehicle dispatching, departure sequence, and route arrangement to achieve efficient and optimized operations.

[0054] For example, a regional vehicle operating company operates five routes: route a, route b, route c, route d, and route e. Route a starts from the city center, passes through eight stops, and reaches its terminus on the city's edge, with a travel time of approximately 60 minutes. Route b connects two major commercial districts, passes through six stops, and has a travel time of 45 minutes, etc.

[0055] The vehicle operating company collected multi-route vehicle information (i.e., the second information) including: a total of 30 vehicles, of which 10 are older vehicles that have always operated on fixed routes and do not need to cross routes, 5 older vehicles can be dispatched across routes as needed, and 15 are new vehicles. The current status of these vehicles is as follows: Vehicle 1 is resting, has been resting for 20 minutes, and is located at the starting control point of Route A; Vehicle 2 is operating on Route B, and has been running for 30 minutes, etc.

[0056] This information on vehicles across multiple routes is input into a previously trained reinforcement learning agent, which then analyzes and processes the data. For example, at a certain departure time, considering a shortage of vehicles on route c while having a relatively large number of vehicles on route d, the agent decides to select an older vehicle that can be transferred from route d to route c to perform the scheduling task. Ultimately, it outputs a target scheduling plan that includes detailed information such as vehicle allocation and departure arrangements for each route, thereby improving vehicle operating efficiency, reducing passenger waiting time, and making rational use of vehicle resources.

[0057] As can be seen from the above, considering the high cost of testing and training methods in real-world scenarios, this embodiment constructs a simulation environment for the multi-line vehicle scheduling problem. This effectively reduces the high cost of testing and training in real-world scenarios, enabling research and development to be carried out efficiently and allowing for rapid iterative optimization of scheduling strategies in a virtual environment.

[0058] Considering the difficulty of simultaneously making heterogeneous decisions regarding vehicle selection and cross-line scheduling in multi-route vehicle scheduling problems, this embodiment integrates these two decisions into a single vehicle selection decision. This greatly simplifies the decision-making process, reduces decision complexity, and allows the agent to focus more intently and efficiently on vehicle allocation when handling multi-route vehicle scheduling.

[0059] Considering the variable action space and the need for additional learning of the correspondence between vehicles in the state information and vehicles in the action space in reinforcement learning, a reinforcement learning network was designed that can process and fuse multiple types of input information and select vehicles based on an attention mechanism. This enhances the system's adaptability and handling capability to complex and changing situations, and improves the accuracy and reliability of the scheduling scheme.

[0060] like Figure 3 As shown, in one embodiment of this disclosure, the first information includes vehicle information, control point information, route information, and timetable information.

[0061] Constructing a feature extraction network includes:

[0062] Vehicle embedding vectors are constructed based on vehicle information, control point embedding vectors are constructed based on control point information, route embedding vectors are constructed based on route information, and timetable embedding vectors are constructed based on timetable information.

[0063] A feature extraction network is derived based on vehicle embedding vectors, control point embedding vectors, route embedding vectors, and timetable embedding vectors.

[0064] In this embodiment, constructing a vehicle selection network includes:

[0065] The target embedding vector is determined based on the vehicle embedding vector, control point embedding vector, route embedding vector, and timetable embedding vector.

[0066] Multi-line vehicle embedding vectors are obtained based on multi-line vehicle vectors and a fully connected network.

[0067] The vehicle selection network is derived from the multi-line vehicle embedding vector and the target embedding vector.

[0068] In this embodiment, the feature extraction network may include a vehicle feature extraction network, a control point feature extraction network, a route feature extraction network, and a timetable feature extraction network. The vehicle feature extraction network extracts vehicle vectors based on vehicle information and concatenates these vectors into a vehicle embedding vector using a fully connected network. The timetable feature extraction network extracts time point vectors based on timetable information and concatenates these vectors into a timetable embedding vector using a fully connected network.

[0069] The control point feature extraction network is used to extract control point vectors based on control point information. The control point vectors are input into a fully connected network to obtain control point embedding vectors. The control point embedding vectors corresponding to all control points are pooled based on the max pooling mechanism to obtain the pooled control point embedding vectors.

[0070] The line feature extraction network is used to extract line vectors based on line information. The line vectors are input into a fully connected network to obtain line embedding vectors. The line embedding vectors corresponding to all lines are pooled based on the max pooling mechanism to obtain the pooled line embedding vectors.

[0071] In this embodiment, pooling the control point embedding vectors corresponding to all control points based on the max pooling mechanism to obtain the pooled control point embedding vectors may include: taking the maximum value of the control point embedding vectors corresponding to all control points in a preset dimension based on the max pooling mechanism to obtain the pooled control point embedding vectors.

[0072] In this embodiment, pooling the line embedding vectors corresponding to all lines based on the max pooling mechanism to obtain the pooled line embedding vectors may include: taking the maximum value of the line embedding vectors corresponding to all lines in a preset dimension based on the max pooling mechanism to obtain the pooled line embedding vectors.

[0073] In this embodiment, the vehicle selection network uses the target embedding vector obtained by the feature extraction network as the Query. A fully connected network is used to extract features from each vehicle in the action space, resulting in a multi-line vehicle vector. The multi-line vehicle vector includes the vehicle embedding vector corresponding to each vehicle. The vehicle embedding vector corresponding to each vehicle in the multi-line vehicle vector is used as the Key. The weight corresponding to each Key is calculated based on the operation between the Query and the Key, and this weight is used as the corresponding vehicle selection probability.

[0074] For example, such as Figure 3As shown, there are 4 selectable vehicles. The vehicle embedding vector of each vehicle corresponds to the Key. The four types of embedding vectors obtained from the feature extraction network are concatenated to obtain the target embedding vector (i.e., Query). The inner product is calculated for each Key and Query: F(Q, K4), F(Q, K3), F(Q, K4), F(Q, K1), and then normalized to obtain the corresponding action probability for each vehicle.

[0075] This embodiment constructs a feature extraction network that includes information on vehicles, control points, routes, and timetables. This network comprehensively and meticulously extracts various key information, enabling the agent to fully learn the complex structure and logic of vehicle operations. The vehicle selection network, generated based on the feature extraction network, determines the vehicle selection probability through specific query and key operations, effectively solving the problem of accurate vehicle selection in multi-route vehicle scheduling. This embodiment can accurately make vehicle allocation decisions based on actual vehicle operation data, improving vehicle utilization and enhancing service reliability and flexibility.

[0076] like Figure 2 As shown, in one embodiment of this disclosure, a reinforcement learning agent is obtained by training the agent based on first information and a reward function, including:

[0077] The first information is input into the intelligent agent to obtain the first action information, and the first state information is determined based on the first action information. The first action information is the vehicle scheduling action, and the first state information is the updated vehicle operation information obtained after executing the first action information.

[0078] The target reward value is determined based on the first action information, the first state information, and the reward function.

[0079] A reinforcement learning agent is obtained by training the agent based on the first state information and the target reward value.

[0080] In this embodiment, the first information is input into the intelligent agent to obtain the first action information, including:

[0081] The preset state information is determined based on the first information.

[0082] The first action information is obtained by inputting the preset state information into the intelligent agent.

[0083] In this embodiment, determining the first state information based on the first action information includes:

[0084] The preset state information is updated based on the first action information and the simulated time period corresponding to the first action information to obtain the first state information.

[0085] In this embodiment, the reward function includes a main reward function and a branch reward function.

[0086] The target reward value is determined based on the first action information, the first state information, and the reward function, including:

[0087] The first reward value is determined based on the first action information, the first state information, and the branch reward function.

[0088] The second reward value is determined based on the first action information, the first state information, and the main reward function.

[0089] The target reward value is determined based on the first reward value and the second reward value.

[0090] In this embodiment, the reward function includes a main reward function and a secondary reward function. The main reward function is used to calculate the second reward value. The secondary reward function is used to calculate the first reward value.

[0091] For example, a first reward value is calculated at each step of the decision sequence to evaluate the suitability of the selected vehicle (i.e., the first action information) in the current decision step. The first reward value may include two first sub-reward values: a vehicle type sub-reward and a vehicle cross-line time sub-reward. For instance, if the vehicle selected at the current departure time is a new vehicle, it means that a new vehicle has been dispatched at the current departure time, increasing the number of vehicles in use. Therefore, the corresponding vehicle type sub-reward is set to -1. In other cases, the vehicle type sub-reward is 0.

[0092] The second reward value is calculated in the last step of the entire decision-making sequence and defaults to 0 in other decision-making steps. The second reward value is used to evaluate the entire scheduling plan. The second reward value can include two second sub-reward values: the total number of vehicles used and the total cross-line time. The two second sub-reward values ​​are weighted to construct the second reward value (i.e., the main line reward), and the corresponding weights can be set to 4 for the total number of vehicles used and 0.01 for the total cross-line time.

[0093] The final target reward value can be determined based on the first reward value and the second reward value. The target reward value is an evaluation score of the first action information and the first state information.

[0094] This embodiment calculates reward values ​​for different stages and dimensions using both a main reward function and a secondary reward function, enabling a comprehensive evaluation of vehicle scheduling actions. The secondary reward function assesses the suitability of the selected vehicle at each decision step, prompting the agent to consider factors such as vehicle type and cross-line time when selecting a vehicle, avoiding unreasonable vehicle selection. The main reward function evaluates the entire scheduling plan at the end of the decision sequence, focusing on the total number of vehicles used and the total cross-line time, guiding the agent to optimize the overall scheduling strategy to reduce costs and improve efficiency. Combining both methods to determine the target reward value effectively trains the agent, enabling it to generate more scientific and reasonable vehicle scheduling plans, thereby improving operational efficiency and resource utilization.

[0095] like Figure 2 As shown, in one embodiment of this disclosure, the intelligent scheduling method for multi-route vehicles further includes:

[0096] A generator is built based on a simulated environment. Initial information is generated based on the generator.

[0097] The first information is input into the reinforcement learning agent. Based on the meta-game matrix, the first meta-game strategy corresponding to the generator and the second meta-game strategy corresponding to the reinforcement learning agent are determined. The meta-game matrix includes the meta-game strategy corresponding to the generator in the row direction and the meta-game strategy corresponding to the reinforcement learning agent in the column direction. The elements in the meta-game matrix represent the payoffs obtained by the generator and the reinforcement learning agent under different combinations of meta-game strategies.

[0098] The target instance generation strategy is determined based on the first-order game strategy and the second-order game strategy, and the generator is updated based on the target instance generation strategy.

[0099] The target agent's policy is determined based on the first-order game strategy and the second-order game strategy, and the reinforcement learning agent is updated based on the target agent's policy.

[0100] In this embodiment, the first information is generated based on the generator, including:

[0101] New time point information is generated based on the generator and the original timetable, and the target timetable is determined based on the new time point information and the original timetable.

[0102] First information is generated based on the target timetable and the simulation environment.

[0103] In this embodiment, the generator is an instance generator used to generate instances (i.e., the first information). To improve the robustness of the trained reinforcement learning agent (i.e., the ability of the reinforcement learning agent to handle various complex problem instances), the intelligent scheduling problem is modeled as a two-player zero-sum game problem. One side is the instance generator, which aims to generate more complex and difficult instances. Different instances can be constructed based on a simulated environment by initializing different departure timetables. The instance generator can generate the number of new departure time points and specific departure time point information. The number of new departure time points can be constructed based on learnable vectors, and the specific departure time point information can be constructed based on a generative model. The other side is the reinforcement learning agent, which aims to better solve the instances generated by the instance generator. The instance generator and the reinforcement learning agent are trained based on a self-game method, competing against each other and co-evolving to obtain a reinforcement learning agent capable of handling various problem instances.

[0104] For example, in a simulated environment, different instances correspond to different departure timetables. Each departure time point in the timetable includes two parts: route number and time. To ensure the rationality of the instances generated by the instance generator, an original departure timetable can be constructed based on historical real data. Adding the newly generated departure time points to this original timetable will yield a new instance.

[0105] For example, the instance generator may include a time point count generator and a departure time point generator. The time point count generator can generate a newly added number of departure time points N, and the departure time point generator can generate detailed information for N departure time points (each departure time point includes two elements: route number and time). The time point count generator can be constructed into a learnable probability vector. (The corresponding parameters are) Each element in the corresponding probability vector represents the probability of the number of additional time points that can be added. Departure Time Point Generator (The corresponding parameters are) Generative models can be generated by common generative models, such as variational autoencoders (VAEs), generative adversarial networks (GANs), and flow-based generative models.

[0106] For example, taking a flow-based model, the details are as follows:

[0107] Flow-based models use reversible neural network transformations to map complex, high-dimensional data onto simple distributions (such as Gaussian distributions), thereby achieving accurate probability estimation and generation. The complex, high-dimensional data can correspond to departure times, and this method can obtain the probability of generating each departure time.

[0108] Self-game training methods can include Vanilla Self-play (VSP), Fictitious Self-play (FSP), and Policy Space Response Oracle (PSRO). Taking PSRO as an example, the details are as follows:

[0109] PSRO maintains a policy set for each player. For the instance generator player, each policy in the policy set corresponds to a specific instance generator responsible for generating instances. For the reinforcement learning agent player, each policy in the policy set corresponds to a specific reinforcement learning agent responsible for resolving instances. The overall PSRO process consists of three steps: simulation, meta-game solving, and optimal reaction solving. These three steps are performed alternately in a loop. After each loop, each player learns a new policy and adds it to its corresponding policy set.

[0110] The purpose of the simulation is to construct a meta-game matrix, where the horizontal and vertical axes correspond to each instance generator and each reinforcement learning agent, respectively. Elements in the meta-game matrix represent the performance of the corresponding reinforcement learning agent in the corresponding instance generator. The instance generator produces several instances, and the reinforcement learning agent S runs in instance I to obtain the corresponding solution. The performance of the corresponding solution can be represented as Performance(S(I)). Based on the exact algorithm, running on the same instance, the performance of the obtained solution can be represented as Performance(Oracle(I)). The formula for calculating the final performance of the reinforcement learning agent in this instance can be expressed as:

[0111]

[0112] The reinforcement learning agent performs the calculation of this formula on all instances generated by the instance generator, calculates the average value, and fills the corresponding position in the matrix with the final average value.

[0113] After obtaining the metagame matrix, the metagame strategy is solved based on the metagame matrix, and the metagame strategy corresponds to the weight of each strategy in the strategy set.

[0114] After obtaining the meta-game strategy of one party, the optimal response is solved for the meta-game strategy of the other party. The optimal response is the new strategy added to the strategy set.

[0115] The new reinforcement learning agent to be trained is trained on the meta-game policy corresponding to the instance generator. Specifically, in each decision sequence, the training is performed according to the meta-game policy of the instance generator. Sample an instance generator The generator includes a generator for generating the number of new time points. and a time point generator The former is responsible for generating the number of new time points to be added, while the latter is responsible for generating the specific time points. A new instance is generated based on this instance generator (first, based on...). The sampling requires N new departure time points to be added, and then based on... Generate the corresponding number of departure time points. (And add it to the original departure timetable), then the reinforcement learning agent performs data sampling and policy updates in the new instance. The objective function of the reinforcement learning agent. The corresponding policy gradient update formula is as follows:

[0116]

[0117] in, To enhance the objective function of the learning agent, To reinforce the parameters of the learning agent, Indicated by Calculate the gradient of the variable with respect to the objective function. Represents the meta-game strategy from the instance generator Medium Sampling Instance Generator And seek expectations, This indicates the generator for generating the number of new time points. Given the number of sampling time points N, calculate the expected value. Indicates the generator from time point Sample N departure times and calculate the expected value. Indicated by Let S be the probability of obtaining a solution for the reinforcement learning agent S in a newly sampled instance. Calculate the gradient. This represents the performance score of the reinforcement learning agent S on the newly sampled instances.

[0118] For the new instance generator to be trained It trains on the meta-game strategy corresponding to the reinforcement learning agent. Specifically, it generates a new instance based on the instance generator (first based on...). The sampling requires N new departure time points to be added, and then based on... Generate the corresponding number of departure time points. (and added to the original departure timetable), based on the meta-game strategy of the reinforcement learning agent. Sample a reinforcement learning agent S, then an instance generator Construct an objective function based on the performance of the reinforcement learning agent S in the instances it generates. The policy gradient method is used for updating.

[0119] The corresponding parameters are Time point generator The update formula is:

[0120]

[0121] in, The objective function of the instance generator. These are the parameters for the time point generator. Indicated by Calculate the gradient of the variable with respect to the objective function. This represents the meta-game strategy of a reinforcement learning agent. The mid-sample reinforcement learning agent S is used to calculate the expectation. This indicates the generator for generating the number of new time points. Given the number of sampling time points N, calculate the expected value. Indicates the generator from time point Sample N departure times and calculate the expected value. Indicated by Calculate the gradient of the variable for the joint probability corresponding to each new time point. This represents the performance score of the reinforcement learning agent S on the newly sampled instances.

[0122] The corresponding parameters are Add a time point number generator The update formula is:

[0123]

[0124] in, The objective function of the instance generator. For the parameters of the new time point number generator, Indicated by Calculate the gradient of the variable with respect to the objective function. This represents the meta-game strategy of a reinforcement learning agent. The mid-sample reinforcement learning agent S is used to calculate the expectation. This indicates the generator for generating the number of new time points. Given the number of sampling time points N, calculate the expected value. Indicates the generator from time point Sample N departure times and calculate the expected value. Indicated by Calculate the gradient of the variable with respect to the probability of N new time points. This represents the performance score of the reinforcement learning agent S on the newly sampled instances.

[0125] This embodiment addresses the limitation that existing technologies can only handle a specific problem instance, often requiring a re-solution when encountering new problem instances. Instead, it employs reinforcement learning combined with self-game training to develop a reinforcement learning agent capable of handling various instances. This avoids the tedious process of re-solving new problem instances, significantly improving the versatility and scalability of the solution, and providing a powerful and flexible solution for the intelligent management of multi-route vehicle operations.

[0126] Corresponding to the intelligent scheduling method for multi-route vehicles in the above embodiment, Figure 4 This is a structural block diagram of a multi-route intelligent vehicle scheduling system provided in one embodiment of the present disclosure. For ease of explanation, only the parts relevant to the embodiment of the present disclosure are shown. References Figure 4 The multi-route vehicle intelligent scheduling system 30 includes: an intelligent agent construction module 31, an intelligent agent training module 32, and an intelligent scheduling module 33.

[0127] The intelligent agent construction module 31 is used to construct a feature extraction network, which extracts feature vectors from the first piece of information. It also constructs a vehicle selection network, which selects vehicles for scheduling. Finally, it constructs a simulation environment to simulate vehicle operation information. The intelligent agent is then constructed based on the feature extraction network and the vehicle selection network.

[0128] The agent training module 32 is used to generate first information based on the simulated environment, and to train the agent based on the first information and the reward function to obtain a reinforcement learning agent. The first information is the historical operating information of the simulated vehicle, and the reward function is the reward value calculation function corresponding to the first information.

[0129] The intelligent scheduling module 33 is used to input the second information into the reinforcement learning agent to obtain the target scheduling plan. The second information is multi-route vehicle information.

[0130] In one embodiment of this disclosure, the first information includes vehicle information, control point information, route information, and timetable information. The intelligent agent construction module 31 is specifically used to construct a vehicle embedding vector based on the vehicle information, a control point embedding vector based on the control point information, a route embedding vector based on the route information, and a timetable embedding vector based on the timetable information.

[0131] A feature extraction network is derived based on vehicle embedding vectors, control point embedding vectors, route embedding vectors, and timetable embedding vectors.

[0132] In one embodiment of this disclosure, the agent construction module 31 is further configured to determine the target embedding vector based on the vehicle embedding vector, control point embedding vector, route embedding vector, and timetable embedding vector.

[0133] Multi-line vehicle embedding vectors are obtained based on multi-line vehicle vectors and a fully connected network.

[0134] The vehicle selection network is derived from the multi-line vehicle embedding vector and the target embedding vector.

[0135] In one embodiment of this disclosure, the agent training module 32 is specifically used to input first information into the agent to obtain first action information, and determine first state information based on the first action information. The first action information is a vehicle scheduling action, and the first state information is the updated vehicle operation information obtained after executing the first action information.

[0136] The target reward value is determined based on the first action information, the first state information, and the reward function.

[0137] A reinforcement learning agent is obtained by training the agent based on the first state information and the target reward value.

[0138] In one embodiment of this disclosure, the agent training module 32 is further configured to determine preset state information based on the first information.

[0139] The first action information is obtained by inputting the preset state information into the intelligent agent.

[0140] The preset state information is updated based on the first action information and the simulated time period corresponding to the first action information to obtain the first state information.

[0141] In one embodiment of this disclosure, the agent training module 32 is further configured to determine a first reward value based on first action information, first state information, and a branch reward function.

[0142] The second reward value is determined based on the first action information, the first state information, and the main reward function.

[0143] The target reward value is determined based on the first reward value and the second reward value.

[0144] In one embodiment of this disclosure, the multi-route vehicle intelligent scheduling system 30 further includes: a self-game module for constructing a generator based on a simulation environment. First information is generated based on the generator.

[0145] The first information is input into the reinforcement learning agent. Based on the meta-game matrix, the first meta-game strategy corresponding to the generator and the second meta-game strategy corresponding to the reinforcement learning agent are determined. The meta-game matrix includes the meta-game strategy corresponding to the generator in the row direction and the meta-game strategy corresponding to the reinforcement learning agent in the column direction. The elements in the meta-game matrix represent the payoffs obtained by the generator and the reinforcement learning agent under different combinations of meta-game strategies.

[0146] The target instance generation strategy is determined based on the first-order game strategy and the second-order game strategy, and the generator is updated based on the target instance generation strategy.

[0147] The target agent's policy is determined based on the first-order game strategy and the second-order game strategy, and the reinforcement learning agent is updated based on the target agent's policy.

[0148] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Figure 5 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned system embodiments, for example... Figure 4 The functions of modules 31 to 33 are shown.

[0149] It should be understood that, in the embodiments of this disclosure, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0150] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0151] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0152] In specific implementations, the processor 301, input device 302, and output device 303 described in this disclosure embodiment can execute the implementation methods described in the first and second embodiments of the multi-line vehicle intelligent scheduling method provided in this disclosure embodiment, or they can execute the implementation methods of the electronic device 300 described in this disclosure embodiment, which will not be repeated here.

[0153] In another embodiment of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to implement these processes. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or system capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0154] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0156] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this disclosure, depending on actual needs.

[0159] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0160] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A multi-line vehicle intelligent scheduling method, characterized in that, The method comprises the following steps: constructing a feature extraction network for extracting a feature vector of first information; constructing a vehicle selection network for selecting a vehicle for scheduling; constructing a simulation environment for simulating running information of the vehicle; constructing an agent based on the feature extraction network and the vehicle selection network; generating the first information based on the simulation environment, and training the agent based on the first information and a reward function to obtain a reinforcement learning agent; wherein the first information is historical running information of the simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information; inputting second information into the reinforcement learning agent to obtain a target scheduling scheme, wherein the second information is multi-line vehicle information; The method further comprises: constructing a generator based on the simulation environment; generating the first information based on the generator; inputting the first information into the reinforcement learning agent, determining a first metagame strategy corresponding to the generator and a second metagame strategy corresponding to the reinforcement learning agent based on a metagame matrix; the metagame matrix includes metagame strategies corresponding to the generator in the row direction, and includes metagame strategies corresponding to the reinforcement learning agent in the column direction, and the elements in the metagame matrix represent the benefits obtained by the generator and the reinforcement learning agent under different metagame strategy combinations; determining a target instance generation strategy based on the first metagame strategy and the second metagame strategy, and updating the generator based on the target instance generation strategy; determining a target agent strategy based on the first metagame strategy and the second metagame strategy, and updating the reinforcement learning agent based on the target agent strategy; generating the first information based on the generator, comprising: generating new time point information based on the generator and the original timetable, and determining a target timetable based on the new time point information and the original timetable; generating the first information based on the target timetable and the simulation environment.

2. The multi-line vehicle assignment method of claim 1, wherein, The first information includes vehicle information, control point information, line information, and timetable information; The method comprises the following steps: constructing a vehicle embedding vector based on vehicle information, constructing a control point embedding vector based on control point information, constructing a line embedding vector based on line information, and constructing a timetable embedding vector based on timetable information; obtaining a feature extraction network based on the vehicle embedding vector, the control point embedding vector, the line embedding vector, and the timetable embedding vector.

3. The multi-line vehicle assignment method of claim 2, wherein, The method comprises the following steps: determining a target embedding vector based on the vehicle embedding vector, the control point embedding vector, the line embedding vector, and the timetable embedding vector; obtaining a multi-line vehicle embedding vector based on a multi-line vehicle vector and a fully connected network; obtaining a vehicle selection network based on the multi-line vehicle embedding vector and the target embedding vector.

4. The multi-line vehicle scheduling method of claim 1, wherein, The method comprises the following steps: inputting the first information into the agent to obtain first action information, determining first state information based on the first action information; wherein the first action information is a vehicle scheduling action, and the first state information is updated vehicle operation information obtained after the first action information is executed; determining a target reward value based on the first action information, the first state information, and the reward function; training the agent based on the first state information and the target reward value to obtain a reinforcement learning agent.

5. The multi-line vehicle assignment method of claim 4, wherein, The method further includes: determining preset state information based on the first information; inputting the preset state information into the agent to obtain first action information; updating the preset state information based on the first action information and a simulation time period corresponding to the first action information to obtain first state information.

6. The multi-line vehicle scheduling method of claim 4, wherein, The reward function includes a main line reward function and a branch reward function. The method further includes: determining a first reward value based on the first action information, the first state information, and the branch reward function; determining a second reward value based on the first action information, the first state information, and the main line reward function; determining a target reward value based on the first reward value and the second reward value.

7. A multi-line vehicle intelligent scheduling system, characterized in that, The method further includes: an agent construction module configured to construct a feature extraction network, the feature extraction network being configured to extract a feature vector of first information; constructing a vehicle selection network, the vehicle selection network being configured to select a vehicle for scheduling; and constructing a simulation environment, the simulation environment being configured to simulate vehicle operation information; and constructing an agent based on the feature extraction network and the vehicle selection network; an agent training module configured to generate first information based on the simulation environment, and train the agent based on the first information and a reward function to obtain a reinforcement learning agent; wherein the first information is historical operation information of a simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information; an intelligent scheduling module configured to input second information into the reinforcement learning agent to obtain a target scheduling scheme, the second information being multi-line vehicle information; a self-game module configured to construct a generator based on the simulation environment, and generate the first information based on the generator; inputting the first information into the reinforcement learning agent, determining a first metagame strategy corresponding to the generator and a second metagame strategy corresponding to the reinforcement learning agent based on a metagame matrix; the metagame matrix includes metagame strategies corresponding to the generator in a row direction, and includes metagame strategies corresponding to the reinforcement learning agent in a column direction, and elements in the metagame matrix represent benefits obtained by the generator and the reinforcement learning agent under different metagame strategy combinations. determine a target instance generation strategy based on the first meta-game strategy and the second meta-game strategy, and update the generator based on the target instance generation strategy; determine a target agent strategy based on the first meta-game strategy and the second meta-game strategy, and update the reinforcement learning agent based on the target agent strategy; generate first information based on the generator, including: generate new time point information based on the generator and the original time table, and determine a target time table based on the new time point information and the original time table; generate the first information based on the target time table and the simulation environment.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images