Multi-line vehicle intelligent scheduling method and system, equipment and storage medium

Abstract

The invention provides a multi-line vehicle intelligent scheduling method and system, equipment and a storage medium, and belongs to the technical field of intelligent vehicle scheduling, and the method comprises the steps: constructing a feature extraction network which is used for extracting a feature vector of first information; and constructing a vehicle selection network, wherein the vehicle selection network is used for selecting vehicles for scheduling. And constructing a simulation environment, wherein the simulation environment is used for simulating operation information of the vehicle. And constructing an intelligent agent based on the feature extraction network and the vehicle selection network. First information is generated based on the simulation environment, the intelligent agent is trained based on the first information and a reward function to obtain a reinforcement learning intelligent agent, the first information is historical operation information of the simulation vehicle, and the reward function is a reward value calculation function corresponding to the first information; and inputting second information into the reinforcement learning agent to obtain a target scheduling scheme, wherein the second information is multi-line vehicle information. According to the multi-line vehicle intelligent scheduling method and system, the equipment and the storage medium provided by the invention, the capability of coping with a complex vehicle scheduling problem can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of intelligent vehicle scheduling, and more specifically, to a method and system, equipment, and storage medium for intelligent scheduling of multi-line vehicles. Background Art

[0002] The key to improving service quality lies in optimizing vehicle scheduling.

[0003] The methods commonly used in the prior art to solve the vehicle scheduling problem are mainly divided into precise algorithms, heuristic methods and reinforcement learning methods. Although precise algorithms and heuristic methods can find the optimal or near-optimal solution for a specific instance, as long as the instance changes, the original solution will become invalid, and these two methods need to be solved again. Since the reinforcement learning method ultimately obtains a strategy, it can better cope with the situation where the instance changes. However, when the instance changes greatly, the generalization performance of the reinforcement learning method is relatively limited.

[0004] Therefore, how to design a vehicle scheduling method that can cope with various complex instances is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present disclosure is to provide a multi-line vehicle intelligent scheduling method and system, equipment, and storage medium to improve the ability to cope with complex vehicle scheduling problems.

[0006] A first aspect of an embodiment of the present disclosure provides a multi-line vehicle intelligent scheduling method, comprising: Constructing a feature extraction network, the feature extraction network is used to extract a feature vector of the first information; constructing a vehicle selection network, the vehicle selection network is used to select a vehicle for scheduling; constructing a simulation environment, the simulation environment is used to simulate the operation information of the vehicle; constructing an intelligent agent based on the feature extraction network and the vehicle selection network; Generate first information based on the simulation environment, and train the agent based on the first information and a reward function to obtain a reinforcement learning agent; wherein the first information is historical operation information of the simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information; The second information is input into the reinforcement learning agent to obtain a target scheduling plan, wherein the second information is multi-line vehicle information.

[0007] A second aspect of the embodiments of the present disclosure provides a multi-line vehicle intelligent scheduling system, including: An intelligent agent building module is used to build a feature extraction network, the feature extraction network is used to extract a feature vector of the first information; build a vehicle selection network, the vehicle selection network is used to select a vehicle for scheduling; build a simulation environment, the simulation environment is used to simulate the operation information of the vehicle; and build an intelligent agent based on the feature extraction network and the vehicle selection network; An agent training module, used to generate first information based on the simulation environment, and train the agent based on the first information and a reward function to obtain a reinforcement learning agent; wherein the first information is historical operation information of the simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information; The intelligent scheduling module is used to input second information into the reinforcement learning agent to obtain a target scheduling plan, wherein the second information is multi-line vehicle information.

[0008] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned multi-line vehicle intelligent scheduling method when executing the computer program.

[0009] In a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned multi-line vehicle intelligent scheduling method are implemented.

[0010] The beneficial effects of the multi-line vehicle intelligent scheduling method and system, device, and storage medium provided by the embodiments of the present disclosure are: by constructing a simulation environment to simulate vehicle operation, the high cost of testing and training in real-life scenarios is effectively reduced, research and development can be carried out efficiently, and scheduling strategies can be quickly iterated and optimized in a virtual environment. The decision-making process is simplified, the complexity of decision-making is reduced, and the intelligent agent can be more focused and efficient in vehicle allocation when dealing with multi-line vehicle scheduling. The special reinforcement learning network designed to address the difficulties of variable action space in reinforcement learning and the need for additional learning of vehicle correspondence enhances the system's adaptability and processing capabilities for complex and changing situations, and improves the accuracy and reliability of scheduling plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1A schematic diagram of a process for intelligent scheduling of multi-line vehicles provided in an embodiment of the present disclosure; Figure 2 A flowchart of a multi-line vehicle intelligent scheduling method provided by another embodiment of the present disclosure; Figure 3 A schematic diagram of a network structure of a reinforcement learning agent provided by an embodiment of the present disclosure; Figure 4 A structural block diagram of a multi-line vehicle intelligent scheduling system provided by an embodiment of the present disclosure; Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0014] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0015] Please refer to Figure 1 , Figure 1 This is a flow chart of a method for intelligent scheduling of multi-line vehicles provided in one embodiment of the present disclosure. The method may include S101 to S103.

[0016] S101: construct a feature extraction network, the feature extraction network is used to extract a feature vector of the first information. Construct a vehicle selection network, the vehicle selection network is used to select a vehicle for scheduling. Construct a simulation environment, the simulation environment is used to simulate the operation information of the vehicle. Construct an intelligent agent based on the feature extraction network and the vehicle selection network.

[0017] like Figure 3 As shown, in this embodiment, the first information may include multiple vehicle information, multiple control point information, multiple route information and schedule information. Among them, the vehicle information may include whether to depart, whether in operation, rest time, control point and cross-line time. The control point information may include the number, short-term vehicle demand, long-term vehicle demand, total number of vehicles and total number of old vehicles. The route information may include the departure control point, arrival control point and running time. The schedule information may include the departure control point, arrival control point, departure time point and departure schedule.

[0018] The feature extraction network can include a vehicle feature extraction network, a control point feature extraction network, a route feature extraction network, and a timetable feature extraction network. Among them, the control point feature extraction network and the route feature extraction network are both feature extraction networks of individual granularity, and the vehicle feature extraction network and the timetable feature extraction network are both feature extraction networks of overall granularity. The vehicle selection network is constructed based on the attention mechanism.

[0019] Exemplarily, the multi-line vehicle scheduling problem is modeled as a sequential decision process. The multi-line vehicle scheduling problem includes multiple lines, and each line has a corresponding departure schedule. The departure schedules corresponding to all lines are merged into a total departure schedule, and two types of decisions, vehicle selection and cross-line, are performed at each departure time point in the total departure schedule. The vehicle selection decision is to select a vehicle to be dispatched at the departure time point to execute the corresponding trip. In order to reduce the operating cost, it is necessary to schedule the vehicles in the lines with a large number of vehicles (the number of vehicles is greater than the first number) to the lines lacking vehicles (the number of vehicles is less than the second number). The vehicle scheduling decision is the cross-line decision. In order to reduce the learning difficulty of the intelligent agent, the two types of heterogeneous decisions, vehicle selection and cross-line, can be merged into one type of vehicle selection decision.

[0020] For example, a Markov decision process model can be constructed. The Markov decision process can include five elements, namely, state space, action space, state transition probability, reward function and discount factor. Among them, the state transition probability is the probability of transitioning from one state to another, and the discount factor is a hyperparameter.

[0021] Among them, the state space contains four parts of information: vehicles, control points, routes and timetables. For vehicle information, the vehicles whose information needs to be extracted are selected based on the vehicle importance sorting mechanism, and a 5-dimensional vehicle vector is designed for each vehicle, which may include: whether to start work, whether to work, rest time, control point and cross-line time. The control point information contains all control points, and a 5-dimensional control point vector is designed for each control point, which may include: number, short-term vehicle demand, long-term vehicle demand, total number of vehicles located at the control point and total number of old vehicles located at the control point. The route information contains all routes, and a 3-dimensional route vector is designed for each route, which may include: departure control point, arrival control point and running time. For the timetable information, for a fixed number of future time points (for example, it can be preset to 15), a 3-dimensional time point vector is designed for each time point, which may include: departure control point, arrival control point and departure time point.

[0022] For the vehicle importance ranking mechanism, the specific execution steps are as follows: vehicles are divided into three types: old vehicles that do not need to cross the line, old vehicles that need to cross the line, and new vehicles, and each type of vehicle is sorted according to the preset rules (old vehicles that do not need to cross the line are sorted from large to small based on rest time, old vehicles that need to cross the line are sorted from small to large based on crossing time, and new vehicles are not sorted). Select from each type of sorted vehicle set in the order of old vehicles that do not need to cross the line, old vehicles that need to cross the line, and new vehicles, until the number of selected vehicles reaches the preset value.

[0023] Action space: An action space is constructed for vehicle selection decisions. The action space contains all available vehicles at the current departure time, which are divided into four categories: old vehicles that do not need to cross the line, old vehicles that need to cross the line, new vehicles that do not need to cross the line, and new vehicles that need to cross the line. Since the effects of selecting all new vehicles are similar, one vehicle type (new vehicle) is used to represent these two types of new vehicles: new vehicles that do not need to cross the line and new vehicles that need to cross the line.

[0024] In this embodiment, the action space is often variable. In addition, the vehicles contained in the vehicle information in the state space often correspond to the vehicles in the action space. In order to select a suitable vehicle based on the information of each vehicle, it is necessary to learn the correspondence between the vehicles in the state space and the vehicles in the action space based on the reinforcement learning network. Therefore, the present disclosure designs a reinforcement learning network that can process and fuse multiple types of input information and select vehicles based on the attention mechanism. The reinforcement learning network includes a feature extraction network and a vehicle selection network.

[0025] S102: Generate first information based on the simulation environment, and train the agent based on the first information and the reward function to obtain a reinforcement learning agent, wherein the first information is historical operation information of the simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information.

[0026] like Figure 2 As shown, in this embodiment, the simulation environment is used to: The preset status information is updated based on the simulation time period.

[0027] The preset status information includes the vehicle operating status.

[0028] If the vehicle operation state is the first operation state, the vehicle information is updated based on the route information corresponding to the vehicle, the vehicle operation time and the simulation time period to obtain the target vehicle information.

[0029] If the vehicle operating state is the second operating state, the vehicle information is updated based on the vehicle rest time and the simulation time period corresponding to the vehicle to obtain the target vehicle information.

[0030] The preset state information includes initial vehicle number information.

[0031] The initial vehicle number information is updated based on the target vehicle information to obtain the target vehicle number information.

[0032] For example, an environment simulator is designed for the multi-line vehicle scheduling problem to construct a simulation environment. The simulation environment is used to simulate the operation process of the multi-line vehicle scheduling problem and realize real-time interaction with the intelligent agent.

[0033] The simulation process of the simulation environment may include: dividing the simulation process into a simulation step sequence based on the departure time points in the timetable information. For each step in the simulation step sequence, constructing first state information based on first information obtained from the simulation environment, and selecting first action information based on the first state information. Inputting the first action information into the simulation environment, obtaining an execution result of the first action information, and updating the first state information in the simulation environment based on the execution result of the first action information to obtain second state information.

[0034] Among them, for each step, the simulation process only simulates the To the next time point The process between (i.e., simulation time period). The simulation environment can manage four types of information: vehicle information, control point information, route information, and timetable information.

[0035] For example, for vehicle information, the simulation environment can manage the specific operation information of the vehicle, including whether the vehicle has started working, whether it is working, rest time, control point, crossing time, route, and elapsed time on the route. The vehicle can be divided into two parts: vehicles in operation and vehicles not in operation. For vehicles in operation, the simulation environment pays attention to the route the vehicle is in and the elapsed time on the route, and calculates the elapsed time on the route based on the vehicle's status. Update the corresponding information. If If the vehicle has completed the operation on the corresponding line within 10 seconds, it will become a non-operating vehicle in the next step. Therefore, it is also necessary to set whether the vehicle is working, the rest time, and the control point. For non-operating vehicles, the environment simulator mainly focuses on the vehicle rest time and Update the corresponding information. If the vehicle is selected by the agent, the vehicle will start to execute the corresponding trip. Therefore, you can also set whether the vehicle is working, the route it is on, and the running time on the route.

[0036] For the control point information, the simulation environment can manage the number of vehicles parked at the control point and the relationship between the control point and the route. For the number of parked vehicles, if a vehicle arrives at the control point during the simulation time period, the corresponding number of vehicles increases accordingly; if a vehicle departs from the control point during the simulation time period, the corresponding number of vehicles decreases accordingly.

[0037] For route information, the simulation environment can manage the route's departure control point, arrival control point, and running time. The running time information can be freely set, and the simulation environment can support hourly adjustments to the running time of each route, and simulate special events such as traffic jams and morning rush hour.

[0038] For timetable information, the simulation environment can freely set different timetables to construct different instances. A timetable contains multiple departure time points, and each departure time point stores two parts of information: the corresponding route and the specific departure time.

[0039] For example, Figure 2 As shown in the figure, the Proximal Policy Optimization (PPO) method is selected as the reinforcement learning algorithm, as follows: First, data collection is performed. The agent performs a series of actions in the simulated environment, collects multi-step sample data (state, action, reward, next state), and uses this data to calculate the advantage function, which is used to measure the value of each action relative to the average level. In addition, in order to construct the objective function, the probability corresponding to the selected action output by the agent is also saved. Next, a policy update is performed to update the policy by optimizing the objective function, where the objective function contains a truncated policy ratio term to limit the magnitude of each update. Specifically, the objective function of PPO balances exploration and stability by applying a truncation operation to the policy ratio to ensure that the change between the new and old policies does not exceed a preset threshold. By repeatedly performing these steps, PPO continuously optimizes the policy and ultimately achieves good task performance.

[0040] S103: Input the second information into the reinforcement learning agent to obtain a target scheduling plan, where the second information is multi-line vehicle information.

[0041] In this embodiment, the second information is multi-line vehicle information. The corresponding vehicle information, control point information, timetable information and line information can be obtained by analyzing the multi-line vehicle information. The reinforcement learning agent has been trained with a large amount of data in advance. After receiving the multi-line vehicle information, the reinforcement learning agent can comprehensively evaluate the distribution, demand and operation status of vehicles on each line, such as predicting the vehicle allocation demand based on the vehicle location and the peak and trough of the line passenger flow, so as to plan a target scheduling plan including vehicle allocation, departure order, line arrangement and other contents to achieve efficient and optimized operation.

[0042] For example, a regional vehicle operating company operates 5 routes, namely route a, route b, route c, route d and route e. Route a starts from the city center, passes through 8 stations, and arrives at the terminal at the edge of the city, with a running time of about 60 minutes. Route b connects two major commercial areas, passes through 6 stations, and runs for 45 minutes, etc.

[0043] The multi-line vehicle information (i.e., the second information) collected by the vehicle operating company includes: there are 30 vehicles in total, 10 of which are old vehicles that have been running on fixed routes without crossing routes, 5 old vehicles that can be dispatched across routes according to demand, and 15 new vehicles. The current status of these vehicles is as follows: Vehicle 1 is resting, has rested for 20 minutes, and is located at the starting control point of route A. Vehicle 2 is running on route b and has been running for 30 minutes, etc.

[0044] The multi-line vehicle information is input into the previously trained reinforcement learning agent, which is then analyzed and processed. For example, at a certain departure time, considering that there is a shortage of vehicles on line C and there are relatively more vehicles on line D, the agent makes a decision to select an old vehicle that can cross lines and transfer it from line D to line C to perform the scheduling task. Finally, it outputs a target scheduling plan including detailed contents such as vehicle allocation and departure arrangements for each line, thereby improving vehicle operation efficiency, reducing passenger waiting time and making rational use of vehicle resources.

[0045] From the above, it can be concluded that considering the high cost of testing and training methods in real scenarios, this embodiment constructs a simulation environment for multi-line vehicle scheduling problems. It effectively reduces the high cost of testing and training in real scenarios, enables research and development to be carried out efficiently, and can quickly iterate and optimize scheduling strategies in a virtual environment.

[0046] Considering the difficulty of making two heterogeneous decisions, vehicle selection and cross-line decision, in the multi-line vehicle scheduling problem, this embodiment combines the two decisions into one vehicle selection decision. This greatly simplifies the decision-making process and reduces the complexity of decision-making, allowing the intelligent agent to be more focused and efficient in vehicle allocation when processing multi-line vehicle scheduling.

[0047] Considering the variable action space in reinforcement learning, and the need for additional learning of the correspondence between the vehicles in the vehicle information in the state and the vehicles in the action space, a reinforcement learning network is designed that can process and fuse multiple types of input information and select vehicles based on the attention mechanism. This enhances the system's adaptability and processing capabilities for complex and changing situations, and improves the accuracy and reliability of the scheduling plan.

[0048] like Figure 3 As shown, in one embodiment of the present disclosure, the first information includes vehicle information, control point information, route information and schedule information.

[0049] Construct a feature extraction network, including: A vehicle embedding vector is constructed based on vehicle information, a control point embedding vector is constructed based on control point information, a route embedding vector is constructed based on route information, and a timetable embedding vector is constructed based on timetable information.

[0050] A feature extraction network is obtained based on the vehicle embedding vector, the control point embedding vector, the route embedding vector and the timetable embedding vector.

[0051] In this embodiment, building a vehicle selection network includes: The target embedding vector is determined based on the vehicle embedding vector, the control point embedding vector, the route embedding vector and the timetable embedding vector.

[0052] A multi-line vehicle embedding vector is obtained based on the multi-line vehicle vector and the fully connected network.

[0053] The vehicle selection network is obtained based on the multi-line vehicle embedding vector and the target embedding vector.

[0054] In this embodiment, the feature extraction network may include a vehicle feature extraction network, a control point feature extraction network, a route feature extraction network, and a schedule feature extraction network. The vehicle feature extraction network is used to extract vehicle vectors based on vehicle information, and to splice vehicle vectors corresponding to all vehicles into vehicle embedding vectors based on a fully connected network. The schedule feature extraction network is used to extract time point vectors based on schedule information, and to splice time point vectors corresponding to all time points into schedule embedding vectors based on a fully connected network.

[0055] The control point feature extraction network is used to extract the control point vector based on the control point information, and the control point vector is input into the fully connected network to obtain the control point embedding vector. The control point embedding vectors corresponding to all control points are pooled based on the maximum pooling mechanism to obtain the pooled control point embedding vector.

[0056] The line feature extraction network is used to extract the line vector based on the line information, input the line vector into the fully connected network to obtain the line embedding vector, and pool the line embedding vectors corresponding to all lines based on the maximum pooling mechanism to obtain the pooled line embedding vector.

[0057] In this embodiment, the control point embedding vectors corresponding to all control points are pooled based on the maximum pooling mechanism to obtain the pooled control point embedding vector, which may include: taking the maximum value of the control point embedding vectors corresponding to all control points in a preset dimension based on the maximum pooling mechanism to obtain the pooled control point embedding vector.

[0058] In this embodiment, the line embedding vectors corresponding to all lines are pooled based on the maximum pooling mechanism to obtain the pooled line embedding vector, which may include: taking the maximum value of the line embedding vectors corresponding to all lines in a preset dimension based on the maximum pooling mechanism to obtain the pooled line embedding vector.

[0059] In this embodiment, the vehicle selection network uses the target embedding vector obtained by the feature extraction network as the query. A fully connected network is used to extract features for each vehicle in the action space to obtain a multi-line vehicle vector. The multi-line vehicle vector includes a vehicle embedding vector corresponding to each vehicle. The vehicle embedding vector corresponding to each vehicle in the multi-line vehicle vector is used as the key. The weight corresponding to each key is calculated based on the operation of the query and the key, and the weight is used as the corresponding vehicle selection probability.

[0060] For example, Figure 3 As shown in the figure, there are 4 optional vehicles. The vehicle embedding vector of each vehicle corresponds to the Key. The four types of embedding vectors obtained from the feature extraction network are concatenated to get the target embedding vector (i.e., Query). The inner product of each Key and Query is solved respectively: F(Q, K4), F(Q, K3), F(Q, K4), F(Q, K1), and normalized to get the corresponding action probability of selecting each vehicle.

[0061] This embodiment can comprehensively and meticulously extract various key information by constructing a feature extraction network that includes vehicle, control point, route and schedule information, so that the intelligent agent can fully learn the complex structure and logic of vehicle operation. The vehicle selection network is generated based on the feature extraction network, and the vehicle selection probability is determined through specific Query and Key operations, which effectively solves the problem of vehicle selection accuracy in multi-route vehicle scheduling. This embodiment can accurately make vehicle allocation decisions based on actual vehicle operation data, improve vehicle utilization, and enhance service reliability and flexibility.

[0062] like Figure 2As shown, in one embodiment of the present disclosure, the agent is trained based on the first information and the reward function to obtain a reinforcement learning agent, including: The first information is input into the agent to obtain the first action information, and the first state information is determined based on the first action information, wherein the first action information is the vehicle scheduling action, and the first state information is the updated vehicle operation information obtained after executing the first action information.

[0063] A target reward value is determined based on the first action information, the first state information, and the reward function.

[0064] The agent is trained based on the first state information and the target reward value to obtain a reinforcement learning agent.

[0065] In this embodiment, inputting the first information into the agent to obtain the first action information includes: Preset state information is determined based on the first information.

[0066] The preset state information is input into the agent to obtain the first action information.

[0067] In this embodiment, determining the first state information based on the first action information includes: The preset state information is updated based on the first action information and the simulation time period corresponding to the first action information to obtain the first state information.

[0068] In this embodiment, the reward function includes a main line reward function and a branch line reward function.

[0069] Determining a target reward value based on the first action information, the first state information, and the reward function includes: A first reward value is determined based on the first action information, the first state information, and the branch reward function.

[0070] A second reward value is determined based on the first action information, the first state information, and the main line reward function.

[0071] The target reward value is determined based on the first reward value and the second reward value.

[0072] In this embodiment, the reward function includes a main line reward function and a branch line reward function. The main line reward function is used to calculate the second reward value. The branch line reward function is used to calculate the first reward value.

[0073] Exemplarily, the first reward value is used to calculate at each step of the decision sequence to evaluate the suitability of the selected vehicle (i.e., the first action information) in the current decision step. The first reward value may include two first sub-reward values, which are the vehicle type sub-reward and the vehicle crossing time sub-reward. For example, if the vehicle selected at the current departure time is a new vehicle, it means that a new vehicle is dispatched at the current departure time, increasing the number of vehicles used, so the corresponding vehicle type sub-reward is set to -1, and in other cases, the vehicle type sub-reward is 0.

[0074] The second reward value is used to calculate the last step of the entire decision sequence. It defaults to 0 in other decision steps. The second reward value is used to evaluate the entire scheduling plan. The second reward value can include two second sub-reward values: the total number of vehicles used and the total cross-line time. The two second sub-reward values ​​are weighted to construct the second reward value (i.e. the main line reward). The corresponding weights can be set to 4 for the total number of vehicles used and 0.01 for the total cross-line time.

[0075] The final target reward value may be determined based on the first reward value and the second reward value. The target reward value is an evaluation score of the first action information and the first state information.

[0076] This embodiment calculates the reward values ​​of different stages and dimensions through the main line reward function and the branch line reward function, which can comprehensively evaluate the vehicle scheduling action. The branch line reward function evaluates the suitability of the selected vehicle at each decision step, prompting the agent to consider factors such as vehicle type and cross-line time every time it selects a vehicle, avoiding unreasonable vehicle selection. The main line reward function evaluates the entire scheduling plan at the end of the decision sequence, paying attention to the total number of vehicles used and the total cross-line time, and guiding the agent to optimize the overall scheduling strategy to reduce costs and improve efficiency. Combining the two to determine the target reward value can effectively train the agent to generate a more scientific and reasonable vehicle scheduling plan, thereby improving operational efficiency and resource utilization.

[0077] like Figure 2 As shown, in one embodiment of the present disclosure, the multi-line vehicle intelligent scheduling method further includes: A generator is constructed based on the simulation environment, and first information is generated based on the generator.

[0078] The first information is input into the reinforcement learning agent, and a first meta-game strategy corresponding to the generator and a second meta-game strategy corresponding to the reinforcement learning agent are determined based on the meta-game matrix. The meta-game matrix includes the meta-game strategies corresponding to the generator in the row direction, and the meta-game matrix includes the meta-game strategies corresponding to the reinforcement learning agent in the column direction. The elements in the meta-game matrix represent the benefits obtained by the generator and the reinforcement learning agent under different meta-game strategy combinations.

[0079] A target instance generation strategy is determined based on the first meta-game strategy and the second meta-game strategy, and a generator is updated based on the target instance generation strategy.

[0080] The target agent strategy is determined based on the first-element game strategy and the second-element game strategy, and the reinforcement learning agent is updated based on the target agent strategy.

[0081] In this embodiment, generating the first information based on the generator includes: New time point information is generated based on the generator and the original timetable, and the target timetable is determined based on the new time point information and the original timetable.

[0082] First information is generated based on the target schedule and the simulation environment.

[0083] In this embodiment, the generator is an instance generator, which is used to generate instances (i.e., the first information). In order to improve the robustness of the trained reinforcement learning agent (i.e., the ability of the reinforcement learning agent to handle various complex problem instances), the intelligent scheduling problem is modeled as a two-person zero-sum game problem, in which one party is an instance generator, with the goal of generating more complex and difficult instances. Different instances can be constructed by initializing different departure schedules based on the simulation environment. The instance generator can generate information such as the number of newly added departure time points and specific departure time points. The number of newly added departure time points can be constructed based on learnable vectors, and the specific departure time point information can be constructed based on a generative model. The other party is a reinforcement learning agent, with the goal of better solving the instances generated by the instance generator. The instance generator and the reinforcement learning agent are trained based on the self-game method, confront each other, and co-evolve to obtain a reinforcement learning agent that can handle various problem instances.

[0084] For example, in a simulation environment, different instances correspond to different departure schedules, and each departure time point in the departure schedule includes two parts of information: line number and time. In order to ensure the rationality of the instance generated by the instance generator, an original departure schedule (i.e., original schedule) can be constructed based on historical real data, and a new instance can be obtained by adding the generated new departure time point to the original departure schedule.

[0085] Exemplarily, the instance generator may include a time point number generator and a departure time point generator. The time point number generator may generate the number of newly added departure time points N, and the departure time point generator may generate specific information of N departure time points (each departure time point includes two elements: line number and time). The time point number generator may be constructed as a learnable probability vector (The corresponding parameters are ), each element in the corresponding probability vector is the probability corresponding to the number of time points that can be increased. Departure time point generator (The corresponding parameters are ) can be generated by common generative models, such as Variational Auto - Encoder (VAE), Generative Adversarial Network (GAN) and Flow -based Generative Model (Flow-based Model).

[0086] For example, taking the Flow-based Model as an example, the details are as follows: The Flow-based Model uses a reversible neural network transformation to map complex high-dimensional data to a simple distribution (such as a Gaussian distribution), thereby achieving accurate probability estimation and generation. The complex high-dimensional data can correspond to the departure time point, and this method can be used to obtain the probability corresponding to each departure time point.

[0087] Self-game training methods can include Vanilla Self-play (VSP), Fictitious Self-play (FSP), and Policy Space Response Oracle (PSRO). Taking PSRO as an example, the details are as follows: PSRO maintains a strategy set for each party. For the instance generator, each strategy in the strategy set corresponds to each instance generator, which is responsible for generating instances. For the reinforcement learning agent, each strategy in the strategy set corresponds to each reinforcement learning agent, which is responsible for solving instances. The overall process of PSRO is divided into three steps: simulation, meta-game solution, and optimal response solution. The three steps are repeated alternately. After each cycle is completed, each party will learn a new strategy and add it to its corresponding strategy set.

[0088] The purpose of the simulation is to construct a meta-game matrix, where the horizontal and vertical axes of the matrix correspond to each instance generator and each reinforcement learning agent, respectively. The elements in the meta-game matrix represent the performance of the corresponding reinforcement learning agent in the corresponding instance generator. The instance generator generates several instances, and the reinforcement learning agent S runs in instance I to obtain the corresponding solution. The performance of the corresponding solution can be expressed as Performance(S(I)). Based on the exact algorithm running on the same instance, the performance of the obtained solution can be expressed as Performance(Oracle(I)). The calculation formula for the final performance of the reinforcement learning agent in this instance can be expressed as:

[0089] The reinforcement learning agent executes the calculation of the calculation formula in all instances generated by the instance generator, calculates the average value, and fills the final average value into the corresponding position in the matrix.

[0090] After obtaining the meta-game matrix, the meta-game strategy is solved according to the meta-game matrix. The meta-game strategy corresponds to the weights corresponding to each strategy in the strategy set.

[0091] After obtaining the meta-game strategy of one party, the optimal response is solved for the meta-game strategy of the other party. The optimal response corresponds to the new strategy added to the strategy set.

[0092] For the new reinforcement learning agent to be trained, it is trained according to the meta-game strategy corresponding to the instance generator. Specifically, in each decision sequence, according to the meta-game strategy of the instance generator Sampling an instance generator , which contains a generator for the number of new time points and a time point generator The former is responsible for generating the number of time points that need to be added, and the latter is responsible for generating specific time points. The sampling requires the number of newly added departure time points N, and then according to Generate the corresponding number of departure time points and added to the original departure schedule), and then the reinforcement learning agent performs data sampling and policy updates in the new instance. The objective function of the reinforcement learning agent The corresponding policy gradient update formula is as follows:

[0093] in, is the objective function of the reinforcement learning agent, are the parameters corresponding to the reinforcement learning agent, Indicates Compute the gradient of the variable with respect to the objective function, Represents the meta-game strategy from the instance generator Medium Sampling Instance Generator And ask for hope, Represents the generator of the number of newly added time points The number of sampling time points N and the expectation, Represents the generator from the time point Sample N departure times and find the expected value. Indicates The probability of the solution obtained by the reinforcement learning agent S in the new instance sampled is Calculate the gradient, Represents the performance score of the reinforcement learning agent S in the new instance sampled.

[0094] For the new instance generator to be trained , which trains the meta-game strategy corresponding to the reinforcement learning agent. Specifically, a new instance is generated according to the instance generator (first according to The sampling requires the number of newly added departure time points N, and then according to Generate the corresponding number of departure time points and added to the original departure schedule), according to the meta-game strategy of the reinforcement learning agent Sample a reinforcement learning agent S, then the instance generator Construct an objective function based on the performance of the reinforcement learning agent S in the instances it generates. , updated using the policy gradient method.

[0095] The corresponding parameters are , time point generator The update formula is:

[0096] in, is the objective function of the instance generator, are the parameters of the time point generator, Indicates Compute the gradient of the variable with respect to the objective function, Represents the meta-game strategy from the reinforcement learning agent Sampling the reinforcement learning agent S and finding the expectation, Represents the generator of the number of newly added time points The number of sampling time points N and the expectation, Represents the generator from the time point Sample N departure times and find the expected value. Indicates Calculate the gradient of the joint probability corresponding to the variables sampled at each new time point, Represents the performance score of the reinforcement learning agent S in the new instance sampled.

[0097] The corresponding parameters are , add a generator for the number of time points The update formula is:

[0098] in, is the objective function of the instance generator, Add parameters for the generator of the number of new time points. Indicates Compute the gradient of the variable with respect to the objective function, Represents the meta-game strategy from the reinforcement learning agent Sampling the reinforcement learning agent S and finding the expectation, Represents the generator of the number of newly added time points The number of sampling time points N and the expectation, Represents the generator from the time point Sample N departure times and find the expected value. Indicates Calculate the gradient of the variable for the probability of adding N new time points, Represents the performance score of the reinforcement learning agent S in the new instance sampled.

[0099] This embodiment takes into account that the existing technology can only handle a specific problem instance. When encountering a new problem instance, it is often necessary to re-solve it. This embodiment uses a reinforcement learning method and combines self-game training to solve a reinforcement learning agent that can handle various instances. It avoids the tediousness of re-solving new problem instances, significantly improves the versatility and scalability of the solution, and provides a powerful and flexible solution for the intelligent management of multi-line vehicle operations.

[0100] Corresponding to the multi-line vehicle intelligent scheduling method of the above embodiment, Figure 4 This is a structural block diagram of a multi-line vehicle intelligent scheduling system provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The multi-line vehicle intelligent scheduling system 30 includes: an intelligent agent construction module 31, an intelligent agent training module 32 and an intelligent scheduling module 33.

[0101] The intelligent agent construction module 31 is used to construct a feature extraction network, which is used to extract the feature vector of the first information. A vehicle selection network is constructed, which is used to select vehicles for scheduling. A simulation environment is constructed, which is used to simulate the operation information of the vehicle. An intelligent agent is constructed based on the feature extraction network and the vehicle selection network.

[0102] The agent training module 32 is used to generate first information based on the simulation environment, and train the agent based on the first information and the reward function to obtain a reinforcement learning agent. The first information is the historical operation information of the simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information.

[0103] The intelligent scheduling module 33 is used to input the second information into the reinforcement learning agent to obtain a target scheduling plan, where the second information is multi-line vehicle information.

[0104] In one embodiment of the present disclosure, the first information includes vehicle information, control point information, route information and schedule information. The agent construction module 31 is specifically used to construct a vehicle embedding vector based on the vehicle information, a control point embedding vector based on the control point information, a route embedding vector based on the route information, and a schedule embedding vector based on the schedule information.

[0105] A feature extraction network is obtained based on the vehicle embedding vector, the control point embedding vector, the route embedding vector and the timetable embedding vector.

[0106] In one embodiment of the present disclosure, the agent construction module 31 is further used to determine a target embedding vector based on a vehicle embedding vector, a control point embedding vector, a route embedding vector and a timetable embedding vector.

[0107] A multi-line vehicle embedding vector is obtained based on the multi-line vehicle vector and the fully connected network.

[0108] The vehicle selection network is obtained based on the multi-line vehicle embedding vector and the target embedding vector.

[0109] In one embodiment of the present disclosure, the agent training module 32 is specifically used to input the first information into the agent to obtain the first action information, and determine the first state information based on the first action information. The first action information is the vehicle scheduling action, and the first state information is the updated vehicle operation information obtained after executing the first action information.

[0110] A target reward value is determined based on the first action information, the first state information, and the reward function.

[0111] The agent is trained based on the first state information and the target reward value to obtain a reinforcement learning agent.

[0112] In one embodiment of the present disclosure, the agent training module 32 is further specifically configured to determine preset state information based on the first information.

[0113] The preset state information is input into the agent to obtain the first action information.

[0114] The preset state information is updated based on the first action information and the simulation time period corresponding to the first action information to obtain the first state information.

[0115] In one embodiment of the present disclosure, the agent training module 32 is further configured to determine a first reward value based on the first action information, the first state information and the branch reward function.

[0116] A second reward value is determined based on the first action information, the first state information, and the main line reward function.

[0117] The target reward value is determined based on the first reward value and the second reward value.

[0118] In one embodiment of the present disclosure, the multi-line vehicle intelligent scheduling system 30 further includes: a self-game module, which is used to construct a generator based on a simulation environment and generate first information based on the generator.

[0119] The first information is input into the reinforcement learning agent, and a first meta-game strategy corresponding to the generator and a second meta-game strategy corresponding to the reinforcement learning agent are determined based on the meta-game matrix. The meta-game matrix includes the meta-game strategies corresponding to the generator in the row direction, and the meta-game matrix includes the meta-game strategies corresponding to the reinforcement learning agent in the column direction. The elements in the meta-game matrix represent the benefits obtained by the generator and the reinforcement learning agent under different meta-game strategy combinations.

[0120] A target instance generation strategy is determined based on the first meta-game strategy and the second meta-game strategy, and a generator is updated based on the target instance generation strategy.

[0121] The target agent strategy is determined based on the first-element game strategy and the second-element game strategy, and the reinforcement learning agent is updated based on the target agent strategy.

[0122] See also Figure 5 , Figure 5 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 5 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned system embodiments, such as Figure 4 The functions of the modules 31 to 33 are shown.

[0123] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0124] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0125] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0126] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the multi-line vehicle intelligent scheduling method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device 300 described in the embodiments of the present disclosure, which will not be repeated here.

[0127] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or system capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0128] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0129] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0133] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A multi-line vehicle intelligent scheduling method, characterized in that: include: Constructing a feature extraction network, wherein the feature extraction network is used to extract a feature vector of the first information; Constructing a vehicle selection network, the vehicle selection network is used to select vehicles for scheduling; constructing a simulation environment, the simulation environment is used to simulate the operation information of the vehicle; constructing an intelligent agent based on the feature extraction network and the vehicle selection network; Generate the first information based on the simulation environment, and train the agent based on the first information and a reward function to obtain a reinforcement learning agent; wherein the first information is historical operation information of the simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information; The second information is input into the reinforcement learning agent to obtain a target scheduling plan, wherein the second information is multi-line vehicle information.

2. The multi-line vehicle intelligent scheduling method according to claim 1, characterized in that: The first information includes vehicle information, control point information, route information and timetable information; The constructing of the feature extraction network comprises: Construct a vehicle embedding vector based on vehicle information, construct a control point embedding vector based on control point information, construct a route embedding vector based on route information, and construct a timetable embedding vector based on timetable information; A feature extraction network is obtained based on the vehicle embedding vector, the control point embedding vector, the route embedding vector and the timetable embedding vector.

3. The multi-line vehicle intelligent scheduling method according to claim 2, characterized in that: The constructing of the vehicle selection network comprises: Determine a target embedding vector based on the vehicle embedding vector, the control point embedding vector, the route embedding vector and the schedule embedding vector; Obtain multi-line vehicle embedding vectors based on multi-line vehicle vectors and a fully connected network; A vehicle selection network is obtained based on the multi-line vehicle embedding vector and the target embedding vector.

4. The multi-line vehicle intelligent scheduling method according to claim 1, characterized in that: The step of training the agent based on the first information and the reward function to obtain a reinforcement learning agent includes: Inputting the first information into the agent to obtain first action information, and determining first state information based on the first action information; wherein the first action information is a vehicle scheduling action, and the first state information is updated vehicle operation information obtained after executing the first action information; determining a target reward value based on the first action information, the first state information, and the reward function; The agent is trained based on the first state information and the target reward value to obtain a reinforcement learning agent.

5. The multi-line vehicle intelligent scheduling method according to claim 4, characterized in that: The step of inputting the first information into the agent to obtain first action information, and determining first state information based on the first action information includes: determining preset status information based on the first information; Inputting the preset state information into the agent to obtain first action information; The preset state information is updated based on the first action information and the simulation time period corresponding to the first action information to obtain first state information.

6. The multi-line vehicle intelligent scheduling method according to claim 4, characterized in that: The reward function includes a main line reward function and a branch line reward function; The determining a target reward value based on the first action information, the first state information and the reward function comprises: determining a first reward value based on the first action information, the first state information, and the branch reward function; determining a second reward value based on the first action information, the first state information, and the main line reward function; A target reward value is determined based on the first reward value and the second reward value.

7. The multi-line vehicle intelligent scheduling method according to claim 1, characterized in that: Also includes: Building a generator based on the simulation environment; generating the first information based on the generator; Inputting the first information into the reinforcement learning agent, and determining a first meta-game strategy corresponding to the generator and a second meta-game strategy corresponding to the reinforcement learning agent based on a meta-game matrix; The meta-game matrix includes the meta-game strategies corresponding to the generator in the row direction, and includes the meta-game strategies corresponding to the reinforcement learning agent in the column direction, and the elements in the meta-game matrix represent the benefits obtained by the generator and the reinforcement learning agent under different meta-game strategy combinations; determining a target instance generation strategy based on the first meta-game strategy and the second meta-game strategy, and updating the generator based on the target instance generation strategy; A target agent strategy is determined based on the first meta-game strategy and the second meta-game strategy, and the reinforcement learning agent is updated based on the target agent strategy.

8. A multi-line vehicle intelligent scheduling system, characterized in that: include: An intelligent agent building module, used to build a feature extraction network, wherein the feature extraction network is used to extract a feature vector of the first information; Constructing a vehicle selection network, the vehicle selection network is used to select vehicles for scheduling; constructing a simulation environment, the simulation environment is used to simulate the operation information of the vehicle; constructing an intelligent agent based on the feature extraction network and the vehicle selection network; An agent training module, used to generate first information based on the simulation environment, and train the agent based on the first information and a reward function to obtain a reinforcement learning agent; wherein the first information is historical operation information of the simulated vehicle, and the reward function is a reward value calculation function corresponding to the first information; The intelligent scheduling module is used to input second information into the reinforcement learning agent to obtain a target scheduling plan, wherein the second information is multi-line vehicle information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images