Vehicle scheduling model training method and device, equipment and storage medium

Abstract

The invention provides a vehicle scheduling model training method and device, equipment and a storage medium, and the method comprises the steps: carrying out the reinforcement learning based on initial state data, obtaining a first data tuple, carrying out the conversion processing of historical scheduling data, obtaining a second data tuple, and carrying out the training of a vehicle scheduling model. And determining loss information of the initial scheduling model according to the first data tuple and the second data tuple in the experience playback pool, and performing iterative optimization of the model to obtain a target scheduling model. According to the method provided by the invention, in a scene of large-scale vehicle scheduling, the generation speed of the vehicle scheduling scheme can be effectively improved, the intelligence and autonomy of the vehicle scheduling system can be realized, and the method has better universality and lower cost.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more particularly, to a method, apparatus, device, and storage medium for training a vehicle scheduling model. Background Art

[0002] Production scheduling is a key link in the manufacturing industry. There are a wide variety of materials required for production. In general, the factory warehouse adopts a mode of multi-warehouse collaborative storage and unified management for material management. When the order pulls the material demand, it is necessary to transport materials from each warehouse to the production factory.

[0003] In the prior art, vehicle scheduling can be performed based on an intelligent scheduling system or a scheduling algorithm. The main scheduling algorithms include mixed integer linear programming and heuristic algorithms. Among them, mixed integer linear programming transforms the scheduling problem into a linear programming problem through mathematical modeling and uses a solver to find the optimal solution to achieve the scheduling plan of the vehicle. The heuristic algorithm can give a feasible scheduling plan for the vehicle based on prior knowledge or experience.

[0004] However, the method of mixed integer linear programming has a high computational complexity and is not applicable to large-scale scheduling problems. The heuristic algorithm depends on prior knowledge and manual experience and needs to be specially developed and adjusted for scenarios and problems, resulting in high development costs and poor universality. Summary of the Invention

[0005] The purpose of the present application is to provide a method, apparatus, device, and storage medium for training a vehicle scheduling model to solve the problems of slow model calculation speed, high development costs, and poor universality in the prior art.

[0006] To achieve the above object, the technical solution adopted by the present application is as follows:

[0007] In a first aspect, the present application provides a method for training a vehicle scheduling model, the method comprising:

[0008] Obtain a plurality of initial state data, and input each initial state data into an initial scheduling model respectively to obtain action data, reward data, and next state data corresponding to each initial state data. The initial state data is used to represent information of the vehicle and information of the task to be scheduled. The information of the vehicle includes: the warehouse location of the current scheduling task of the vehicle, the remaining capacity of the vehicle, the end time of the current scheduling task of the vehicle, the path information of the vehicle, and the planned time information of the vehicle. The information of the task to be scheduled includes: the quantity of material required, the delivery time of the material required, and the warehouse information of the material. The action data is used to represent at least one vehicle for executing the task to be scheduled and the scheduling path of each vehicle;

[0009] Combine each initial state data, as well as the corresponding action data, reward data, and next state data of each initial state data, into a first data tuple respectively, and store each data tuple in an experience replay pool. The experience replay pool also includes multiple second data tuples, and each second data tuple is converted according to the historical scheduling data of the vehicle;

[0010] Extract training tuples from the experience replay pool, where the training tuples include first data tuples and / or second data tuples;

[0011] Input the training tuples into the initial scheduling model. The initial scheduling model determines loss information according to the training tuples, corrects the initial scheduling model according to the loss information, and regenerates the first data tuple by the initial scheduling model again. Iteratively execute until the initial scheduling model meets the preset conditions, and use the initial scheduling model that meets the preset conditions as the target scheduling model.

[0012] Optionally, the step of inputting each initial state data into the initial scheduling model to obtain the corresponding action data, reward data, and next state data of each initial state data includes:

[0013] Input the initial state data into the initial scheduling model, and the initial scheduling model determines at least one optional action based on the initial state data;

[0014] The initial scheduling model determines a target action from the at least one optional action based on a preset selection strategy;

[0015] The initial scheduling model takes the target action as the action data corresponding to the state data, takes the reward value of the target action as the reward data, and after executing the target action, takes the updated state data as the next state data.

[0016] Optionally, the step of the initial scheduling model determining a target action from the at least one optional action based on a preset selection strategy includes:

[0017] The initial scheduling model determines the value of each optional action according to the reward value of each optional action;

[0018] The initial scheduling model selects the optional action with the largest value from each optional action as the target action based on a greedy strategy, or randomly selects an optional action from each optional action as the target action based on a random selection strategy.

[0019] Optionally, the process of converting the second data tuple according to the historical scheduling data of the vehicle includes:

[0020] Traverse the path information of the vehicles in the historical scheduling data, and convert the scheduling vehicles at each node and the order in which the scheduling vehicles visit the nodes into action data;

[0021] Generate state data corresponding to the action data according to the information of the tasks to be scheduled at each node and the information of the vehicles;

[0022] Use the state data of the next node of each node as the next state data, and determine the reward data of each node according to the next state data of each node;

[0023] Take the action data, the state data, the next state data, and the reward data as a second data tuple.

[0024] Optionally, the initial scheduling model determines loss information according to the training tuple, including:

[0025] The initial scheduling model determines the predicted value of each training tuple according to the action data, the reward data, and the state data in the training tuple;

[0026] The initial scheduling model determines the value loss value of each training tuple according to the target value corresponding to the next state data in the training tuple and the predicted value of each training tuple;

[0027] The initial scheduling model determines the loss information according to the value loss value of each training tuple.

[0028] Optionally, after obtaining the target scheduling model, the method further includes:

[0029] Obtain the information of the schedulable vehicles and the task information of the target task. The information of the schedulable vehicles includes: the warehouse location of the vehicle's current scheduling task, the remaining capacity of the vehicle, the end time of the vehicle's current scheduling task, the path information of the vehicle, and the planned time information of the vehicle. The task information of the task to be scheduled includes: the quantity of material requirements, the delivery time of the material requirements, and the warehouse information of the material;

[0030] Input the information of the schedulable vehicles and the task information into the target scheduling model to obtain the target scheduling plan for the task to be scheduled. The target scheduling plan includes: at least one target vehicle for the task to be scheduled, and the scheduling paths of each target vehicle.

[0031] Optionally, the method further includes:

[0032] After executing the target scheduling plan, obtain the actual scheduling result of the target scheduling plan, and correct the target scheduling model based on the actual scheduling result.

[0033] In a second aspect, the present application provides a vehicle scheduling model training device, the device comprising:

[0034] An acquisition module, configured to acquire a plurality of initial state data, and input each of the initial state data into an initial scheduling model respectively, to obtain action data, reward data, and next state data corresponding to each of the initial state data. The initial state data is used to characterize information of a vehicle and information of a to-be-scheduled task. The information of the vehicle includes: the warehouse location of the vehicle's current scheduling task, the remaining capacity of the vehicle, the end time of the vehicle's current scheduling task, the path information of the vehicle, and the planned time information of the vehicle. The information of the to-be-scheduled task includes: the quantity of material required, the delivery time of the material required, and the warehouse information of the material. The action data is used to characterize at least one vehicle for executing the to-be-scheduled task and the scheduling paths of each vehicle;

[0035] A storage module, configured to combine each of the initial state data and the action data, reward data, and next state data corresponding to each of the initial state data into a first data tuple respectively, and store each data tuple into an experience replay pool. The experience replay pool further includes a plurality of second data tuples, and each second data tuple is converted according to the historical scheduling data of the vehicle;

[0036] An extraction module, configured to extract training tuples from the experience replay pool, where the training tuples include first data tuples and / or second data tuples;

[0037] A training module, configured to input the training tuples into the initial scheduling model, and the initial scheduling model determines loss information according to the training tuples, and corrects the initial scheduling model according to the loss information, and regenerates first data tuples by the initial scheduling model again, and iteratively execute until the initial scheduling model meets a preset condition, and use the initial scheduling model that meets the preset condition as the target scheduling model.

[0038] Optionally, the training module is configured to:

[0039] Input the initial state data into the initial scheduling model, and the initial scheduling model determines at least one optional action based on the initial state data;

[0040] Determine a target action among the at least one optional action based on a preset selection strategy;

[0041] Use the target action as the action data corresponding to the state data, use the reward value of the target action as the reward data, and after executing the target action, use the updated state data as the next state data.

[0042] Optionally, the training module is configured to:

[0043] Determine the value of each of the optional actions according to the reward value of each of the optional actions;

[0044] Select the optional action with the maximum value from each of the optional actions as the target action based on a greedy strategy, or randomly select an optional action from each of the optional actions as the target action based on a random selection strategy.

[0045] Optionally, the obtaining module is further configured to:

[0046] Traverse the path information of the vehicles in the historical scheduling data, and convert the scheduling vehicles of each node and the order in which the scheduling vehicles visit the nodes into action data;

[0047] Generate state data corresponding to the action data according to the information of the tasks to be scheduled for each node and the information of the vehicles;

[0048] Use the state data of the next node of each node as the next state data, and determine the reward data of each node according to the next state data of each node;

[0049] Take the action data, the state data, the next state data, and the reward data as a second data tuple.

[0050] Optionally, the training module is configured to:

[0051] Determine the predicted value of each of the training tuples according to the action data, the reward data, and the state data in the training tuple;

[0052] Determine the value loss value of each of the training tuples according to the target value corresponding to the next state data in the training tuple and the predicted value of each of the training tuples;

[0053] Determine the loss information according to the value loss value of each of the training tuples.

[0054] Optionally, the obtaining module is further configured to:

[0055] Obtain the information of the schedulable vehicles and the task information of the target task. The information of the schedulable vehicles includes: the warehouse location of the current scheduling task of the vehicle, the remaining capacity of the vehicle, the end time of the current scheduling task of the vehicle, the path information of the vehicle, and the planned time information of the vehicle. The task information of the task to be scheduled includes: the quantity of material required, the delivery time of the material required, and the warehouse information of the material;

[0056] The device further includes a scheduling module, configured to input the schedulable vehicle information and the task information into the target scheduling model to obtain a target scheduling plan for the to-be-scheduled task, where the target scheduling plan includes: at least one target vehicle for the to-be-scheduled task, and a scheduling path for each target vehicle.

[0057] Optionally, the device further includes a correction module, configured to:

[0058] After executing the target scheduling plan, obtain the actual scheduling result of the target scheduling plan, and correct the target scheduling model based on the actual scheduling result.

[0059] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of a vehicle scheduling model training method as described in any one of the first aspects.

[0060] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it performs the steps of a vehicle scheduling model training method as described in any one of the first aspects.

[0061] The beneficial effects of the present application are as follows: By constructing the second data tuple based on historical data and generating the first data tuple through reinforcement learning, the diversity and quantity of training data can be improved. Moreover, in the training phase, by learning the vehicle scheduling rules, the present application can automatically explore and select the optimal scheduling strategy, realizing the intelligence and autonomy of the vehicle scheduling system. When performing vehicle scheduling based on the target scheduling model of the present application, compared with the mixed integer programming method and heuristic calculation in the prior art, the calculation speed of the present application is greatly improved, so that the vehicle scheduling system can process a larger number of order delivery data, be applicable to the scenario of large-scale vehicle scheduling, and provide better support for recalculation of data changes.

[0062] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0064] Figure 1 The flowchart of a vehicle scheduling model training method provided by an embodiment of the present application is shown;

[0065] Figure 2 The flowchart of an initial scheduling model predicting state data provided by an embodiment of the present application is shown;

[0066] Figure 3 The flowchart of determining a target action provided by an embodiment of the present application is shown;

[0067] Figure 4 The flowchart of generating a second data tuple provided by an embodiment of the present application is shown;

[0068] Figure 5 The flowchart of determining loss information provided by an embodiment of the present application is shown;

[0069] Figure 6 The flowchart of vehicle scheduling based on a target scheduling model provided by an embodiment of the present application is shown;

[0070] Figure 7 The structural schematic diagram of a vehicle scheduling model training device provided by an embodiment of the present application is shown;

[0071] Figure 8 The structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. Detailed implementation manners

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0073] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the features stated thereafter, but does not exclude the addition of other features.

[0074] In large-scale factory production, a large number of materials are required. In many scenarios, a multi-warehouse collaborative storage and unified management model is adopted. When production is needed, materials need to be transported from each supporting warehouse to the production factory. In the prior art, multi-vehicle scheduling can generally be carried out through an intelligent scheduling system.

[0075] In the scheduling system, a vehicle group may need to access multiple warehouses to pick up materials. A certain warehouse may also require multiple vehicles to deliver materials simultaneously. And since the delivery time of materials is closely related to the production plan, it is also necessary to ensure the timeliness of material delivery.

[0076] In the current scheduling system, vehicle scheduling is generally implemented through a mixed-integer linear programming algorithm or a heuristic algorithm. Among them, the mixed-integer linear programming algorithm can convert the scheduling problem into a linear programming problem, by minimizing or maximizing a certain index in the objective function, such as a time index, and restricting the value range of variables such as vehicles and tasks in the constraint conditions, so as to find the optimal vehicle scheduling plan.

[0077] However, this method requires a large amount of calculation time, especially for large-scale scheduling problems, there is a problem of too long calculation time, so it will seriously affect the efficiency of vehicle scheduling.

[0078] The heuristic algorithm refers to developing or adjusting an algorithm based on expert experience or prior knowledge for the vehicle scheduling scenario. When performing vehicle scheduling, it is necessary to combine heuristic search logic for global search to generate a vehicle scheduling plan.

[0079] However, the heuristic algorithm needs to be specifically developed for a specific scenario, and during the process of vehicle scheduling, it also takes several minutes to dozens of minutes of calculation time, and there is still a problem of relatively long calculation time. And the heuristic algorithm needs to specifically write a specific heuristic search logic for the specific business of the application scenario, which also increases the development cost.

[0080] In summary, the scheduling algorithms in the current intelligent scheduling system have problems of long calculation time and high development cost.

[0081] Based on this, the present application proposes a method for training a vehicle scheduling model. By improving the vehicle scheduling model based on the DQN algorithm, not only the time for generating a vehicle scheduling plan is shortened, but also an automated decision-making process can be realized, and there is no need to specifically develop a search logic for specific business, which greatly reduces the development cost.

[0082] Next, in combination with Figure 1A vehicle scheduling model training method of the present application will be described. The vehicle scheduling model is a model for scheduling vehicles for production material distribution. This method can be applied to an intelligent scheduling system, and the execution entity can be, for example, an electronic device, such as Figure 1 As shown, the method includes:

[0083] S101. Obtain a plurality of initial state data, and respectively input each initial state data into an initial scheduling model to obtain action data, reward data, and next state data corresponding to each initial state data. The initial state data is used to characterize the information of the vehicle and the information of the task to be scheduled. The information of the vehicle includes: the warehouse location of the vehicle's current scheduling task, the remaining capacity of the vehicle, the end time of the vehicle's current scheduling task, the path information of the vehicle, and the planned time information of the vehicle. The information of the task to be scheduled includes: the material demand quantity, the material demand delivery time, and the warehouse information to which the material belongs. The action data is used to characterize at least one vehicle for executing the task to be scheduled and the scheduling paths of each vehicle.

[0084] Before training the initial scheduling model, environmental modeling can be performed first to define concepts such as state data, action data, and reward data in the vehicle scheduling scenario. The state data can characterize the information of all current vehicles and the information of the task to be scheduled. The information of the vehicle can be the information of schedulable vehicles and vehicles being scheduled.

[0085] Exemplarily, the information of the vehicle can be shown in Table 1 below. Table 1 shows a scenario where the vehicle departs from the factory to each warehouse for loading materials and finally returns to the factory for unloading. Table 1 includes information such as the order and planned time for the vehicle to depart from the factory to multiple warehouses for loading materials. Among them, the path node can be the warehouse passed through in the vehicle scheduling path. The scheduling task can characterize the warehouse location passed through in the scheduling path and the access order of each warehouse, and indicate the time to reach each warehouse and the time to leave each warehouse. And the scheduling task can also indicate the remaining capacity of the vehicle, as well as the type and quantity of materials to be scheduled in each warehouse.

[0086] Table 1 Example of Vehicle Information

[0087]

[0088]

[0089] Among them, the current vehicle scheduling task can be the scheduling task that the vehicle is currently executing or the to-be-executed scheduling task that has been assigned to the vehicle. The warehouse location of the current vehicle scheduling task can be the unique number of the warehouse where the vehicle is located when executing the current scheduling task. The remaining vehicle capacity is used to indicate the amount of materials that the vehicle can still load. The end time of the current vehicle scheduling task can be the estimated time after the vehicle finishes executing the current scheduling task, and this end time is constrained by the material demand delivery time, that is, the end time should be before the material demand delivery time. The path information of the vehicle can be the warehouses that the vehicle needs to reach on the scheduling path and the order of each warehouse. The planned time information of the vehicle includes the planned arrival time of the vehicle and the planned departure time of the vehicle.

[0090] It should be noted that after the initial environment is modeled, the travel time of the vehicle between each warehouse can be roughly estimated, and the loading time of the vehicle can be preset or estimated. Therefore, based on the path information of the vehicle, the planned time information for the vehicle to reach each warehouse node can be determined, and the time when the vehicle finally returns to the factory can be used as the end time of the vehicle scheduling task.

[0091] Optionally, the initial scheduling model can simulate the vehicle scheduling process based on the initial state data for data sampling to generate action data, reward data, and next state data corresponding to the initial state data, so as to increase the number of available samples.

[0092] It should be understood that the process of data sampling based on the initial scheduling model can be iteratively executed during the process of training the vehicle scheduling model. As the number of training times increases, the generated initial state data can be closer to the vehicle scheduling situation in the real scenario.

[0093] Optionally, the action data is used to represent the vehicle executing the to-be-scheduled task and the scheduling path of the vehicle. The scheduling path of the vehicle includes the warehouses visited by the vehicle when executing the scheduling task and the order in which the vehicle visits each warehouse.

[0094] The information of the to-be-scheduled task includes: the material demand quantity, the material demand delivery time, and the warehouse information of the material. The material demand quantity can be the quantity or weight of the required material. Exemplarily, the to-be-scheduled task can be: "Warehouse A - Material M01 - 1t, Warehouse B - Material M03 - 2t, 2025.02.09.14:00", which means that before 14:00 on February 9, 2025, 1 ton of Material M01 needs to be scheduled from Warehouse A and 2 tons of Material M03 need to be scheduled from Warehouse B to the factory.

[0095] Optionally, the reward data is used to indicate the effect of the current action. When the current action has a positive effect on vehicle scheduling, the reward value is greater; when the current action has a negative effect on vehicle scheduling, the reward value is smaller. The reward data may include: vehicle path reward data, vehicle time reward data, and vehicle load reward data.

[0096] Among them, if after executing the action data, the warehouse node successfully joins the vehicle scheduling path, a vehicle path reward value can be obtained. The vehicle path reward value can be the opposite of the increased path length after the warehouse node joins the vehicle scheduling path. Exemplarily, assume that node n joins between nodes i and j of path v, and the increased path length is L1 = iv + vj - ij. Since the smaller the increased path length, the better the result, the reward takes the negative value of the increased length L1 of path v, and the node represents the warehouse.

[0097] After executing the action data, due to the change in the vehicle path, the path end time of the vehicle will also be updated accordingly. The path end time of the vehicle can be compared with the demand delivery time. If the path end time exceeds the demand delivery time, the vehicle time reward data can be the opposite of the exceeded time. Among them, if the scheduling task is completed by multiple vehicles together, the latest path end time of all vehicles can be compared with the demand delivery time, and the vehicle time reward data can be obtained. Among them, the weight coefficient of the vehicle time reward data can be determined based on the impact degree of the material on production. For example, a larger weight coefficient is set for critical materials. If the path end time of critical materials exceeds the demand delivery time, a larger weight coefficient can be multiplied by the vehicle time reward data, so that the model can improve the timeliness of critical material delivery in the subsequent scheduling process.

[0098] The vehicle load reward data can be a penalty when the vehicle exceeds the restricted load. If the vehicle load is exceeded, it means that the scheduling plan is infeasible, and a large negative value can be used as the vehicle load reward.

[0099] S102: Combine each initial state data, the corresponding action data, reward data, and next state data of each initial state data into a first data tuple respectively, and store each data tuple in the experience replay pool. The experience replay pool also includes multiple second data tuples, and each second data tuple is converted according to the historical scheduling data of the vehicle.

[0100] It should be noted that the initial state data includes unscheduled task information and schedulable vehicle information. The initial scheduling model can assign vehicles to unscheduled tasks, that is, determine the action data and record the reward data. After each execution of the action data, the updated unscheduled task information and schedulable vehicle information are recorded as the next state data. The initial state data, action data, reward data, and next state data of this time are recorded as a first data tuple. Then, the next state data is used as the new initial state data, and the new initial state data is re - input into the initial scheduling model to obtain the action data, reward data, and next state data corresponding to the new initial state data, and they are recorded as a first data tuple. When the unscheduled task is completed, that is, the last node of the unscheduled task is the target node, or all the loaded materials of the unscheduled task have been assigned vehicles, the scheduling of the unscheduled task is completed.

[0101] As a possible implementation, during the execution of the unscheduled task, the action data, reward data, next state data, and initial state data obtained each time can be assembled to obtain a first data tuple and recorded.

[0102] Optionally, the experience replay pool is used to store the first data tuples generated by reinforcement learning sampling and the second data tuples obtained by converting historical scheduling data. The formats of the first data tuple and the second data tuple can be expressed as (s t ,a t ,r t ,s t+1 ), where s t represents the initial state data, a t represents the action data, r t represents the reward data, and s t+1 represents the next state data.

[0103] The sources of historical scheduling data can include: the historical scheduling data of the system and the manual scheduling data. The historical scheduling data includes: material distribution task information, vehicle allocation for tasks, and vehicle path data. Among them, the material distribution task information includes the quantity of material requirements, the delivery time of material requirements, and the warehouse to which the materials belong. By pre - processing and converting the historical scheduling data, multiple second data tuples can be obtained.

[0104] Optionally, the second data tuple includes: initial state data, action data, reward data, and next state data. It should be understood that the first data tuple and the second data tuple have different sources but the same form.

[0105] S103. Extract training tuples from the experience replay pool. The training tuples include the first data tuple and / or the second data tuple.

[0106] Optionally, some data can be randomly sampled from the experience replay pool as training tuples. For example, multiple first data tuples can be sampled, or multiple second data tuples can be sampled, or multiple first data tuples and second data tuples can be sampled.

[0107] S104. Input the training tuple into the initial scheduling model. The initial scheduling model determines the loss information based on the training tuple, corrects the initial scheduling model according to the loss information, and regenerates the first data tuple by the initial scheduling model again. Iterate until the initial scheduling model meets the preset conditions, and use the initial scheduling model that meets the preset conditions as the target scheduling model.

[0108] Optionally, the initial scheduling model can determine the loss information based on the initial state data, action data, reward data, and next state data in the training tuple, update the weight parameters of the initial scheduling model based on the loss information to obtain the updated initial scheduling model, and use the updated initial scheduling model as the initial scheduling model again, and repeat the above step of inputting the initial state data into the initial scheduling model until the initial scheduling model meets the preset conditions. It should be understood that the initial state data input into the initial scheduling model can be different in each iteration process.

[0109] It should be noted that an initial scheduling model can be copied and saved as the target model at the beginning. After the training of the initial scheduling model is completed, the weight parameters of the target model can be updated based on the trained initial scheduling model, and the updated target model is used as the target scheduling model.

[0110] In the embodiments of the present application, by constructing the second data tuple based on historical data and generating the first data tuple through reinforcement learning, the diversity and quantity of training data can be improved. Moreover, in the training stage, by learning the vehicle scheduling rules, the present application can automatically explore and select the optimal scheduling strategy to realize the intelligence and autonomy of the vehicle scheduling system. When performing vehicle scheduling based on the target scheduling model of the present application, compared with the mixed integer programming method and heuristic calculation in the prior art, the calculation speed of the present application is greatly improved, so that the vehicle scheduling system can process a larger number of order delivery data, be applicable to the scenario of large-scale vehicle scheduling, and provide better support for the recalculation of data changes.

[0111] The following is a further description of the above steps of inputting each initial state data into the initial scheduling model respectively to obtain the action data, reward data, and next state data corresponding to each initial state data. As Figure 2 shown, the above step S101 includes:

[0112] S201. Input the initial state data into the initial scheduling model, and the initial scheduling model determines at least one optional action based on the initial state data.

[0113] Among them, the optional actions include the vehicles that can be allocated in the current state and the access order of the warehouses in the paths of the allocated vehicles. After inputting the initial state data into the initial scheduling model, the initial scheduling model can, based on the pre-established environment model, determine the warehouse location to which the materials of the vehicles to be scheduled in the initial state data belong, determine the allocable vehicles based on the currently schedulable vehicles, and determine the order of the warehouses in the paths of the allocable vehicles.

[0114] S202. The initial scheduling model determines the target action from at least one optional action based on the preset selection strategy.

[0115] Optionally, the preset selection strategy may include a first strategy and a second strategy. In this application, an initial exploration rate can be set for the initial scheduling model, and the selection strategy is determined based on the initial exploration rate. Exemplarily, assuming the initial exploration rate is ε, it means that the initial scheduling model selects the first strategy with a probability of ε or selects the second strategy with a probability of 1 - ε. The first strategy can be to randomly select an action, and the second strategy can be to select the action with the maximum value. The action with the maximum value can be the action with the best scheduling effect in the current state, where the value can be the Q-value of the action.

[0116] S203. The initial scheduling model takes the target action as the action data corresponding to the state data, takes the reward value of the target action as the reward data, and after executing the target action, takes the updated state data as the next state data.

[0117] After the initial scheduling model selects the target action, it can take the target action as the action data, take the reward value of the target action as the reward data, and execute the target action. After the execution is completed, the information of the currently schedulable vehicles and the information of the tasks to be scheduled are updated, and the updated state data is taken as the next state data.

[0118] Optionally, when the initial scheduling model selects the target action, it can calculate the Q-value of each optional action, take the optional action with the maximum Q-value as the target action, and add the Q-value to the next state data as the target Q-value of the next state.

[0119] The process in which the initial scheduling model determines the target action from at least one optional action based on the preset selection strategy, as Figure 3 shown, includes:

[0120] S301. The initial scheduling model determines the value of each optional action according to the reward value of each optional action.

[0121] As a possible implementation, the value of an optional action can be the Q-value of the optional action, representing the long-term reward value that can be obtained in the next state after executing the action. The calculation method of the Q-value can be shown as the following formula (1), where r represents the reward value of the action, Q’(s’, a’) represents the maximum reward value under the current action sequence, and y is a set parameter.

[0122] Q = r + y * max Q’(s’, a’) (1)

[0123] S302. The initial scheduling model selects the optional action with the maximum value from each optional action as the target action based on the greedy strategy, or randomly selects an optional action from each optional action as the target action based on the random selection strategy.

[0124] Optionally, the initial scheduling model randomly selects one from each optional action as the target action with a probability of ε based on the initial exploration rate, and selects the one with the maximum Q-value as the target action with a probability of 1 - ε based on the greedy strategy.

[0125] The following is a further description of the process of converting the above-mentioned historical scheduling data of the vehicle into the second data tuple, as Figure 4 shown, this process includes:

[0126] S401. Traverse the path information of the vehicle in the historical scheduling data, and convert the scheduling vehicle of each node and the order of the vehicle accessing the node into action data.

[0127] Among them, the node can be a warehouse or a factory passed through in the path information. In a possible implementation, the nodes in the path can be traversed in reverse order. For the currently traversed node, the scheduling vehicle of the node and the sequential position of the node in the vehicle's scheduling path are converted into action data.

[0128] S402. Generate state data corresponding to the action data according to the information of the tasks to be scheduled for each node and the information of the vehicle.

[0129] For the currently traversed node, state data corresponding to the action data can be generated according to the untraversed nodes, the states of each node on the vehicle path, and the information of the vehicle.

[0130] S403. Use the state data of the next node of each node as the next state data, and determine the reward data of each node according to the next state data of each node.

[0131] Since the next node of each node in the historical scheduling data is a known node, the next state data of the current state data can be determined according to the order of the nodes in the vehicle path, and the reward data of each node can be determined according to the next state data.

[0132] It should be noted that since the next state data of each node is known information, the historical scheduling data can be re-evaluated. If the scheduling path has good effects, a relatively large positive reward value can be assigned to the reward data corresponding to the state data related to the scheduling path. If the scheduling path has poor effects, such as timeout problems, a relatively large negative reward value can be assigned to the reward value corresponding to the state data related to the scheduling path. The state data related to the scheduling path includes: the state data obtained by converting the nodes included in the scheduling path and the state data obtained by converting the vehicle to which the scheduling path belongs.

[0133] S404. Take the action data, state data, next state data, and reward data as a second data tuple.

[0134] Optionally, during the process of traversing each node, the action data, state data, and next state data can be converted for the currently traversed node, the reward data can be calculated, and then the action data, state data, next state data, and reward data of the currently traversed node are taken as a second data tuple and stored in the experience replay pool.

[0135] Furthermore, the process of the initial scheduling model determining the loss information according to the training tuple, as Figure 5 shown, includes:

[0136] S501. The initial scheduling model determines the predicted value of each training tuple according to the action data, reward data, and state data in the training tuple.

[0137] Among them, the predicted value can be the predicted Q value, and the predicted value can be calculated based on the above formula (1).

[0138] S502. The initial scheduling model determines the value loss value of each training tuple according to the target value corresponding to the next state data in the training tuple and the predicted value of each training tuple.

[0139] Optionally, the target value can be the actual value after executing the action, and the predicted value can be the value predicted by the model after executing the action. Therefore, the model can be optimized based on the target value and the predicted value. Exemplarily, the value loss value can be the difference between the target value and the predicted value.

[0140] S503. The initial scheduling model determines the loss information according to the value loss value of each training tuple.

[0141] In a possible implementation manner, the mean square error loss function can be used to calculate the value loss value of each training tuple to obtain the loss information.

[0142] The following is a further description of vehicle scheduling based on the obtained target scheduling model above, as Figure 6 shown, including:

[0143] S601. Obtain the information of schedulable vehicles and the task information of target tasks. The information of schedulable vehicles includes: the warehouse location of the vehicle's current scheduling task, the remaining capacity of the vehicle, the end time of the vehicle's current scheduling task, the path information of the vehicle, and the planned time information of the vehicle. The task information of the to-be-scheduled task includes: the quantity of material requirements, the delivery time of material requirements, and the warehouse information of the material.

[0144] It should be noted that at the beginning, the path information of the vehicle only includes the factory node at the starting point, the planned time information of the vehicle can be the planned departure time of the vehicle from the factory node, and the end time of the vehicle's current scheduling task can be empty.

[0145] S602. Input the information of schedulable vehicles and task information into the target scheduling model to obtain the target scheduling plan for the to-be-scheduled task. The target scheduling plan includes: at least one target vehicle for the to-be-scheduled task, and the scheduling path of each target vehicle.

[0146] During the execution of the target scheduling model, the information of schedulable vehicles and the task information of target tasks can be continuously updated, the action data is determined according to the currently input state data, and the reward data and the next state data are recorded after executing the action data. Then, the next state data is used as the new input information and the execution is repeated until the scheduling of all input tasks is completed.

[0147] Optionally, the method of the present application further includes:

[0148] After executing the target scheduling plan, obtain the actual scheduling result of the target scheduling plan, and correct the target scheduling model based on the actual scheduling result.

[0149] The actual scheduling result includes the information of the vehicle and the information of the scheduling task. In one possible implementation, the actual scheduling result can be used as historical scheduling data, and the actual scheduling result is converted into a second data tuple through the above steps S401 - S403, and then the target scheduling model is corrected based on the second data tuple.

[0150] Based on the same inventive concept, the present application embodiment also provides a vehicle scheduling model training device corresponding to the vehicle scheduling model training method. Since the principle of solving problems by the device in the present application embodiment is similar to the above vehicle scheduling model training method in the present application embodiment, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0151] Figure 7The figure shows a schematic structural diagram of a vehicle scheduling model training device provided by an embodiment of the present application.

[0152] An acquisition module 701, configured to acquire a plurality of initial state data, and input each of the initial state data into an initial scheduling model respectively, to obtain action data, reward data, and next state data corresponding to each of the initial state data. The initial state data is used to characterize information of a vehicle and information of a to-be-scheduled task. The information of the vehicle includes: the warehouse location of the vehicle's current scheduling task, the remaining capacity of the vehicle, the end time of the vehicle's current scheduling task, the path information of the vehicle, and the planned time information of the vehicle. The information of the to-be-scheduled task includes: the quantity of material required, the delivery time of the material required, and the warehouse information of the material. The action data is used to characterize at least one vehicle for executing the to-be-scheduled task and the scheduling paths of each vehicle;

[0153] A storage module 702, configured to combine each of the initial state data and the action data, reward data, and next state data corresponding to each of the initial state data into a first data tuple respectively, and store each data tuple into an experience replay pool. The experience replay pool further includes a plurality of second data tuples, and each of the second data tuples is converted according to the historical scheduling data of the vehicle;

[0154] An extraction module 703, configured to extract training tuples from the experience replay pool. The training tuples include first data tuples and / or second data tuples;

[0155] A training module 704, configured to input the training tuples into the initial scheduling model. The initial scheduling model determines loss information according to the training tuples, and corrects the initial scheduling model according to the loss information, and regenerates the first data tuple by the initial scheduling model again, and iteratively executes until the initial scheduling model meets a preset condition, and takes the initial scheduling model that meets the preset condition as a target scheduling model.

[0156] Optionally, the training module 704 is configured to:

[0157] Input the initial state data into the initial scheduling model, and the initial scheduling model determines at least one optional action based on the initial state data;

[0158] Determine a target action among at least one optional action based on a preset selection strategy;

[0159] Take the target action as the action data corresponding to the state data, take the reward value of the target action as the reward data, and after executing the target action, take the updated state data as the next state data.

[0160] Optionally, the training module 704 is configured to:

[0161] Determine the value of each optional action according to the reward value of each optional action;

[0162] Select the optional action with the maximum value from each optional action as the target action based on the greedy strategy, or randomly select an optional action from each optional action as the target action based on the random selection strategy.

[0163] Optionally, the acquisition module 701 is further configured to:

[0164] Traverse the path information of the vehicle in the historical scheduling data, and convert the scheduling vehicle of each node and the order of the scheduling vehicle accessing the node into action data;

[0165] Generate state data corresponding to the action data according to the information of the tasks to be scheduled at each node and the information of the vehicle;

[0166] Use the state data of the next node of each node as the next state data, and determine the reward data of each node according to the next state data of each node;

[0167] Take the action data, state data, next state data, and reward data as a second data tuple.

[0168] Optionally, the training module 704 is used to:

[0169] Determine the predicted value of each training tuple according to the action data, reward data, and state data in the training tuple;

[0170] Determine the value loss value of each training tuple according to the target value corresponding to the next state data in the training tuple and the predicted value of each training tuple;

[0171] Determine the loss information according to the value loss value of each training tuple.

[0172] Optionally, the acquisition module 701 is further configured to:

[0173] Obtain the information of the schedulable vehicle and the task information of the target task. The schedulable vehicle information includes: the warehouse location of the vehicle's current scheduling task, the remaining capacity of the vehicle, the end time of the vehicle's current scheduling task, the path information of the vehicle, and the planned time information of the vehicle. The task information of the task to be scheduled includes: the quantity of material required, the delivery time of the material required, and the warehouse information of the material;

[0174] The device further includes a scheduling module, configured to input the schedulable vehicle information and task information into the target scheduling model to obtain the target scheduling plan for the task to be scheduled. The target scheduling plan includes: at least one target vehicle for the task to be scheduled, and the scheduling path of each target vehicle.

[0175] Optionally, the device further includes a correction module, configured to:

[0176] After executing the target scheduling scheme, obtain the actual scheduling result of the target scheduling scheme, and correct the target scheduling model based on the actual scheduling result.

[0177] Figure 8 The figure shows a schematic structural diagram of an electronic device provided by an embodiment of the present application, including: a processor 801, a storage medium 802, and a bus 803. The storage medium 802 stores machine-readable instructions executable by the processor 801. When the electronic device runs a vehicle scheduling model training method as in the embodiment, the processor 801 communicates with the storage medium 802 through the bus 803, and the processor 801 executes the machine-readable instructions, which are the preamble part of the method item of the processor 801 to execute the steps of the vehicle scheduling model training method.

[0178] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the processor executes the steps of the vehicle scheduling model training method.

[0179] In the embodiment of the present application, when the computer program is run by a processor, it can also execute other machine-readable instructions to execute other methods as described in the embodiment. For the specific method steps and principles of execution, refer to the description of the embodiment, and details will not be elaborated here.

[0180] In the embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some communication interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0181] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, the functional units in the embodiments provided by the present application can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0183] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0184] It should be noted that: similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0185] Finally, it should be noted that: the above-mentioned embodiments are only specific implementation manners of this application, used to illustrate the technical solution of this application, rather than limiting it. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed in this application can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A vehicle scheduling model training method, characterized in that: include: Acquire multiple initial state data, and input each initial state data into the initial scheduling model respectively, to obtain action data, reward data and next state data corresponding to each initial state data, wherein the initial state data is used to characterize the information of the vehicle and the information of the task to be scheduled, wherein the information of the vehicle includes: the warehouse location of the current scheduling task of the vehicle, the remaining capacity of the vehicle, the end time of the current scheduling task of the vehicle, the path information of the vehicle and the planned time information of the vehicle, and the information of the task to be scheduled includes: the required quantity of materials, the required delivery time of materials, and the warehouse information to which the materials belong, and the action data is used to characterize at least one vehicle used to execute the task to be scheduled and the scheduling path of each vehicle; Each initial state data and the action data, reward data and next state data corresponding to each initial state data are respectively combined into a first data tuple, and each data tuple is stored in an experience replay pool, wherein the experience replay pool also includes a plurality of second data tuples, each second data tuple is converted according to the historical dispatch data of the vehicle; Extracting a training tuple from the experience replay pool, wherein the training tuple includes a first data tuple and / or a second data tuple; The training tuple is input into the initial scheduling model, the initial scheduling model determines the loss information according to the training tuple, and the initial scheduling model is corrected according to the loss information, and the first data tuple is regenerated by the initial scheduling model, and the execution is iteratively performed until the initial scheduling model meets the preset conditions, and the initial scheduling model that meets the preset conditions is used as the target scheduling model.

2. The method according to claim 1, characterized in that: The initial state data are input into the initial scheduling model respectively to obtain the action data, reward data and next state data corresponding to each initial state data, including: Inputting the initial state data into the initial scheduling model, and allowing the initial scheduling model to determine at least one optional action based on the initial state data; Determining a target action from the at least one optional action by the initial scheduling model based on a preset selection strategy; The initial scheduling model uses the target action as the action data corresponding to the state data, uses the reward value of the target action as the reward data, and after executing the target action, uses the updated state data as the next state data.

3. The method according to claim 2, characterized in that The determining of the target action from the at least one optional action by the initial scheduling model based on a preset selection strategy includes: The initial scheduling model determines the value of each optional action according to the reward value of each optional action; The initial scheduling model selects the optional action with the greatest value as the target action from among the optional actions based on a greedy strategy, or randomly selects an optional action as the target action from among the optional actions based on a random selection strategy.

4. The method according to claim 1, characterized in that The process of converting the historical dispatch data of the vehicle to obtain the second data tuple includes: Traversing the path information of the vehicles in the historical dispatch data, and converting the dispatch vehicles of each node and the order in which the dispatch vehicles visit the nodes into action data; Generate status data corresponding to the action data according to the information of the tasks to be scheduled at each node and the information of the vehicle; The state data of the next node of each node is used as the next state data, and the reward data of each node is determined according to the next state data of each node; The action data, the state data, the next state data and the reward data are taken as a second data tuple.

5. The method according to claim 1, characterized in that The determining of loss information according to the training tuple by the initial scheduling model includes: Determining, by the initial scheduling model, a predicted value of each of the training tuples according to the action data, reward data, and state data in the training tuples; The initial scheduling model determines the value loss value of each training tuple according to the target value corresponding to the next state data in the training tuple and the predicted value of each training tuple; The loss information is determined by the initial scheduling model according to the value loss value of each training tuple.

6. The method according to claim 1, characterized in that After obtaining the target scheduling model, the method further includes: Obtaining the information of dispatchable vehicles and the task information of the target task, wherein the dispatchable vehicle information includes: the warehouse location of the vehicle's current dispatch task, the vehicle's remaining capacity, the end time of the vehicle's current dispatch task, the vehicle's path information, and the vehicle's planned time information; the task information of the task to be dispatched includes: the required quantity of materials, the required delivery time of materials, and the warehouse information to which the materials belong; The dispatchable vehicle information and the task information are input into the target scheduling model to obtain a target scheduling solution for the task to be scheduled, wherein the target scheduling solution includes: at least one target vehicle for the task to be scheduled, and a scheduling path for each target vehicle.

7. The method according to claim 6, characterized in that The method further comprises: After executing the target scheduling scheme, an actual scheduling result of the target scheduling scheme is obtained, and the target scheduling model is modified based on the actual scheduling result.

8. A vehicle dispatch model training device, characterized in that: include: An acquisition module is used to acquire a plurality of initial state data, and input each initial state data into the initial scheduling model respectively, to obtain action data, reward data and next state data corresponding to each initial state data, wherein the initial state data is used to characterize the information of the vehicle and the information of the task to be scheduled, wherein the information of the vehicle includes: the warehouse location of the current scheduling task of the vehicle, the remaining capacity of the vehicle, the end time of the current scheduling task of the vehicle, the path information of the vehicle and the planned time information of the vehicle, and the information of the task to be scheduled includes: the required quantity of materials, the required delivery time of materials, and the warehouse information to which the materials belong, and the action data is used to characterize at least one vehicle used to execute the task to be scheduled and the scheduling path of each vehicle; A storage module, for combining each initial state data and the action data, reward data and next state data corresponding to each initial state data into a first data tuple, and storing each data tuple in an experience replay pool, wherein the experience replay pool also includes a plurality of second data tuples, each second data tuple being converted according to historical dispatch data of the vehicle; An extraction module, used for extracting a training tuple from the experience replay pool, wherein the training tuple includes a first data tuple and / or a second data tuple; A training module is used to input the training tuple into the initial scheduling model, and the initial scheduling model determines the loss information according to the training tuple, and corrects the initial scheduling model according to the loss information, and regenerates the first data tuple by the initial scheduling model, and iterates until the initial scheduling model meets the preset conditions, and the initial scheduling model that meets the preset conditions is used as the target scheduling model.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of a vehicle scheduling model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a vehicle scheduling model training method as described in any one of claims 1 to 7 are executed.

Images