Scheduling model training method and device and scheduling method and device
Through the reinforcement learning method, the scheduling model is trained, combined with simulation and planning modules, the road area division and individual scheduling strategies are optimized, and the problem of low scheduling efficiency in the existing technology is solved, achieving more efficient distribution efficiency and accuracy.
Patent Information
- Application Number
- CN202310619871.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2043-05-29
AI Technical Summary
When scheduling couriers, it is difficult to effectively consider the division of orders in the road area, resulting in slow model solving speed, easy performance to fall into local optimality, and unable to effectively improve the delivery efficiency of the last mile.
The scheduling model is trained using reinforcement learning method. By obtaining training sample sets and pre-trained simulation models, simulate road zone division and individual scheduling strategies, optimize the decision model to improve the efficiency of road zone division and individual scheduling strategies, combined with reinforcement grouping modules and planning module, generate optimized road zone division and individual scheduling strategies.
Through simulation models, the behavior of distribution individuals is simulated, reliable reward-driven, and individual scheduling strategies are optimized, which improves the distribution efficiency of distribution individuals in different road areas, and improves the accuracy and generation efficiency of the scheduling model.
Smart Images

Figure CN119047535A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology, and especially to a scheduling model training method and device, a scheduling method and device, an electronic device, and a computer-readable storage medium. Background Art
[0002] With the development of e-commerce, higher requirements are placed on the efficiency of logistics and distribution services. The last mile of delivery is the link where the logistics chain interacts with customers, so it is imperative to reduce costs and improve efficiency. The last mile of delivery is mainly completed by couriers, so the scheduling of couriers determines the efficiency of delivery. The scheduling of couriers includes road zone division and route planning. Road zone division refers to dividing the jurisdiction of a delivery station into several road zones, and each courier is responsible for picking up and delivering orders in that road zone. Route planning refers to determining the order of routes for couriers to deliver goods.
[0003] The existing technology mainly models the courier dispatch problem as a vehicle routing problem and uses models or algorithms to solve it. The vehicle routing problem refers to a vehicle starting from a warehouse, visiting several customers and finally returning to the warehouse under the condition of satisfying constraints, and the road division of orders cannot be considered. In addition, the solution speed of the model or algorithm is slow, and the performance is prone to fall into local optimality. Summary of the invention
[0004] The embodiments of the present disclosure provide a scheduling model training method and device, a scheduling method and device, an electronic device, and a computer-readable storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a scheduling model training method, the method comprising: obtaining a training sample set, the training sample set comprising at least one training sample, each training sample comprising delivery individual information of multiple delivery individuals and road area information of multiple road areas; obtaining a pre-trained simulation model and a decision model to be trained, the simulation model being used to simulate the action sequence of the delivery individual under the road area division and individual scheduling strategy, and obtain the simulated efficiency value of the action sequence; the decision model adopts a reinforcement learning method to obtain the road area division and individual scheduling strategy of the dispatched delivery individual at the next moment based on the delivery individual information, road area information, road area division and individual scheduling strategy at the current moment, and the efficiency value corresponding to the road area division and individual scheduling strategy; performing the following training steps: selecting training samples from the training sample set, controlling the decision model to provide the road area division and individual scheduling strategy for the simulation model based on the training samples and the sample efficiency values, obtaining the simulated efficiency value corresponding to the road area division and individual scheduling strategy from the simulation model, and replacing the sample efficiency value with the simulated efficiency value; in response to the decision model satisfying the training completion condition, obtaining a scheduling model corresponding to the decision model.
[0006] In some embodiments, the above-mentioned decision model includes: an enhanced grouping module and a planning module, wherein the grouping module groups the delivery individuals into road areas based on the delivery individual information and road area information, and generates a road area set for the delivery individuals based on the road area division and individual scheduling strategy at the current moment and the efficiency value corresponding to the road area division and individual scheduling strategy; the planning module obtains the road area division and individual scheduling strategy for the delivery individuals based on the road area set.
[0007] In some embodiments, the above method further includes: in response to the decision model not satisfying the training completion condition, adjusting the model parameters of the decision model so that the reward function of the decision model is corrected in an increasing direction, and continuing to execute the above training steps.
[0008] In some embodiments, the above method also includes: recording the number of iterations of the training step; the model parameters include: a probability table, the probability table is used to measure the probability of each road area being assigned to each delivery individual; adjusting the model parameters of the decision model includes: in response to the number of iterations being less than a preset value, assigning a random value to the probability table.
[0009] In some embodiments, the above-mentioned adjusting the model parameters of the decision model further includes: in response to the number of iterations being greater than a preset value, using a learning automaton algorithm to adjust the values in the probability table.
[0010] In some embodiments, the above method also includes: adjusting the parameters related to the delivery individual in the simulation model based on the courier's work record; when the courier has a collection demand, based on the collection demand, adding a collection action and a timestamp of the collection action to the action sequence.
[0011] In a second aspect, an embodiment of the present disclosure provides a scheduling method, the method comprising: obtaining order information and road area information and courier information related to the order information; based on the order information and road area information, using a scheduling model obtained by the scheduling model training method of any embodiment of the first aspect, generating road area divisions and individual scheduling strategies for each courier in the courier information.
[0012] In some embodiments, the above-mentioned order information includes: all orders within a set time period, and the buildings where each order is located; the road area information includes: the road area where each building in the order is located; the above-mentioned road area division and individual scheduling strategy include: the buildings in the road area corresponding to each courier in the courier information and the order in which each courier visits the buildings in their respective road areas.
[0013] In a third aspect, an embodiment of the present disclosure provides a scheduling model training device, which includes: a sample acquisition unit, configured to acquire a training sample set, the training sample set includes at least one training sample, each training sample includes delivery individual information of multiple delivery individuals and road area information of multiple road areas; a model acquisition unit, configured to acquire a pre-trained simulation model and a decision model to be trained, the simulation model is used to simulate the action sequence of the delivery individual under the road area division and individual scheduling strategy, and obtain the simulated efficiency value of the action sequence; the decision model is based on the delivery individual information, the road area information, the road area division and individual scheduling strategy at the current moment ... third aspect, an embodiment of the present disclosure provides a scheduling model training device, which includes: a sample acquisition unit, configured to acquire a training sample set, The efficiency values corresponding to the road zone division and individual dispatching strategies are obtained by using a reinforcement learning method to obtain the road zone division and individual dispatching strategies of the dispatching and distribution individuals at the next moment; a selection unit is configured to select training samples from a training sample set; a control unit is configured to control the decision model to provide the road zone division and individual dispatching strategies for the simulation model based on the training samples and the sample efficiency values; a replacement unit is configured to obtain the simulation efficiency values corresponding to the road zone division and the individual dispatching strategies from the simulation model, and replace the sample efficiency values with the simulation efficiency values; an obtaining unit is configured to obtain a dispatching model corresponding to the decision model in response to the decision model satisfying a training completion condition.
[0014] In some embodiments, the above-mentioned decision model includes: an enhanced grouping module and a planning module, wherein the grouping module groups the delivery individuals into road areas based on the delivery individual information and road area information, and generates a road area set for the delivery individuals based on the road area division and individual scheduling strategy at the current moment and the efficiency value corresponding to the road area division and individual scheduling strategy; the planning module obtains the road area division and individual scheduling strategy for the delivery individuals based on the road area set.
[0015] In some embodiments, the above-mentioned device also includes: an adjustment unit, which is configured to adjust the model parameters of the decision model in response to the decision model not satisfying the training completion conditions so that the reward function of the decision model is corrected in an increasing direction, and continue to control the above-mentioned selection unit, control unit, replacement unit and obtaining unit to work.
[0016] In some embodiments, the above-mentioned device also includes: a recording unit, configured to record the number of iterations of the training step; the model parameters include: a probability table, the probability table is used to measure the probability of each road area being assigned to each delivery individual; the above-mentioned adjustment unit is further configured to: in response to the number of iterations being less than a preset value, assign a random value to the probability table.
[0017] In some embodiments, the adjustment unit is further configured to adjust the value in the probability table using a learning automaton algorithm in response to the number of iterations being greater than a preset value.
[0018] In some embodiments, the above-mentioned device also includes: a courier simulation unit, which is configured to adjust the parameters related to the delivery individual in the simulation model based on the courier's work record; when the courier has a collection requirement, based on the collection requirement, a collection action and a timestamp of the collection action are added to the action sequence.
[0019] In a fourth aspect, an embodiment of the present disclosure provides a scheduling device, comprising: an information acquisition unit, configured to acquire order information and road area information and courier information related to the order information; a generation unit, configured to generate road area divisions and individual scheduling strategies for each courier in the courier information based on the order information and road area information, using a scheduling model obtained by a scheduling model training device of any embodiment of the third aspect.
[0020] In some embodiments, the above-mentioned order information includes: orders within a set time period and the buildings where each order is located; the road area information includes: the road area where each building in the order is located; the road area division and individual scheduling strategy include: the buildings in the road area corresponding to each courier in the courier information and the order in which each courier visits the buildings in their respective road areas.
[0021] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any embodiment of the first aspect or the second aspect.
[0022] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the first aspect or the second aspect.
[0023] The scheduling model training method and device provided by the embodiments of the present disclosure firstly obtain a training sample set, the training sample set includes at least one training sample, each training sample includes the delivery individual information of multiple delivery individuals and the road area information of multiple road areas; secondly, obtain a pre-trained simulation model and a decision model to be trained, the simulation model is used to simulate the action sequence of the delivery individual under the road area division and individual scheduling strategy, and obtain the simulated efficiency value of the action sequence; the decision model adopts a reinforcement learning method to obtain the road area division and individual scheduling strategy of the dispatched delivery individual at the next moment based on the delivery individual information, the road area information, the road area division and individual scheduling strategy at the current moment, and the efficiency value corresponding to the road area division and the individual scheduling strategy; thirdly, select training samples from the training sample set; then, control the decision model to provide the road area division and individual scheduling strategy for the simulation model based on the training samples and the sample efficiency values, obtain the simulated efficiency value corresponding to the road area division and the individual scheduling strategy from the simulation model, and replace the sample efficiency value with the simulated efficiency value; finally, in response to the decision model satisfying the training completion condition, obtain the scheduling model corresponding to the decision model. Therefore, by simulating the behavioral actions of delivery individuals through the simulation model, it is possible to effectively simulate and characterize the real behavior of delivery individuals in different road areas, and provide a reliable reward drive for the decision-making model; by optimizing the dispatching strategy of delivery individuals through the decision-making model, it is possible to provide the delivery individuals with the optimal road area division strategy, thereby improving the efficiency of delivery individuals in delivering ordered goods in different road areas; by coordinating the simulation module with the decision-making model, the accuracy of dispatching model generation is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0025] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;
[0026] Figure 2 is a flowchart of an embodiment of a scheduling model training method according to the present disclosure;
[0027] Figure 3 It is a structural diagram of the cooperation between the simulation model and the decision model disclosed in the present invention;
[0028] Figure 4 is a flow chart of an embodiment of a scheduling method according to the present disclosure;
[0029] Figure 5 is a structural schematic diagram of an embodiment of a scheduling model training device according to the present disclosure;
[0030] Figure 6is a structural schematic diagram of an embodiment of a scheduling device according to the present disclosure;
[0031] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION
[0032] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0034] Figure 1 An exemplary system architecture 100 is shown to which the scheduling model training method or scheduling method of the present disclosure can be applied.
[0035] like Figure 1 As shown, the system architecture 100 may include terminals 101, 102, a network 103, a database server 104, and a server 105. The network 103 is used to provide a medium for communication links between the terminals 101, 102, the database server 104, and the server 105. The network 103 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0036] The user 110 can use the terminals 101 and 102 to interact with the server 105 through the network 103 to receive or send messages, etc. Various client applications can be installed on the terminals 101 and 102, such as model training applications, image recognition applications, shopping applications, payment applications, web browsers, and instant messaging tools.
[0037] The terminals 101 and 102 here can be hardware or software. When the terminals 101 and 102 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Dynamic Image Experts Compression Standard Audio Layer 3), laptops and desktop computers, etc. When the terminals 101 and 102 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0038] The database server 104 may be a database server that provides various services. For example, a training sample set may be stored in the database server. The training sample set includes a large number of training samples, each of which is different. The training samples may include the delivery individual information of multiple delivery individuals and the road area information of multiple road areas. The delivery individuals of all training samples in the training sample set are the delivery individuals in the same delivery individual set, and the road areas in the road area information of all training samples in the training sample set are the road areas in the same road area set. The user 110 may also select training samples from the training sample set stored in the database server 104 through the terminals 101 and 102.
[0039] The server 105 may also be a server that provides various services, such as a background server that provides support for various applications displayed on the terminals 101 and 102. The background server may train the scheduling model using the training samples in the training sample set sent by the terminals 101 and 102, and may send the trained scheduling model to the terminals 101 and 102. In this way, the user may use the generated scheduling model to generate the road area division and individual scheduling strategy for scheduling and distribution individuals.
[0040] The database server 104 and the server 105 here can also be hardware or software. When they are hardware, they can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When they are software, they can be implemented as multiple software or software modules (for example, for providing distributed services), or as a single software or software module. No specific limitation is made here.
[0041] It should be noted that the scheduling model training method or scheduling method provided in the embodiments of the present disclosure is generally executed by the server 105. Accordingly, the scheduling model training device or scheduling device is generally also provided in the server 105.
[0042] It should be pointed out that, in the case where the server 105 can implement the related functions of the database server 104 , the database server 104 may not be provided in the system architecture 100 .
[0043] It should be understood that Figure 1 The number of terminals, networks, database servers and servers in the embodiment is only for illustration. Any number of terminals, networks, database servers and servers may be provided according to the implementation requirements.
[0044] In traditional technology, the courier scheduling problem is modeled as a vehicle routing problem and solved using exact methods and heuristic methods. Among them, the exact method refers to mathematically modeling the problem and using precise calculation methods to solve it, such as the branch and bound method. The heuristic method is to use heuristic algorithms to continuously iterate and optimize the current solution. The heuristic method and the exact method have a slow solution speed, and the vehicle routing model performance is prone to fall into local optimality. In addition, since the last mile delivery scenario is too complex, it is difficult to simply solve it by modeling it as a vehicle routing problem. Simply modeling it as a vehicle routing problem will simplify many details of the real scenario and the optimization goals, making the accuracy of the vehicle routing model obtained by modeling low.
[0045] In view of the above problems, the present disclosure provides a scheduling model training method, which can effectively solve the courier scheduling problem. In particular, for the last mile delivery scenario, the scheduling model obtained by the present disclosure can effectively improve the courier's delivery efficiency, such as Figure 2 , shows a process 200 of an embodiment of a scheduling model training method according to the present disclosure, the scheduling model training method comprising the following steps:
[0046] Step 201: Obtain a training sample set.
[0047] In this embodiment, the training sample set includes at least one training sample, and each training sample includes delivery individual information of a plurality of delivery individuals and road area information of a plurality of road areas.
[0048] In this embodiment, in order to make the training samples in the training sample set more closely match the actual delivery scenarios where the delivery individuals are located, the delivery individuals and road areas in the training samples can be generated based on the jurisdiction of the delivery station and the order requirements. For example, one training sample corresponds to a delivery station, which is divided into N (N>2) road areas and has K (K>2) delivery individuals.
[0049] In this embodiment, the delivery individual is the subject who picks up or puts items in each road area, and the delivery individual information includes: the delivery individual name, delivery individual number, delivery individual quantity, etc. Among them, the delivery individual can be a delivery person or an automatic delivery vehicle; when the delivery individual is a delivery person, the scheduling model trained by the present disclosure can generate a road area division and individual scheduling strategy to guide the delivery person's delivery, so that the delivery person can effectively deliver based on the road area division and individual scheduling strategy after obtaining the road area division and individual scheduling strategy. When the delivery individual is an automatic delivery vehicle, the road area division and individual scheduling strategy provided by the present disclosure for the automatic delivery vehicle can enable the automatic delivery vehicle to automatically pick up or deliver items on its delivery route.
[0050] In this embodiment, the road area is an area obtained by dividing the jurisdiction of the delivery station according to predetermined rules (for example, areas under the jurisdiction of the same community), and the road area information includes: delivery station name, delivery station number, delivery jurisdiction area location, etc.
[0051] In this embodiment, the training samples in the training sample set are different, and the delivery individuals of any two training samples may be the same but the road areas are different, or the delivery individuals of any two training samples may be different but the road areas are the same. All training samples in the training sample set select delivery individuals from the same delivery individual set, select road areas from the same road area set, and the road area set may be included in one delivery station.
[0052] In this embodiment, the execution subject of the scheduling model training method (for example Figure 1 The server 105 shown in FIG. 104 may obtain the training sample set in a variety of ways. For example, the execution subject may obtain the training sample set from a database server (e.g. Figure 1 For example, a user may obtain a training sample set stored in the database server 104 as shown in FIG. Figure 1 In this way, the execution subject can receive the training samples collected by the terminal and store these training samples locally, thereby generating a training sample set.
[0053] Step 202, obtaining a pre-trained simulation model and a decision model to be trained.
[0054] In this embodiment, the simulation model is used to simulate the action sequence of the delivery individuals under the road zone division and individual dispatching strategy, and obtain the simulated efficiency value of the action sequence.
[0055] In this embodiment, the simulation model is a model built based on simulation technology. Simulation technology refers to an experimental method that uses similarity principles to establish a model of the research object (such as an image model, a description model, a mathematical model), and indirectly studies the regularity of the prototype through the model. The research object of the simulation model is the delivery individual. The simulation model is used to simulate the action sequence of the delivery individual in different road areas, and each action sequence is an action that the delivery individual can perform.
[0056] Optionally, when the delivery individual is a delivery person, the simulation model aims to simulate the work action sequence of the delivery person, and to make the actions and the time points of the actions in the action sequence as realistic as possible. The simulation model will be coupled with the decision model after pre-training to optimize the personalized parameters of its internal delivery personnel, and the output of the simulation model will be used as the reward function module of reinforcement learning in the decision model. When the delivery individual is a delivery person, the input, output, parameters required for pre-training, and optimization method of the simulation model pre-training process are as follows:
[0057] Simulation model input: the order in which each delivery personnel visits the buildings in their respective routes in a day (when the simulation model is pre-trained, this input can be provided by a certain sample; after the simulation model pre-training is completed, this input is provided by the decision model).
[0058] The simulation model outputs: the daily actions of each delivery personnel (including going to the route area, picking up goods, delivering and collecting goods, etc.) and the simulated efficiency value of each delivery personnel. The simulated efficiency value is the value obtained by dividing the total number of orders completed by the delivery personnel by the time it takes to complete the total orders after the simulated delivery personnel actions are completed.
[0059] Parameters that need to be optimized for simulation model pre-training: behavioral characteristic parameters of delivery personnel, such as climbing speed, picking time, rest time, delivery and collection time, and walking time in the community.
[0060] Data required for pre-training of the simulation model: i) Fixed facility data (this part of data is used to construct the basic parameters of the simulation model): road network information within the sales department (given in a topological structure); building information (including whether it is an elevator room, building boundaries, etc.) ii) Historical work records of delivery personnel (used to optimize the behavioral parameters of delivery personnel in the simulation model): records the order and time nodes in which delivery personnel handle each order.
[0061] In the simulation model pre-training optimization method: any of the Bayesian parameter adjustment method, grid search and random search parameter adjustment methods can be used.
[0062] In this embodiment, when the delivery individual is an automatic delivery vehicle, the pre-training process of the simulation model is the same as the pre-training process of the delivery personnel. At this time, the parameters that need to be optimized during the pre-training of the simulation model no longer need to consider human factors (for example, rest time).
[0063] In this embodiment, the decision model is based on the distribution individual information, road area information, the road area division and individual dispatching strategy at the current moment, and the efficiency value corresponding to the road area division and individual dispatching strategy, and uses the reinforcement learning method to obtain the road area division and individual dispatching strategy of the dispatch distribution individual at the next moment. Among them, the efficiency value corresponding to the road area division and individual dispatching strategy can be provided by the simulation model as the reward function of the decision model.
[0064] In this embodiment, the road area information is information related to the road area. The decision model allocates the road area in the road area information to each delivery individual in the delivery individual information, and after the allocation is completed, determines the order in which each delivery individual visits their respective road areas, and uses the road area division and the order in which the delivery individuals visit their respective road areas as the road area division and individual scheduling strategy.
[0065] Optionally, the road area information also includes: building information. The decision model allocates the buildings under the road area in the road area information to each delivery individual in the delivery individual information, and after the allocation is completed, determines the order in which each delivery individual visits the buildings in their respective road areas. The division of buildings in the road area and the order in which the delivery individuals visit the buildings in their respective road areas are used as the road area division and individual scheduling strategies.
[0066] In some optional implementations of the present disclosure, the decision model includes: an enhanced grouping module and a planning module, wherein the grouping module groups the delivery individuals into road areas based on the delivery individual information and road area information, and generates a road area set for the delivery individuals based on the road area division and individual scheduling strategy at the current moment and the efficiency values corresponding to the road area division and individual scheduling strategy; the planning module obtains the road area division and individual scheduling strategy for the delivery individuals based on the road area set.
[0067] In this optional implementation, the decision model needs to do two things. First, the grouping module is used to divide the road areas for all delivery entities, for example, N road areas are divided into K delivery entities. Second, after each delivery entity clearly defines the road area it is responsible for, the planning module is used to plan the order in which each delivery entity visits each road area. Specifically, the decision model uses reinforcement learning to train the grouping module's road area division decisions, while the planning module uses the LKH-3 solver to calculate the order in which the delivery entity visits each road area. The LKH-3 solver is an integrated solver for solving paths.
[0068] The decision model provided by this optional implementation method converts the road zone division into a grouping problem through a grouping module; selects the optimal grouping result from the grouping results of the grouping model through a planning module, and plans the optimal access path to the road zone for the distribution individual, thereby improving the accuracy of the road zone division and individual scheduling strategy.
[0069] Optionally, when the route area information is the building information within the business department (the business department has M (M>2) buildings), since the order is within the building, the grouping module first uses the reinforcement learning method to divide the business department into buildings, that is, assigning M buildings to K delivery individuals; the planning module plans the order in which each delivery individual visits their respective buildings after each delivery individual clearly identifies the building he is responsible for.
[0070] The various elements of road area division trained by reinforcement learning method are as follows:
[0071] (1) Status: The current allocation of each building to each delivery entity (this allocation can also be understood as the division of each building in the current road area).
[0072] (2) Action: Adjust the way buildings are allocated to each delivery unit (this method is the building division and individual dispatching strategy).
[0073] (3) Reward function: The reward function can be measured by the simulated efficiency value of the delivery individual calculated by the simulation model (for example, if the simulated efficiency value of the delivery individual increases after building 1 is assigned to delivery individual 2, the probability of building 1 being assigned to other delivery individuals decreases, while the probability of building 1 being assigned to delivery individual 2 increases).
[0074] The parameters of the grouping model will be modified and adjusted, which will affect the parameters of the reinforcement learning model, thereby affecting the elements (1) and (2) of the reinforcement learning, that is, affecting the state and action, so that the road area division plan is adjusted and different buildings are assigned to each delivery individual.
[0075] Step 203: Select training samples from the training sample set.
[0076] In this embodiment, the execution subject may select training samples from the training sample set obtained in step 201, and perform the training steps of step 204 to step 206. The selection method and the number of training samples are not limited in this application. For example, at least one training sample may be randomly selected, or a training sample that is easier to allocate to a delivery individual and a road area may be selected.
[0077] Step 204: The control decision model provides a road zone division and an individual scheduling strategy for the simulation model based on the training samples and the sample efficiency values.
[0078] In this embodiment, the sample efficiency value of the decision model is the reward value in the reinforcement learning environment where the decision model is located. When the execution subject initially controls the decision model to provide the simulation model with road area division and individual scheduling strategies, the sample efficiency value is zero. In the subsequent multiple iterations of the training step, the sample efficiency value can be directly obtained through the simulation efficiency value of the simulation model.
[0079] In this embodiment, after obtaining the training sample, the decision model allocates the road area in the road area information corresponding to the training sample to each delivery individual in the delivery individual information, and after the allocation is completed, determines the order in which each delivery individual visits their respective road areas, and uses the road area division and the order in which the delivery individuals visit their respective road areas as the road area division and individual scheduling strategy, and provides the simulation model with the road area division and individual scheduling strategy.
[0080] Step 205, obtaining a simulation efficiency value corresponding to the road area division and the individual scheduling strategy from the simulation model, and using the simulation efficiency value to replace the sample efficiency value.
[0081] In this embodiment, after the decision model provides the simulation model with road area division and individual scheduling strategies, the simulation model will generate an action sequence of the delivery individuals and a simulated efficiency value of the action sequence. After the simulation model generates the simulated efficiency value, the simulated efficiency value is used to replace the sample efficiency value, so that the sample efficiency value can be made the same as the simulated efficiency value, that is, the simulation model provides the decision model with a reinforcement learning reward value.
[0082] In this embodiment, after replacing the sample efficiency value with the simulation efficiency value, the decision model can perform training for the next iteration based on the replaced sample efficiency value (each iteration executes steps 203 to 206 in sequence, and returns to execute step 203 when step 206 is not satisfied), that is, after the decision model obtains the sample efficiency value, it provides the simulation model with road area division and individual scheduling strategies based on the training samples and sample efficiency values of the next iteration.
[0083] It should be noted that in the training step of the current iteration, if the decision model has met the training completion conditions, the execution subject will no longer input training samples to the decision model, and the decision model will stop providing road area division and individual scheduling strategies for the simulation model. If the decision model does not meet the training completion conditions, the execution subject will continue to select training samples from the training sample set, input the training samples and sample efficiency values into the decision model, and the decision model will perform training for the current iteration.
[0084] Step 206, in response to the decision model satisfying the training completion condition, obtaining a scheduling model corresponding to the decision model.
[0085] In this embodiment, the training completion condition includes at least one of the following: the number of training iterations reaches a predetermined iteration threshold, and the rate of change of the model parameters of the decision model is less than a predetermined threshold. For example, the training iterations reach 5,000. The rate of change of the model parameters of the decision model is less than 0.05. After the training is completed, only the decision model is retained as the scheduling model. Setting the training completion condition can speed up the model convergence speed.
[0086] In step 204, after the decision model outputs the road zone division and individual scheduling strategy to the simulation model, the model parameters of the decision model can be collected in real time. If the model parameters of the current decision model do not change much compared with the model parameters under the historical iteration number and are less than the predetermined threshold, it is determined that the model parameters converge and the scheduling model training is completed.
[0087] In some optional implementations of the present disclosure, in response to the decision model not satisfying the training completion condition, the model parameters of the decision model are adjusted so that the reward function of the decision model is corrected in an increasing direction, and the above steps 203 to 206 are continued to be executed.
[0088] In this embodiment, if the training is not completed, the model parameters of the decision model are adjusted to make the model parameters converge. Overall, the simulation efficiency value (i.e., the reward function set in the present disclosure) output by the simulation model is used to evaluate the quality of the training effect of the decision module, and compared with the result of the previous iteration. If the effect is better, it means that the parameter optimization direction is correct at this time, and it can continue to be adjusted; if the effect is worse, it means that the parameter optimization direction is wrong at this time and needs to be adjusted.
[0089] The method for adjusting the model parameters of the decision model provided by this optional implementation, when the training completion conditions are not met, adjusts the model parameters of the decision model in the direction of increasing the reward function, so as to maximize the reward value obtained by the decision model and ensure that the trained decision model is the optimal model.
[0090] The scheduling model training method provided by the embodiment of the present disclosure comprises the following steps: first, obtaining a training sample set, wherein the training sample set comprises at least one training sample, and each training sample comprises the delivery individual information of a plurality of delivery individuals and the road area information of a plurality of road areas; second, obtaining a pre-trained simulation model and a decision model to be trained, wherein the simulation model is used to simulate the action sequence of the delivery individual under the road area division and the individual scheduling strategy, and obtain the simulated efficiency value of the action sequence; the decision model adopts a reinforcement learning method to obtain the road area division and the individual scheduling strategy of the dispatched delivery individual at the next moment based on the delivery individual information, the road area information, the road area division and the individual scheduling strategy at the current moment, and the efficiency value corresponding to the road area division and the individual scheduling strategy; third, selecting a training sample from the training sample set; and third, controlling the decision model to provide the road area division and the individual scheduling strategy for the simulation model based on the training sample and the sample efficiency value, obtaining the simulated efficiency value corresponding to the road area division and the individual scheduling strategy from the simulation model, and replacing the sample efficiency value with the simulated efficiency value; finally, in response to the decision model satisfying the training completion condition, obtaining the scheduling model corresponding to the decision model. Therefore, by simulating the behavioral actions of delivery individuals through the simulation model, it is possible to effectively simulate and characterize the real behavior of delivery individuals in different road areas, and provide a reliable reward drive for the decision-making model; by optimizing the dispatching strategy of delivery individuals through the decision-making model, it is possible to provide the delivery individuals with the optimal road area division strategy, thereby improving the efficiency of delivery individuals in delivering ordered goods in different road areas; by coordinating the simulation module with the decision-making model, the accuracy of dispatching model generation is improved.
[0091] In some embodiments of the present disclosure, the above-mentioned scheduling model training method also includes: recording the number of iterations of the training step; the model parameters include: a probability table, the probability table is used to measure the probability of each road area being assigned to each delivery individual; adjusting the model parameters of the decision model includes: in response to the number of iterations being less than a preset value, assigning a random value to the probability table.
[0092] In this embodiment, the above-mentioned recording of the number of iterations of the training step includes: when it is determined that the decision model does not meet the training completion condition, the number of iterations of the repeated iterations of the above-mentioned steps 203 to 206 is recorded in real time.
[0093] Optionally, the above-mentioned recording of the number of iterations of the training step further includes: setting the number of iterations; in response to the simulation efficiency value having replaced the sample efficiency value, adding 1 to the value of the number of iterations.
[0094] The model parameters of the above-mentioned adjustment decision model include: monitoring in real time whether the value of the number of iterations is equal to a preset value, and in response to the number of iterations being less than the preset value, assigning a random value to the probability table. The preset value can be set according to R&D requirements, for example, the preset value is 3.
[0095] In this embodiment, an N*K probability table is constructed to measure the probability of assigning a road area to a delivery individual (the initial values in the probability table are all equal): in the first three iterations of the decision model training, the selection of each road area assigned to the delivery individual is random, which can ensure that the model parameters of the decision model will not be polled at fixed values.
[0096] The scheduling model training method provided in this embodiment sets the model parameters of the decision model as a probability table, thereby converting the road area division problem into a grouping problem; in the first preset number of cycles of the iteration number, by randomly setting the value of the probability table, the decision model will not fall into the local optimum, thereby improving the reliability of the decision model training.
[0097] In another embodiment of the present disclosure, the scheduling model training method further includes: in response to the number of iterations being greater than a preset value, using a learning automaton algorithm to adjust the values in the probability table.
[0098] In this embodiment, when the number of iterations of the decision model is greater than 3, it is determined that the decision model has entered the third iteration cycle, and the values in the probability table are updated and iterated based on the simulation efficiency values output by the simulation model. The maximum values in the probability table are used to determine the correspondence between the road area and the delivery individual, and the road area allocation is performed based on the correspondence; the probability table is adjusted based on each road area allocation and the simulation efficiency value output by the simulation model until the probability table converges, and it is determined that the decision model meets the training completion conditions.
[0099] In this embodiment, reinforcement learning is to adjust the probability of each building being assigned to different delivery individuals. This is a probability table. If this probability table converges, that is, the probability change range of this probability table is not large, it means that a road area allocation method with the maximum expected benefit has been found.
[0100] In this embodiment, the learning automaton adjusts itself by constantly interacting with the random environment, that is, it gains experience through continuous communication with the environment to improve its own behavior, thereby selecting the best action in the environment from the selectable actions, and the best action is the action with the highest probability of obtaining environmental rewards in the current environment. The decision model of this embodiment uses a learning automaton to continuously communicate with the sample efficiency value output by the simulation model to adjust the value in the probability table; when the probability table changes in a gradually increasing direction, the value in the probability table is increased. When the probability table changes in a gradually decreasing direction, the value in the probability table is reduced.
[0101] The scheduling model training method provided in this embodiment uses a learning automaton algorithm to adjust the values in the probability table, which can make the reinforcement learning of the decision model completely synchronized with environmental changes, thereby improving the accuracy of the decision model.
[0102] In some embodiments of the present disclosure, the above-mentioned model training method also includes: adjusting the parameters related to the delivery individual in the simulation model based on the courier's work record; when the courier has a collection demand, based on the collection demand, adding a collection action and a timestamp of the collection action to the action sequence.
[0103] In this embodiment, the delivery individual is a courier, that is, the scenario in which the scheduling model is located is the courier's delivery scenario. When the scheduling model needs to be trained, the parameters related to the delivery individual in the simulation model are adjusted based on the courier's work record, so that the delivery individual in the simulation model can be fully consistent with the courier's actual work.
[0104] Furthermore, since the courier's collection task is generated in real time and is uncertain, in order to ensure that the simulation model fits the courier's actual collection task, the collection action and the timestamp of the collection action are added to the action sequence based on the courier's collection needs. After the action sequence changes, the simulation model automatically calculates the simulation efficiency value of the action sequence.
[0105] The scheduling model training method provided in this embodiment adjusts the relevant parameters of the simulation model based on the courier's work records in the courier delivery and collection order scenarios, so that the simulation model is more suitable for the courier delivery and collection order scenarios; further, when the courier has a collection demand, the collection action and the timestamp of the collection action are added to the action sequence, which can facilitate the simulation model to effectively calculate the simulation efficiency value of the action sequence.
[0106] The scheduling model training method provided in this disclosure can be applied to the last mile delivery scenario of logistics. Figure 3 The scheduling model training process in this scenario is described as follows:
[0107] Based on rule training, a simulation model of the courier in the last mile delivery is obtained. The simulation model simulates the courier's actions by essentially generating a series of courier actions and corresponding timestamps. Considering that couriers need to deliver and collect goods during their work, and the goods that need to be delivered on a certain day are known, the collection tasks are generated in real time. In this scheme, the courier's work sequence for the delivery person is first generated, and then the courier's work sequence is optimized based on the real-time generated collection tasks. Specifically, the ABM (Agent-Based Modeling) method can be used for modeling. Since the courier's actions include:
[0108] The courier's actions in the building: going up and down stairs, taking the elevator, collecting and delivering items -> set the duration of the corresponding actions and the order of the actions (for example, going up and down stairs before and after delivery is necessary, and whether to go up and down stairs or take the elevator depends on the building type) to ensure that the simulated actions conform to the real behavior logic.
[0109] The courier's actions of shuttling between roads -> set the courier's riding and walking speed The courier's actions of picking up goods, taking a break, etc. -> set the courier's riding and walking speed
[0110] The above actions constitute the main set of equal actions of the courier's work. Therefore, after a delivery task is given, the courier will generate an action sequence based on the road area division and individual scheduling strategy sent by the decision model. The parameters involved in the above actions will be adjusted based on real data to meet the work characteristics of each courier as much as possible. In this way, the courier's action sequence for delivery has been generated, including the order of each action and the corresponding timestamp. On this basis, if there is a real-time collection demand, the delivery action and the timestamp of the delivery action will be inserted at a reasonable location according to the delivery sequence.
[0111] The simulation model parameters are adjusted based on the real couriers' work records. As shown above, the duration of each action varies among couriers, so real data is used to adjust the parameters of each courier. The Bayesian parameter adjustment method is used here.
[0112] Road zone division is a complex division problem. Here we consider grouping the buildings in the area (road zones and buildings correspond to each other), which can be transformed into a grouping problem, that is, M buildings are assigned to K couriers. The method based on learning automata is used to solve this grouping problem. An M*K probability table is constructed to measure the probability of buildings being assigned to couriers (the initial values are all equal): in the first three iterations, the selection of each building assigned to a courier is random; after the third iteration, since the probability table has been updated and iterated based on the human efficiency gain of the simulation model, the allocation method corresponding to the maximum probability is used; based on the sample efficiency value of each road zone division and individual dispatching strategy (calculated by the simulation model), the probability table is adjusted until the probability table converges to obtain a trained decision model.
[0113] The LKH-3 solver is used to plan the courier route. The LKH-3 solver is a solver for the TSP (Traveling Salesman Problem). After the road area is divided, the order in which the courier visits the buildings in each road area is a TSP problem. Therefore, the LKH-3 solver can be used to calculate the order in which each courier visits the buildings in his road area.
[0114] like Figure 3 As shown in the figure, the overall framework includes a simulation model (courier action simulation) and a decision model (road area division and courier route optimization). The two parts interact with each other and continuously optimize the decision content. Figure 3 In the process, the road division and route planning calculated by the decision model will be used as the road division and individual scheduling strategy L input into the simulation model, and the simulated efficiency value X related to the courier calculated by the simulation module will be fed back to the decision model to promote its performance improvement.
[0115] See also Figure 4 , which shows a process 400 of an embodiment of a dispatching method provided by the present disclosure. The dispatching method can be used to assign different road areas and couriers to orders, and dispatch couriers to pick up and dispatch orders in different road areas. The dispatching method may include the following steps:
[0116] Step 401, obtaining order information and route information and courier information related to the order information.
[0117] In this embodiment, the execution subject on which the scheduling method runs can communicate with the terminal (such as Figure 1 The terminals 101 and 102 communicate with each other to obtain the order information sent by the terminals and the route information and courier information related to the order information.
[0118] In this embodiment, the order information includes: order code, order product information, the delivery station to which the order belongs, etc.; the road area information related to the order information includes: the road area divided by the delivery station corresponding to the order information, and the buildings in each road area; the courier information related to the order information includes: the courier corresponding to the delivery station in the order information, and the number of the couriers.
[0119] Step 402, based on the order information and the route zone information, a dispatching model obtained by a dispatching model training method is used to generate the route zone division and individual dispatching strategy of each courier in the courier information.
[0120] In this embodiment, the scheduling model is trained based on the scheduling model training method of this embodiment. The scheduling model training method of the above embodiment is used to generate a scheduling model, wherein the specific generation process of the scheduling model can be referred to in Figure 2 The relevant description of the embodiments will not be repeated here.
[0121] It should be noted that the scheduling method of this embodiment can be used to test the scheduling model generated by the above embodiments. Then, the scheduling model can be continuously optimized according to the test results. This method can also be a practical application method of the scheduling model generated by the above embodiments. Using the scheduling model generated by the above embodiments to identify the entity type in the test sentence helps to improve the recognition efficiency of the entity.
[0122] The scheduling method provided by the embodiment of the present disclosure first obtains order information and road area information and courier information related to the order information; finally, the order information and road area information are input into the scheduling model obtained by the scheduling model training method, and the road area division and individual scheduling strategy of the couriers visiting their respective road areas in the courier information output by the scheduling model are obtained. Thus, the road area information and courier information are grouped and the optimal path is calculated by the scheduling model obtained by pre-training, and the warehouse area division and individual scheduling strategy of each courier are obtained, so that the couriers can be quickly and effectively scheduled to improve the efficiency of courier allocation.
[0123] In some optional implementations of the present disclosure, order information includes: orders within a set time period and the buildings where each order is located; route information includes: the route where each building in the order is located; route division and individual scheduling strategies include: the buildings in the route corresponding to each courier in the courier information and the order in which each courier visits the buildings in their respective routes.
[0124] In this optional implementation, the time period can be set according to demand, for example, the time period is set to one day.
[0125] In this optional implementation, the order has a fixed location, and the fixed location is only within the building. Each order will be assigned to the corresponding building. Therefore, the order corresponds to the building one by one. The way of picking up or delivering orders in different road areas can be converted into the order of visiting the buildings in the road area. After obtaining the order information and road area information, the scheduling model divides the buildings in the order information into multiple road areas based on the building where the order is located and the road area where the building is located, and makes each courier correspond to the buildings in different road areas; after the division of the buildings in the road areas corresponding to each courier is completed, the scheduling model sets the order for each courier to visit the buildings in their respective road areas.
[0126] The scheduling method provided by this optional implementation adopts a scheduling model to generate buildings in the corresponding road area for each courier based on the buildings in the order information, and the order in which each courier visits the buildings in their respective road areas, thereby improving the efficiency of courier delivery.
[0127] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a scheduling model training device, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0128] like Figure 5As shown, an embodiment of the present disclosure provides a scheduling model training device 500, and the device 500 includes: a sample acquisition unit 501, a model acquisition unit 502, a selection unit 503, a control unit 504, a replacement unit 505, and a obtaining unit 506. The sample acquisition unit 501 can be configured to obtain a training sample set, the training sample set includes at least one training sample, and each training sample includes the distribution individual information of multiple distribution individuals and the road area information of multiple road areas. The model acquisition unit 502 can be configured to obtain a pre-trained simulation model and a decision model to be trained, the simulation model is used to simulate the action sequence of the distribution individual under the road area division and individual scheduling strategy, and obtain the simulation efficiency value of the action sequence; the decision model is based on the distribution individual information, the road area information, the road area division and individual scheduling strategy at the current moment, and the efficiency value corresponding to the road area division and individual scheduling strategy, and adopts a reinforcement learning method to obtain the road area division and individual scheduling strategy of the dispatched distribution individual at the next moment. The selection unit 503 can be configured to select a training sample from the training sample set. The control unit 504 can be configured to control the decision model to provide the simulation model with road zone division and individual scheduling strategy based on the training sample and sample efficiency value. The replacement unit 505 can be configured to obtain the simulation efficiency value corresponding to the road zone division and individual scheduling strategy from the simulation model, and use the simulation efficiency value to replace the sample efficiency value. The obtaining unit 506 can be configured to obtain the scheduling model corresponding to the decision model in response to the decision model meeting the training completion condition.
[0129] In this embodiment, in the scheduling model training device 500, the specific processing of the sample acquisition unit 501, the model acquisition unit 502, the selection unit 503, the control unit 504, the replacement unit 505, and the obtaining unit 506 and the technical effects thereof can be referred to respectively. Figure 2 Corresponding to step 201, step 202, step 203, step 204, step 205, and step 206 in the embodiment.
[0130] In some embodiments, the above-mentioned decision model includes: an enhanced grouping module and a planning module, wherein the grouping module groups the delivery individuals into road areas based on the delivery individual information and road area information, and generates a road area set for the delivery individuals based on the road area division and individual scheduling strategy at the current moment and the efficiency value corresponding to the road area division and individual scheduling strategy; the planning module obtains the road area division and individual scheduling strategy for the delivery individuals based on the road area set.
[0131] In some embodiments, the above-mentioned device 500 also includes: an adjustment unit (not shown in the figure); the above-mentioned adjustment unit can be configured to adjust the model parameters of the decision model in response to the decision model not satisfying the training completion condition so that the reward function of the decision model is corrected in an increasing direction, and continue to control the above-mentioned selection unit 503, control unit 504, replacement unit 505 and obtaining unit 506 to work.
[0132] In some embodiments, the above-mentioned device 500 also includes: a recording unit (not shown in the figure), configured to record the number of iterations of the training step; the model parameters include: a probability table, the probability table is used to measure the probability of each road area being assigned to each delivery individual; the above-mentioned adjustment unit is further configured to: in response to the number of iterations being less than a preset value, assign a random value to the probability table.
[0133] In some embodiments, the adjustment unit is further configured to adjust the value in the probability table using a learning automaton algorithm in response to the number of iterations being greater than a preset value.
[0134] In some embodiments, the above-mentioned device 500 also includes: a courier simulation unit (not shown in the figure); the above-mentioned courier simulation unit can be configured to adjust the parameters related to the delivery individual in the simulation model based on the courier's work record; when the courier has a collection demand, based on the collection demand, a collection action and a timestamp of the collection action are added to the action sequence.
[0135] The scheduling model training device provided by the embodiment of the present disclosure, first, the sample acquisition unit 501 acquires a training sample set, the training sample set includes at least one training sample, each training sample includes the delivery individual information of multiple delivery individuals and the road area information of multiple road areas; secondly, the model acquisition unit 502 acquires a pre-trained simulation model and a decision model to be trained, the simulation model is used to simulate the action sequence of the delivery individual under the road area division and individual scheduling strategy, and obtain the simulated efficiency value of the action sequence; the decision model is based on the delivery individual information, the road area information, the road area division and individual scheduling strategy at the current moment, the road area division and individual scheduling The efficiency value corresponding to the strategy is obtained by using the reinforcement learning method to obtain the next-moment road zone division and individual scheduling strategy of the dispatched distribution individual; again, the selection unit 503 selects training samples from the training sample set; from this, the control unit 504 controls the decision model to provide the road zone division and individual scheduling strategy for the simulation model based on the training samples and sample efficiency values, and then the replacement unit 505 obtains the simulation efficiency value corresponding to the road zone division and individual scheduling strategy from the simulation model, and uses the simulation efficiency value to replace the sample efficiency value; finally, the obtaining unit 506 obtains the scheduling model corresponding to the decision model in response to the decision model meeting the training completion condition. Thus, by simulating the behavior of the distribution individual through the simulation model, it is possible to effectively simulate and characterize the real behavior of the distribution individual in different road zones, and provide a reliable reward drive for the decision model; by optimizing the scheduling strategy of the distribution individual through the decision model, it is possible to provide the distribution individual with an optimal road zone division strategy, and improve the efficiency of the distribution individual in distributing ordered goods in different road zones; by cooperating with the simulation module and the decision model, the accuracy of the generation of the scheduling model is improved.
[0136] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a scheduling device, which is similar to Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0137] like Figure 6 As shown, an embodiment of the present disclosure provides a scheduling device 600, which includes: an information acquisition unit 601 and a generation unit 602. The information acquisition unit 601 can be configured to acquire order information and route information and courier information related to the order information. The generation unit 602 can be configured to generate the route division and individual scheduling strategy of each courier in the courier information based on the order information and route information and the scheduling model obtained by the scheduling model training device.
[0138] In this embodiment, the scheduling model is trained using a scheduling model training device.
[0139] In this embodiment, the specific processing of the information acquisition unit 601 and the generation unit 602 and the technical effects thereof can be referred to in Figure 4 This corresponds to step 401 and step 402 in the embodiment.
[0140] Reference below Figure 7 , which shows a schematic structural diagram of an electronic device 700 suitable for implementing an embodiment of the present disclosure.
[0141] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0142] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0143] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0144] It should be noted that the computer-readable medium of the embodiment of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiment of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0145] The computer-readable medium may be included in the server, or may exist independently without being installed in the server. The computer-readable medium carries one or more programs. When the one or more programs are executed by the server, the server: obtains a training sample set, the training sample set includes at least one training sample, each training sample includes the delivery individual information of multiple delivery individuals and the road area information of multiple road areas; obtains a pre-trained simulation model and a decision model to be trained, the simulation model is used to simulate the action sequence of the delivery individual under the road area division and individual scheduling strategy, and obtains the simulated efficiency value of the action sequence; the decision model uses a reinforcement learning method to obtain the road area division and individual scheduling strategy of the dispatched delivery individual at the next moment based on the delivery individual information, the road area information, the road area division and individual scheduling strategy at the current moment, and the efficiency value corresponding to the road area division and individual scheduling strategy; performs the following training steps: selects training samples from the training sample set, controls the decision model to provide the road area division and individual scheduling strategy for the simulation model based on the training samples and the sample efficiency values, obtains the simulated efficiency value corresponding to the road area division and individual scheduling strategy from the simulation model, and replaces the sample efficiency value with the simulated efficiency value; in response to the decision model meeting the training completion condition, obtains the scheduling model corresponding to the decision model.
[0146] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0148] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor comprising a sample acquisition unit, a model acquisition unit, a selection unit, a control unit, a replacement unit, and a obtaining unit. The names of these units do not constitute limitations on the units themselves in certain circumstances. For example, the sample acquisition unit may also be described as a unit "configured to acquire a training sample set, the training sample set comprising at least one training sample, each training sample comprising delivery individual information of a plurality of delivery individuals and road area information of a plurality of road areas".
[0149] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.
Claims
1. A scheduling model training method, the method include: Acquire a training sample set, wherein the training sample set includes at least one training sample, and each training sample includes delivery individual information of a plurality of delivery individuals and road area information of a plurality of road areas; Obtaining a pre-trained simulation model and a decision model to be trained, wherein the simulation model is used to simulate the action sequence of the distribution individual under the road zone division and individual dispatching strategy, and obtain the simulated efficiency value of the action sequence; The decision model is based on the distribution individual information, road area information, the road area division and individual dispatching strategy at the current moment, and the efficiency value corresponding to the road area division and individual dispatching strategy, and uses a reinforcement learning method to obtain the road area division and individual dispatching strategy of the dispatch distribution individual at the next moment; Execute the following training steps: select training samples from the training sample set, control the decision model to provide road zone division and individual scheduling strategy for the simulation model based on the training samples and sample efficiency values, obtain the simulation efficiency value corresponding to the road zone division and individual scheduling strategy from the simulation model, and use the simulation efficiency value to replace the sample efficiency value; in response to the decision model meeting the training completion condition, obtain the scheduling model corresponding to the decision model.
2. The method according to claim 1, in, The decision model includes: an enhanced grouping module and a planning module, wherein the grouping module groups the delivery individuals into road areas based on the delivery individual information and road area information, and generates a road area set of the delivery individuals based on the road area division and individual scheduling strategy at the current moment and the efficiency values corresponding to the road area division and individual scheduling strategy; the planning module obtains the road area division and individual scheduling strategy of the delivery individuals based on the road area set.
3. The method according to claim 1, further comprising: include: In response to the decision model not satisfying the training completion condition, the model parameters of the decision model are adjusted so that the reward function of the decision model is corrected in an increasing direction, and the above training steps are continued.
4. The method according to claim 3, further comprising: include: Recording the number of iterations of the training step; The model parameters include: a probability table, which is used to measure the probability of each road area being assigned to each delivery individual; The step of adjusting the model parameters of the decision model comprises: In response to the number of iterations being less than a preset value, a random value is assigned to the probability table.
5. The method according to claim 4, in, The step of adjusting the model parameters of the decision model further comprises: In response to the number of iterations being greater than the preset value, a learning automaton algorithm is used to adjust the values in the probability table.
6. The method according to claim 1, further comprising: include: Based on the courier's work record, adjusting the parameters related to the delivery individual in the simulation model; When the courier has a collection requirement, based on the collection requirement, a collection action and a timestamp of the collection action are added to the action sequence.
7. A scheduling method, the method comprising: include: Obtain order information and route information and courier information related to the order information; Based on the order information and the route zone information, a dispatching model obtained by the dispatching model training method according to any one of claims 1 to 6 is used to generate the route zone division and individual dispatching strategy of each courier in the courier information.
8. The method according to claim 7, in, The order information includes: orders within a set time period, and the buildings where each order is located; the road area information includes: the road area where each building in the order is located; The road zone division and individual dispatching strategy include: the buildings in the road zone corresponding to each courier in the courier information and the order in which each courier visits the buildings in the respective road zone.
9. A scheduling model training device, the device include: A sample acquisition unit is configured to acquire a training sample set, wherein the training sample set includes at least one training sample, each training sample includes delivery individual information of a plurality of delivery individuals and road area information of a plurality of road areas; A model acquisition unit is configured to acquire a pre-trained simulation model and a decision model to be trained, wherein the simulation model is used to simulate the action sequence of the distribution individual under the road zone division and the individual dispatching strategy, and obtain the simulated efficiency value of the action sequence; The decision model is based on the distribution individual information, road area information, the road area division and individual dispatching strategy at the current moment, and the efficiency value corresponding to the road area division and individual dispatching strategy, and uses a reinforcement learning method to obtain the road area division and individual dispatching strategy of the dispatch distribution individual at the next moment; A selection unit, configured to select a training sample from the training sample set; A control unit is configured to control the decision model to provide road zone division and individual scheduling strategy for the simulation model based on the training sample and the sample efficiency value; A replacement unit is configured to obtain a simulation efficiency value corresponding to the road area division and the individual scheduling strategy from the simulation model, and use the simulation efficiency value to replace the sample efficiency value; The obtaining unit is configured to obtain a scheduling model corresponding to the decision model in response to the decision model satisfying a training completion condition.
10. A scheduling device, the device include: An information acquisition unit, configured to acquire order information and route information and courier information related to the order information; The generation unit is configured to generate the road zone division and individual scheduling strategy of each courier in the courier information based on the order information and the road zone information, using the scheduling model obtained by the scheduling model training device according to claim 9.
11. An electronic device, include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.
12. A computer readable medium having a computer program stored thereon, in, When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Multi-machine collaborative air combat planning method and system based on deep reinforcement learning
CN112861442A
Resource scheduling intelligent decision-making method and device, electronic equipment and storage medium
CN114595970A
Predicted arrival time acquisition method and device, electronic equipment and storage medium
CN115222316A
Real-time distribution and real-time cooperation scheduling system based on hierarchical reinforcement learning
CN115510664A
Logistics trolley energy-saving dispatching system based on deep learning
CN115685800A
