A logistics robot scheduling method based on deep reinforcement learning

By applying the combination of deep reinforcement learning and KM combination optimization algorithms in hospital logistics scheduling, a deep state value network is built, which solves the limitations of solving reliability and computing efficiency in multi-logistics robot scheduling, and achieves efficient and accurate logistics scheduling , suitable for complex and dynamic hospital logistics needs.

CN118536783BActive Publication Date: 2024-11-01ZHEJIANG UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411009101.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-01
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

The prior art has limitations in solving reliability and computing efficiency in multi-logistics robot scheduling. Especially in hospital logistics scheduling tasks, time and space are limited, and traditional heuristic methods are difficult to effectively deal with complex and dynamic scheduling problems. .

Method used

Using a method based on deep reinforcement learning, a deep state value network (DVN) is built and combined with KM combination optimization algorithm, the state value function is updated through offline learning and timing difference method, and the logistics robot and order are matched in real time, and the idle logistics robot is dispatched to the highest state value. area.

Benefits of technology

Improve the accuracy and resource utilization of logistics robot scheduling, optimize order processing, reduce calculation overhead, adapt to logistics needs in different time periods, significantly improve scheduling efficiency and accuracy, and enhance the robustness and adaptability of the system. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118536783B_ABST
    Figure CN118536783B_ABST
Patent Text Reader

Abstract

The present invention discloses a logistics robot scheduling method based on deep reinforcement learning. The method comprises: first, constructing and training a deep state value network DVN, updating the state value function by using the time series difference method through offline learning of historical order data, and optimizing the scheduling strategy by using the deep reinforcement learning method of multiple logistics robots; then, realizing the real-time matching of logistics robots and orders through the combined optimization algorithm of DVN and KM; finally, scheduling idle logistics robots to go to areas with higher expected returns to wait for the next round of scheduling. The present invention realizes the efficient scheduling of logistics orders by hospital logistics robots by constructing a deep state value network DVN and adopting a combined optimization method. The present invention has significant advantages in improving order response speed and reducing order waiting time, and is suitable for various hospital scenarios, and can effectively improve logistics scheduling efficiency and optimize resource allocation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and logistics scheduling, and in particular to a logistics robot scheduling method based on deep reinforcement learning. Background Art

[0002] The hospital logistics robot dispatching system consists of multiple logistics robots working together, so its dispatching control directly affects the transportation efficiency. Although there are few studies on logistics robot task dispatching strategies in hospital scenarios, solving logistics dispatching problems through algorithm optimization has always been a widely discussed topic and has been widely used in other application scenarios such as online car-hailing, factories, docks, and airports.

[0003] (Thanos E, Wauters T, Vanden Berghe G. Dispatch and conflict-free routing of capacitated vehicles with storage stack allocation[J]. Journal of the Operational Research Society, 2021, 72(8): 1780-1793.) A heuristic method is proposed to combine the scheduling and conflict-free routing of load vehicles with the storage allocation problem to minimize the total completion time of pickup, delivery, and relocation requests in a warehouse environment. They evaluated different metaheuristic methods of the proposed method using two methods and demonstrated its effectiveness in dealing with large-scale instances. (Gayathri Devi K, Mishra RS, Madan A K. Combined Objective Optimization of FMS Scheduling by a Hybrid Genetic Algorithm [C] / / Advances in Mechanical and Materials Technology: Select Proceedings of EMSME 2020. Springer Singapore, 2022: 125-139.) A new hybrid method called GAPSOTS is proposed to minimize machine idle time and overall penalty cost through combinatorial optimization. This hybrid technique is a fusion of three meta-heuristic methods: genetic algorithm (GA), taboo search (TS) and particle swarm optimization (PSO). By comparing the experimental results of other hybrid techniques, it is found that the GAPSOTS algorithm outperforms other existing meta-heuristic techniques in performance, thus proving its feasibility and adaptability. Aiming at the scheduling problem of logistics robots in automated container terminals, a two-level planning model was designed (YongshengYang, Meisu Zhong, Yasser Dessouky and Octavian Postolache, 2018, An integrated scheduling method for AGV routing in automated containerterminals, Computers & Industrial Engineering, 126, pp.482-493.), and the rolling range process (RHP) and the anti-blocking rule-based two-level genetic algorithm (CPR-BGA) were studied and compared in depth.In general, since the multi-AGV task scheduling problem is an NP-hard problem, most studies use intelligent optimization algorithms or hybrid optimization algorithms to solve it. Although these studies have made great progress, as the scale of application increases, these methods still have limitations in solution reliability and computational efficiency. Therefore, future research may need to explore more effective algorithms or combine different methods to address this challenge.

[0004] Multi-logistics robot deep reinforcement learning (DRL) combines reinforcement learning and deep neural networks. It can handle complex environments with large amounts of data and has strong generalization capabilities. It has been successfully applied to games, chess competitions, autonomous driving, logistics robot control, logistics scheduling and other fields, proving its strong potential and feasibility in practical problems. Compared with traditional heuristic methods, the biggest advantage of DRL lies in the offline parameter training process, which learns the characteristics of the solution space and builds a parameterized computational graph to simulate the constraints of the original problem. This means that when a new scheduling problem arises, the optimized scheduling solution can be obtained by simply inputting the problem into a well-trained reinforcement learning model, while traditional heuristic methods must start from scratch to find the best solution.

[0005] Since the decision-making task of logistics scheduling can be naturally modeled as a Markov decision problem, many recent studies have been devoted to using deep reinforcement learning algorithms of multiple logistics robots that are independent of environmental models to solve logistics scheduling problems, such as (Singh B, Kumar R, Singh V P. Reinforcement learning in robotic applications: a comprehensive survey[J]. Artificial Intelligence Review, 2022: 1-46.), (Vahdat-Nejad H, Tamadon T, Salmani F, et al. A Survey on Crowdsourcing Applications in Smart Cities[J]. Recent Innovations in Artificial Intelligence and Smart Applications, 2022: 239-253.), etc., among which the application effects in the fields of online car-hailing and delivery are significant. (Oda T, Joe-Wong C. MOVI: A model-free approach to dynamic fleet management[C] / / Proc of IEEE INFOCOM Conf on Computer Communications. Piscataway, NJ: IEEE, 2018: 2708-2716)Use convolutional neural network to extract supply and demand distribution features, and use distributed DQN strategy to learn the optimal dispatching action of individual vehicles. (Liu Zhidan, Li Jiangzhou, Wu Kaishun. Context-aware taxi dispatching at city-scale using deep reinforcement learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, DOI: 10.1109 / TITS.2020.3030252)Use clustering algorithm to divide the road network into multiple areas for online car-hailing dispatching, and then calculate the regional level supply and demand through the demand prediction model of environment perception and real-time online car-hailing status, and finally add these supply and demand information to the status to improve the performance of DQN.

[0006] In summary, the research on multi-AGV task scheduling has received more and more attention. Compared with ordinary scheduling tasks, hospital logistics scheduling tasks have more time and space constraints, requiring logistics robots to appear in a specific spatial location within a specific time range to perform tasks. Summary of the invention

[0007] In view of the shortcomings of the prior art, the object of the present invention is to provide a logistics robot scheduling method based on deep reinforcement learning.

[0008] The objective of the present invention is achieved through the following technical solution: A logistics robot scheduling method based on deep reinforcement learning, comprising the following steps:

[0009] 1. Build and train the deep state value network (DVN) model, use the temporal difference method to update the state value function by offline learning of historical order data;

[0010] 2. Real-time matching of logistics robots and orders based on KM combination optimization algorithm and DVN model;

[0011] 3. Dispatch the idle logistics robot to the area with the highest status value in the nearest n road sections, where: .

[0012] Furthermore, the step 1 includes the following sub-steps:

[0013] 1.1. Markov decision process modeling: The actual physical process of logistics robots delivering orders is modeled as a Markov decision process, and the state, action, reward function and reward discount coefficient are defined;

[0014] 1.2.DVN model design: Design a deep state value network DVN model so that the DVN model can fit the state value function of the logistics robot. The deep state value network DVN includes:

[0015] Input layer: mainly responsible for inputting the state characteristics of the current logistics robot into the hidden layer. The state characteristics are ; The current time period of the logistics robot Use one-hot encoding to convert it into a 144-dimensional one-hot vector and number the current section of the logistics robot. Use embedding technology to transform it into a 64-dimensional road segment vector;

[0016] Hidden layer: It mainly consists of three fully connected layers, with 256, 256 and 64 neurons in each layer. Each fully connected layer uses the rectified linear unit ReLU as the activation function to avoid the gradient vanishing problem caused by the sigmoid function.

[0017] Output layer: mainly responsible for outputting the status value of the current state of the logistics robot;

[0018] 1.3.DVN model training: The historical driving trajectories of the logistics robots are collected and converted into a series of state transition tuples as training data for the model. The state value function is updated using the temporal difference method, and the long-term return expectation of each state is learned through a deep neural network to improve the prediction accuracy and generalization ability of the DVN model.

[0019] Furthermore, the step 1.1 includes the following sub-steps:

[0020] 1.1.1. Define the state of the logistics robot: The state of the logistics robot is modeled as a three-dimensional vector of spatiotemporal state and context features. The specific time is divided into time periods. A day is divided into 144 time periods of 10 minutes, and the section number is the section number of the logistics robot. ,in Indicates a specific time period. Indicates the road section number. Indicates context features, wherein the context features include date and day of the week;

[0021] 1.1.2. Set the actions of each online logistics robot in each round of scheduling: When the logistics robot does not match any order and does not perform any task, it waits for the next round of scheduling in the current area. At this time, the status of the logistics robot changes from Transfer to , and the logistics robot will not receive any reward; when the logistics robot matches an order, the logistics robot goes to the designated order starting point to load the goods, and then delivers the goods to the order destination along the established route, and receives the order reward;

[0022] 1.1.3. Setting rewards and reward discount factors: rewards for a single order The function is expressed as follows:

[0023]

[0024] in, The actual time taken for the order is used in the offline learning phase, while the estimated delivery time of the logistics robot is used in the actual dispatch of the hospital dispatch center. In addition, the reward discount coefficient is taken into account. , for a lasting The reward for orders in a certain time period is The function is expressed as follows:

[0025] .

[0026] Furthermore, the step 1.3 includes the following sub-steps:

[0027] The historical data of the hospital logistics robot is decomposed into a set of state-action pairs, including the actions of the logistics robot accepting orders and transporting, and the actions of idling and waiting. All logistics robots share the learned state value function.

[0028] For an idle logistics robot, the current state is , For the time period, is the road section number. is the context feature; after a period of time, the state of the logistics robot changes from Updated to , the time period is updated to ; Because the logistics robot has not received any orders, the reward it receives is 0. At this time, its state value function The updates are as follows:

[0029]

[0030] in, represents the learning rate, represents the reward discount factor;

[0031] For a logistics robot that receives an order, the current status is , For the time period, is the number of the road section; if the order takes 20 minutes, then after two periods, the status of the logistics robot changes from Updated to , the time period is updated to ; The reward that the logistics robot receives for accepting the order is , at this time its state value function The updates are as follows:

[0032]

[0033] General state value function The updated formula is as follows:

[0034]

[0035] in, Indicates the number of time slots taken for state transfer, Indicates the state of the logistics robot at time t+k;

[0036] Using this general state value function The updated formula can iteratively update the state value function network ;

[0037] After collecting the historical driving trajectories of all logistics robots, they are converted into a series of state transition tuples, where each tuple represents a logistics robot from state Convert to The time step is , and the order reward is calculated according to whether the logistics robot accepts the order; the training is divided into the original state value function network and the target state value function network ,in, represents the weights of all trainable neural networks, Represents the weight of the target state value function network; the target state value function network and the main network regularly synchronize parameters, and the target state value function network is used to evaluate the state; the loss function selects Huber Loss, which is specifically expressed as follows:

[0038]

[0039] in, represents the loss function, represents the true value, represents the predicted value of the model, Represents the smoothing parameter of Huber Loss, which determines which interval uses the square error MSE and which interval uses the absolute error MAE; when the absolute value of the prediction error is less than or equal to When the absolute value of the prediction error is greater than When , the Huber loss switches to the absolute error form to reduce the impact of the model on outliers.

[0040] Furthermore, the step 2 includes the following sub-steps:

[0041] 2.1. Modeling of bipartite graph matching problem: The allocation process between logistics robots and orders is modeled as a bipartite graph matching problem, and the objective function of order scheduling is defined;

[0042] 2.2. Collect idle logistics robots and unassigned orders: monitor the status of all logistics robots in the hospital in real time, and obtain all currently idle logistics robots and pending logistics orders;

[0043] 2.3. Calculate the advantage function based on the DVN network: Calculate the advantage function of each logistics robot-order pair based on the DVN model trained in step 1;

[0044] 2.4. Use combinatorial optimization algorithm for matching: Use KM combinatorial optimization algorithm to solve the bipartite graph matching problem and obtain the matching order that can maximize the objective function;

[0045] 2.5. Dispatch logistics robots to perform order tasks: Dispatch logistics robots to perform corresponding order pick-up and delivery tasks according to the matching orders.

[0046] Furthermore, the step 2.1 is specifically as follows:

[0047] For each order that has not been accepted by a logistics robot, an idle logistics robot is selected to match and accept the order; if there is no idle logistics robot in this round of matching, the order will continue to be matched in the next round; a logistics robot can only match one order at most in a single round, and an order can only be transported by one logistics robot to avoid cross-contamination between drugs; the matching of logistics robots and orders is modeled as a bipartite graph matching problem;

[0048] Each logistics robot seeks the best behavior to achieve the best match between logistics robots and orders to optimize the overall future benefits; the objective function of the centralized order scheduling algorithm is expressed as:

[0049]

[0050] in, Indicates the matching situation between logistics robot i and order j, represents the order number, n represents the maximum order number, represents the logistics robot number, m represents the maximum logistics robot number, Indicates order Assigned to logistics robots , Indicates order Not assigned to a logistics robot , All available logistics robots corresponding to this timestamp, and For unfilled orders, when or Indicates that the logistics robot or order failed to be successfully matched in this round of dispatch; Indicates order and logistics robots The matching weight value of the logistics robot Receive Order And execute the action value function of the transportation task; the DVN network learns the state value function through the temporal difference method. According to the Bellman equation, the two have the following conversion formula:

[0051]

[0052] in, represents the action, s represents the current state, Indicates selection action The next state of post-logistics robots, represents the action value function of taking action a in state s, represents the reward function for taking action a in state s, Indicates the next state The state value function of .

[0053] Furthermore, the above 2.3 is specifically as follows:

[0054] Calculate the expected reward for each logistics robot-order pair , and then each idle logistics robot Current status Pass it into the DVN state value network to calculate its state value , where t represents the current time period, g represents the current road section number, and v represents the context feature composed of date and week features; according to the estimated order delivery time, each logistics robot Complete the order After the status Pass it into the DVN state value network to calculate its state value Finally, the advantage function corresponding to each logistics robot-order pair is calculated ,in, Indicates the time period after the order is completed. Indicates the segment number after the order is completed.

[0055] Furthermore, the step 2.4 includes the following sub-steps:

[0056] 2.4.1. Top label initialization: Initialize the top label lx[u] for each order vertex u∈U, and initialize the top label ly[v] for each logistics robot vertex v∈V;

[0057] Among them, U represents the vertex set of orders to be matched, and V represents the vertex set of idle logistics robots;

[0058] 2.4.2. Finding augmenting paths: Use depth-first search to find augmenting paths;

[0059] 2.4.3. Top label adjustment: When an augmenting path does not exist, the weight difference between vertices is reduced by adjusting the top labels to find a minimum value δ so that the augmenting path exists in the adjusted path;

[0060] 2.4.4. Update matching relationship: Update the matching relationship through alternating paths to ensure that the matching is the maximum weight matching, and finally obtain the matching pair between the order and the logistics robot.

[0061] Furthermore, the step 3 includes the following sub-steps:

[0062] 3.1. Obtain idle logistics robots and calculate the status value of nearby road sections: Analyze the current location and possible future order demand of all idle logistics robots in the hospital, and calculate the status value of nearby road sections;

[0063] 3.2. Dispatch idle logistics robots to the road sections with high status values: According to the calculated status values, dispatch idle logistics robots to the road sections with the highest status values ​​among the nearest n road sections and wait for the next round of dispatch to improve the response efficiency and resource utilization of logistics robots. .

[0064] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the above-mentioned logistics robot scheduling method based on deep reinforcement learning.

[0065] The beneficial effects of the present invention are:

[0066] The present invention realizes efficient scheduling of hospital logistics robots by constructing a deep state value network (DVN) and adopting a combinatorial optimization algorithm. DVN updates the state value function through offline learning and temporal difference method, which effectively improves the accuracy of the scheduling strategy and resource utilization. The combinatorial optimization algorithm can quickly match logistics robots and orders in real-time scheduling, optimize resource allocation, and reduce order processing delays. The present invention proposes for the first time a hospital logistics scheduling system that combines deep reinforcement learning and combinatorial optimization of multiple logistics robots, which performs well in coping with complex and changeable hospital logistics needs. While reducing computational overhead, the present invention can dynamically adjust the scheduling strategy to adapt to logistics needs in different time periods, significantly improving the efficiency and accuracy of logistics scheduling. Compared with traditional logistics scheduling methods, the present invention not only improves the robustness and adaptability of the system, but also has a wide range of application prospects, is suitable for hospital scenarios of various sizes, and has the advantages of strong versatility and ease of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is the overall architecture diagram of the scheduling model of the present invention;

[0068] Figure 2 This is a DVN network structure diagram of the present invention;

[0069] Figure 3 A schematic diagram of updating the state value function using the temporal difference method of the present invention;

[0070] Figure 4 This is a schematic diagram of the matching between the logistics robot and the order of the present invention;

[0071] Figure 5This is a schematic diagram of the present invention using advantage function instead of action value function. DETAILED DESCRIPTION

[0072] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0073] Example 1

[0074] like Figure 1 As shown, a logistics robot scheduling method based on deep reinforcement learning includes the following steps:

[0075] The present invention uses the total time consumption of long-term orders to quantify the dispatching efficiency of the hospital logistics robot dispatching center. In order to maximize the efficiency of the dispatching center, the total time consumption of all orders needs to be minimized, which can be expressed by the following optimization objective function.

[0076]

[0077] where t ij represents the time it takes for order i to be delivered by logistics robot j, a ij =1 means order i is assigned to logistics robot j, a ij =0 means that order i has not been assigned to logistics robot j, and each order can only be assigned to one logistics robot.

[0078] 1. Build a deep state value network (DVN), conduct offline learning on historical order data, and use the temporal difference method to update the state value function.

[0079] 1.1. Markov decision process modeling.

[0080] The present invention formalizes the logistics order scheduling problem into a continuous, sequential decision-making process, in which each decision point involves deciding whether to assign a logistics robot to a specific order. Unlike focusing only on the current optimal solution (such as simply minimizing the distance that all logistics robots travel to pick up items), this model fully considers the impact of each scheduling decision on the future spatiotemporal distribution of supply and demand, and strives to achieve global optimality. To this end, the present invention uses reinforcement learning to construct a centralized scheduling algorithm, which aims to discover a set of behavioral strategies that can maximize future total returns. In the logistics scheduling scenario of hospital logistics robots, the total return is reflected in the maximization of the overall operating efficiency of the scheduling center. Considering the functional homogeneity and mutual independence of each logistics robot, the global state space of the entire scheduling center can be constructed by performing Cartesian product operations on the state spaces of all individual logistics robots. The decisions of all logistics robots are based on the unified information provided by the scheduling center, and they work together towards the same optimization goal. In this way, reinforcement learning can guide the scheduling center to dynamically adjust its strategy in a changing environment and improve the overall scheduling performance.

[0081] Markov decision process definition: Markov decision process (MDP) is usually used to simulate sequential decision problems. In MDP, the behavior of a logistics robot in the environment is based on a policy, which specifies how the logistics robot chooses to act in each state of the MDP. The present invention models a single logistics robot as a logistics robot, and each logistics robot shares the same scheduling policy.

[0082] State definition: In real scenarios, the dynamic scheduling of the dispatch center not only needs to obtain the location of each logistics robot, but also needs to take into account the busyness of each area of ​​the hospital at the current moment. Therefore, the state of the logistics robot is modeled as a three-dimensional vector of spatiotemporal state and contextual features. In order to avoid the inconvenience in training caused by time continuity, the present invention divides the specific time into time periods. A day is divided into 144 time periods of 10 minutes, and the section number is the section number where the logistics robot is currently located. That is, ,in Indicates a specific time period. Indicates the road section number. Represents contextual features, which are composed of static features such as date and day of the week.

[0083] Action definition: Each online logistics robot has only two actions in each round of scheduling. The first action is that the logistics robot does not match any order, does not perform any task, and waits for the next round of scheduling in the current area. At this time, the status of the logistics robot changes from Transfer to , at this time, the logistics robot will not receive any reward. The second action is that the logistics robot matches an order. In this case, the logistics robot will go to the designated order starting point to load the goods, and then deliver the goods to the order end point according to the established route and receive the order reward.

[0084] Reward and Reward Discount Coefficient Definition: The definition of reward determines the optimization goal of the entire reinforcement learning system. The goal of the hospital's logistics dispatch center is to reasonably assign tasks to designated logistics robots so that all medicines can be delivered to patients or nurses as quickly as possible, so that the overall logistics efficiency is maximized, that is, the total time required for all orders should be the shortest. Therefore, the design of the reward function is related to the time required for a single order. The reward for a single order The function is expressed as follows:

[0085]

[0086] in The actual delivery time of the order is used in the offline learning stage. However, when the hospital dispatch center is actually dispatching, the estimated delivery time of the logistics robot is used because the order has not been completed. In addition, the reward discount factor needs to be considered. , because the goal of reinforcement learning is to maximize the expected reward of the logistics robot, which includes not only the immediate reward but also the reward obtained in the future. The reward for orders in a certain time period is The function is expressed as follows:

[0087]

[0088] 1.2. DVN model design.

[0089] Designing a good neural network structure is crucial to solving complex multi-logistics robot deep reinforcement learning problems, because a good neural network can model data and solve problems more efficiently and reduce the required computing resources and training time. In addition, it can help the model generalize better to unseen data.

[0090] like Figure 2 As shown in the figure, the DVN network mainly consists of three parts: input layer, hidden layer and output layer.

[0091] Input layer: The input layer is mainly responsible for inputting the state characteristics of the current logistics robot into the hidden layer. ,in Indicates a specific time period. Indicates the road section number. Represents context features. For the logistics robot, that is, the current time period of the logistics robot Use one-hot encoding to convert it into a 144-dimensional one-hot vector and number the current section of the logistics robot. The embedding technology is used to transform it into a 64-dimensional road segment vector. Represents contextual features, such as date, day of the week, and other static features.

[0092] Hidden layer: The hidden layer is mainly composed of three fully connected layers, and the number of neurons in each layer is 256, 256 and 64 respectively. Each fully connected layer uses the rectified linear unit ReLU as the activation function, which can avoid the gradient vanishing problem caused by the sigmoid function.

[0093] Output layer: The output layer is mainly responsible for outputting the state value of the current state of the logistics robot.

[0094] 1.3. DVN model training.

[0095] The present invention decomposes the historical data of hospital logistics robots into a set of state-action pairs, which includes not only the actions of logistics robots accepting orders and transporting, but also the actions of logistics robots waiting in idle state, and does not distinguish any logistics robots. That is, the learned state value function is common to all logistics robots. Figure 3 As shown, for an idle logistics robot, assuming the current state , For the time period, is the road section number. is the context feature. After a period of time, the state of the logistics robot changes from Updated to , the time period is updated to Since the logistics robot has not received any orders, the reward it receives is 0. At this time, its state value function The updates are as follows:

[0096]

[0097] in represents the learning rate, Represents the reward discount coefficient. For a logistics robot that receives an order, assuming the current state , For the time period, is the number of the road section. The order takes 20 minutes. After two periods, the status of the logistics robot changes from Updated to , the time period is updated to The reward that the logistics robot receives for accepting orders is , at this time its state value function The updates are as follows:

[0098]

[0099] General state value function The updated formula is as follows:

[0100]

[0101] in, Indicates the number of time slots taken for state transfer, Represents the state of the logistics robot at time t+k. This formula can be used to iteratively update the state value function network .

[0102] The present invention collects the historical driving trajectories of all logistics robots and converts them into a series of state transition tuples, where each tuple represents an independent logistics robot from state Convert to The time step is , and the order reward is calculated according to whether the logistics robot accepts the order. In order to ensure the stability of DVN network training, the training adopts a structure similar to DQN, which is divided into the original state value function network ( represents the weights of all trainable neural networks) and the target state value function network ( Represents the weight of the target state value function network). The target network and the main network periodically synchronize parameters, and the target network is used to evaluate the state. The loss function is Huber Loss, also known as smoothed absolute error, which is a loss function used in regression problems and is particularly suitable for processing data with outliers. Its specific representation is as follows:

[0103]

[0104] in, represents the loss function, represents the true value, represents the predicted value of the model, Represents the smoothing parameter of Huber Loss, which determines which interval uses the square error (MSE) and which interval uses the absolute error (MAE). When the Huber loss function is greater than , the square error form is used. This is because the square error has better stability and simplicity than the absolute error in the range of small errors, and can provide enough gradients for model learning. When , the Huber loss switches to the absolute error form, because in the interval with large error (that is, the interval where outliers may exist), the absolute error is less sensitive to outliers than the squared error, thereby reducing the impact of the model on outliers.

[0105] 2. Real-time matching of logistics robots and orders based on combinatorial optimization algorithms.

[0106] 2.1. Modeling of bipartite graph matching problem.

[0107] The scheduling system described in the present invention needs to perform a round of matching at fixed time intervals. In each round of matching, the function of the scheduling algorithm is: for each order that has not been accepted by the logistics robot, select an idle and suitable logistics robot to accept the order. If there is no idle logistics robot in this round of matching, the order will continue to be matched in the next round. It should be noted that in view of the limited actual transportation capacity of hospital logistics robots and the fact that most orders involve drug transportation, in order to ensure that drugs do not contaminate each other, it is stipulated that the logistics robot can only match one order in a single round, and one order can only be transported by one logistics robot. The matching of logistics robots and orders can be modeled as a bipartite graph matching problem, such as Figure 4 As shown, on the left are the orders to be matched issued by medical staff, on the right are the currently idle logistics robots, the middle edge represents the matching between the order and the logistics robot, and its weight represents the value function of the matching pair. The present invention realizes the allocation between orders and logistics robots by solving this bipartite graph matching problem.

[0108] In order to achieve the best match between logistics robots and orders, in the modeling of scheduling algorithm problems, this goal can also be interpreted as finding the best behavior for each logistics robot to optimize the overall future benefits. The objective function of the centralized order scheduling algorithm can be expressed in mathematical expressions as:

[0109]

[0110] in, Indicates the matching situation between logistics robot i and order j, represents the order number, n represents the maximum order number, represents the logistics robot number, m represents the maximum logistics robot number, Indicates order Assigned to logistics robots , When indicating an order Not assigned to a logistics robot .here, All available logistics robots corresponding to this timestamp, and Corresponds to unfilled orders. or Indicates that the logistics robot or order has not been successfully matched in this round of dispatch. The order and logistics robots The matching weight value of the logistics robot Receive Order And execute the action value function of the transport task. The DVN network proposed in the present invention learns the state value function through the temporal difference method. According to the Bellman equation, the two have the following conversion formula:

[0111]

[0112] in, represents the action, s represents the current state, Indicates selection action The next state of post-logistics robots, represents the action value function of taking action a in state s, represents the reward function for taking action a in state s, Indicates the next state In order to convert the above matching weight from action value function to state value function, the advantage function in the A2C algorithm is used to replace the action value function, such as Figure 5 shown.

[0113] 2.2. Collect idle logistics robots and unassigned orders.

[0114] After medical staff initiate an order pickup and delivery task, the hospital information management system will generate an order. The logistics dispatch center will collect all unassigned orders and idle logistics robots at fixed time intervals to obtain the estimated time required for the logistics robots to complete the delivery of the order.

[0115] 2.3. Calculate the advantage function based on the DVN network.

[0116] Calculate the expected reward for each logistics robot-order pair , and then each idle logistics robot Current status Pass it into the DVN state value network to calculate its state value , where t represents the current time period, g represents the current road section number, and v represents the context feature composed of date and week features; according to the estimated order delivery time, each logistics robot Complete the order After the status Pass it into the DVN state value network to calculate its state value Finally, the advantage function corresponding to each logistics robot-order pair is calculated ,in, Indicates the time period after the order is completed. Indicates the segment number after the order is completed.

[0117] 2.4. Use combinatorial optimization algorithm for matching.

[0118] The KM algorithm is used to calculate the maximum matching problem of the weighted bipartite graph, and the weight value of each edge is the advantage function .

[0119] In the specific implementation, the KM algorithm is mainly completed through the following parts:

[0120] 2.4.1. Top label initialization: Initialize the top label lx[u] for each order vertex u ∈ U, and initialize the top label ly[v] for each logistics robot vertex v ∈ V, where U represents the vertex set of orders to be matched, and V represents the vertex set of idle logistics robots.

[0121] 2.4.2. Finding augmenting paths: Use depth-first search (DFS) to find augmenting paths.

[0122] 2.4.3. Top label adjustment: When an augmenting path does not exist, the weight gap between vertices is narrowed by adjusting the top labels. Usually, a minimum value δ needs to be found so that an augmenting path exists on the adjusted path.

[0123] 2.4.4. Update matching relationship: Update the matching relationship through alternating paths to ensure that the matching is the maximum weight matching, and finally obtain the matching pair between the order and the logistics robot.

[0124] 2.5. Dispatch logistics robots to execute order tasks.

[0125] The matching information calculated in the above steps is forwarded to each logistics robot. After receiving the order notification, the logistics robot will go to the departure node of the order according to the preset route to pick up the goods and deliver the goods to the designated destination.

[0126] 3. Dispatch idle logistics robots to areas with higher expected returns.

[0127] 3.1. Obtain idle logistics robots and calculate their status values.

[0128] From the definition of reinforcement learning, we can know that the state value function in the present invention is , which actually means the expected future revenue of the logistics robot under the road section number in the current period. State value function The larger the value, the greater the expected future revenue of the road section in the current period. The expected future revenue directly affects the efficiency of the dispatching platform. Therefore, if the idle logistics robots at the current moment are located near the road section with higher expected revenue, the overall efficiency of the dispatching platform can be improved.

[0129] When the matching of logistics robots and orders described above is completed, the dispatch center will obtain all idle logistics robots. For each idle logistics robot, the status of each nearby road section is calculated according to the current time period. , and pass the state features into the DVN state value network to calculate the state value of each nearby road segment .

[0130] 3.2. Dispatch idle logistics robots to sections with high status value.

[0131] Based on the calculated status value, idle logistics robots are dispatched to the road section with the highest status value among the nearest 5-10 road sections and wait for the next round of scheduling to improve the response efficiency and resource utilization of logistics robots. Limiting to 5-10 road sections can save robot power and avoid congestion caused by a large number of robots going to one area.

[0132] Example 2

[0133] An implementation example of the present invention is implemented on a machine equipped with an Intel Core i7-3770 central processor, an NVidia GTX760 graphics processor and 32GB of memory. The present invention designs a logistics robot operation simulator and uses historical data of the real hospital logistics robot operation environment. The simulator fully simulates the operation map of the hospital logistics robot, including the corresponding road sections and nodes, and can simulate and generate corresponding orders based on the completed historical orders in the hospital database. Each logistics robot can specify an initial position and move according to the set route, and when encountering congestion on a two-way road section, it will perform avoidance operations at nearby nodes. In addition, the simulator also simulates the elevator operation system for the logistics robot to go up and down the stairs.

[0134] In order to compare the advantages and disadvantages of different algorithms, this paper uses the total time consumption of all orders to quantify the efficiency of the scheduling algorithm. The lower the total time consumption, the higher the utilization rate of the logistics robot; conversely, if the total time consumption is higher, the lower the utilization rate of the logistics robot, which is also consistent with the design of the reward function in the previous article.

[0135] The algorithms compared this time include: naive distance greedy matching, distance greedy matching based on combinatorial optimization, matching + scheduling based on state value function table method, Double-DQN matching based on combinatorial optimization, and DVN matching based on combinatorial optimization and DVN matching + DVN scheduling algorithm based on combinatorial optimization proposed by the present invention. The detailed introduction of each algorithm is as follows:

[0136] Based on naive distance greedy matching (greedy): Naive distance greedy means matching each unassigned order with the idle logistics robot closest to the order starting point at a fixed time interval. If there is no idle logistics robot available for matching, it will be matched again in the next time interval. Naive distance greedy is the original scheduling allocation algorithm currently used in hospitals.

[0137] Distance Greedy Matching (KM-greedy) based on combinatorial optimization: This algorithm combines the KM algorithm with the naive distance greedy algorithm. The naive distance greedy algorithm searches for the nearest idle logistics robot for all unassigned orders in each time interval. The KM algorithm matches all unassigned orders with idle logistics robots at the same time, which can make the total distance of all logistics robot-order matching pairs in the current round the shortest.

[0138] DVN matching based on combinatorial optimization: This algorithm only uses the KM algorithm and the deep state value network DVN for matching. After the matching is completed, the idle logistics robot will not be dispatched to the nearby road section with higher expected benefits.

[0139] DVN matching + DVN scheduling based on combinatorial optimization: This algorithm is the complete DVN network real-time scheduling algorithm proposed in the present invention, which includes real-time matching of logistics robots and orders and scheduling of idle logistics robots.

[0140] Matching + Scheduling Based on State-Value Function Table Method (V-table): This algorithm is similar to the traditional reinforcement learning algorithm Q-Learning. The difference is that Q-Learning uses a table method to save the action value function , while the state-value function table method saves the state-value function In this experiment, the state By time period and section number Composition, that is .

[0141] Double-DQN matching based on combinatorial optimization: This algorithm uses the DQN network to learn the action-value function , and As the weight value matched by the KM algorithm. The network structure of Double-DQN ensures the stability of DQN network training.

[0142] The total time-consuming experimental results are shown in Table 1.

[0143] Table 1: Comparison of time consumption between a single day and a week

[0144] It can be seen from the experimental results of the simulator that the DVN matching + DVN scheduling algorithm based on combinatorial optimization proposed in the present invention is superior to other algorithms, whether it is a single day or a week.

[0145] Although the implementation of the naive distance greedy matching algorithm is relatively simple, it only matches the idle logistics robot closest to the starting point of each unassigned order at each match, but it cannot guarantee the shortest total distance of all single matches, resulting in the longest total time consumption and the lowest efficiency. The distance greedy matching (KM-greedy) algorithm based on combinatorial optimization solves this problem. At each match, KM-greedy will make the total time consumption of all single matches the shortest. However, KM-greedy cannot predict the spatiotemporal distribution of future orders and logistics robots, so each match can only achieve local optimization, and cannot achieve global optimization. Whether it is the Double-DQN matching algorithm based on combinatorial optimization or the DVN matching algorithm based on combinatorial optimization proposed in the present invention, this type of scheduling problem is modeled as a sequential decision problem of reinforcement learning, and the future rewards are calculated by designing a discount coefficient, thereby taking into account the future distribution of orders and logistics robots and achieving a better matching effect. Although the Double-DQN matching algorithm can achieve relatively good matching results, it cannot quantify the state value of each road section in a specified period. The actual meaning of the state value is to indicate the expected future benefits of the logistics robot under the road section number in the current period. Therefore, Double-DQN cannot realize the scheduling of idle logistics robots, which may cause some logistics robots to stay in the area with low state value for a long time and fail to match orders for a long time, thus reducing the overall efficiency to a certain extent. The matching + scheduling (V-table) based on the state value function table method and the DVN matching + DVN scheduling based on combinatorial optimization can solve this problem well. The scheduling center only needs to schedule the logistics robots in the area with low state value to the area with high state value. From the experimental results, it can be seen that the time consumption of V-table matching + scheduling and DVN matching + DVN scheduling is less than that of other algorithms. However, because the V-table algorithm stores the state value function in the form of a table in the memory, the size of the table will increase exponentially with the growth of the state space, which is not conducive to storage and processing, that is, the so-called dimensional disaster problem occurs. In addition, the V-table algorithm cannot learn more contextual features and it is difficult to generalize to unseen states. The deep state value network DVN algorithm proposed in this invention uses a neural network to parameterize the value function. Unlike the V-table algorithm, which uses simple time and space to represent the state, the DVN algorithm not only learns the road segment features of the map through the embedding matrix, but also adds other contextual features, solving the shortcomings of the V-table algorithm.

[0146] The above experimental results show that in the field of hospital logistics scheduling, the DVN matching + DVN scheduling algorithm based on combinatorial optimization proposed in the present invention has better effects. Compared with the best performing V-table algorithm in the baseline model, the DVN matching + DVN scheduling algorithm achieves a 2.8% and 3.6% reduction in total time consumption in a single day and a week, respectively.

[0147] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement a logistics robot scheduling method based on deep reinforcement learning of any of the above embodiments.

[0148] It should be noted that the computer-readable storage medium embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

Claims

1. A logistics robot scheduling method based on deep reinforcement learning, characterized in that: The steps include: (1) Build and train a deep state value network (DVN) model, perform offline learning on the historical order data of the logistics robot, and use the temporal difference method to update the state value function of the logistics robot; The step (1) comprises the following sub-steps: (1.1) Markov decision process modeling: The actual physical process of the logistics robot delivering orders is modeled as a Markov decision process, and the state, action, reward function and reward discount coefficient are defined; (1.2) DVN model design: Design a deep state value network DVN model so that the DVN model can fit the state value function of the logistics robot. The deep state value network DVN includes: Input layer: responsible for inputting the state features of the current logistics robot into the hidden layer, the state features are s = (t, g, v); the current time period t of the logistics robot is converted into a 144-dimensional one-hot vector using one-hot encoding, and the current section number g of the logistics robot is converted into a 64-dimensional section vector using embedding technology; Hidden layer: It consists of three fully connected layers, with 256, 256 and 64 neurons in each layer respectively. Each fully connected layer uses the rectified linear unit ReLU as the activation function; Output layer: responsible for outputting the status value of the current state of the logistics robot; (1.3) DVN model training: The historical driving trajectories of the logistics robot are collected and converted into a series of state transition tuples as the training data of the DVN model. The state value function is updated using the temporal difference method, and the long-term return expectation value of each state is learned through a deep neural network. The step (1.3) includes the following sub-steps: The historical data of the hospital logistics robot is decomposed into a set of state-action pairs, including the actions of the logistics robot accepting orders and transporting, and the actions of idling and waiting. All logistics robots share the learned state value function. For an idle logistics robot, the current state is S0(T0,X0,V), T0 is the time period, X0 is the road section number, and V is the context feature; after a time period, the state of the logistics robot is updated from S0(T0,X0,V) to S1(T1,X0,V), and the time period is updated to T1; because the logistics robot has not received any orders, the reward it receives is 0, and the update of its state value function V(S0) is as follows: V(S0)←V(S0)+α(0+γV(S1)-V(S0)) Among them, α represents the learning rate and γ represents the reward discount coefficient; For a logistics robot that receives an order, its current state is S0(T0,X0,V), where T0 is the time period and X0 is the road section number. If the order takes 20 minutes, then after two time periods, the logistics robot's state is updated from S0(T0,X0,V) to S2(T2,X2,V), and the time period is updated to T2. The reward the logistics robot receives for accepting the order is R γ , at this time, the update of its state value function V(S0) is as follows: V(S0)←V(S0)+α(R γ +γ 2 V(S2)-V(S0)) The general state value function V(S t )The updated formula is as follows: V(S t )←V(S t )+α(R γ +γ k V(S t+k )-V(S t )) Where k represents the number of time periods spent on state transfer, S t+k Indicates the state of the logistics robot at time t+k; Using this general state value function V(S t ) The updated formula can iteratively update the state value function network V π (s); After collecting the historical driving trajectories of all logistics robots, they are converted into a series of state transition tuples, where each tuple represents a logistics robot transitioning from state s to s ′ The time step is , and the order reward is calculated according to whether the logistics robot accepts the order; the training is divided into the original state value function network V π (s|θ) and the target state value function network Among them, θ represents the weights of all trainable neural networks, Represents the weight of the target state value function network; the target state value function network and the main network regularly synchronize parameters, and the target state value function network is used to evaluate the state; the loss function selects Huber Loss, which is specifically expressed as follows: in, represents the loss function, y represents the true value, f(x) represents the predicted value of the model, δ represents the smoothing parameter of Huber Loss, and the smoothing parameter determines which interval uses the square error MSE and which interval uses the absolute error MAE; when the absolute value of the prediction error is less than or equal to δ, the Huber loss function uses the square error form to provide sufficient gradient for the model to learn; when the absolute value of the prediction error is greater than δ, the Huber loss switches to the absolute error form to reduce the impact of the model on outliers; (2) Real-time matching of logistics robots and orders based on KM combination optimization algorithm and DVN model; The step (2) comprises the following sub-steps: (2.1) Modeling of bipartite graph matching problem: The allocation process between logistics robots and orders is modeled as a bipartite graph matching problem, and the objective function of order scheduling is defined; (2.2) Collect idle logistics robots and unassigned orders: monitor the status of all logistics robots in the hospital in real time, and obtain all currently idle logistics robots and pending logistics orders; (2.3) Calculating the advantage function based on the DVN network: Calculate the advantage function of each logistics robot-order pair based on the DVN model trained in step (1); (2.4) Matching using combinatorial optimization algorithm: Use KM combinatorial optimization algorithm to solve the bipartite graph matching problem and obtain the matching order that can maximize the objective function; (2.5) Dispatching logistics robots to execute order tasks: Dispatching logistics robots to execute corresponding order pickup and delivery tasks according to the matched orders; The step (2.4) includes the following sub-steps: (2.4.1) Top label initialization: Initialize the top label lx[u] for each order vertex u∈U, and initialize the top label ly[v] for each logistics robot vertex v∈V; Among them, U represents the vertex set of orders to be matched, and V represents the vertex set of idle logistics robots; (2.4.2) Finding the augmenting path: Use depth-first search to find the augmenting path; (2.4.3) Top scale adjustment: When an augmenting path does not exist, the weight difference between vertices is reduced by adjusting the top scale to find a minimum value δ so that an augmenting path exists on the adjusted path; (2.4.4) Update matching relationship: Update the matching relationship through alternating paths to ensure that the matching is the maximum weight matching, and finally obtain the matching pair between the order and the logistics robot; (3) Dispatch the idle logistics robot to the area with the highest state value among the nearest n road sections, where 5≤n≤10.

2. A logistics robot scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The step (1.1) comprises the following sub-steps: (1.1.1) Define the state of the logistics robot: Model the state of the logistics robot as a three-dimensional vector of spatiotemporal state and context features. Divide the specific time into time periods, and divide a day into 144 time periods of 10 minutes each. The section number is the section number of the logistics robot, that is, s = (t, g, v) ∈ S, where t represents the specific time period, g represents the section number, and v represents the context features, including date and day of the week; (1.1.2) Set the actions of each online logistics robot in each round of scheduling: When the logistics robot does not match any order and does not perform any task, it waits for the next round of scheduling in the current area. At this time, the state of the logistics robot is transferred from s0 = (t, g, v) to s1 = (t+1, g, v), and the logistics robot will not receive any reward; when the logistics robot matches an order, the logistics robot goes to the designated order starting point to load the goods, and then delivers the goods to the order end point along the established route, and receives the order reward; (1.1.3) Set the reward and reward discount coefficient: The reward P for a single order is expressed as a function as follows: Among them, Δt is the time consumption of the order. In the offline learning stage, the actual time consumption of the order is used, while in the actual dispatching of the hospital dispatch center, the estimated delivery time of the logistics robot is used. In addition, considering the reward discount coefficient γ, for an order that lasts for T time periods, its reward R γ The function is expressed as follows:

3. A logistics robot scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The step (2.1) is specifically: For each order that has not been accepted by a logistics robot, an idle logistics robot is selected to match and accept the order; if there is no idle logistics robot in this round of matching, the order will continue to be matched in the next round; a logistics robot can only match one order in a single round, and one order can only be transported by one logistics robot to avoid cross-contamination between drugs; Each logistics robot seeks the best behavior to achieve the best match between logistics robots and orders to optimize the overall future benefits; the objective function of the centralized order scheduling algorithm is expressed as: Among them, a ij represents the matching situation between logistics robot i and order j, j represents the order number, n represents the maximum order number, i represents the logistics robot number, m represents the maximum logistics robot number, a ij =1 means order j is assigned to logistics robot i, a ij =0 means that order j has not been assigned to logistics robot i, i∈[1,…,m] corresponds to all available logistics robots at this timestamp, and j∈[1,…,n] corresponds to unaccepted orders. When i=0 or j=0, it means that the logistics robot or order has not been successfully matched in this round of scheduling; Q(i,j) represents the matching weight value of order j and logistics robot i, which represents the action value function of logistics robot i receiving order j and performing the delivery task; the DVN network learns the state value function through the temporal difference method. According to the Bellman equation, the two have the following conversion formula: Q(s,a)=R(s,a)+γV(s ′ ) Among them, a represents the action, s represents the current state, and s ′ represents the next state of the logistics robot after selecting action a, Q(s,a) represents the action value function of taking action a in state s, R(s,a) represents the reward function of taking action a in state s, V(s ′ ) represents the state value function of the next state s′.

4. According to the logistics robot scheduling method based on deep reinforcement learning in claim 1, it is characterized in that: The (2.3) is specifically: Calculate the expected reward R for each pair of logistics robots and orders, and then pass the current state s = (t, g, v) of each idle logistics robot i into the DVN state value network to calculate its state value V (s i ), where t represents the current time period, g represents the current road segment number, and v represents the context feature composed of date and week features; according to the estimated order delivery time, the state s′=(t′,g′,v) of each logistics robot i after completing order j is passed to the DVN state value network to calculate its state value V(s′ ij ), and finally the advantage function A(i,j) corresponding to each logistics robot-order pair is calculated, where t′ represents the time period after the order is completed, g ′ Indicates the segment number after the order is completed.

5. According to the logistics robot scheduling method based on deep reinforcement learning in claim 1, it is characterized in that: The step (3) comprises the following sub-steps: (3.1) Obtain idle logistics robots and calculate the state value of nearby road sections: Analyze the current positions and possible future order demands of all idle logistics robots in the hospital and calculate the state value of nearby road sections; (3.2) Dispatching idle logistics robots to sections with high status values: According to the calculated status values, dispatch idle logistics robots to the section with the highest status value among the nearest n sections and wait for the next round of dispatch, where 5≤n≤10.

6. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement a logistics robot scheduling method based on deep reinforcement learning as described in any one of claims 1-5.

Citation Information

Cited By

  • Coal sample transfer robot scheduling method based on reinforcement learning

    CN121365856A

  • A coal sample transfer robot scheduling method based on reinforcement learning

    CN121365856B