Automated laboratory deadlock-free dynamic scheduling method based on monte carlo tree search

Abstract

The application is an automated laboratory non-deadlock dynamic scheduling method based on Monte Carlo tree search, and belongs to the technical field of industrial intelligent scheduling. The method models the tasks of the automated laboratory as a directed acyclic graph, searches the task scheduling decision sequence for the graph by using the Monte Carlo tree search method, the state of the tree node includes the state of whether the task is executed and the state of whether the device is occupied, the action of the edge represents scheduling which task to execute on which device, the variables of the reward value, the access frequency and the number of deadlocks are set for the tree node, the improved action selection function is designed, the branch and bound strategy is used to perform the pruning operation on the expanded child nodes, etc.; the list scheduling mode is used for actual scheduling, and the task directed acyclic graph is reconstructed when a dynamic event occurs. The method can not only complete the optimization target of minimizing the maximum task completion time, but also effectively solve the scheduling deadlock problem caused by the blocking of the task to the device and the resource competition.

Description

TECHNICAL FIELD

[0001] The present application relates to a laboratory automation deadlock-free dynamic scheduling method based on improved Monte Carlo tree search (MCTS), belonging to the field of industrial intelligent scheduling and artificial intelligence. BACKGROUND

[0002] With the new round of industrial 4.0 technology revolution, industrial scenarios (such as intelligent factory, intelligent laboratory, intelligent logistics, etc.) are rapidly developing towards informatization, automation and intelligentization. As an important application scenario of intelligent production, automated laboratories use modern technology and equipment to automate various operations and processes in the laboratory, liberate manpower, intelligently design experiments, reduce human error, and improve experimental throughput, speed, accuracy, etc. The implementation of laboratory automation relies on various automated equipment, such as automatic samplers, liquid handling equipment, robotic systems, etc. These devices need to work together to complete various tasks in the laboratory, thereby achieving fast, stable and accurate experimental processes.

[0003] In the laboratory automation system, the scheduling algorithm is the core module, which is used to control and allocate specific tasks to be executed on specific devices at a specific time, and to generate a plan that meets all scenario constraints and has the shortest total execution time. How to design a good scheduling algorithm is the most important and difficult problem in the development of laboratory automation towards high throughput, large scale and overall automation. Especially in the field of life science laboratory automation, customers are small and scattered and immature, and scheduling tasks have the characteristics of "non-standard, small batch, and multiple varieties". Traditional heuristic scheduling cannot handle the demand for large-scale flexible customization tasks, so there is an urgent need for an artificial intelligence approach to build a general intelligent scheduling method.

[0004] The scheduling problem in the automated laboratory scenario can be abstracted as a dynamic scheduling problem of flexible blocking flow shop, which is an NP-hard combinatorial optimization problem. As the problem size increases, the search space grows exponentially, and the solution to large-scale optimization scheduling problems cannot be obtained within a reasonable polynomial time. Current scheduling techniques can be mainly divided into rule-based scheduling methods and meta-heuristic techniques. Rule-based scheduling methods include list scheduling methods such as SPT (shortest processing time), SLT (system level test), etc. These methods allocate ready tasks in each decision step according to the determined priority rules. Meta-heuristic-based scheduling methods include particle swarm optimization, genetic algorithm, simulated annealing and tabu search, etc. These methods first construct a feasible solution, and then use the idea of swarm intelligence to search iteratively in the solution space to obtain a better solution that meets the problem constraints.

[0005] Although the existing intelligent scheduling algorithm can solve the scheduling problem under the constraint of general scene, in the actual production environment, the production of jobs has a blocking constraint, that is, when a workpiece is completed by a device, if the next processing device of the workpiece is not idle, the workpiece will continue to block the device and cannot be released. Due to the blocking constraint, the concurrency and interaction of jobs compete for limited device resources, which may lead to a deadlock condition, that is, a cyclic wait for resources, and thus a deadlock occurs. In the case of deadlock, the automated manufacturing system is blocked indefinitely and cannot complete the task.

[0006] In addition, in the process of automated laboratory scheduling, unexpected events such as machine failure, emergency task arrival, device resource state and task change often occur, which requires modification of the original static scheduling scheme and rapid generation of a rescheduling scheme. In the field of life science laboratory automation, the manufacturing system scheduling is prone to deadlock, the calculation time of rescheduling is required to be high, the list heuristic scheduling algorithm encounters deadlock when generating a scheduling scheme and needs to backtrack, the solving time is long and uncertain in a large-scale task, and the random search algorithm based on genetic and evolution needs a long time of iteration and cannot solve the deadlock problem in scheduling.

[0007] In summary, the automated scheduling in the laboratory scene prone to deadlock not only needs to complete the optimization of the objective function such as the shortest task period, but also involves the handling of the deadlock problem and the demand for dynamic scheduling. The current scheduling algorithm cannot complete high-quality scheduling in a high dynamic, flexible environment and complex task prone to deadlock to support the intelligent manufacturing demand of the rapidly developing automated laboratory. Therefore, it is crucial to develop a blocking mixed flow shop dynamic scheduling method in a complex scene to effectively avoid scheduling deadlock and optimize the scheduling performance of the system, which is the key to realizing intelligent scheduling in the automated laboratory scene. SUMMARY

[0008] To solve the above problems, the present application provides an automated laboratory deadlock-free dynamic scheduling method based on Monte Carlo tree search to solve the problem that the prior art cannot meet the demand for easy deadlock and rapid rescheduling in automated laboratory scheduling.

[0009] To solve the above technical problems, the automated laboratory deadlock-free dynamic scheduling method based on Monte Carlo tree search includes:

[0010] Step one: the modeling calculation module models the experimental tasks and equipment resources of the automated laboratory. The laboratory tasks are modeled as a directed acyclic graph according to the experimental process, the nodes of the graph represent operation sub-tasks, the dependence relationship between tasks is represented by a directed edge, and the node information records the required equipment type and quantity of the task and the task execution time. An equipment state matrix is established for the equipment resource pool to record all equipment resources and the state of whether the equipment is occupied. The directed acyclic graph of the currently obtained laboratory task and the equipment state matrix are input into the scheduling decision module.

[0011] Step two: the scheduling decision module calculates the task scheduling decision sequence using the Monte Carlo tree search method. The nodes of the Monte Carlo tree represent system states, which include the state of whether all tasks are executed and the state of whether the equipment is occupied; the edges between the nodes of the Monte Carlo tree represent actions, which represent scheduling which task to execute on which equipment; for each node in the Monte Carlo tree, a variable set [Q, N, D] is set, where Q is the reward value, initially 0; N is the number of visits to the node, initially 1; D is the number of times of deadlocks that have occurred in the node during the search and simulation process, initially 0.

[0012] The scheduling decision module obtains the current directed acyclic graph and equipment state matrix, constructs the root node of the Monte Carlo tree according to the current laboratory task state and equipment state, combines the set of candidate actions, and starts the Monte Carlo tree search from the root node to perform a preset number of iterations to obtain the task scheduling decision sequence.

[0013] Step three: the task allocation module obtains the task scheduling decision sequence, generates a Gantt chart of the task scheduling using the list scheduling method, and actually schedules the to-be-allocated operation sub-tasks to the equipment according to the Gantt chart according to the earliest available principle, and the system state at the current decision time is the root node of the decision sequence.

[0014] Step four: after the execution of the operation sub-tasks at the current decision time, the system state perception module perceives the environmental state, including the state of all laboratory tasks and all equipment resource pools, detects whether there is a dynamic event of change in the state of the laboratory task and / or the equipment, if there is a dynamic event, returns to step one for execution; if not, constructs the root node from the current laboratory task state and equipment state, and then goes to step two for execution until all laboratory tasks are scheduled to be completed.

[0015] In step two, the Monte Carlo tree search includes the following steps:

[0016] Step 21: construct the root node from the current system state, initialize the variable set [Q, N, D] for the root node, set the search round number I, and set the pruning upper bound β based on the branch and bound strategy = +∞;

[0017] Step 22: performing a selection operation; starting from the current root node, searching down to a child node with the largest UCB value, if the searched leaf node is a non-terminal node, performing an expansion operation on the leaf node;

[0018] Step 23: performing an expansion operation; obtaining all executable actions from the system state of the expanded node, expanding new child nodes, and pruning the expanded child nodes using a branch-and-bound strategy; an executable action refers to all predecessor tasks of the task being scheduled to be completed and the required device resources being released;

[0019] Step 24: performing a simulation operation; after each expansion operation, starting from the state of the expanded child node, randomly selecting an action to perform until all laboratory tasks are scheduled to be completed to reach a terminal node, calculating the makespan from the expanded child node to the terminal node, and updating β to the makespan when the makespan is less than β; when the last selected node in the simulation process does not reach the terminal node and the subsequent action has no executable task candidate, judging that a deadlock state is reached, interrupting and marking.

[0020] Step 25: performing a backpropagation operation; the backpropagation operation is performed only when the simulation reaches the terminal node or the deadlock state; the tree nodes on the scheduling decision sequence of the simulation reaching the terminal node or the deadlock state are backpropagated, and the variable parameters Q, N, and D of the nodes are updated.

[0021] In the step 22, the selection operation of the Monte Carlo tree search selects the action a with the largest UCB value according to the following calculation rule * As follows:

[0022]

[0023] Let s be a Monte Carlo tree node ′ The selection operation is performed on the child node of s ′ to reach s', and the above formula, argmax[.] represents selecting the action with the largest value in the brackets, Q(s,a) represents the cumulative reward value of the tree node s, the larger the value of Q(s,a), the more valuable the action a is to execute, N(s,a) represents the number of visits to the tree node s, D(s,a) represents the number of deadlocks occurring in the search and simulation process of the tree node s; N(s') represents the number of visits to the parent node s' of the node s; C is a hyperparameter for balancing the weight between exploration and utilization; F is a hyperparameter for controlling the size of the deadlock penalty term.

[0024] The step 23 adopts a branch and bound strategy to perform pruning operation on the extended child node, including: obtaining the completion time of the scheduled task on the device, i.e. span time α, from the root node to the extended child node system state; setting the remaining unscheduled task set from the extended child node system state as CP last , obtaining the cumulative time c required by the tasks on the critical path in the set CP last on the earliest available device; if α+c is not less than the pruning upper bound β, the extended child node is pruned, and the node is marked as no longer expanded or accessed.

[0025] In the step 25, for the scheduling decision sequence simulating reaching the termination node, an evaluation score V(makespan) = e*(1 / makespan) is calculated according to the span time makespan of the scheduling decision sequence, wherein e is a hyperparameter for controlling the coefficient of the backtracking reward;

[0026] Setting a deadlock flag k, k=1 represents that the simulation reaches a deadlock state, and k=0 represents that the simulation reaches a termination node;

[0027] Updating the reward value, the access frequency and the deadlock frequency of each node on the path of the scheduling decision sequence simulating reaching the termination node or reaching the deadlock state, as follows:

[0028] Updating the reward value of the node s as follows:

[0029]

[0030] Wherein, Q new (s,a) is the updated reward value of the node s, Q old (s,a) is the reward value of the node s before updating;

[0031] Updating the access frequency of the node s as follows:

[0032]

[0033] Wherein, N old (s,a) is the access frequency of the node s before updating, N new (s,a) is the updated access frequency of the node s;

[0034] Updating the deadlock frequency of the node s as follows:

[0035]

[0036] Wherein, D old (s,a) is the deadlock frequency of the node s before updating, D new (s,a) is the updated deadlock frequency of the node s.

[0037] Compared with the prior art, the present application has the following advantages and positive effects:

[0038] (1) The method solves the task and equipment scheduling problem in laboratory automation manufacturing by using the Monte Carlo tree search algorithm, uses the MCTS algorithm to search on the DAG graph constructed by the experiment task, and cyclically executes the selection, expansion, simulation and back propagation process to construct the search tree. In the process of constructing the search tree, the pruning strategy during expansion is further designed, which further improves the search efficiency of the MCTS algorithm, and can effectively meet the frequent dynamic rescheduling demand in the actual production scene.

[0039] (2) The method improves the Monte Carlo tree search algorithm, and according to the node state, deadlock processing, back transmission strategy and selection function designed according to the problem to be solved by the present application, a forward-looking decision can be made in each scheduling process, which can avoid the deadlock caused by the resource circular waiting that may occur subsequently, and effectively solve the deadlock problem in laboratory automation scheduling.

[0040] (3) The method adopts the mode of scheduling and executing simultaneously, when the task changes dynamically along with the process flow, the changed task is re-modeled DAG, and MCTS is used for further exploration; through the pruning operation in the expansion stage, the search of the unfavorable path is prevented, the search speed is improved, and a better task scheduling decision sequence is obtained in a limited time, which improves the search effect of the scheduling strategy. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is the overall implementation framework diagram of the laboratory automation scheduling method of the present application;

[0042] Figure 2 is an example diagram of a deadlock state in the embodiment of the present application;

[0043] Figure 3 is a flowchart of the scheduling decision using Monte Carlo tree in the embodiment of the present application;

[0044] Figure 4 is a step schematic diagram of the Monte Carlo tree search in the embodiment of the present application;

[0045] Figure 5 is a root node conversion schematic diagram. DETAILED DESCRIPTION

[0046] The present application will be further described in detail below in combination with the drawings and embodiments.

[0047] The meta-heuristic scheduling method based on the priority rule heuristic method and the group optimization technology is a commonly used method for the deadlock-prone automated laboratory dynamic scheduling problem, but these methods cannot effectively solve the no-deadlock fast scheduling scheme generation requirement in the dynamic scene. In order to solve this problem, the method of the application makes a forward-looking decision at each scheduling decision step by improving the node state in the Monte Carlo tree, simulating the deadlock processing at the time, the back propagation strategy, the selection function and the like, so as to avoid the deadlock caused by resource circular waiting. The method of the application can not only complete the optimization target of minimizing the maximum task completion time, but also effectively process the scheduling deadlock problem caused by the blocking of tasks to devices and resource competition.

[0048] The application provides a laboratory automation scheduling method based on an improved Monte Carlo tree search, which is mainly used for laboratory automation scheduling prone to deadlock and high dynamics, and the overall framework of the system is as shown in Figure 1 The method of the application can be realized as a functional module in the laboratory automation system, and the whole method is modeled by modeling the experimental process tasks and the device pool resources, constructs a directed acyclic graph model, constructs a search tree through the Monte Carlo tree search method, a large number of selection, expansion, simulation and back propagation, effectively searches the scheduling decision sequence path to obtain a no-deadlock scheduling scheme.

[0049] The application is used in all scheduling scenes with the blocking flow shop property, and can solve the scheduling process deadlock problem in the scene through the deadlock avoidance mechanism. The generation example of the deadlock is as shown in Figure 2 The scheduling task example starts from the initial state, and the element S ij =(k,l) in the state matrix represents that the operation O i of the job JOB j needs to be performed on the machine M k for l seconds. The following sequence of scheduling is performed from the initial state of the system: (1,1,3), (4,1,2), (4,2,1), (2,1,2), wherein (i,j,k) represents that the operation O i of the job JOB j is allocated to the resource M k to perform the job. As shown in Figure 2 In the embodiment of the application, the system enters the state, and the ready jobs at this time are: (JOB1, O2), (JOB2, O2), (JOB3, O1), (JOB4, O3), (JOB5, O1), (JOB6, O1). As shown in Figure 2It can be seen that the resource M2 required by the job JOB6 is occupied by the job JOB2, the resources M3 required by the jobs JOB2, JOB3, JOB4 and JOB5 are occupied by the job JOB1, and the resource M1 required by the job JOB1 is occupied by the job JOB4. At this time, the cyclic waiting state of job to resource demand occurs, the system enters the deadlock, and all operations cannot proceed to the next step. The algorithm using the priority rule is prone to deadlock state in the actual production of no-buffer blocking scheduling task, resulting in production interruption. However, the meta-heuristic method can avoid the generation of scheduling decision scheme that causes deadlock by judging the deadlock of the solution, but the calculation time of these optimization algorithms is longer, which is difficult to meet the rapid rescheduling demand after the dynamic event occurs. The dynamic deadlock-free scheduling method of the easy deadlock laboratory of the application solves the above problems by using the Monte Carlo tree search algorithm, and the Monte Carlo tree search algorithm is improved. By using the hysteresis of the deadlock, the search path of the easy deadlock is punished by using the deadlock control item, so as to avoid selecting the operation that may cause future deadlock.

[0050] The method models the scheduling problem of an automated laboratory as a search decision problem in a directed acyclic graph (DAG), first models the tasks according to the experimental process, models the resources, and models the experimental jobs as a task graph using a DAG. Each node in the task graph represents a subtask in the experimental task process, including the required equipment type, quantity, and task execution time of the task, and the dependency relationship between tasks is represented by a directed edge. The improved Monte Carlo tree search method is used to search for an optimal scheduling decision path. The Monte Carlo tree search (MCTS) method is a tree-based data structure that can effectively balance exploration and utilization. A search tree is constructed by repeatedly selecting, expanding, simulating, and backpropagating, and the highest value action under the current root node is selected as the real decision of the current step when a scheduling action is actually performed. Monte Carlo tree search can make decisions that are more optimal for the global goal by simulating a large number of operations before actually performing each step to construct the value evaluation of each state node in the search tree. In view of the deadlock problem occurring in the scheduling process, the Monte Carlo tree search algorithm is further improved, the hysteresis of the deadlock is utilized, and the deadlock control item is used to punish the search path prone to deadlock to avoid selecting operations that may cause future deadlock. In view of the dynamic change of the environment, the method of scheduling and executing is adopted, the system state is detected after each allocation operation is executed, and when the task changes dynamically with the process flow, the changed system state is modified and input into the MCTS for further exploration. In addition, in order to improve the search speed and obtain a better scheduling decision in a shorter time, the pruning method of the expansion and simulation stages is further proposed, which prevents the search of unfavorable paths to improve the search effect within a limited time.

[0051] The Monte Carlo tree search-based automated laboratory deadlock-free dynamic scheduling method of the embodiment of the application realizes modeling calculation modules, scheduling decision modules, task allocation modules, and system state sensing modules through computer programs. The method includes the following four steps.

[0052] Step 1) The modeling calculation module establishes a static task model for the laboratory task process and the equipment pool according to the current environment state, represents the laboratory task as a directed acyclic graph, and establishes an equipment state matrix for the equipment pool resources.

[0053] For example, an experimental process in an automated laboratory is represented as a directed acyclic graph G=(V,E), each node in the graph is a task, V={n1,n2,…,n m} represents the set of m task nodes in the task process, E={e1,e2,…,e s} represents a set of directed edges of task nodes, and a directed edge e(i,j) E represents a dependency constraint relationship of tasks, that is, task n i must be completed before task n j . The node information records the type and number of devices required by the task, and the task execution time. Corresponding Figure 2 , the embodiment of the present application establishes a directed acyclic graph for each job operation as a task node.

[0054] The device state matrix contains all device resources and the current state of whether it is occupied, such as Figure 2 M1-M6, the current availability of the device can be represented by 0 / 1, if occupied, represented by 0, if not occupied, currently available, represented by 1.

[0055] Step 2) The scheduling decision module uses the Monte Carlo tree search method to build a search tree, and calculates the task scheduling decision sequence, as shown in Figure 3 . The nodes in the Monte Carlo search tree represent the state of the system, and the edges represent actions. Each node is defined by the following variable set: [Q, N, D], where Q represents the reward value, N represents the number of visits to the state node, and D represents the number of times the node has occurred Deadlock in the search and simulation process.

[0056] The scheduling decision module obtains the directed acyclic graph of the laboratory task, the device state matrix, and the task demand matrix for the device, and then inputs MCTS. In the embodiment of the present application, the task of the directed acyclic graph is a job operation. In order to facilitate description, the operation O i of the job JOB j is represented by the character O ij , such as Figure 3 O 11 represents the operation O1 of the job JOB1. For the problem to be solved by the present application, the system state records the current task state and the device resource state, the task state refers to the state of whether all tasks are executed, and the device resource state refers to the state of whether the device is occupied; the action refers to the executed job operation, that is, which job operation is executed on which device. The set of candidate actions can be obtained from the current system state and the directed acyclic graph.

[0057] The Monte Carlo tree search method of the embodiment of the present application includes the following sub-steps 2.1-2.5.

[0058] Step 2.1) First, construct the initial node S0 based on the current task state and device state, initialize the variable set [Q,N,D] of the initial node, where Q=0, D=0, and N=1, set the search round number I, and initialize the branch-and-bound pruning upper bound β=+∞ for pruning during expansion. Constructing a Monte Carlo tree requires I iterations and four actions: Selection, Expand, Simulation, and Backup.

[0059] like Figure 4 As shown, the root node state of the Monte Carlo search tree is initialized, and the selection, expansion, simulation, and backpropagation process is executed I times in a loop to construct the search tree until the set number of iterations is reached. Each iteration includes the following steps 2.2-2.5.

[0060] Step 2.2) Perform the selection operation.

[0061] An operation selection tree is a state tree obtained by performing different alternative operations based on the state of the current root node. Starting from the root node, the child node with the highest value is selected cyclically until a leaf node in the operation selection tree is selected.

[0062] Suppose we are currently selecting a child node with the highest value for tree node s′, and the corresponding action a is... * as follows:

[0063]

[0064] Where argmax[.] represents the action that maximizes the UCB value within the parentheses. Let's say we choose action a from state s′ to reach state s. Q(s,a) represents the cumulative reward value of tree node s. The larger the value of Q(s,a), the more valuable the action a has, and the more the search tree will tend to search deeper in that direction; N(s ′ Let N(s,a) represent the number of times node s' is visited; N(s,a) represents the number of times node s is visited; C is a hyperparameter used to balance the weight between exploration and utilization. If the number of visits to node s is low, the value of the exploration term will increase, meaning that paths that are not explored or have been explored less may have greater access value. Compared to the traditional MCTS method, this invention sets an additional state label D for each node in MCTS to record the number of deadlocks that occur when the scheduling strategy solution passes through that node. When a deadlock occurs during the simulation phase, the deadlock state variable D(s,a) of all nodes on the path controlled by the strategy changes, and the deadlock count of the corresponding node is increased by 1. When performing the selection operation, the deadlocked path is penalized, and the size of the deadlock penalty term is controlled by the hyperparameter F.

[0065] The child node with the maximum UCB (upper confidence bound) is searched down in the built search tree in turn until a leaf node is reached. If the leaf node is a non-terminal node, an expansion operation is performed on the leaf node.

[0066] Step 2.3) performing an expansion operation. The expansion operation obtains all eligible task scheduling actions under the current state node through the simulator. The expansion of a node means that the task is selected for scheduling, and the system enters the next state. After the expansion of a node is completed, the simulation is performed from the current expanded node. The simulation operation starts from the current state node, and a task is randomly selected each time until all tasks are scheduled to be completed, and back propagation is performed to update the state node values of the corresponding paths in the search tree. If the current node is a non-terminal node and there is no executable action, it is determined that the system enters a deadlock state and the deadlock control parameters are directly returned to re-enter the selection stage.

[0067] In the embodiment of the application, when entering the expansion operation, the executable actions are calculated according to the system state of the current node of the search, an alternative action set is obtained, and then a maximum value executable action is selected to expand a new node, that is, the system reaches a new state after the action. The initial variables of the new system state node are set as Q=0, D=0 and N=1. Specifically, the executable action represents a candidate task set that can be scheduled under the current scheduled process, and the tasks need to meet two conditions: first, all predecessor tasks of the task have been scheduled to be completed, and second, the required occupied resources of the task are currently released.

[0068] In particular, the method of the application proposes a pruning method based on the branch and bound strategy in the expansion stage, which can effectively reduce the search space of the search tree, increase the search efficiency of the task scheduling decision sequence solution within a limited time, and specifically as follows:

[0069] The span cumulative time required by all tasks on the path from the root node state to the expansion child node state is α, and is as follows:

[0070]

[0071] Wherein, P represents a device pool set; avail[j] represents the earliest available time of the device p j used for executing the task; max represents the maximum value, which here refers to the latest time of completing the tasks on the path from the root node to the expansion child node, that is, the earliest available time of the device used for executing the tasks.

[0072] Let the set of unscheduled tasks remaining from the expansion child node state be CP lastFrom the expanded child node, the span time cumulative value required for executing the critical path task node of the unscheduled task set on the earliest available device is c, as follows:

[0073]

[0074] Wherein, w i,j represents the execution time of task n i on device p j ; min represents taking the minimum value, which refers to the critical task n i least completion time here.

[0075] It can be concluded that the span time m obtained from the simulation of all task scheduling completion starting from the current expanded child node meets:

[0076] m≥α+c

[0077] If α+c≥β, then m≥β, that is, the execution time m of any scheduling decision solution searched from the current expanded node is greater than or equal to the execution time β of the optimal solution searched at present, then the node is selected to be pruned, and is marked and will not be expanded and accessed in the future, the search range is controlled in other sub-trees, and the root node is returned to re-enter the selection phase. Therefore, the embodiment of the present application obtains two span times α and c for the expanded child node, and if α+c is not less than the pruning upper bound β, the expanded child node is pruned.

[0078] Step 2.4) performing simulation operation. After the expansion operation, a new state is reached by randomly selecting an action from all executable actions and executing the action, starting from the system state of the expanded child node, and this process is repeatedly executed until all laboratory tasks are scheduled to be completed, and the execution is terminated at the termination node. Note that here the action is not actually selected and executed.

[0079] For the simulated scheduling decision sequence in which the experimental tasks are scheduled to be completed, the earliest start time EST and the earliest end time EFT of the tasks are calculated, and the list scheduling method of the present application is used, so that the span time makespan of the simulated scheduling decision sequence can be obtained. When the makespan of the simulation to the termination state is <β, β is updated as follows:

[0080] β=makespan

[0081] The method of the present application calculates the earliest start time EST(n i , p j ) of task n i on device p j in the following way:

[0082]

[0083] Where, pred(n i ) represents task n i The set of all predecessor tasks, AFT(n m ) represents task n i The task n that is completed latest among all predecessor nodes m The final completion time for task n. i In device p j The earliest start time depends on the maximum value of the set consisting of the latest completion time of all its predecessor task nodes and the earliest available time of the device.

[0084] Correspondingly, task n i In device p j The earliest end time EFT(n) i ,p j The calculation is as follows:

[0085] EFT(n i ,p j ) = EST(n i ,p j )+w i,j

[0086] The maximum completion time obtained after all tasks in the constructed directed acyclic graph is recorded as the completion time makespan, defined as:

[0087] makespan = max{AFT(n z )}

[0088] Where, n z Indicates the terminal node. max{AFT(n z )} represents the final completion time of all tasks in the laboratory.

[0089] Furthermore, during the simulation of the scheduling algorithm, if the current system state has not reached the termination node and there are no ready candidate executable tasks, a deadlock can be identified. When a deadlock occurs, the simulation is interrupted and marked, and the process directly enters the backpropagation phase.

[0090] Step 2.5) Perform backpropagation. Backpropagation is performed if and only if the simulation operation has completed the entire task scheduling process, or if a deadlock state is reached.

[0091] For a simulated schedule decision sequence reaching the termination node, the present application back-propagates the value evaluation obtained after the search from the termination leaf node to the root node according to the makespan value obtained in the simulation stage, and updates the parameter values of the path nodes along the way, including the reward value Q, the number of visits N, and the deadlock control state variable D. The objective of the scheduling method of the present application is to minimize the maximum completion time. Therefore, the score evaluation function of a simulated schedule decision sequence obtained by the Monte Carlo tree search is calculated by the following formula:

[0092] V(makespan) = e * (1 / makespan)

[0093] wherein V(makespan) is the evaluation value of the schedule decision sequence, and e is a hyperparameter for controlling the coefficient of the back-propagated reward.

[0094] If the simulation is interrupted and the termination node is not reached, a deadlock occurs, and in this case, the evaluation value does not need to be calculated.

[0095] The reward value of each node on the path of the current schedule decision sequence is updated, and the formula is as follows:

[0096]

[0097] wherein k represents the deadlock flag, k = 1 indicates that a deadlock occurs during the simulation, and k = 0 indicates that no deadlock occurs during the simulation; Q old (s,a) and Q enw (s,a) are the reward values of node s before and after the update, respectively, and the initial reward value is 0.

[0098] The update formula for the number of visits of each node on the path of the current schedule decision sequence is as follows:

[0099]

[0100] wherein N old (s,a) and N new (s,a) are the number of visits of node s before and after the update, respectively. When a deadlock occurs, the number of visits of the node is not updated, and when no deadlock occurs, the number of visits of the node is increased by 1.

[0101] The update formula for the number of deadlocks of each node on the path of the current schedule decision sequence is as follows:

[0102]

[0103] wherein D old (s,a) and D new (s,a) are the number of deadlocks of node s before and after the update, respectively.

[0104] When the back propagation is completed, return to sub-step 2.2, and the selection, expansion, simulation and back propagation steps are repeatedly executed from the root node until the iteration number reaches the set threshold I. The core idea of the present application for solving the deadlock-prone laboratory automation scheduling problem is that the occurrence of deadlock is hysteresis, and the occurrence of deadlock at a certain moment may be caused by a step in the decision path, and it is expected to have foresight when selecting the execution of the task, that is, to try to avoid selecting the operation that may cause the subsequent deadlock.

[0105] Step 3) converting the dynamic scheduling problem of the task into a static scheduling task, generating a Gantt chart of the task scheduling by using the list scheduling mode for the task scheduling decision sequence; and actually scheduling the operation sub-tasks to be allocated at the current decision time to the devices for execution according to the earliest available principle.

[0106] After step 2 is iteratively executed I times, a search tree is obtained, from the root node, the action with the maximum UCB value is actually executed one step, that is, a task is selected to be scheduled to the device with the earliest start time in the candidate available device pool for execution. And the new state node reached after executing the scheduling action is taken as the root node, and step 2 is converted to rebuild the search tree from the current system state, as shown in Figure 5

[0107] The method of the present application adopts the list scheduling mode, generates a Gantt chart of the task scheduling according to the scheduling decision sequence obtained by the improved Monte Carlo tree search method of the present application, and then can actually schedule according to the task scheduling Gantt chart. The steps of generating the Gantt chart of the task scheduling by using the list scheduling mode are as follows:

[0108] Step 31) inputting the scheduling decision sequence obtained by the Monte Carlo tree search method, and setting the current decision time as 0;

[0109] Step 32) constructing and maintaining an earliest available time array avail[j],p j ∈P, device p j is a device in the device pool set P;

[0110] Step 33) cyclically traversing the operation sub-tasks to be allocated at the current decision time according to the scheduling decision sequence;

[0111] Step 34) querying all device resources required by the operation sub-task, allocating the operation sub-task to the device with the earliest start time according to the earliest available principle, and updating the earliest available time of the device;

[0112] Step 35) the current decision time is incremented by 1, and steps 33 to 34 are repeated until all task nodes are scheduled and allocated, and the processing start time EST(n i ,p​j ) and end time EFT(n i ,p j ), a Gantt chart of the experimental flow scheme and the earliest completion time makespan of the current scheduling scheme are generated.

[0113] Step 4) In actual operation, the current task state is perceived by the system state perception module, and each time a real execution step scheduling operation is performed to determine whether a dynamic event occurs in the environment state including equipment, tasks, etc. If no dynamic event occurs, steps 2 and 3 are repeatedly executed until all experimental tasks are scheduled to be completed. If a dynamic event occurs, step 1 is returned to perform modeling on a new task flow, and steps 2 and 3 are repeatedly executed until all tasks in the laboratory are completed.

[0114] The method of the present application adopts a list scheduling mode for actual scheduling. Let N be equal to the number of task nodes in the task graph, then the length of the scheduling decision sequence obtained is N, that is, the depth of the search tree is N. In particular, when no dynamic event occurs, a subtree reservation strategy is adopted, and the state node reached by the current execution step action is taken as the root node, and the root node is the current decision time. According to the reserved subtree and the corresponding Gantt chart, the operation subtask at the current decision time can be actually scheduled. When a dynamic event occurs, the directed acyclic graph of the task needs to be changed, and the MCTS search tree constructed last time will no longer be applicable. At this time, the subtree reservation strategy is not adopted, but a new search tree needs to be reconstructed from the current root node.

[0115] In addition to the technical features described in the specification, they are known to those skilled in the art. The present application omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present application. The embodiments described in the above embodiments do not represent all embodiments consistent with the present application. Various modifications or variations made by those skilled in the art without creative labor on the basis of the technical solutions of the present application are still within the protection scope of the present application.

Claims

1. A Monte Carlo tree search based automated laboratory deadlock-free dynamic scheduling method, characterized in that, The method comprises the following steps: Step 1: a modeling calculation module models laboratory tasks as a directed acyclic graph according to an experimental process, and establishes a device state matrix for a device resource pool; nodes in the directed acyclic graph represent operation subtasks, and a dependency relationship between tasks is represented by a directed edge; node information records a required device type and quantity of the task, and a task execution time; the device state matrix records all device resources and a state of whether the device is occupied; Step 2: a scheduling decision module searches a task scheduling decision sequence by using a Monte Carlo tree search method; A node of the Monte Carlo tree represents a system state, and the system state comprises a state of whether all tasks are executed and a state of whether a device is occupied; an edge between nodes of the Monte Carlo tree represents an action, and the action represents scheduling which task to which device for execution; a variable set [Q, N, D] is set for the node in the Monte Carlo tree, wherein Q is a reward value, is initially 0, and the reward is calculated with a minimum maximum completion time as an optimization target; N is a visit number of the node, and is initially 1; D is a number of deadlocks that occur in the node in a search and simulation process, and is initially 0; The scheduling decision module obtains a current directed acyclic graph and a device state matrix, constructs a root node of the Monte Carlo tree from a current laboratory task state and a device state, performs the Monte Carlo tree search from the root node, combines an action set, performs an iterative search for a preset number of rounds, and obtains a task scheduling decision sequence; Step 3: a task allocation module obtains the task scheduling decision sequence, generates a Gantt chart of task scheduling by using a list scheduling mode for the task scheduling decision sequence, and actually schedules a to-be-allocated operation subtask at a current decision time to a device for execution according to the Gantt chart according to an earliest available principle; Step 4: after the operation subtask at the current decision time is executed, a system state perception module perceives an environmental state, detects whether a dynamic event of a change in a laboratory task and / or a device state occurs, constructs a root node from a current laboratory task state and a device state if no dynamic event occurs, and then performs step 2 until all laboratory tasks are scheduled to be completed; if the dynamic event occurs, step 1 is returned to continue to be performed.

2. The method of claim 1, wherein, In step 2, performing the Monte Carlo tree search comprises the following steps: Step 21: a root node is constructed from a current system state, a variable set [Q, N, D] is initialized for the root node, a search round number I is set, and a pruning upper limit β based on a branch and bound strategy is set as +∞; Step 22: a selection operation is performed; starting from the current root node, a child node with a maximum value is searched downward, and an expansion operation is performed on a leaf node if the searched leaf node is a non-terminating node; Step 23: an expansion operation is performed; all executable actions are obtained from a system state of the expanded node, a new child node is expanded, and a branch and bound strategy is used to prune the expanded child node; the executable action refers to that all predecessor tasks of the task have been scheduled to be completed, and required device resources have been released; Step 24: perform a simulation operation; after each expansion operation, starting from the system state of the expanded child node, randomly select an action to perform until all laboratory tasks are scheduled to be completed to reach the terminal node, calculate the makespan of the task scheduling decision sequence obtained after the simulation operation, and if the makespan is less than β, update β as the makespan; when the last selected node in the simulation process does not reach the terminal node and there is no candidate executable action afterwards, it is judged that a deadlock state is reached; Step 25: perform a backpropagation operation; the backpropagation operation is performed only when the simulation reaches the terminal node or the deadlock state; the variable parameters Q, N, and D of the tree nodes on the scheduling decision sequence of the simulation reaching the terminal node or the deadlock state are updated.

3. The method according to claim 1 or 2, characterized in that, In step 2, when performing the selection operation on the child nodes under the Monte Carlo tree node s', the action a* with the maximum value is selected according to the following calculation rule: where argmax[.] represents selecting the action that maximizes the value in the brackets; let s be the system state reached by performing the action a from the system state s', Q(s, a) represent the cumulative reward value of the tree node s, the greater the value of Q(s, a), the more valuable the action a is to perform, N(s, a) represent the number of visits to the tree node s, and D(s, a) represent the number of deadlocks that occur to the tree node s during the search and simulation processes; N(s') represents the number of visits to the node s'; C is a hyperparameter used to balance the weight between exploration and exploitation; when a deadlock occurs during the simulation stage, the number of deadlocks of all nodes on the path of the scheduling strategy is increased by 1; F is a hyperparameter used to control the size of the deadlock penalty term.

4. The method according to claim 1 or 2, characterized in that, In step 2, during the Monte Carlo tree search process, when performing the expansion operation, a branch and bound strategy is used to prune the expanded child nodes, including: obtaining the completion time of the scheduled tasks on the devices, i.e., the makespan α, from the root node to the expanded child node; Let the set of unscheduled tasks remaining from the expanded child node system state be CP last , obtain the cumulative time c required by the tasks on the critical path in the set CP last on the earliest available device; if α + c is not less than the pruning upper bound β, the expanded child node is pruned, and the node is marked as not being expanded or visited.

5. The method according to claim 1 or 2, characterized in that, In step 2, all laboratory tasks are scheduled to be completed by starting from the expanded child node through the simulation operation, and a simulation scheduling decision sequence is obtained; the makespan of the simulation scheduling decision sequence is determined by calculating the earliest start time and the earliest end time of the tasks; The earliest start time of a task is calculated as follows: Let task n i be scheduled for execution on device p j , the earliest start time EST(n i , p j ) of task n i on device p j is calculated as follows: where avail[j] represents the earliest available time of device p j , pred(n i ) represents the set of all predecessor tasks of task n i , AFT(n m ) represents the completion time of predecessor task n i of task n m , and max means to take the latest time. The earliest end time of a task is calculated as follows: task n i The earliest end time EFT(n j , p i ) on device p j is calculated as follows: EFT(n i , p j ) = EST(n i , p j ) + W i,j where W i，j represents the task n i on the device p j execution time; The span time makespan of the simulated dispatching decision sequence is obtained as makespan = max{AFf(n z )}, n z denotes the termination node.

6. The method according to claim 1 or 2, characterized in that, In step 2, when the simulation reaches the terminal node or the deadlock state, the backpropagation is performed; For the scheduling decision sequence that reaches the terminal node, an evaluation score y(makespan) = e*(1 / makespan) is calculated based on the makespan of the scheduling decision sequence, where e is a hyperparameter used to control the coefficient of the backpropagation reward; A deadlock marker k is set, where k = 1 represents that the simulation reaches the deadlock state, and k = 0 represents that the simulation reaches the terminal node; The reward value, the number of visits, and the number of deadlocks of each node on the path of the scheduling decision sequence that reaches the terminal node or the deadlock state are updated as follows: The reward value of the node s is updated as follows: wherein Q new Q (s, a) is the updated reward value of node s, Q old Q (s, a) is the reward value of node s before updating; The access times of the node s are updated as follows: wherein No ld (s, a) is the number of accesses of node s before the update, N new (s, a) is the number of accesses of node s after the update; The deadlock times of the node s are updated as follows: where D old (s, a) is the number of deadlocks before the update of node s, D new (s, a) is the number of deadlocks after the update of node s.

7. The method of claim 1, wherein, In the step 3, the Gantt chart of the task scheduling is generated by using a list scheduling mode, and the Gantt chart comprises: Step 3.1: obtaining the task scheduling decision sequence output by the scheduling decision module, and setting the current decision time as 0; Step 3.2: Construct and maintain an earliest available time array avail[j], p for all devices in the device resource pool j ∈ P, device p j for devices in the pool set P Step 3.3: according to the decision sequence, the operation sub-tasks to be allocated at the current decision time are looped and traversed; Step 3.4: the required equipment resources of the operation sub-tasks are inquired, the operation sub-tasks are allocated to the earliest available equipment according to the earliest available principle, and the earliest available time of the equipment is updated; Step 3.5: the current decision time is increased by 1, then steps 3.3 and 3.4 are repeated until all the operation sub-tasks of the laboratory tasks are scheduled and allocated, the processing start time and end time of all the task nodes on the allocated equipment are obtained, the Gantt chart of the task scheduling scheme of the experimental process is generated, and the earliest completion time of the scheduling scheme is obtained.

8. The method according to claim 1 or 7, characterized in that, In the step 4, when no dynamic event occurs, the state node after one action of the step 3 is executed is taken as the current decision time, the search tree constructed by the last step 2 is subjected to a subtree reservation strategy, and one action at the current decision time is actually executed according to the corresponding Gantt chart.

Images