Industrial robot path planning and execution method based on layered Monte Carlo tree search

Abstract

The invention discloses an industrial robot path planning and execution method based on layered Monte Carlo tree search. By converting a global complex linear sequential logic task into a finite state automaton, simplification and clear representation of the task are achieved, and the industrial robot can understand and execute the task more easily. On the basis, a cross product model of a finite state automaton and a multi-agent Markov decision process is further constructed, so that the collaboration problem among a plurality of industrial robots is processed more effectively. Furthermore, in order to compress the historical state space, a hierarchical Monte Carlo tree search (MCTS) algorithm including task selection and task allocation is provided, the algorithm includes two levels of task selection and task allocation, the algorithm complexity can be remarkably reduced, and the search efficiency can be improved. In conclusion, an efficient, accurate and flexible online path planning and execution solution is provided for the industrial robot, and the method is particularly suitable for various long-range tasks with sequential logic characteristics.

Description

Technical Field

[0001] The invention relates to an industrial robot path planning and execution method based on hierarchical Monte Carlo tree search, belonging to the field of information technology. Background Art

[0002] Industrial robots are widely used in manufacturing, logistics, automated production and other fields, and can efficiently complete tasks such as assembly, welding, painting, and handling. With technological advances, industrial robots are developing towards higher flexibility and adaptability, especially when faced with complex tasks that require coordination of multiple robots for long-range collaborative tasks. Multi-agent planning technology is committed to finding a strategy for each agent that can maximize the long-term reward function. However, tasks in real scenarios often contain time, space, and logical constraints, which makes it difficult for traditional planning methods based on scalar reward functions to effectively solve such tasks. For example, in industrial production tasks, when a production line detects a device failure, the industrial robot must perform tasks in a specific order within each time step, perform maintenance or replacement operations within a limited time, and must ensure that other key equipment continues to operate normally after the fault is repaired to avoid violating timing constraints. In warehouse management tasks, robots need to move items based on real-time data and complete tasks within a specified time window, ensuring that each handling operation is performed in order while avoiding obstacles and avoiding task conflicts. These tasks require strict adherence to the constraints of linear temporal logic to ensure that the task is completed smoothly and meets the dual requirements of time and quality. Existing multi-agent planning methods have difficulty handling these tasks with complex constraints, and recent studies have attempted to use linear temporal logic languages ​​to model such tasks.

[0003] In order to model linear temporal logic constraints, existing multi-agent planning methods attempt to use mixed integer linear programming (MILP) to model the entire action sequence of each agent. On the other hand, in unknown environments, many studies in the field of machine learning in recent years, such as reinforcement learning (RL) methods, consider how to learn multi-agent strategies to meet linear temporal logic goals. Since the complexity of multi-agent planning problems grows exponentially with the number of agents and the length of linear temporal logic tasks, existing MILP methods and RL methods have low scalability. Therefore, some scholars have proposed to decompose the global linear temporal logic task into multiple independent linear temporal logic subtasks, and then assign each subtask to each agent for independent execution, thereby improving computational efficiency. However, global long-range collaborative linear temporal logic tasks may be highly coupled and difficult to decompose into independent subtasks. Some scholars have proposed to solve linear temporal logic tasks based on heuristic search mechanisms, but this method is only applicable to single robot scenarios. Therefore, although the existing technical solutions can solve the planning problem for linear temporal logic tasks, they cannot meet the requirements of ensuring the correct completion of tasks and the quality of solutions. Summary of the invention

[0004] Purpose of the invention: The present invention aims to solve the problem of how to plan a solution for an industrial robot that can meet the task requirements and take into account real-time performance in long-range tasks facing linear temporal logic constraints. To this end, the present invention proposes an industrial robot path planning and execution method based on hierarchical Monte Carlo tree search. This method can support and implement complex long-range planning tasks, especially when linear temporal logic constraints are involved, to improve the overall efficiency of multi-robot collaboration and the benefits of task execution. At the same time, the method also ensures the real-time responsiveness and good scalability of the system, and adapts to changing task requirements and dynamic environments.

[0005] Technical solution: The present invention proposes an industrial robot path planning and execution method based on hierarchical Monte Carlo tree search, thereby compressing the historical state space of multi-agent long-range collaboration. First, the global linear temporal logic task is equivalently converted into a finite state automaton, and the complex task is converted into a set of states and transitions. Then, in order to conveniently represent the joint transfer function of the automaton state and the robot state, a cross-multiplication model of the finite state automaton and the multi-agent Markov decision process is constructed. Furthermore, a hierarchical MCTS algorithm including task selection and task allocation is proposed, in which the task selection hierarchy represents the state selection of the automaton, and the task allocation hierarchy represents the allocation of different subtasks to different robots, thereby reducing the complexity through stratification and thereby improving the search efficiency. Specifically, the following steps are included:

[0006] (1) Constructing a cross-product model: Given a multi-agent Markov decision process model that represents the robot environment characteristics and a linear temporal logic task LTL φ The corresponding automaton constructs a cross-product model. This model limits the space for subsequent searches by comprehensively considering the environmental state and task constraints, so that subsequent exploration meets the requirements of temporal logic tasks. The key to this step is that the cross-product model provides a clear structural framework for multi-robot path planning, so that task and environmental features can be effectively integrated;

[0007] (2) Decision node selection: By selecting appropriate branches in the tree nodes, a path from the root node to the leaf node is formed. The nodes are divided into automaton sub-cross product state nodes and task allocation nodes. Through the hierarchical selection strategy, all subsequent sub-task situations are still considered in the task selection stage, but only a small number of tasks are explored for allocation, so that more exploration can be carried out in a limited time. This process is a key step in the hierarchical Monte Carlo tree search algorithm, which can effectively reduce the search space and improve computational efficiency;

[0008] (3) Tree structure expansion stage: When encountering unvisited sub-cross product automaton state nodes or task assignment nodes during the selection process, the algorithm will expand the search tree by adding leaf nodes to the tree. This stage ensures the integrity of the tree structure and provides more potential paths for subsequent decision-making;

[0009] (4) Strategy simulation and evaluation: Randomly select cross-product state nodes and task assignment nodes, and execute simulation strategies. The simulation strategy not only takes environmental factors into account, but also integrates task constraints. The simulation process considers the collaboration and conflict avoidance between multiple robots, thus providing more accurate feedback for subsequent decision-making;

[0010] (5) Backtracking and value updating: After initializing the value of the newly added node, the values ​​of all nodes in the tree are updated by backpropagating the value from the leaf node to the root node, ensuring that the value of each node can reflect the optimization result of the global task;

[0011] (6) Task execution synchronization stage: To ensure the smooth execution of tasks, the actions of industrial robots are coordinated so that some robots wait at the appropriate time to ensure that all robots can complete their tasks at the same time, thus improving the overall task execution efficiency and accuracy.

[0012] Furthermore, the implementation process of step (1) is as follows:

[0013] Constructing a cross-product model, given a multi-agent Markov model and automata Define the cross product cMMDP φ for and Represent the joint state and joint action respectively. Among them:

[0014] ·

[0015] ·

[0016] ·

[0017] ·

[0018] ·s 0φ =(s0,q0).

[0019] in represents the MDP model, φ represents the atomic proposition set The LTL formula, represents the labeling function. Q represents the finite state set of DFA, represents the power set of propositional sets (i.e., the range of labeling functions), represents the transfer function, q0 represents the initial state, and F represents the final state.

[0020] Furthermore, the implementation process of step (2) is as follows:

[0021] In hierarchical Monte Carlo tree search, selecting branch tree nodes to form a path from the root node to the leaf node is a key step that determines the direction and scope of the search. The nodes in the search tree are mainly divided into two types: cross-product state nodes and task assignment nodes.

[0022] The cross product state node can be represented as (s,q), where s represents the state of the robot and q represents the automaton DFA. φ The parent node and child node of the cross product state node are both task allocation nodes. The task allocation node is the child node of the cross product state node. Given the target of the parent node and the robot’s state, task assignment refers to the specific robot performing a specific task, for example: Γ(s,P)=<(1,p1),(2,p2),…,(k,p k )>.

[0023] Automaton subgoal extraction: Given a cross product state (s, q), define the automaton subgoal as: definition It represents the current minimum cost of satisfying the linear temporal logic task specification φ with (s,q) as the root node, and is calculated as follows:

[0024]

[0025] Among them, -c(Γ(s,P)) represents the cost of completing the task assignment Γ(s,P), and Children(s,q) represents all task assignment child nodes of node (s,q).

[0026] Task allocation of automaton subgoals: Given the current state s of the robot, the subgoals of the automaton These subtasks (i.e., subgoals) can be assigned to agents with different execution costs. In order to evaluate the effect of task assignment, it is necessary to calculate the cost of executing these tasks. Given the state s of an agent and the task assignment Γ(s,P)=<(1,p1),(2,p2),…,(k,p k )>, robot i executes sub-goal p i The cost of i (s,p i ) is calculated as the transition from the current state s to the target state p iTherefore, for the task assignment Γ(s,P), the global task execution cost is the maximum cost of the individual agent to perform the subtask, that is: in, represents the robots involved in the task assignment Γ(s,P).

[0027] Given a task assignment node (t,q), the current minimum cost C(s,q) is calculated as:

[0028]

[0029] Among them, (t,q′) represents a cross product node, Children(t,q) represents all possible cross product child nodes of the task assignment node (t,q), and q′ represents the φ The corresponding automaton transitions from state q to state.

[0030] Drawing on the UCB idea, the node selection process is regarded as a multi-armed bandit problem. Let N(s,q) be the number of times the node (s,q) is visited during the search process. If the current node is a cross-product state node, the UCB value can be calculated as follows:

[0031]

[0032] Among them, β is a constant variable. Select the child node with the largest UCB value, that is, (t * ,q * ) = argmax (t,q′)∈Children(s,q) UCB(t,q′). Intuitively, when the number of visits N(t,q′) of all child nodes is equal, UCB will choose However, when some child nodes are visited less frequently than others, the selection is more inclined to those with higher Therefore, the UCB value can well balance exploration and utilization.

[0033] Furthermore, the implementation process of step (3) is as follows:

[0034] The process of expanding the search tree is mainly done by adding new leaf nodes to the tree. When expanding the tree, based on the flag of the current node, the system will decide whether to perform action expansion at the automaton state layer or to perform task assignment at the task assignment layer. If the flag indicates that the state corresponding to the current tree node is a "cross product state node", the expansion operation at this time includes selecting a new action and updating the state, and finally adding the new state as a child node in the tree to the search tree. Specifically, the system needs to select actions according to the requirements of the LTL task specification, which may involve the robot's movement, operation, or execution of other tasks in the environment. If the flag indicates that the current node belongs to a "task assignment node", task assignment needs to be performed in the robot at this time.

[0035] Furthermore, the implementation process of step (4) is as follows:

[0036] By simulating different paths, the final execution effect of each path is evaluated. (i.e., the state that satisfies the linear temporal logic specification) is terminated, thereby evaluating the cost and possible results of task execution. Similar to the Monte Carlo tree search simulation process used in AlphaGo, a random simulation strategy is adopted in the path planning process. By simulating the random selection of cross-product state nodes or task assignment nodes, it is determined that if (s, q) is a cross-product state node, the cost is updated, that is, the execution cost of task assignment is increased.

[0037] Furthermore, the implementation process of step (5) is as follows:

[0038] Back propagation updates the number of node visits and costs. The cost is updated as follows: By adding the value Back propagates from the leaf node q' to the root node to update the values ​​of all parent nodes in the tree. For the nodes in the propagation path, if the node (t, q') is a cross product state node, then If it is a task allocation node, then

[0039] Furthermore, the implementation process of step (6) is as follows:

[0040] Task execution synchronization is a key step in industrial robot path planning, especially in multi-agent collaborative tasks. First, determine whether it is a single-task or multi-task scenario. If it is a single-task (Flag = 0), the system selects the robot with the lowest execution cost; if it is a multi-task scenario (Flag = 1), the longest time required for all robots to perform the task is calculated, and the execution time of the slowest robot is selected as the total time of the task. On this basis, a waiting strategy is adopted to ensure that all robots complete the task synchronously. Specifically, when some robots complete the task faster, the start of their subsequent tasks will be delayed until the tasks of all robots are completed, ensuring that each task is completed in the same time. This strategy is crucial in industrial robot collaboration and resource sharing, and can optimize execution efficiency and improve the overall performance of task completion.

[0041] Beneficial effects: Compared with the existing offline MILP and online real-time search methods, the industrial robot path planning method based on hierarchical Monte Carlo tree search proposed in the present invention has significant advantages in linear temporal logic task solving benefits and scalability. The method of the present invention can dynamically adjust task allocation in a real-time environment to improve the efficiency and accuracy of task completion. Secondly, the existing real-time search methods face a large computational burden when dealing with complex constraints and multi-agent collaboration, while the present method significantly improves the computational efficiency and scalability of the system through effective state space pruning and task allocation optimization, especially in large-scale industrial robot collaboration tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the industrial robot path planning method based on hierarchical Monte Carlo tree search. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0044] like Figure 1 As shown, the industrial robot path planning and execution method based on hierarchical Monte Carlo tree search provided by the present invention specifically includes the following steps:

[0045] Step 1: Construct a cross-product model, given a multi-agent Markov model and automata Define the cross product cMMDP φ for and Represent joint state and joint action respectively.

[0046] Step 2: Form a path from the root node to the leaf node by selecting branch tree nodes. The search tree nodes include two types: cross product state nodes and task assignment nodes. The cross product state node can be represented as (s, q), where s represents the state of the robot and q represents the automaton DFA. φ The parent node and child node of the cross product state node are both task allocation nodes. The task allocation node is the child node of the cross product state node. Given the target of the parent node and the robot state s, task assignment refers to the specific robot performing a specific task, for example: Γ(s,P)=<(1,p1),(2,p2),…,(k,p k )>.

[0047] Automaton subgoal extraction: Given a cross product state (s, q), define the automaton subgoal as: definition It represents the current minimum cost of satisfying the linear temporal logic task specification φ with (s,q) as the root node, and is calculated as follows:

[0048]

[0049] Among them, -c(Γ(s,P)) represents the cost of completing the task assignment Γ(s,P), and Children(s,q) represents all task assignment child nodes of node (s,q).

[0050] Task assignment of automaton subgoals: Given the current state s of the robot, the subgoals of the automaton These subtasks (i.e., subgoals) can be assigned to agents with different execution costs. In order to evaluate the effect of task assignment, it is necessary to calculate the cost of executing these tasks. Given the state s of an agent and the task assignment Γ(s,P)=<(1,p1),(2,p2),…,(k,p k )>, robot i executes sub-goal p i The cost of i (s,p i ) is calculated as the transition from the current state s to the target state p i Therefore, for the task assignment Γ(s,P), the global task execution cost is the maximum cost of the individual robot to perform the subtask, that is: in, represents the robots involved in the task assignment Γ(s,P).

[0051] Given a task assignment node (t,q), the current minimum cost C(s,q) is calculated as:

[0052]

[0053] Among them, (t,q′) represents a cross product node, Children(t,q) represents all possible cross product child nodes of the task assignment node (t,q), and q′ represents the φ The corresponding automaton transitions from state q to state.

[0054] Drawing on the UCB idea, the node selection process is regarded as a multi-armed bandit problem. Let N(s,q) be the number of times the node (s,q) is visited during the search process. If the current node is a cross-product state node, the UCB value can be calculated as follows:

[0055]

[0056] Among them, β is a constant variable. Select the child node with the largest UCB value, that is, (t * ,q * ) = argmax (t,q′)∈Children(s,q) UCB(t,q′). Intuitively, when the number of visits N(t,q′) of all child nodes is equal, UCB will choose However, when some child nodes are visited less frequently than others, the selection is more inclined to those with higher Therefore, the UCB value can well balance exploration and utilization.

[0057] Step 3: The process of expanding the search tree is mainly done by adding new leaf nodes to the tree. When expanding the tree, based on the flag of the current node, the system will decide whether to perform action expansion at the automaton state layer or to perform task assignment at the task assignment layer. If the flag indicates that the state corresponding to the current tree node is a "cross product state node", the expansion operation at this time includes selecting a new action and updating the state, and finally adding the new state as a child node in the tree to the search tree. Specifically, the system needs to select actions according to the requirements of the LTL task specification, which may involve the robot's movement, operation, or execution of other tasks in the environment. If the flag indicates that the current node belongs to a "task assignment node", task assignment needs to be performed in the robot at this time.

[0058] Step 4: Evaluate the final execution effect of each path by simulating different paths. (i.e., the state that satisfies the linear temporal logic specification) is terminated, thereby evaluating the cost and possible results of task execution. Similar to the Monte Carlo tree search simulation process used in AlphaGo, a random simulation strategy is adopted in the path planning process. By simulating the random selection of cross-product state nodes or task assignment nodes, it is determined that if (s, q) is a cross-product state node, the cost is updated, that is, the execution cost of task assignment is increased.

[0059] Step 5: Back propagate to update the number of visits and cost of the node. The cost is updated as follows: By adding the value Back propagates from the leaf node q' to the root node to update the values ​​of all parent nodes in the tree. For the nodes in the propagation path, if the node (t, q') is a cross product state node, then If it is a task allocation node, then

[0060] Step 6: Task execution synchronization is a key step in industrial robot path planning, especially in multi-agent collaborative tasks. First, determine whether it is a single-task or multi-task scenario. If it is a single-task (Flag = 0), the system selects the robot with the lowest execution cost; if it is a multi-task scenario (Flag = 1), the longest time required for all robots to perform the task is calculated, and the execution time of the slowest robot is selected as the total time of the task. On this basis, a waiting strategy is adopted to ensure that all robots complete the task synchronously. Specifically, when some robots complete the task faster, the start of their subsequent tasks will be delayed until the tasks of all robots are completed, ensuring that each task is completed in the same time. This strategy is crucial in industrial robot collaboration and resource sharing, and can optimize execution efficiency and improve the overall performance of task completion.

[0061] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A method for industrial robot path planning and execution based on hierarchical Monte Carlo tree search, characterized in that: The following steps are involved: (1) Constructing a cross-product model: Given a multi-agent Markov decision process model that represents the robot environment characteristics and an automaton corresponding to a linear temporal logic task, construct a cross-product model. (2) Decision node selection: By selecting appropriate branches in the tree nodes, a path from the root node to the leaf node is formed. The nodes are divided into automaton sub-cross product state nodes and task allocation nodes. (3) Tree structure expansion: When an unvisited sub-cross product automaton state node or task assignment node is encountered during the selection process, the algorithm will expand the search tree by adding leaf nodes to the tree. (4) Strategy simulation and evaluation: Randomly select cross-product state nodes and task allocation nodes, and execute simulation strategies. (5) Backtracking and value updating: After initializing the value of the newly added node, the values ​​of all nodes in the tree are updated by backpropagating the value from the leaf node to the root node, ensuring that the value of each node can reflect the optimization result of the global task; (6) Task execution synchronization phase: By coordinating the actions between industrial robots, some robots are made to wait at the appropriate time to ensure that all robots can complete their respective tasks at the same time.

2. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 1 is characterized in that: The implementation process of step (1) is as follows: Construct a cross-product model, given a multi-agent Markov decision process model in represents the MDP model, φ represents the atomic proposition set The LTL formula, Represents the labeling function, given the automaton corresponding to the linear temporal logic task Where: Q represents the finite state set of DFA, represents the power set of propositional sets (i.e., the range of labeling functions), represents the transfer function, q0 represents the initial state, F represents the final state, and the cross product cMMDP is defined φ for and Represent the joint state and joint action respectively, where: · · · · ·s 0φ =(s0,q0)。 3. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 1, characterized in that: The implementation process of step (2) is as follows: In hierarchical Monte Carlo tree search, selecting branch tree nodes to form a path from the root node to the leaf node is a key step. The nodes in the search tree are mainly divided into two types: cross-product state nodes and task assignment nodes. Cross-product state node: The cross-product state node represents the combination of the physical state of the industrial robot and the state of the automaton, denoted as (s, q), where s represents the current physical state of the industrial robot and q represents the automaton DFA φ status, Task allocation node: The task allocation node is a child node of the cross product state node, which is used to indicate the specific task allocation situation. and the robot state s, task assignment refers to the specific robot performing a specific task. Task assignment Γ(s,P) can be expressed as <(1,p1),(2,p2),…,(k,p k )>, where each (i,p i ) indicates that robot i performs subtask p i , Automaton subgoal extraction: Given a cross product state (s, q), define the automaton subgoal as: definition It represents the current minimum cost of satisfying the linear temporal logic task specification φ with (s,q) as the root node, and is calculated as follows: Among them, -c(Γ(s,P)) represents the cost of completing the task assignment Γ(s,P), Children(s,q) represents all task assignment child nodes of node (s,q), Task assignment of automaton subgoals: Given the current state s of the robot, the subgoals of the automaton These subtasks (i.e., subgoals) are assigned to robots with different execution costs. In order to evaluate the effect of task assignment, it is necessary to calculate the cost of executing these tasks. Given the robot state s and the task assignment Γ(s,P)=<(1,p1),(2,p2),…,(k,p k )>, robot i executes sub-goal p i The cost of i (s,p i ) is calculated as the transition from the current state s to the target state p i Therefore, for the task assignment Γ(s,P), the global task execution cost is the maximum cost of the individual agent to perform the subtask, that is: in, represents the robots involved in the task assignment Γ(s,P), Given a task assignment node (t,q), the current minimum cost C(s,q) is calculated as: Among them, (t,q′) represents a cross product node, Children(t,q) represents all possible cross product child nodes of the task assignment node (t,q), and q′ represents the φ The corresponding automaton transfers from state q to the state, The node selection process is regarded as a multi-armed bandit problem, and N(s,q) is the number of times the node (s,q) is visited during the search process. If the current node is a cross-product state node, the UCB value can be calculated as follows: Among them, β is a constant variable, and the child node with the largest UCB value is selected, that is, (t * ,q * )=argmax (t,q′)∈Children(s,q) UCB(t,q′), when the number of visits N(t,q′) of all child nodes is equal, UCB will choose When some child nodes are visited less frequently than other child nodes, the selection is more inclined to those with higher Value, that is, the node with higher exploration potential, therefore, the UCB value can balance exploration and utilization well.

4. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 1, characterized in that: The implementation process of step (3) is as follows: If the flag indicates that the state corresponding to the current tree node is a "cross product state node", the expansion operation at this time includes selecting a new action and updating the state, and finally adding the new state as a child node in the tree to the search tree. The system needs to select actions according to the requirements of the LTL task specification. These actions may involve the robot's movement, operation or execution of other tasks in the environment. If the flag indicates that the current node belongs to a "task allocation node", task allocation needs to be performed in the robot at this time.

5. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 1, characterized in that: The implementation process of step (4) is as follows: By simulating different paths, evaluating the final execution effect of each path, simulating the The path planning process is terminated when (s, q) reaches a state that satisfies the linear temporal logic specification, thereby evaluating the cost and possible results of task execution. A random simulation strategy is used in the path planning process to simulate the random selection of cross-product state nodes or task assignment nodes. If (s, q) is a cross-product state node, the cost is updated, that is, the execution cost of task assignment is increased.

6. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 1, characterized in that: The implementation process of step (5) is as follows: Back propagation updates the number of node visits and costs. The cost is updated as follows: By adding the value Back propagates from the leaf node q′ to the root node to update the values ​​of all parent nodes in the tree. For the nodes in the propagation path, if the node (t, q′) is a cross product state node, then If it is a task allocation node, then 7. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 1, characterized in that: The implementation process of step (6) is as follows: First, determine whether it is a single-task or multi-task scenario. If it is a single-task scenario (Flag=0), the system selects the robot with the lowest execution cost; if it is a multi-task scenario (Flag=1), the longest time required for all robots to perform the task is calculated, and the execution time of the slowest robot is selected as the total task time.

Images