Industrial robot path planning and execution method based on hierarchical monte carlo tree search

Abstract

The application discloses an industrial robot path planning and execution method based on hierarchical Monte Carlo tree search. By converting the global complex linear temporal logic task into a finite state automaton, the task is simplified and clearly represented, making it easier for the industrial robot to understand and execute. On this basis, a cross product model of finite state automaton and multi-agent Markov decision process is further constructed to more effectively handle the coordination problem between multiple industrial robots. Further, in order to compress the history state space, a hierarchical Monte Carlo tree search (MCTS) algorithm containing task selection and task allocation is proposed, which contains two levels of task selection and task allocation, can significantly reduce the algorithm complexity and improve the search efficiency. In summary, the application provides an efficient, accurate and flexible online path planning and execution solution for industrial robots, especially suitable for various long-range tasks with temporal logic characteristics.

Description

TECHNICAL FIELD

[0001] The application relates to an industrial robot path planning and execution method based on hierarchical Monte Carlo tree search, and belongs to the technical field of information. BACKGROUND

[0002] Industrial robots are widely used in manufacturing, logistics, automated production and other fields, and can efficiently complete tasks such as assembly, welding, painting and transportation. With the advancement of technology, industrial robots are developing towards higher flexibility and adaptability, especially when facing complex tasks, multiple robots need to be coordinated to perform long-range collaborative tasks. Multi-agent planning technology aims to find a strategy for each agent that can maximize the long-term reward function. However, tasks in real-world scenarios often involve time, space and logical constraints, which makes it difficult for traditional planning methods based on scalar reward functions to effectively solve such tasks. For example, in an industrial production task, when a production line detects a device failure, the industrial robot must perform tasks in a specific order at each time step, perform maintenance or replacement operations within a limited time, and must ensure that other critical devices continue to operate normally after the fault is repaired, avoiding violations of the timing constraints. In a warehouse management task, robots need to transport goods based on real-time data and complete tasks within a specified time window, ensuring that each transport operation is performed in sequence while avoiding obstacles and task conflicts. These tasks require strict adherence to linear temporal logic constraints to ensure successful task completion and meet both time and quality requirements. Existing multi-agent planning methods are difficult to handle these tasks with complex constraints, and recent research attempts to use linear temporal logic language to model such tasks.

[0003] To model linear temporal logic constraints, existing multi-agent planning methods attempt to use mixed integer linear programming (MILP) to model the entire action sequence of each agent. On the other hand, in unknown environments, many recent studies in the field of machine learning, such as reinforcement learning (RL) methods, consider how to learn the strategy of multiple agents to meet linear temporal logic goals. Due to the complexity of multi-agent planning problems, which grows exponentially with the number of agents and the length of linear temporal logic tasks, existing MILP methods and RL methods have low scalability. Therefore, some scholars propose to decompose the global linear temporal logic task into multiple independent linear temporal logic subtasks, and then assign each subtask to each agent for independent execution, thereby improving computational efficiency. However, global long-range collaborative linear temporal logic tasks may have high coupling and are difficult to decompose into independent subtasks. Some scholars propose to solve linear temporal logic tasks based on heuristic search mechanisms, but this method is only applicable to single robot scenarios. Therefore, although existing technical solutions can solve planning problems for linear temporal logic tasks, they cannot meet the demand of ensuring task completion and solving quality. SUMMARY

[0004] The present application aims to solve the problem of how to plan a solution for industrial robots in long-range tasks facing linear temporal logic constraints that can meet task requirements and also consider real-time. To this end, the present application proposes an industrial robot path planning and execution method based on hierarchical Monte Carlo tree search. This method can support and implement complex long-range planning tasks, especially when linear temporal logic constraints are involved, improving the overall efficiency of multi-robot collaboration and the benefits of task execution. At the same time, this method also ensures the real-time response capability and good scalability of the system, adapting to changing task requirements and dynamic environment.

[0005] Technical solution: The present application proposes an industrial robot path planning and execution method based on hierarchical Monte Carlo tree search, thereby compressing the history state space of multi-agent long-range collaboration. First, the global linear temporal logic task is equivalent to a finite state automaton, which converts complex tasks into a set of states and transitions. Then, in order to facilitate the representation of the joint transition function of automaton state and robot state, a cross product model of finite state automaton and multi-agent Markov decision process is constructed. Further, a hierarchical MCTS algorithm is proposed, which includes task selection and task allocation. The task selection level represents the state selection of the automaton, and the task allocation level represents the allocation of different sub-tasks to different robots, thereby reducing complexity and improving search efficiency through hierarchical reduction. Specifically, the following steps are included:

[0006] (1) Constructing a cross product model: Given a multi-agent Markov decision process model representing the characteristics of the robot environment and a linear temporal logic task LTL φ corresponding automaton, a cross product model is constructed. This model limits the subsequent search space by considering environmental state and task constraints, so that subsequent exploration meets the requirements of temporal logic tasks. The key to this step is that the cross product model provides a clear structural framework for multi-robot path planning, allowing task and environmental features to be effectively integrated;

[0007] (2) Decision node selection: By selecting appropriate branches in the tree nodes, a path from the root node to the leaf node is formed. The nodes are divided into automaton sub-cross product state nodes and task allocation nodes. Through hierarchical selection strategy, all subsequent sub-tasks are still considered in the task selection stage, but only a small number of task exploration allocation methods are explored, thereby exploring more in a limited time. This process is the key step of the hierarchical Monte Carlo tree search algorithm, which can effectively reduce the search space and improve computational efficiency;

[0008] (3) Tree structure expansion phase: When encountering unvisited sub-cross product automaton state nodes or task allocation nodes during the selection process, the algorithm expands the search tree by adding leaf nodes to the tree. This phase ensures the integrity of the tree structure and provides more potential paths for subsequent decision-making;

[0009] (4) Strategy simulation and evaluation: Randomly select cross product state nodes and task allocation nodes and execute the simulation strategy. The simulation strategy not only considers environmental factors, but also integrates task constraints, taking into account multi-robot cooperation and conflict avoidance during simulation, providing more accurate feedback for subsequent decision-making;

[0010] (5) Backtracking and value updating: After initializing the values of the newly added nodes, the values of all nodes in the tree are updated by propagating the values from the leaf nodes to the root node, ensuring that the value of each node reflects the global task optimization result;

[0011] (6) Task execution synchronization phase: Task. To ensure the smooth execution of tasks, coordinate the actions of industrial robots so that some robots wait at the right time to ensure that all robots can complete their tasks at the same time, improving the overall task execution efficiency and accuracy.

[0012] Further, the step (1) is implemented as follows:

[0013] Construct a cross product model, given a multi-agent Markov model and automaton Define cross product cMMDP φ For and represent joint state and joint action respectively. Among them:

[0014] ·

[0015] ·

[0016] ·

[0017] ·

[0018] ·s 0φ = (s0, q0).

[0019] Where represents the MDP model, φ represents the LTL formula based on the atomic proposition set , is the marking function. Q represents the finite state set of DFA, Power set representing the set of propositions (i.e. the range of the labeling function), Transition function, q0 represents the initial state, and F represents the final state.

[0020] Further, the step (2) is implemented as follows:

[0021] In hierarchical Monte Carlo tree search, the key step is to select the branch tree node to form a path from the root node to the leaf node, which determines the direction and range of the search. The nodes in the search tree are mainly divided into two types: cross product state nodes and task allocation nodes.

[0022] The cross product state node can be represented as (s, q), where s represents the state of the robot, and q represents the state of the automaton DFA φ The parent node and child node of the cross product state node are both task allocation nodes. The task allocation node is the child node of the cross product state node. Given the target of the parent node and the state of the robot, the task allocation refers to the specific task performed by the specific robot, for example: Γ(s, P) = <(1, p1), (2, p2), …, (k, pk k )>.

[0023] Automaton subgoal extraction: Given the cross product state (s, q), define the automaton subgoal as: Definition C(s, q) represents the current minimum cost of satisfying the linear temporal logic task specification φ with (s, q) as the root node, and the calculation method is:

[0024]

[0025] Where -c(Γ(s, P)) represents the cost of completing the task allocation Γ(s, P), and Children(s, q) represents all task allocation child nodes of the node (s, q).

[0026] Automaton subgoal task allocation: Given the current state s of the robot, the subgoal of the automaton can be assigned to the Agent with different execution costs. In order to evaluate the effect of task allocation, the cost of executing these tasks needs to be calculated. Given the state s of the Agent, and the task allocation Γ(s, P) = <(1, p1), (2, p2), …, (k, pk k >, the cost c i (s, p i ) of the robot i executing the subgoal p i is calculated as the cost from the current state s to the target state p ithe shortest path. Therefore, for a task assignment Γ(s, P), the global task execution cost is the maximum cost of the individual agents executing the sub-tasks, i.e., where, represents the robots involved in the task assignment Γ(s, P).

[0027] Given a task assignment node (t, q), the current minimum cost C(s, q) is calculated as:

[0028]

[0029] where, (t, q') represents the cross product node, Children(t, q) represents all possible cross product child nodes of the task assignment node (t, q), and q' represents the LTL φ the state that the corresponding automaton moves from state q to.

[0030] Inspired by the UCB idea, the node selection process is regarded as a multi-armed bandit problem. Let N(s, q) be the number of times that the node (s, q) is visited in the search process. If the current node is a cross product state node, the UCB value can be calculated as follows:

[0031]

[0032] where β is a constant variable. The child node with the maximum UCB value is selected, i.e., (t * , q * ) = argmax (t,q′)∈Children(s,q) UCB(t, q'). Intuitively, when the visit times N(t, q') of all child nodes are equal, UCB selects the child node with the maximum value. However, when the visit frequency of some child nodes is lower than that of other child nodes, the selection is more inclined to those nodes with higher values. Therefore, the UCB value can well balance exploration and exploitation.

[0033] Further, the step (3) is implemented as follows:

[0034] The process of expanding the search tree is mainly done by adding new leaf nodes to the tree. When expanding the tree, based on the flag of the current node, the system will decide whether to perform action expansion at the automaton state layer or to perform task assignment at the task assignment layer. If the flag indicates that the state corresponding to the current tree node is a "cross product state node", the expansion operation at this time includes selecting a new action and updating the state, and finally adding the new state as a child node in the tree to the search tree. Specifically, the system needs to select actions according to the requirements of the LTL task specification. These actions may involve the robot's movement, operation, or execution of other tasks in the environment. If the flag indicates that the current node belongs to a "task assignment node", task assignment needs to be performed in the robot at this time.

[0035] Furthermore, the implementation process of step (4) is as follows:

[0036] By simulating different paths, the final execution effect of each path is evaluated. The algorithm terminates when the task reaches a state that satisfies the linear temporal logic specification, thereby evaluating the cost and possible outcomes of task execution. Similar to the Monte Carlo tree search simulation process used in AlphaGo, the path planning process employs a random simulation strategy, simulating the random selection of cross-product state nodes or task assignment nodes. If (s,q) is a cross-product state node, the cost is updated, increasing the execution cost of the task assignment.

[0037] Furthermore, the implementation process of step (5) is as follows:

[0038] Back propagation updates the number of visits and cost of the node. The cost is updated as follows: By adding the value Back propagates from the leaf node q' to the root node to update the values ​​of all parent nodes in the tree. For the nodes in the propagation path, if the node (t,q') is a cross product state node, then If it is a task allocation node, then

[0039] Furthermore, the implementation process of step (6) is as follows:

[0040] Task execution synchronization is a key step in industrial robot path planning, especially in multi-agent collaborative tasks. First, determine whether it is a single task or a multi-task scenario. If it is a single task (Flag = 0), the system selects the robot with the lowest execution cost; if it is a multi-task scenario (Flag = 1), calculate the longest time required for all robots to execute the task, and select the execution time of the slowest robot as the total time of the task. On this basis, the waiting strategy is adopted to ensure that all robots complete the task synchronously. Specifically, when some robots complete the task faster, the start of the subsequent task is delayed until all robots complete the task, ensuring that each task is completed within the same time. This strategy is crucial in industrial robot collaboration and resource sharing, optimizing execution efficiency and improving overall performance in task completion.

[0041] Beneficial effects: Compared with existing offline MILP and online real-time search methods, the industrial robot path planning method based on hierarchical Monte Carlo tree search proposed by the present application has significant advantages in linear temporal logic task solving benefits and scalability. The method of the present application can dynamically adjust task allocation in real-time environment, improve the efficiency and accuracy of task completion. Second, existing real-time search methods face a large computational burden when dealing with complex constraints and multi-agent collaboration, while the present method significantly improves the computational efficiency and scalability of the system through effective state space pruning and task allocation optimization, especially in large-scale industrial robot collaboration tasks. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A schematic diagram of the industrial robot path planning method based on hierarchical Monte Carlo tree search. DETAILED DESCRIPTION

[0043] The present application will be further described in detail below in conjunction with the drawings and specific embodiments:

[0044] As Figure 1 shown, the industrial robot path planning and execution method based on hierarchical Monte Carlo tree search provided by the present application specifically includes the following steps:

[0045] Step 1: Construct a cross product model, given a multi-agent Markov model and an automaton Define cross product cMMDP φ for and represent joint states and joint actions, respectively.

[0046] Step 2: Form a path from the root node to the leaf node by selecting the branch tree nodes. The search tree nodes include two types: cross product state nodes and task assignment nodes. The cross product state node can be represented as (s,q), where s represents the state of the robot and q represents the automaton DFA. φ The parent node and child node of the cross product state node are both task assignment nodes. The task assignment node is the child node of the cross product state node. Given the target of the parent node and the robot’s state s, task assignment refers to the specific robot performing a specific task, for example: Γ(s,P)=<(1,p1),(2,p2),…,(k,p k )>.

[0047] Automaton subgoal extraction: Given a cross-product state (s,q), define the automaton subgoal as: definition It represents the current minimum cost of the linear temporal logic task specification φ with (s,q) as the root node, and is calculated as follows:

[0048]

[0049] Among them, -c(Γ(s,P)) represents the cost of completing the task assignment Γ(s,P), and Children(s,q) represents all task assignment child nodes of node (s,q).

[0050] Task assignment of automaton subgoals: Given the current state s of the robot, the subgoals of the automaton These subtasks (i.e., subgoals) can be assigned to agents with different execution costs. In order to evaluate the effectiveness of task assignment, it is necessary to calculate the cost of executing these tasks. Given the state s of the agent and the task assignment Γ(s,P)=<(1,p1),(2,p2),…,(k,p k )>, robot i executes sub-goal p i The cost of c i (s,p i ) is calculated as the transition from the current state s to the target state p i Therefore, for the task assignment Γ(s,P), the global task execution cost is the maximum cost of the individual robot to perform the subtask, that is: in, represents the robots involved in the task assignment Γ(s,P).

[0051] Given a task assignment node (t,q), the current minimum cost C(s,q) is calculated as:

[0052]

[0053] where (t, q') denotes a cross-product node, Children(t, q) denotes all possible cross-product children of task assignment node (t, q), and q' denotes the cross-product of LTL formula q and the current state of the automaton. φ the state that the corresponding automaton transitions from state q.

[0054] UCB idea, the node selection process is regarded as a multi-armed bandit problem. Let N(s, q) denote the number of times that node (s, q) is visited in the search process. If the current node is a cross-product state node, the UCB value can be calculated as follows:

[0055]

[0056] where β is a constant variable. The child node with the maximum UCB value is selected, i.e., (t * , q * ) = argmax (t,q′)∈Children(s,q) UCB(t, q'). Intuitively, when the visit frequencies of all child nodes are equal, UCB will select the child node with the maximum value. However, when the visit frequencies of some child nodes are lower than those of other child nodes, the selection is more inclined to those nodes with higher values. Therefore, the UCB value can well balance exploration and exploitation.

[0057] Step 3: The process of expanding the search tree mainly adds new leaf nodes to the tree. When expanding the tree, according to the flag of the current node, the system will decide whether to perform action expansion in the automaton state layer or to perform task assignment in the task assignment layer. If the flag (Flag) indicates that the state corresponding to the current tree node is a "cross-product state node", the expansion operation at this time includes selecting a new action and updating the state, and finally adding the new state as a child node in the search tree. Specifically, the system needs to select actions according to the requirements of the LTL task specification, which may involve the movement, operation or execution of other tasks of the robot in the environment. If the flag (Flag) indicates that the current node belongs to the "task assignment node", task assignment needs to be performed in the robot at this time.

[0058] Step 4: By simulating different paths, the final execution effect of each path is evaluated. Simulation is performed until the acceptance state The algorithm terminates when the task reaches a state that satisfies the linear temporal logic specification, thereby evaluating the cost and possible outcomes of task execution. Similar to the Monte Carlo tree search simulation process used in AlphaGo, the path planning process employs a random simulation strategy, simulating the random selection of cross-product state nodes or task assignment nodes. If (s,q) is a cross-product state node, the cost is updated, increasing the execution cost of the task assignment.

[0059] Step 5: Back propagate to update the number of visits and cost of the node. The cost is updated as follows: By adding the value Back propagates from the leaf node q' to the root node to update the values ​​of all parent nodes in the tree. For the nodes in the propagation path, if the node (t,q') is a cross product state node, then If it is a task allocation node, then

[0060] Step 6: Task execution synchronization is a key step in industrial robot path planning, especially in multi-agent collaborative tasks. First, determine whether it is a single-task or multi-task scenario. If it is a single-task (Flag = 0), the system selects the robot with the lowest execution cost; if it is a multi-task scenario (Flag = 1), the longest time required for all robots to perform the task is calculated, and the execution time of the slowest robot is selected as the total time of the task. On this basis, a waiting strategy is adopted to ensure that all robots complete the task synchronously. Specifically, when some robots complete the task faster, the start of their subsequent tasks will be delayed until the tasks of all robots are completed, ensuring that all tasks are completed in the same time. This strategy is crucial in industrial robot collaboration and resource sharing, and can optimize execution efficiency and improve the overall performance of task completion.

[0061] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions fall within the scope of protection of the claims of the present invention.

Claims

1. A method for industrial robot path planning and execution based on hierarchical Monte Carlo tree search, characterized in that: The following steps are involved: (1) Constructing a cross-product model: Given a multi-agent Markov decision process model that represents the robot's environment characteristics and an automaton corresponding to a linear temporal logic task, construct a cross-product model. (2) Decision node selection: By selecting appropriate branches in the tree nodes, a path from the root node to the leaf node is formed. The nodes are divided into automaton sub-cross product state nodes and task allocation nodes. (3) Tree structure expansion: When encountering an unvisited sub-cross product automaton state node or task assignment node during the selection process, the algorithm will expand the search tree by adding a leaf node to the tree. (4) Strategy simulation and evaluation: Randomly select cross-product state nodes and task allocation nodes, and execute simulation strategies. (5) Backtracking and value updating: After initializing the value of the newly added node, the values ​​of all nodes in the tree are updated by backpropagating the value from the leaf node to the root node, ensuring that the value of each node can reflect the optimization result of the global task; (6) Task execution synchronization stage: By coordinating the actions of industrial robots, some robots are made to wait at the appropriate time to ensure that all robots can complete their tasks at the same time; The implementation process of step (1) is as follows: Construct a cross-product model, given a multi-agent Markov decision process model in Represents the MDP model, φ represents the set of atomic propositions The LTL formula, Represents a labeling function, given a linear temporal logic task corresponding to the automaton Where: Q represents the finite state set of DFA, The power set of the proposition set is the range of the labeling function, represents the transfer function, q0 represents the initial state, F represents the final state, and the cross product cMMDP is defined φ for and Represent joint states and joint actions respectively, where: ● ● ● ● ●s 0φ =(s0,q0)。 2. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 1 is characterized in that: The implementation process of step (2) is as follows: In hierarchical Monte Carlo tree search, selecting branch tree nodes to form a path from the root node to the leaf node is a key step. The nodes in the search tree are mainly divided into two types: cross-product state nodes and task assignment nodes. Cross-product state node: The cross-product state node represents the combination of the physical state of the industrial robot and the state of the automaton, denoted as (s,q), where s represents the current physical state of the industrial robot and q represents the automaton DFA φ status, Task assignment node: The task assignment node is a child node of the cross product state node, which is used to represent the specific task assignment situation. and the robot’s state s, task assignment refers to the specific robot performing a specific task, and the task assignment Γ(s,P) is expressed as <(1,p1),(2,p2),…,(k,p k )>, where each (i,p i ) indicates that robot i performs subtask p i , Automaton subgoal extraction: Given a cross-product state (s, q), define the automaton subgoal as: definition It represents the current minimum cost of satisfying the linear temporal logic task specification φ with (s, q) as the root node, and is calculated as follows: Among them, -c(Γ(s, P)) represents the cost of completing the task assignment Γ(s, P), Children(s, q) represents all task assignment child nodes of node (s, q), Task assignment of automaton subgoals: Given the current state s of the robot, the subgoals of the automaton Assign these subgoals to robots with different execution costs and calculate the cost of executing these tasks, given the robot's state s and the task assignment Γ(s, P) = <(1, p1),(2, p2),…,(k, p k )>, robot i executes sub-goal p i The cost of c i (s,p i ) is calculated as the transition from the current state s to the target state p i For the shortest path of task assignment Γ(s, P), the global task execution cost is the maximum cost of individual Agent to perform subtasks, that is: in, represents the robots involved in the task assignment Γ(s, P), Given a task assignment node (t, q), the current minimum cost C(s, q) is calculated as: Among them, (t, q′) represents the cross product node, Children(t, q) represents all possible cross product child nodes of the task assignment node (t, q), and q′ represents the φ The corresponding automaton transitions from state q to the state, The node selection process is regarded as a multi-armed bandit problem. N(s, q) is the number of times the node (s, q) is visited during the search process. If the current node is in the cross-product state, the UCB value is calculated as follows: Among them, β is a constant variable, and the child node with the largest UCB value is selected, that is, (t * ,q * )=argmax (t，q′)∈Children(s，q) UCB(t,q′), when the number of visits N(t,q′) of all child nodes is equal, UCB will choose When some child nodes are visited less frequently than other child nodes, the selection is more inclined to those with the largest value. Higher values ​​indicate nodes with higher exploration potential.

3. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 1, characterized in that: The implementation process of step (3) is as follows: If the flag indicates that the state corresponding to the current tree node is a "cross product state node", the expansion operation at this time includes selecting a new action and updating the state, and finally adding the new state as a child node in the tree to the search tree. The system needs to select actions according to the requirements of the LTL task specification. These actions involve the robot's movement, operation or execution of other tasks in the environment. If the flag indicates that the current node belongs to a "task assignment node", task assignment needs to be performed in the robot at this time.

4. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 3 is characterized in that: The implementation process of step (4) is as follows: By simulating different paths, evaluating the final execution effect of each path, simulating the final execution effect of each path, and simulating the final execution effect of each path before reaching the acceptance state. That is, it terminates when the state satisfies the linear temporal logic specification, thereby evaluating the cost and possible results of task execution. A random simulation strategy is adopted in the path planning process. By simulating the random selection of cross-product state nodes or task assignment nodes, it is determined that if (s, q) is a cross-product state node, the cost is updated, that is, the execution cost of the task assignment is increased.

5. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 4, characterized in that: The implementation process of step (5) is as follows: Back propagation updates the number of visits and cost of the node. The cost is updated as follows: By adding the value Back propagates from the leaf node q' to the root node to update the values ​​of all parent nodes in the tree. For the nodes in the propagation path, if the node (t, q') is a cross product state node, then If it is a task allocation node, then 6. The industrial robot path planning and execution method based on hierarchical Monte Carlo tree search according to claim 5, characterized in that: The implementation process of step (6) is as follows: First, determine whether it is a single-task or multi-task scenario. If it is a single-task scenario, Flag = 0, and the system selects the robot with the lowest execution cost. If it is a multi-task scenario, Flag = 1, then calculate the longest time required for all robots to perform the task, and select the execution time of the slowest robot as the total task time.

Images