Robot path planning method based on partial order analysis and reinforcement learning

Abstract

The invention provides a robot path planning method based on partial order analysis and reinforcement learning, which combines partial order analysis and hierarchical reinforcement learning, depicts a complex task through linear sequential logic, decomposes the whole task into sub-tasks by using partial order analysis, extracts a key sequential logic relationship, and finally performs robot path planning. A strategy for reaching a specific target point is obtained by using a low-layer path planning module of hierarchical reinforcement learning, and a high-layer task decision-making module makes a decision to a target point conforming to logic constraints by using information obtained by partial order analysis; in addition, according to a partial order boundary function, partial order analysis and hierarchical reinforcement learning are combined for the first time, a task set conforming to logic constraints is updated, a dynamically changing reward mechanism is constructed, and the reward density is improved. According to the method, the algorithm sample efficiency and the path planning success rate can be improved under the conditions of large environment state space and high task complexity, the core problems that robot path planning is long in planning time, the obtained path violates logic constraints and the like under the condition are solved, and the application prospect of the mobile robot in the real world is expanded.

Description

Technical Field

[0001] This invention relates to the field of multi-task path planning, and in particular to a robot path planning method based on partial order analysis and reinforcement learning. Background Technology

[0002] With social development and technological advancements, mobile robots are increasingly being widely applied in various fields such as logistics, deep space exploration, and the military. The fundamental goal of path planning is to find a feasible path from the starting point to the destination. Existing path planning algorithms, such as Model Predictive Control (MMCC) and Particle Swarm Optimization (PSO), can effectively solve point-to-point path planning problems, but they cannot handle path planning for complex logical tasks in practice. They also cannot understand the temporal and logical constraints between user-defined tasks, which limits the application prospects of mobile robots in performing complex tasks. For example, a drone might need to go to area A to pick up goods and then to area B to deliver them. Choosing to go to area B first and then to area A simply because the overall path is shorter violates the logical constraint that goods must be picked up before delivery. This requires further introduction of linear temporal logic to characterize complex tasks and combining it with existing reinforcement learning algorithms. However, the complexity of logical tasks and sparse rewards significantly increase the difficulty of path planning. As the task size and complexity increase, and the environment scale expands, the low sample efficiency of existing reinforcement learning methods becomes apparent, making it difficult to obtain feasible paths that conform to logical constraints. Summary of the Invention

[0003] The purpose of this invention is to provide a robot path planning method based on partial order analysis and reinforcement learning to solve the single robot path planning problem with logical constraints between tasks.

[0004] The technical solution to achieve the purpose of this invention is: a robot path planning method based on partial order analysis and reinforcement learning, comprising the following steps:

[0005] Environmental modeling steps: Construct a generalized Markov decision process M describing the gridded environment, wherein the generalized Markov decision process M = (S, A, s0, P, AP, L, R), where:

[0006] S is a finite set of states, where each state corresponds to a physical location of the robot, usually in two-dimensional or three-dimensional discrete spatial coordinates.

[0007] A is a finite set of actions, each of which controls the robot's movement to adjacent physical locations, including basic actions such as moving up, down, left, right, or remaining stationary;

[0008] s0∈S is the robot's initial state, corresponding to the robot's physical position at the start time;

[0009] P:S×A→S is a deterministic state transition function that describes the change in the robot's physical position after it performs an action;

[0010] AP is a set of atomic propositions, where atomic propositions declare whether the robot is located in a specific task area. Each element α∈AP corresponds one-to-one with a predefined task area in the map, representing the task of the robot "arriving" or "visiting" a specific area. An atomic proposition is a Boolean variable; it is true if and only if the robot is in the physical position state corresponding to that area. Atomic propositions corresponding to other physical positions outside the task area are considered empty elements.

[0011] Tag function L:S→2 AP Assign an atomic proposition that is true at each physical location state s;

[0012] R:S×A→R is the reward function, which describes the immediate reward that the robot can obtain by performing an action at the current physical position. The immediate reward includes the cost of negative constants such as time and energy, as well as the normal reward for completing the task.

[0013] Task parsing steps: Receiving linear sequential logic formula The described complex task, constructing the corresponding The Butch automaton B, wherein the Butch automaton B = (Q, q0, Δ, q f ),in:

[0014] Q is the set of automaton states, where each state q∈Q corresponds to the achievement of a sub-goal of the task, representing an abstract logical stage when a certain part of the linear sequential logic formula is satisfied, rather than the specific physical position of the robot;

[0015] q0 is the initial state, corresponding to the logical phase at which the task begins at the initial moment;

[0016] Δ is a deterministic state transition function, which defines how the task logic stage should change when the robot reaches a specific task area under the current abstract task stage q. The state transition function encodes the task progress logic specified by the linear sequential logic formula.

[0017] q f The "accept" state corresponds to the logical stage where all linear sequential logic tasks have been completed.

[0018] The Butch automaton B was analyzed using partial order analysis. The process involves decomposing the data into all subtasks and extracting the logical constraints between them to obtain a partially ordered set. in This is a set of subtasks, where each subtask corresponds to a specific task area that the robot needs to reach; subtask ω lIt consists of an atomic proposition label σ and a unique index l. This represents the set of sequential relationships between subtasks.

[0019] Dynamic task set and reward construction steps: Construct and initialize a feasible subtask set A, representing the set of subtasks that satisfy the linear temporal logic constraints at the current time. The preceding task of each subtask in the set has been completed by the robot. If the robot completes the subtasks in the set, it will not violate the logic constraints. The initial value of the feasible subtask set A is the task corresponding to all subtasks without preceding constraints at the initial time. Construct the partial order boundary function Pgf: Ω×2 Ω →2 Ω This is used to dynamically update the feasible subtask set A after the robot completes each subtask. If the subtask is in the current feasible subtask set, it is removed from A according to the partial order boundary function, and subsequent tasks that require the subtask as a prerequisite are added to the feasible subtask set A. If the subtask is not in the current feasible subtask set, the feasible subtask set A remains unchanged. Based on the updated feasible subtask set A, a dynamic high-level reward function R is constructed. high ;

[0020] The hierarchical decision model construction steps are as follows: The generalized Markov decision process M is multiplied by the Butch automaton B to construct a product semi-Markov decision process M. Θ =(S×Q,A,AP,O,L,P×Δ,R) high The constructed product semi-Markov decision process can characterize how the robot makes task decisions. The robot will need to decide which specific task area to proceed to based on the logical constraints between sub-tasks. The product semi-Markov decision process will serve as the input to the high-level task decision module in the hierarchical learning and planning steps, where:

[0021] S×Q represents each state in the state space, which consists of two parts: a generalized Markov decision process state representing the physical location of the robot, and a Butch automaton state representing the logical stage in which the robot completes a linear sequential logic task.

[0022] A is a finite set of actions, each action used to control the robot's movement to adjacent physical locations, including basic actions such as moving up, down, left, right, or remaining stationary;

[0023] AP is a set of atomic propositions, in which atomic propositions are used to declare whether the robot is located in a specific task area. Each element α∈AP in the set of atomic propositions corresponds one-to-one with a predefined task area in the map, and is used to represent the task of the robot "arriving" or "visiting" a specific area.

[0024] O is the sub-task set The exported option set, where each option o∈O is associated with a subtask ω. l This is associated with and corresponds to a sub-strategy π learned by the underlying module for reaching the target region of the task. o ;

[0025] L:S→2 AP Assign the label function an atomic proposition that is true at each physical location;

[0026] P×Δ is the state transition function, which consists of the low-level generalized Markov decision process transition function P and the Butch automaton transition Δ, and simultaneously describes the changes in the robot's physical position and logical stage after the robot performs an action.

[0027] R high The reward function is used to characterize the immediate reward that the robot can obtain by performing an action at its current physical location;

[0028] Hierarchical learning and planning steps: A hierarchical reinforcement learning framework is used for path planning. A high-level task decision module determines the logical order of travel to different task regions, while a low-level path planning module determines the physical path to a specific task region. Specifically, the low-level path planning module, based on a generalized Markov decision process M, uses reinforcement learning algorithms to pre-learn or online learn the sub-policy π for each option o. o The robot can reach a specific task area using this strategy; the high-level task decision module uses a product semi-Markov decision process M Θ Above, based on the current feasible subtask set A and the dynamic high-level reward function R. high Learning and decision-making are performed; at each decision point, the higher-level module performs learning and decision-making based on the current product state. Select option o from option set O and execute it, utilizing the sub-policy π of the underlying module. o Upon reaching the corresponding sub-objective, the robot triggers an update of the feasible subtask set A and a high-level reward calculation for each sub-objective reached, repeating this process until the linear time-series logic formula is completed. All tasks described.

[0029] Furthermore, it also includes a task definition step: describing a composite task with logical constraints between subtasks using a linear temporal logic formula. This formula consists of an atomic proposition set AP, several Boolean operators, and temporal operators. The Boolean operators include: AND... express and Both tasks need to be completed; or Indicates completion Any task will suffice; not This indicates that completion is prohibited. The task involves using the final temporal operator α, which represents the robot's requirement to reach a specific task area within a finite time. Linear temporal logic formulas transform ambiguous natural language instructions into precise mathematical objects that can be automatically verified and reasoned about by the computer, providing a rigorous input specification for subsequent task analysis and strategy synthesis.

[0030] Furthermore, the robot path planning method based on partial order analysis and reinforcement learning is characterized in that the partial order analysis method is used to analyze the Butch automaton B, from... The process involves decomposing all subtasks and extracting the logical constraints between them. A partial order analysis is then performed on the finite running trajectory of the automaton. The order of the subtasks in the trajectory is changed. If the automaton can still accept the subtasks, then there is no requirement for the order between the two subtasks.

[0031] Furthermore, the robot path planning method based on partial order analysis and reinforcement learning is characterized in that the partial order boundary function is:

[0032]

[0033] The symbolically feasible subtask set The feasible subtask set represents the set of subtasks that start from subtask ω and point to according to their sequence; if there is no subtask that has a sequence relationship with subtask ω, then the set is empty.

[0034] Furthermore, the robot path planning method based on partial order analysis and reinforcement learning is characterized by the dynamic high-level reward function R. high for:

[0035] R high (s,q,o)=R o (s)+R p ;

[0036] Where R o (s) is an in-option reward negatively correlated with path length, to incentivize the robot to find the shortest path, R p The reward is based on a set of feasible subtasks, and its assignment rule is as follows:

[0037]

[0038] Where r p >0 represents a positive reward, r n =0 represents a neutral reward. When the robot completes an option associated with a task in A, and this task conforms to the logical constraints, the robot receives a positive reward for completing the correct subtask.

[0039] Furthermore, the robot path planning method based on partial order analysis and reinforcement learning is characterized in that the high-level task decision module uses the Q-learning algorithm to learn the high-level policy, and its Q-value function update formula is:

[0040]

[0041] in The current product state of the robot, 0 < μ ≤ 1 is the learning rate, and γ is the discount factor. To determine the product state reached after executing option o, the higher-level policy makes a decision by selecting the option that maximizes the Q value.

[0042] Furthermore, the robot path planning method based on partial order analysis and reinforcement learning is characterized in that the sub-policy π learned by the underlying path planning module is... o The termination condition is when the robot reaches a state that makes the proposition label σ of the corresponding subtask ω true.

[0043] This invention also proposes a robot path planning system based on partial order analysis and reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot path planning method based on partial order analysis and reinforcement learning.

[0044] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot path planning method based on partial order analysis and reinforcement learning.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot path planning method based on partial order analysis and reinforcement learning.

[0046] Compared with the prior art, the significant advantages of this invention are:

[0047] 1) The partial order analysis method is used to decompose the overall task into subtasks and associate them with options, which solves the problem that traditional hierarchical reinforcement learning relies on manual division and formulation of subtasks and improves the interpretability of the algorithm.

[0048] 2) This paper introduces a novel approach that combines partial order analysis with a hierarchical reinforcement learning framework using a partial order boundary function. The partial order boundary function dynamically updates the logically consistent task set at the current time step, constructing a reward function to guide the high-level task decision-making module. This reward function is highly dense and easily learned by the robot. It can improve the sample efficiency of the high-level task decision-making module even in situations with complex tasks and a rapidly expanding state space. Attached Figure Description

[0049] Figure 1 This is an algorithm flowchart of a robot path planning method based on partial order analysis and reinforcement learning proposed in this invention;

[0050] Figure 2 This is a pseudocode algorithm diagram of a robot path planning method based on partial order analysis and reinforcement learning proposed in this invention;

[0051] Figure 3 This is a comparison table of simulation test data for a robot path planning method based on partial order analysis and reinforcement learning proposed in this invention; Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] Reference Figure 1 and Figure 2 As an embodiment of the present invention, a robot path planning method based on partial order analysis and reinforcement learning is provided, characterized by comprising the following steps:

[0054] Step 1, Environment Modeling Step: Construct a generalized Markov decision process M describing the gridded environment, wherein the generalized Markov decision process M = (S, A, s0, P, AP, L, R), where:

[0055] S is a finite set of states, where each state corresponds to a physical location of the robot, usually in two-dimensional or three-dimensional discrete spatial coordinates.

[0056] A is a finite set of actions, each of which controls the robot's movement to adjacent physical locations, including basic actions such as moving up, down, left, right, or remaining stationary;

[0057] s0∈S is the robot's initial state, corresponding to the robot's physical position at the start time;

[0058] P:S×A→S is a deterministic state transition function that describes the change in the robot's physical position after it performs an action;

[0059] AP is a set of atomic propositions, where atomic propositions declare whether the robot is located in a specific task area. Each element α∈AP in the atomic proposition set corresponds one-to-one with a predefined task area in the map, representing the task of the robot "arriving" or "visiting" a specific area. An atomic proposition is a Boolean variable, and it is true if and only if the robot is in the physical position state corresponding to that area. Atomic propositions corresponding to other physical positions outside the task area are considered empty elements.

[0060] Tag function L:S→2 AP Assign an atomic proposition that is true at each physical location state s;

[0061] R:S×A→R is the reward function, which describes the immediate reward that the robot can obtain by performing an action at the current physical position. The immediate reward includes the cost of negative constants such as time and energy, as well as the normal reward for completing the task.

[0062] The constructed generalized Markov decision process M will serve as... Figure 2 The input to the main program reflects the interaction process between the robot and the workspace, and is used for training and learning the low-level path planning module, as well as for input to the hierarchical decision model construction steps.

[0063] Step 2, Task Definition Step: A composite task with logical constraints between subtasks is described using a linear temporal logic formula. This formula consists of an atomic proposition set AP, several Boolean operators, and temporal operators. The Boolean operators include: AND... express and Both tasks need to be completed; or Indicates completion Any task will suffice; not This indicates that completion is prohibited. The task involves using the final temporal operator α, which represents the robot's goal of reaching a specific task area within a finite time. Linear temporal logic formulas transform ambiguous natural language instructions into precise mathematical objects that can be automatically verified and reasoned about by the computer. This corresponds to the robot's path planning objective and provides strictly unambiguous input for subsequent task parsing steps.

[0064] Step 3, Task parsing steps: Receiving the task definition step using linear sequential logic formulas. The described complex task, constructing the corresponding The Butch automaton B, wherein the Butch automaton B = (Q, q0, Δ, q f ),in:

[0065] Q is the set of automaton states, where each state q∈Q corresponds to the achievement of a sub-goal of the task, representing an abstract logical stage when a certain part of the linear sequential logic formula is satisfied, rather than the specific physical position of the robot;

[0066] q0 is the initial state, corresponding to the logical phase at which the task begins at the initial moment;

[0067] Δ is a deterministic state transition function, which defines how the task logic stage should change when the robot reaches a specific task area under the current abstract task stage q. The state transition function encodes the task progress logic specified by the linear sequential logic formula.

[0068] q f The "accept" state corresponds to the logical stage where all linear sequential logic tasks have been completed.

[0069] Furthermore, the Butch automaton B was analyzed using a partial order analysis method, from... The process involves decomposing the data into all subtasks and extracting the logical constraints between them to obtain a partially ordered set. in This is a set of subtasks, where each subtask corresponds to a specific task area that the robot needs to reach; subtask ω l It consists of an atomic proposition label σ and a unique index l. The set representing the sequential relationship between subtasks; the partial order set obtained from task parsing. This will serve as input to the dynamic task set and reward construction steps, and is also Figure 2 The initial input of the main program;

[0070] Furthermore, the partial order analysis method is used to analyze the Butch automaton B, from... The process involves decomposing all subtasks and extracting the logical constraints between them. A partial order analysis is then performed on the finite running trajectory of the automaton. The order of the subtasks in the trajectory is changed. If the automaton can still accept the subtasks, then there is no requirement for the order between the two subtasks.

[0071] Step 4, Dynamic Task Set and Reward Construction Steps: Based on the partially ordered set Construct and initialize a feasible subtask set A; the feasible subtask set represents the set of subtasks that satisfy the linear temporal logic constraints at the current time. Each subtask in the set represents a specific region that the robot still needs to visit, and the preceding tasks of these subtasks have all been completed by the robot; the robot completing the subtasks in the set will not violate the linear temporal logic constraints; the initial value of the feasible subtask set A is the task corresponding to all subtasks without preceding constraints at the initial time. The initialization of the feasible subtask set A corresponds to... Figure 2 Line 9 of the main program;

[0072] Construct the partial order boundary function Pgf:Ω×2 Ω →2 Ω This is used to dynamically update the feasible subtask set A after the robot completes each subtask. If the subtask is in the current feasible subtask set, it is removed from A according to the partial order boundary function, and subsequent tasks that require the subtask as a prerequisite are added to the feasible subtask set A. If the subtask is not in the current feasible subtask set, the feasible subtask set A remains unchanged. Based on the updated feasible subtask set A, a dynamic high-level reward function R is constructed. high ;exist Figure 2 The dynamic high-level reward function R in lines 18 and 19 of the main program high The accuracy depends on the feasible subtask set A. After each reward is given, the feasible subtask set A needs to be updated according to the partial order boundary function.

[0073] Furthermore, the partial order boundary function mentioned in line 19 of the main program is:

[0074]

[0075] Among the symbols This represents the set of subtasks that originate from subtask ω and are linked according to their sequence.

[0076] Furthermore, the dynamic high-level reward function R described in line 18 of the main program... high for:

[0077] R high (s,q,o)=R o (s)+R p ;

[0078] Where R o (s) is an in-option reward negatively correlated with path length, to incentivize the robot to find the shortest path, R p The reward is based on a set of feasible subtasks, and its assignment rule is as follows:

[0079]

[0080] Where r p >0 represents a positive reward, r n =0 represents a neutral reward. When the robot completes an option associated with a task in A, and this task conforms to the logical constraints, the robot receives a positive reward for completing the correct subtask. At this time, the feasible subtask set needs to be updated through the partial order boundary function. The positive reward for completing the correct subtask will accelerate the convergence speed of the Q-learning algorithm training in line 20 of the main program.

[0081] Step 5, Hierarchical decision model construction steps: Multiply the generalized Markov decision process M with the Butch automaton B to construct the product semi-Markov decision process M. Θ =(S×Q,A,AP,O,L,P×Δ,R) high ),in:

[0082] The state space is S×Q. Each state consists of two parts: a generalized Markov decision process state representing the physical location of the robot, and a Butch automaton state representing the logical stage in which the robot completes a linear sequential logic task.

[0083] A is a finite set of actions, each action used to control the robot's movement to adjacent physical locations, including basic actions such as moving up, down, left, right, or remaining stationary;

[0084] AP is a set of atomic propositions, in which atomic propositions are used to declare whether the robot is located in a specific task area. Each element α∈AP in the set of atomic propositions corresponds one-to-one with a predefined task area in the map, and is used to represent the task of the robot "arriving" or "visiting" a specific area.

[0085] O is the sub-task set The exported option set, where each option o∈O is associated with a subtask ω. l This is associated with and corresponds to a sub-strategy π learned by the underlying module for reaching the target region of the task. o ;

[0086] Tag function L:S→2 AP Assign an atomic proposition that is true at each physical location;

[0087] P×Δ is the state transition function, which consists of the low-level generalized Markov decision process transition function P and the Butch automaton transition Δ, and simultaneously describes the changes in the robot's physical position and logical stage after the robot performs an action.

[0088] R high The reward function is used to characterize the immediate reward that the robot can obtain by performing an action at its current physical location;

[0089] The constructed product semi-Markov decision process can characterize how the robot makes task decisions. The robot will need to decide which task area to go to first according to the logical constraints between sub-tasks. The product semi-Markov decision process will serve as the basis for the high-level task decision module in the hierarchical learning and planning steps.

[0090] Step 6, Hierarchical Learning and Planning: A hierarchical reinforcement learning framework is used for path planning, where: Figure 2Lines 1-7 of the main program are the underlying path planning module. This module uses reinforcement learning algorithms to pre-learn or learn online the sub-policy π for each option o on the generalized Markov decision process M. o The robot can reach different task areas by using different options and strategies; Figure 2 Lines 8-23 of the main program, the high-level task decision module is in the product semi-Markov decision process M Θ Above, based on the current feasible subtask set A and the dynamic high-level reward function R. high Learning and decision-making are performed; at each decision point, the higher-level module performs learning and decision-making based on the current product state. Select option o from option set O and execute it, utilizing the sub-policy π of the underlying module. o Reach the corresponding sub-target; Figure 2 Lines 18 and 19 of the main program represent the robot reaching a sub-goal, triggering an update of the feasible subtask set A and a high-level reward calculation, and this cycle continues until the linear time-series logic formula is completed. The robot describes all tasks. A high-level task decision module determines the logical order of travel to different task areas, while a low-level path planning module determines the physical path to a specific task area. Combining the strategies of both modules yields a feasible path capable of completing the sequential logical task.

[0091] Furthermore, Figure 2 In line 20 of the main program, the high-level task decision module uses the Q-learning algorithm to learn the high-level strategy. Its Q-value function update formula is:

[0092]

[0093] in The current product state of the robot, 0 < μ ≤ 1 is the learning rate, and γ is the discount factor. To determine the product state reached after executing option o, Q converges after multiple rounds of robot training iterations. The higher-level policy decides the next task region to proceed by selecting the option that maximizes the Q value.

[0094] Figure 2 The sub-strategy π learned by the underlying path planning module, as described in line 5 of the main program. o The termination condition is when the robot reaches a state that makes the proposition label σ of the corresponding subtask ω true.

[0095] This invention also proposes a robot path planning system based on partial order analysis and reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot path planning method based on partial order analysis and reinforcement learning.

[0096] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot path planning method based on partial order analysis and reinforcement learning.

[0097] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot path planning method based on partial order analysis and reinforcement learning.

[0098] Example

[0099] To verify the effectiveness of the present invention, the following experimental design was conducted.

[0100] This embodiment was verified through simulation experiments in a 40×40 two-dimensional rasterized environment, and the standard option-based hierarchical reinforcement learning algorithm was implemented using Python. Figure 3 (Blue curve in the middle) Logical option framework algorithm ( Figure 3 The green curve in the middle) and the path planning algorithm based on partial order analysis and reinforcement learning of this invention ( Figure 3 (Middle red curve).

[0101] Figure 3 The average statistical data for each of the three algorithms after 100 simulation trials is shown. The horizontal axis represents the number of rounds, and the vertical axis represents the normalized average reward. Figure 3 The robot task in this paper is a strictly sequential task, requiring it to patrol and visit eight locations—regions A, B, C, D, E, F, G, and H—in a strict order. If the robot disrupts the visiting order, the task is considered incomplete. It can be seen that under large-scale complex tasks and MDP scales, the proposed path planning algorithm based on partial order analysis and reinforcement learning converges faster and has a higher average reward. The logical option framework algorithm and the option-based hierarchical reinforcement learning algorithm converge more slowly and are prone to converging to suboptimal paths. This is because the algorithm proposed in this invention makes fuller use of the knowledge given by linear temporal logic; whenever the robot makes a decision that conforms to the logical constraints, it receives an additional positive reward, which can greatly accelerate the training speed of the algorithm.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot path planning method based on partial order analysis and reinforcement learning, characterized in that, Includes the following steps: Environmental modeling steps: Construct a generalized Markov decision process M describing the gridded environment, wherein the generalized Markov decision process M = (S, A, s0, P, AP, L, R), where: S is a finite set of states, where each state corresponds to a physical position of the robot and is a two-dimensional or three-dimensional discrete spatial coordinate. A is a finite set of actions, each of which controls the robot's movement to adjacent physical locations, including moving up, down, left, right, or remaining stationary; s0∈S is the robot's initial state, corresponding to the robot's physical position at the start time; P:S×A→S is a deterministic state transition function that describes the change in the robot's physical position after it performs an action; AP is a set of atomic propositions used to declare whether the robot is located in a specific task area. Each element α∈AP corresponds one-to-one with a predefined task area in the map, indicating that the task of the robot reaching or visiting a specific area is true if and only if the robot is in the physical location state corresponding to that area. This atomic proposition is considered to indicate that the robot has completed the corresponding sub-task of visiting the specific area. Atomic propositions corresponding to other physical locations outside the task area are considered as empty elements. L:S→2 AP The label function assigns an atomic proposition that is true at each physical location state s; R:S×A→R is the reward function, used to characterize the immediate reward that the robot can obtain by performing an action at the current physical position; Task parsing steps: Receiving linear sequential logic formula The described complex task, constructing the corresponding The Butch automaton B, wherein the Butch automaton B = (Q, q0, Δ, q f ),in: Q is the set of states of the automaton. Each state q∈Q corresponds to the achievement of a sub-goal of the task. It represents an abstract logical stage when a certain part of the linear sequential logic formula is satisfied, rather than the specific physical position of the robot. q0 is the initial state, corresponding to the logical phase at which the task begins at the initial moment; Δ is a deterministic state transition function, which defines how the task logic stage should change when the robot reaches a specific task region under the current abstract task stage q. q f The "accept" state corresponds to the logical stage where all linear sequential logic tasks have been completed. Furthermore, the Butch automaton B was analyzed using a partial order analysis method, from... The process involves decomposing the data into all subtasks and extracting the logical constraints between them to obtain a partially ordered set. in For a task set, each subtask ω l The robot needs to reach a specific task area, which consists of a proposition label σ and an index l. This represents a set of partial order relations between task events; Dynamic task set and reward construction steps: Construct and initialize a feasible subtask set A, whose initial value is the task corresponding to all subtasks without pre-constraints at the initial time; construct the partial order boundary function Pgf: Ω×2 Ω →2 Ω This is used to dynamically update the feasible subtask set A after the robot completes each subtask. If the subtask is in the current feasible subtask set, it is removed from A according to the partial order boundary function, and subsequent tasks that require the subtask as a prerequisite are added to the feasible subtask set A. If the subtask is not in the current feasible subtask set, the feasible subtask set remains unchanged. Based on the updated feasible subtask set A, a dynamic high-level reward function R is constructed. high ; The hierarchical decision model construction steps are as follows: The generalized Markov decision process M is multiplied by the Butch automaton B to construct a product semi-Markov decision process M. Θ =(S×Q,A,AP,O,L,P×Δ,R) high ),in: The state space is S×Q. Each state consists of two parts: a generalized Markov decision process state representing the physical location of the robot, and a Butch automaton state representing the logical stage in which the robot completes a linear sequential logic task. O is the sub-task set The exported option set, where each option o∈O is associated with a subtask ω. l This is associated with and corresponds to a sub-strategy π learned by the underlying module for reaching the target region of the task. o ; P×Δ is the state transition function, which consists of the low-level generalized Markov decision process transition function P and the Butch automaton transition Δ, and simultaneously describes the changes in the physical position and logical stage after the robot performs an action. Hierarchical learning and planning steps: A hierarchical reinforcement learning framework is used for path planning. A high-level task decision module determines the logical order of travel to different task regions, while a low-level path planning module determines the physical path to a specific task region. Specifically, the low-level path planning module, based on a generalized Markov decision process M, uses reinforcement learning algorithms to pre-learn or online learn the sub-policy π for each option o. o The robot can reach different task areas by using different option strategies; the high-level task decision module uses the product semi-Markov decision process M Θ Above, based on the current feasible subtask set A and the dynamic high-level reward function R. high Learning and decision-making are performed; at each decision point, the higher-level module performs learning and decision-making based on the current product state. Select option o from the corresponding options in option set O and execute it, utilizing the sub-strategy π of the underlying module. o Upon reaching the corresponding sub-objective, the robot triggers an update of the feasible subtask set A and a high-level reward calculation for each sub-objective reached, repeating this process until the linear time-series logic formula is completed. All tasks described.

2. The robot path planning method based on partial order analysis and reinforcement learning according to claim 1, characterized in that, The linear temporal logic formula consists of an atomic proposition set AP, several Boolean operators, and temporal operators, wherein: the Boolean operators include: AND... express and Both tasks need to be completed; or Indicates completion Any task will suffice; not This indicates that completion is prohibited. The task; the timing operator is the final ◇α, which means that the robot must reach a specific task area within a finite time.

3. The robot path planning method based on partial order analysis and reinforcement learning according to claim 1, characterized in that, The partial order analysis method is used to analyze the Butch automaton B, from... The process involves decomposing all subtasks and extracting the logical constraints between them. A partial order analysis is then performed on the finite running trajectory of the automaton. The order of the subtasks in the trajectory is changed. If the automaton can still accept the subtasks, then there is no requirement for the order between the two subtasks.

4. The robot path planning method based on partial order analysis and reinforcement learning according to claim 1, characterized in that, The partial order boundary function is: The symbolically feasible subtask set The feasible subtask set represents the set of subtasks that start from subtask ω and point to according to their sequence; if there is no subtask that has a sequence relationship with subtask ω, then the set is empty.

5. The robot path planning method based on partial order analysis and reinforcement learning according to claim 1, characterized in that, The dynamic high-level reward function R high for: R high (s,q,o)=R o (s)+R p ; Where R o (s) is an in-option reward negatively correlated with path length, to incentivize the robot to find the shortest path, R p The reward is based on a set of feasible subtasks, and its assignment rule is as follows: Where r p >0 represents a positive reward, r n =0 represents a neutral reward. When the robot completes an option associated with a task in A, and this task conforms to the logical constraints, the robot receives a positive reward for completing the correct subtask.

6. The robot path planning method based on partial order analysis and reinforcement learning according to claim 1, characterized in that, The high-level task decision-making module uses the Q-learning algorithm to learn high-level strategies, and its Q-value function update formula is as follows: in The current product state of the robot, 0 < μ ≤ 1 is the learning rate, and γ is the discount factor. To determine the product state reached after executing option o, the higher-level policy makes a decision by selecting the option that maximizes the Q value.

7. The robot path planning method based on partial order analysis and reinforcement learning according to claim 1, characterized in that, The sub-strategy π learned by the underlying path planning module o The termination condition is when the robot reaches a state that makes the proposition label σ of the corresponding subtask ω true.

8. A robot path planning system based on partial order analysis and reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the robot path planning method based on partial order analysis and reinforcement learning as described in any one of claims 1-7.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the robot path planning method based on partial order analysis and reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, wherein when executed by a processor, the computer program implements the robot path planning method based on partial order analysis and reinforcement learning as described in any one of claims 1-7.

Images