Robot hierarchical learning method based on linear temporal logic and parallel waypoint planning

Abstract

The application discloses a kind of robot hierarchical learning method based on linear temporal logic and parallel waypoint planning, including mapping expert demonstration trajectory into action primitive sequence by inverse dynamics model, automatically dividing subtask boundary and determining waypoint quantity benchmark, while converting primitive sequence into linear temporal logic formula and giving text logical constraint;The text semantic features of logical formula are extracted in advance using pre-training encoder;Based on the current observation state, use parallel neural network architecture to synchronously predict multiple target waypoints in the whole process, eliminate time sequence dependence and solve strategy error accumulation;Based on the current state, text semantic features and target waypoints, output hybrid action instruction to drive robot to execute tasks.The application realizes the automatic decomposition from demonstration to structured task, guides policy learning through formal logical constraint, and significantly improves the robustness and sample efficiency of long-time sequence operation through parallel waypoint planning, suitable for application in object stacking, precision assembly and other tasks.

Description

Technical Field

[0001] This invention relates to the field of robot operation and reinforcement learning technology, specifically to a robot hierarchical learning method based on linear temporal logic and parallel waypoint planning. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, enabling robots to autonomously complete long-horizon manipulation tasks in complex environments has become a core research goal in the field of robotics. Long-horizontal manipulation tasks (such as precision industrial assembly, multi-step object stacking, and tidying up in a home environment) are typically characterized by long task cycles, continuous and high-dimensional state spaces, and complex action sequences. These tasks not only require robots to possess low-level motion control capabilities but also high-level logical reasoning and task planning abilities, posing a significant challenge to existing technologies.

[0003] Currently, the mainstream solution to this problem is Hierarchical Reinforcement Learning (HRL). HRL alleviates the curse of dimensionality and sparse reward problems to some extent by decomposing complex, long-sequential tasks into several easily tractable subtasks or action primitives. However, in practical applications, existing hierarchical learning frameworks still face the following three major technical bottlenecks:

[0004] First, subtask partitioning lacks autonomy. Existing hierarchical frameworks often rely heavily on prior human knowledge when decomposing tasks. Researchers typically need to predefine the boundaries, number, and specific objectives of each subtask. While this "hard-coded" decomposition method is effective in specific experimental environments, it lacks generalization ability. When the task environment or object pose changes, the pre-defined partitioning logic often fails. Furthermore, how to automatically extract action primitive sequences that conform to task logic from limited human expert demonstrations, and achieve automatic mapping from demonstrations to structured task descriptions, remains a pressing challenge in current technology. Existing techniques, such as time-interval-based segmentation methods, struggle to capture the true semantic boundaries of the task, while reward-change-based methods are limited by the scarcity of sparse reward signals.

[0005] Second, there is the compounding error effect during the execution of long-term tasks. This is a fatal flaw in traditional hierarchical planning schemes. Current subgoal or waypoint planning often uses a serial generation logic, that is, predicting the first waypoint based on the current state, then predicting the second waypoint based on the expected state of the first waypoint, and so on to form a waypoint chain. Under this chain-dependent structure, the robustness of the system is extremely low. In actual operation, due to perceived noise or control disturbances, the robot often produces a small pose deviation when performing the first stage action. This deviation will produce a "domino effect" in the serial structure, causing the subsequently generated waypoints to be further and further away from the real target point. Especially in high-precision contact tasks such as "peg-in-hole", even a millimeter-level error in the initial alignment stage can cause the subsequent insertion action to fail completely. Existing technology lacks a planning mechanism that can decouple the temporal dependencies between stages and improve the fault tolerance of long-term decision-making.

[0006] Third, the semantic sparsity and logical consistency verification of task instructions are lacking. In human-computer interaction, humans typically provide macroscopic task instructions (such as "complete the nut assembly"), while robots need to translate these into low-level motion trajectories. Existing methods are relatively general in semantic alignment and lack fine-grained constraints on intermediate task processes. When tasks become extremely complex, numerical reward functions alone are often insufficient to guide the policy to learn the correct logical sequence. For example, a robot might attempt to perform a movement action without grasping an object, or attempt a placement operation before reaching the target location. Although Linear Temporal Logic (LTL) has been introduced to provide logical constraints, how to efficiently integrate LTL with action primitives in reinforcement learning and achieve automated feature encoding in large-scale state spaces remains a weak point in current research.

[0007] In conclusion, developing a robot learning framework that can autonomously divide task structures, provide fine-grained semantic guidance, and mitigate the effects of accumulated errors is of significant theoretical and practical value for achieving truly intelligent and robust automated robot operations. Summary of the Invention

[0008] To address the problems in existing technologies, such as subtask division relying on prior human knowledge, error accumulation due to sequential planning, and lack of semantic constraints, this invention provides a robot hierarchical learning method based on linear temporal logic and parallel waypoint planning. By automatically parsing the task structure from a single expert demonstration, using linear temporal logic to provide global constraints, and combining a parallel waypoint generation mechanism to alleviate error accumulation, this method significantly improves the exploration efficiency and execution robustness of robots performing long-sequential operation tasks.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A hierarchical learning method for robots based on linear temporal logic and parallel waypoint planning includes the following steps:

[0011] S10. Input the expert demonstration trajectory into the trajectory division module, use the inverse dynamics model to map the expert demonstration trajectory into an action primitive sequence, realize the automatic definition of the sub-task boundary, and convert the action primitive sequence into a linear temporal logic formula to give each sub-task a text logic constraint.

[0012] S20. The semantic encoding module uses a pre-trained encoder to extract the textual semantic features of the linear temporal logic formula and generate a semantic representation with temporal logic constraints.

[0013] S30. The parallel waypoint planning module, based on the current observation state, uses a parallel neural network architecture to synchronously predict multiple target waypoints throughout the process. The target waypoints correspond one-to-one with each sub-task stage in the action primitive sequence, thereby mitigating the accumulation of decision errors by eliminating temporal dependencies.

[0014] S40, the primitive execution module, based on the current state, text semantic features and target waypoints of the corresponding stage, outputs discrete action primitives and their continuous parameters to form mixed action instructions to drive the robot to perform operation tasks.

[0015] Specifically, step S10 includes the following process:

[0016] S11. A task-independent inverse dynamics model is trained through self-supervised training. The inverse dynamics model predicts the probability distribution of the corresponding action primitives based on the initial and final states of the trajectory segments of the expert demonstration trajectory.

[0017] S12. Using a dynamic programming algorithm, the expert demonstration trajectory is optimally segmented by maximizing the accumulated probability product, and the action primitive sequence is parsed out. The action primitive sequence is the optimal sequence composed of predefined action primitives, and the benchmark for the number of waypoints generated is determined accordingly.

[0018] S13. The parsed sequence of action primitives is transformed into a cooperative safe linear temporal logic formula based on temporal logic, and the remaining tasks are dynamically updated using an online advancement mechanism during execution.

[0019] Specifically, the objective function of the dynamic programming algorithm in step S12 is expressed as:

[0020]

[0021] In the above formula, f(i) represents the state-value function of dynamic programming, f(t) represents the historical optimal value of dynamic programming, and p represents the action primitive type. To adjust the parameter, s t and s i Let represent the start and end states of the i-th subtask segment, respectively. This represents the output of the inverse dynamics model (IDM).

[0022] Specifically, in step S20, the pre-trained encoder is a Transformer encoder, the linear temporal logic formula is input into the pre-trained encoder as a text sequence, and the output text semantic features are used as a global semantic feature vector.

[0023] Specifically, step S30 includes the following process:

[0024] S31. Extract the pose data of the robot and the target object from the observation information of the current observation state, and use it as the hidden state input based on the parallel neural network architecture of the gated recurrent unit.

[0025] S32. The parallel neural network architecture based on gated recurrent units outputs the mean and variance of N target waypoints simultaneously in parallel, where N matches the number of action primitives in the action primitive sequence; by coordinating the optimization of all waypoints at the global level, it provides a stable guidance signal for the underlying control.

[0026] Specifically, in step S32, the waypoint generation strategy is configured as a random strategy. The mean of waypoints The variance is output by a parallel neural network architecture based on gated recurrent units. It is obtained through a linear transformation of the hidden state. Here, s is the policy parameter, and s0 is the initial observation state. The linear time-series logic formula and the waypoint generation loss function are given. Represented as:

[0027]

[0028] In the above formula, E represents the expectation operation. This represents the waypoint generation strategy, where s is the current system state. This represents the weighted cumulative return.

[0029] Specifically, the primitive execution module in step S40 includes a primitive selection layer and a primitive parameter strategy layer. The primitive selection layer selects the most suitable discrete primitive k based on the current state, text semantic features, and the target waypoint at the current stage. Under the condition of selecting the discrete primitive k, the primitive parameter strategy layer predicts the continuous action parameters x required to execute the discrete primitive. Finally, the synthesized hybrid action h=(k,x) is output to interact with the environment.

[0030] Specifically, the primitive selection layer is configured as a strategy. , where s t Indicates the current state. Representing the semantic features of the text. Indicates the target waypoint at the current stage. Policy parameters are selected for primitives; the policy loss function of the primitive selection layer. Represented as:

[0031]

[0032] In the above formula, E represents the expectation operation. Indicates primitive selection strategy, The entropy coefficient chosen for the primitive. Represents a state-action value function.

[0033] Specifically, the primitive parameter strategy layer is configured as a strategy ,in For primitive parameters and policy parameters, the policy loss function of the primitive parameter policy layer. Represented as:

[0034]

[0035] In the above formula, Indicates primitive parameter strategy, is the entropy coefficient of the primitive parameter.

[0036] Furthermore, the method is based on a hierarchical reinforcement learning architecture and constructs a decision framework that integrates formal logic and parallel objectives. This decision framework includes a trajectory partitioning module, a semantic encoding module, a parallel waypoint planning module, and a primitive execution module. The trajectory partitioning module extracts the task structure from the demonstration, the semantic encoding module implements the formal representation and feature extraction of the task logic, the parallel waypoint planning module generates the full-process stage objectives in parallel, and the primitive execution module combines high-level semantics, mid-level waypoints, and low-level action states to form the final control command.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) This invention uses the inverse dynamics model (IDM) and dynamic programming to automatically and optimally parse the action primitive sequence and subtask boundary from a single expert demonstration, reducing the reliance on human prior knowledge and automatically converting the action primitive sequence into a linear temporal logic (LTL) formula. This provides interpretable and verifiable textual semantic constraints for each subtask stage, enhances the logical alignment between high-level task instructions and low-level execution, ensures logical consistency during task execution, avoids illegal action sequences, and significantly reduces the difficulty of robot exploration in long-term tasks.

[0039] (2) This invention designs a waypoint generation model based on a parallel gated loop unit (GRU) to synchronously output all staged target waypoints throughout the process, breaking the chain-like time-series dependency of traditional serial planning. This effectively alleviates the problem of decision error accumulation caused by perception and control errors, greatly enhances the interpretability and execution robustness of the decision chain, effectively improves the success rate of tasks such as precision assembly in high-precision contact tasks, and has a stronger tolerance to initial disturbances.

[0040] (3) The present invention adopts a hierarchical reinforcement learning architecture for end-to-end optimization, which enables semantic coding features to adaptively assist decision-making. In long-term operation tasks such as object stacking and precision assembly, the sample efficiency and convergence stability are significantly improved.

[0041] (4) The framework proposed in this invention has good generalization ability and can adapt to different task scenarios and object pose changes. It does not require redesigning the sub-task division logic for new environments, which significantly reduces the deployment cost and maintenance difficulty of the robot system. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall frame structure in an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram illustrating the working principle of the inverse dynamics model in an embodiment of the present invention. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0045] Example

[0046] like Figures 1 to 2As shown, this embodiment provides a robot hierarchical learning method based on linear temporal logic and parallel waypoint planning. Based on a hierarchical reinforcement learning architecture, a decision framework integrating formal logic and parallel objectives is constructed, comprising four modules: trajectory segmentation module, semantic encoding module, parallel waypoint planning module, and primitive execution module. The trajectory segmentation module extracts the task structure from the demonstration, the semantic encoding module realizes the formal representation and feature extraction of the task logic, the parallel waypoint planning module realizes the parallel generation of the entire process stage objectives, and the primitive execution module realizes the combination of high-level semantics, mid-level waypoints, and low-level action states to form the final control command.

[0047] This embodiment describes each module in detail by enabling the robot to autonomously complete a long-sequential task of "stacking red blocks on top of green blocks". This task typically requires the sequential execution of multiple sub-tasks such as "approaching the red block", "grabbing the red block", "moving on top of the green block", and "placing the red block", exhibiting typical long-sequential characteristics.

[0048] 1. Trajectory Division Module

[0049] First, collect some expert tracks demonstrated by human remote control operations, for example, 5 tracks, each expert demonstration track being a state-action sequence. State s includes the angles of each joint of the robotic arm, the pose of the end effector, and the pose of the target object (obtained through visual perception). Action a is the robot's low-level control command (such as joint velocity). A set of discrete action primitives P = {approach, grasp, move, place, release...} is predefined, and each primitive corresponds to a short-term and semantically clear subtask.

[0050] To automatically extract primitive sequences from the demonstration, a task-independent inverse dynamics model (IDM) is first trained. The IDM takes state pairs (s, s') as input and outputs the probability distribution of action primitive p. The IDM is implemented using a Multilayer Perceptron (MLP), and is self-supervised trained by maximizing the log-likelihood of the corresponding real primitives across all states in the training set. The training data is generated as follows: the demonstration trajectory is divided into segments by time windows (variable window length), and each segment is labeled with its corresponding primitive category (using a small number of manually labeled samples, existing datasets, or automatically generated pseudo-labels through clustering; in this embodiment, a small number of manually labeled samples are used to ensure accuracy). After training, the IDM can predict the most likely intermediate action primitive based on any two states.

[0051] Then, a dynamic programming algorithm is used to optimally segment a new demonstration trajectory. Define f(i) as the segment from the initial state to the final state s. i If the objective function of the optimal segmentation cumulative probability is a function, then the objective function expression of the dynamic programming is:

[0052]

[0053] In the above formula, f(i) represents the state-value function of dynamic programming, f(t) represents the historical optimal value of dynamic programming, and p represents the action primitive type. To adjust the parameter, a value of 0.95 can be selected to penalize excessively long segmentation segments and prevent primitive sequences from becoming too long. t and s i Let represent the start and end states of the i-th subtask segment, respectively. This represents the output of the inverse dynamics model (IDM).

[0054] The optimal primitive sequence can be obtained by backtracking the path to the maximum value of f(t). And the corresponding split points. In this task, the automatically parsed primitive sequence may be (close to red, grab red, move to green, place red), N=4, and this number N is the benchmark for subsequent parallel waypoint planning.

[0055] Simultaneously, the parsed primitive sequence is automatically converted into cooperatively secure linear-time-sequential logic (sc-LTL) formulas. The transformation rules are based on a predefined mapping table: for example, the primitive "approaching red" corresponds to logic atom π1, "grabbing red" corresponds to logic atom π2, "moving to green" corresponds to logic atom π3, and "placing red" corresponds to logic atom π4. The LTL formula for the entire task is then expressed as follows: This indicates that the events occur sequentially. An online propagation mechanism using sc-LTL is employed, dynamically updating the remaining task requirements based on current observations during execution. For example, after the robot completes π1, it will... Updated to This ensures real-time tracking of task logic.

[0056] 2. Semantic encoding module

[0057] sc-LTL formula As a text sequence, it is input into a pre-trained Transformer encoder (such as BERT-base) and outputs text semantic features. This serves as a global semantic feature vector. This feature encodes the logical intent of the high-level task and will be used as conditional input in subsequent modules. Because the Transformer encoder is pre-trained on a large-scale corpus, it can capture the semantic information of the LTL formula, and its parameters can be fine-tuned through gradient backpropagation during reinforcement learning, enabling the semantic representation to adaptively assist decision-making.

[0058] 3. Parallel waypoint planning module

[0059] This module aims to generate phased target waypoints throughout the entire process. Waypoints are defined as the states that the robot should reach at the end of each subtask, such as the desired pose of the end effector.

[0060] A parallel neural network architecture based on gated recurrent units (GRUs) is adopted, using the current observation state s0 and text semantic features. As input, N target waypoints are generated synchronously. Specifically, this parallel neural network architecture consists of a shared encoding layer and N parallel GRU units. The shared encoding layer combines the current observation state s0 with the text semantic features. After concatenation, the hidden state is mapped to the initial hidden state h0 through a fully connected layer. For the i-th branch (i=1,2,…,N), its GRU unit uses h0 as the initial state and outputs the waypoint mean w after several time steps of looping (single-step output is used in this embodiment). i and variance Therefore, the waypoint generation strategy is modeled as a Gaussian-distributed stochastic strategy. , These are strategy parameters. All target waypoints. They are generated synchronously, with no temporal dependencies between them.

[0061] During training, the waypoint generation strategy is optimized by maximizing the expected cumulative reward. In the reinforcement learning task, the reward corresponding to a target waypoint w is defined as... It measures the cumulative reward earned by performing a complete mission guided by the target waypoint. The loss function of the waypoint generation strategy is expressed as:

[0062]

[0063] In the above formula, E represents the expectation operation. This represents the waypoint generation strategy, where s is the current system state. This represents the weighted cumulative return. This loss function encourages the generation of waypoints that lead to high returns.

[0064] 4. Primitive Execution Module

[0065] Obtain semantic text features and the target waypoint w at the current stage i Subsequently, the primitive execution module adopts a hierarchical strategy to output hybrid actions, including a primitive selection layer and a primitive parameter strategy layer. The primitive selection layer selects the most suitable discrete primitive k based on the current state, text semantic features, and the target waypoint at the current stage. Under the condition of selecting the discrete primitive k, the primitive parameter strategy layer predicts the continuous action parameters x (such as movement distance, grasping force, etc.) required to execute the discrete primitive. Finally, the hybrid action h=(k,x) is synthesized and output to interact with the environment.

[0066] Primitive selection layer configured as strategy , in the current state s t Textual semantic features and the current target waypoint As input, the primitives are processed through a fully connected layer and a softmax layer, and the output is the probability distribution of the primitive categories. Select policy parameters for primitives. The policy loss function of the primitive selection layer. Represented as:

[0067]

[0068] In the above formula, E represents the expectation operation. Indicates primitive selection strategy, The entropy coefficient chosen for the primitive. Represents a state-action value function.

[0069] Primitive parameter strategy layer configuration as policy Given a discrete primitive k, output the Gaussian distribution mean and variance of the continuous parameters required by the primitive. For primitive parameters and policy parameters, the policy loss function of the primitive parameter policy layer. Represented as:

[0070]

[0071] In the above formula, Indicates primitive parameter strategy, is the entropy coefficient of the primitive parameter.

[0072] The entire framework is jointly trained using the SAC (Soft Actor-Critic) algorithm. The Critic network is updated based on actual interaction data, while the Actor network (including waypoint generation, primitive selection, and primitive parameters) is updated using the aforementioned loss function. The parameters of the semantic encoder are fine-tuned through policy gradient backpropagation, enabling the semantic representation to better guide target waypoint generation and primitive execution.

[0073] 5. Experimental setup and results

[0074] This embodiment establishes a "block stacking" task within the RoboSuite physics simulation environment. The robot utilizes a Pandas robotic arm, and the environment contains two cubes, one red and one green. The reward function is set as follows: +1 for successfully grasping a red cube, +1 for successfully placing and stacking a cube, no reward for other steps, and termination with a -1 penalty if a cube falls or the task fails. Training was performed using this method, with 5 random seed experiments and 2 million steps per experiment. For comparison, a traditional cascaded hierarchical reinforcement learning method (such as Maple) was used as the baseline.

[0075] The experimental results are as follows: After 1 million training steps, the average success rate of this method stabilized at over 85%, eventually reaching 92%; while the baseline method only achieved a success rate of 65% after 1 million steps, and the training process showed significant fluctuations. In terms of sample efficiency, the sample size required for this method to reach an 80% success rate is approximately 900,000 steps, a reduction of about 40% compared to the baseline. Furthermore, by introducing parallel waypoint planning, this method is more robust to initial pose perturbations: when the initial position is randomly shifted by ±5cm, the success rate of this method still maintains 78%, while the baseline method drops to 35%. This indicates that parallel waypoint planning effectively mitigates the error accumulation effect.

[0076] In summary, this embodiment verifies the effectiveness of the method of the present invention, realizes the automatic learning of task structure from expert demonstrations, and significantly improves the execution success rate and learning efficiency of long-sequence operation tasks through LTL semantic constraints and parallel waypoint planning.

[0077] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any changes made based on the design principles of the present invention, or any non-creative modifications made thereon, shall fall within the scope of protection of the present invention.

Claims

1. A hierarchical learning method for robots based on linear sequential logic and parallel waypoint planning, characterized in that, Includes the following steps: S10. Input the expert demonstration trajectory into the trajectory division module, use the inverse dynamics model to map the expert demonstration trajectory into an action primitive sequence, realize the automatic definition of the sub-task boundary, and convert the action primitive sequence into a linear temporal logic formula to give each sub-task a text logic constraint. S20. The semantic encoding module uses a pre-trained encoder to extract the textual semantic features of the linear temporal logic formula and generate a semantic representation with temporal logic constraints. S30. The parallel waypoint planning module, based on the current observation state, uses a parallel neural network architecture to synchronously predict multiple target waypoints throughout the process. The target waypoints correspond one-to-one with each sub-task stage in the action primitive sequence, thereby mitigating the accumulation of decision errors by eliminating temporal dependencies. S40, the primitive execution module, based on the current state, text semantic features and target waypoints of the corresponding stage, outputs discrete action primitives and their continuous parameters to form mixed action instructions to drive the robot to perform operation tasks.

2. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to claim 1, characterized in that, The process of step S10 includes: S11. A task-independent inverse dynamics model is trained through self-supervised training. The inverse dynamics model predicts the probability distribution of the corresponding action primitives based on the initial and final states of the trajectory segments of the expert demonstration trajectory. S12. Using a dynamic programming algorithm, the expert demonstration trajectory is optimally segmented by maximizing the accumulated probability product, and the action primitive sequence is parsed out. The action primitive sequence is the optimal sequence composed of predefined action primitives, and the benchmark for the number of waypoints generated is determined accordingly. S13. The parsed sequence of action primitives is transformed into a cooperative safe linear temporal logic formula based on temporal logic, and the remaining tasks are dynamically updated using an online advancement mechanism during execution.

3. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to claim 2, characterized in that, The objective function of the dynamic programming algorithm in step S12 is expressed as follows: In the above formula, f(i) represents the state-value function of dynamic programming, f(t) represents the historical optimal value of dynamic programming, and p represents the action primitive type. To adjust the parameter, s t and s i Let represent the start and end states of the i-th subtask segment, respectively. This represents the output of the inverse dynamics model (IDM).

4. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to claim 1, characterized in that, In step S20, the pre-trained encoder uses a Transformer encoder. The linear temporal logic formula is used as the text sequence input into the pre-trained encoder, and the output text semantic features are used as the global semantic feature vector.

5. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to claim 1, characterized in that, The process of step S30 includes: S31. Extract the pose data of the robot and the target object from the observation information of the current observation state, and use it as the hidden state input based on the parallel neural network architecture of the gated recurrent unit. S32. The parallel neural network architecture based on gated recurrent units outputs the mean and variance of N target waypoints simultaneously in parallel, where N matches the number of action primitives in the action primitive sequence; by coordinating the optimization of all waypoints at the global level, it provides a stable guidance signal for the underlying control.

6. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to claim 5, characterized in that, In step S32, the waypoint generation strategy is configured as a random strategy. The mean of waypoints The variance is output by a parallel neural network architecture based on gated recurrent units. It is obtained through a linear transformation of the hidden state. Here, s is the policy parameter, and s0 is the initial observation state. The linear time-series logic formula and the waypoint generation loss function are given. Represented as: In the above formula, E represents the expectation operation. This represents the waypoint generation strategy, where s is the current system state. This represents the weighted cumulative return.

7. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to any one of claims 1 to 6, characterized in that, In step S40, the primitive execution module includes a primitive selection layer and a primitive parameter strategy layer. The primitive selection layer selects the most suitable discrete primitive k based on the current state, text semantic features, and the target waypoint at the current stage. Under the condition of selecting the discrete primitive k, the primitive parameter strategy layer predicts the continuous action parameters x required to execute the discrete primitive. Finally, the synthesized hybrid action h=(k,x) is output to interact with the environment.

8. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to claim 7, characterized in that, The primitive selection layer is configured as a strategy. , where s t Indicates the current state. Representing the semantic features of the text. Indicates the target waypoint at the current stage. Policy parameters are selected for primitives; the policy loss function of the primitive selection layer. Represented as: In the above formula, E represents the expectation operation. Indicates primitive selection strategy, The entropy coefficient chosen for the primitive. Represents a state-action value function.

9. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to claim 7, characterized in that, The primitive parameter strategy layer is configured as a strategy. ,in For primitive parameters and policy parameters, the policy loss function of the primitive parameter policy layer. Represented as: In the above formula, Indicates primitive parameter strategy, is the entropy coefficient of the primitive parameter.

10. The robot hierarchical learning method based on linear temporal logic and parallel waypoint planning according to claim 7, characterized in that, The proposed method is based on a hierarchical reinforcement learning architecture, which constructs a decision framework that integrates formal logic and parallel objectives. This decision framework includes a trajectory segmentation module, a semantic encoding module, a parallel waypoint planning module, and a primitive execution module. The trajectory segmentation module extracts the task structure from the demonstration, the semantic encoding module implements the formal representation and feature extraction of the task logic, the parallel waypoint planning module generates the objectives of the entire process in parallel, and the primitive execution module combines high-level semantics, mid-level waypoints, and low-level action states to form the final control command.

Images