A robot manipulation method based on a visual language model

Abstract

This invention discloses a robot manipulation method based on a visual language model, comprising: acquiring multimodal information; performing cross-modal understanding based on the visual language model to obtain visual language semantic representations characterizing the task objective, the manipulated object, and the scene state, and fusing them with the robot's own state to obtain a multimodal conditional representation; inputting the multimodal conditional representation and the stage semantic information into a hierarchical hybrid expert-enhanced diffusion strategy model, and generating strategy features that match the current manipulation task stage and current manipulation requirements through expert selection or weighting at the task-level expert layer and action-level expert layer; generating continuous action segments oriented towards future time windows based on the strategy features; and performing rolling execution and replanning to drive the robotic arm to complete the manipulation task guided by the task language instructions. This invention effectively improves the stage modeling capability and action generation quality in long-term robotic arm manipulation.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically to a robot manipulation method based on a visual language model. Background Technology

[0002] With the continuous development of intelligent manufacturing, smart warehousing, and flexible assembly technologies, robots, especially robotic arms, are gradually evolving from repetitive execution units in traditional fixed workstations into autonomous manipulation platforms for open environments. Addressing the issue of insufficient autonomous manipulation capabilities of robotic arms in complex environments, the rapid development of large-scale modeling and embodied intelligence research in recent years has provided new technological pathways for robot control. Existing research shows that large-scale model-driven embodied intelligence is propelling robots from a separate "perception-planning-control" architecture towards a unified semantic understanding and decision generation approach. Particularly in robot manipulation scenarios, the joint modeling of vision, language, and action is gradually becoming an important trend in solving complex tasks. Against this backdrop, robot systems are beginning to possess the ability to extract semantic information from image observations and natural language commands, and further generate executable actions, laying the foundation for autonomous decision-making in complex scenarios.

[0003] In existing technologies, significant progress has been made in the research of visual language models and visual language action models to address the shortcomings in task semantic understanding and scene constraint representation. Works such as QwenGrasp and Bi-VLA attempt to combine large visual language models with grasping decisions for target-oriented grasping and dual-arm dexterity manipulation. Researches such as CodeasPolicies, PromptBook, and RobotGPT further explore the feasibility of using large language models for robot task decomposition, skill generation, and manipulation decisions. Meanwhile, cross-modal pre-trained models such as CLIP and BLIP-2 provide the technical foundation for unified image-language representation, while works such as Instruct2Act, Zero-shot Autonomous RobotManipulation, and VoxPoser drive the evolution of robots from "target recognition" to "understanding constraints and executing tasks." Overall, language-guided robot manipulation is gradually evolving from single-target recognition or single-step action control into a multimodal decision-making problem oriented towards complex task chains.

[0004] To address the challenge of motion generation in high-dimensional continuous motion spaces, diffusion-based strategies offer a novel modeling paradigm for continuous robot control. DiffusionPolicy enhances multimodal representation in complex continuous control problems by progressively denoising the motion distribution. Building upon this, works such as RT-1, RT-2, and RT-H continue to advance integrated vision-language-motion modeling and improve robot adaptability to complex tasks through hierarchical motion structures and network knowledge transfer. Octo further leverages large-scale, diverse data to train general robot policies, improving the model's generalization ability across platforms and tasks. OpenVLA emphasizes the efficient fine-tuning capabilities of open-source vision-language-motion models on new tasks. RDT-1B demonstrates the application potential of diffusion-based foundational models in complex dual-arm manipulation. In summary, existing technologies primarily focus on single-stage manipulation, short-term motion prediction, or the construction of general foundational models, lacking explicit modeling of stage semantics in long-term tasks, improvement of stage switching stability, and enhancement of execution-phase self-recovery capabilities. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned shortcomings and propose a robot manipulation method based on a visual language model that can effectively improve the stage modeling capability, motion generation quality and system execution stability in long-term robotic arm manipulation.

[0006] The present invention provides a robot manipulation method based on a visual language model, comprising the following steps:

[0007] Step 1, Multimodal Information Acquisition: The multimodal information includes task language instructions, scene visual observations, and robot body state; the robot is a robotic arm;

[0008] Step 2, Visual Language Conditional Understanding: Based on the visual language model, cross-modal understanding is performed on the task language instructions and scene visual observations to obtain visual language semantic representations that characterize the task objectives, operation objects, and scene states;

[0009] Step 3, Ontology State Fusion: The visual language semantic representation is fused with the robot ontology state to represent a multimodal conditional representation of the current task semantics, environmental state, and robotic arm motion state;

[0010] Step 4, Task Stage Determination: Based on the multimodal conditional representation, determine the current task stage of the manipulation task and obtain the semantic information of the current stage of the manipulation task.

[0011] Step 5, hierarchical expert strategy generation: Input the multimodal conditional representation and the stage semantic information into the hierarchical hybrid expert-enhanced diffusion strategy model, and perform expert selection or expert weighting through the task-level expert layer and action-level expert layer in the hierarchical hybrid expert-enhanced diffusion strategy model to generate strategy features that match the current manipulation task stage and the current manipulation requirements.

[0012] Step 6, motion segment generation: Generate continuous motion segments oriented towards future time windows based on the strategy features. The continuous motion segments are used to control the execution actions in the end effector, joints, and end effector of the robotic arm.

[0013] Step 7, Rolling Execution and Replanning: Execute the actions in the continuous action segments in a rolling time-domain manner, and update the multimodal conditional representation based on the scene visual observation and robot body state after execution; when a change in task stage, execution deviation, target state change or action stagnation is detected, regenerate the continuous action segments based on the updated multimodal conditional representation to drive the robotic arm to complete the manipulation task guided by the task language instructions.

[0014] The above-mentioned robot manipulation method based on visual language model, wherein in step 5, the hierarchical hybrid expert-enhanced diffusion strategy model: on the diffusion robot strategy backbone network, a hierarchical hybrid expert structure is constructed, including a task-level expert layer for explicitly modeling the current manipulation task stage and an action-level expert layer for adjusting the action generation feature transformation under stage condition constraints.

[0015] The above-mentioned robot manipulation method based on a visual language model, wherein the hierarchical hybrid expert structure of the diffusion strategy model with hierarchical hybrid expert enhancement employs conditional routing computation, and the sparse activation form corresponding to any input feature x is:

[0016]

[0017] in, Indicates the first A network of experts, This indicates the corresponding route weight. The set of activated experts is given as input to feature x; the routing weights are obtained through a linear mapping and Function calculation:

[0018]

[0019] in, This is the routing weight matrix. Temperature parameters used to adjust the sharpness of route distribution.

[0020] In the aforementioned robot manipulation method based on a visual language model, the training of the hierarchical hybrid expert-enhanced diffusion strategy model incorporates a load balancing auxiliary loss term during model training to suppress expert collapse and balance expert invocation. :

[0021]

[0022] in, Indicates the first Average route importance of an expert in a batch This indicates the percentage of the expert actually selected in the batch. The weights for load balancing items.

[0023] The above-mentioned robot manipulation method based on a visual language model includes the following specific construction and optimization steps for the hierarchical hybrid expert-enhanced diffusion strategy model:

[0024] S1: The multimodal conditional representation obtained using multimodal input encoding As input to the task-level expert layer, high-level stage features are output through adaptive aggregation by the task-level expert layer. :

[0025]

[0026] in, Indicates the first A task-level expert, Indicates the corresponding first Routing weights for task-level experts For The set of activated experts is the input;

[0027] Will Input the stage classification head and calculate the posterior probability distribution of the semantic stage the task is in at the current time. :

[0028]

[0029] in, Indicates time Corresponding stage variables, The parameters for the stage classification header are used; the cross-entropy loss function is employed. Supervised training is performed on the stage prediction results, using the cross-entropy loss function. for:

[0030]

[0031] in, Indicates the actual stage label;

[0032] S2: The action-level expert layer in the stage variables Under high-level semantic constraints, parameterized specialized modeling is performed on heterogeneous action modes such as approach, grasp, transfer, alignment and placement; the action-level expert layer is embedded through an equivalent implementation method.

[0033] In the aforementioned robot manipulation method based on a visual language model, the action-level expert layer is embedded through an equivalent implementation: action-level experts are introduced into the feedforward neural network module of the diffusion denoising network in the hierarchical hybrid expert-enhanced diffusion strategy model, and the enhanced feedforward mapping... Represented as:

[0034]

[0035] in, It is stage embedding. This represents the routing input after concatenating or fusing stage conditions with the current features. Indicates the first A top-level action expert, Indicates the corresponding first Routing weights for each action-level expert Indicates The set of activated experts is the input.

[0036] The aforementioned robot manipulation method based on a visual language model, wherein the action-level expert layer is embedded through an equivalent implementation: an action-level expert is introduced at the final action output head position of the diffusion-based policy backbone network in the hierarchical hybrid expert-enhanced diffusion policy model, and the action fragment output under stage conditions is represented as follows:

[0037]

[0038] in, The final output representation of the denoising network backbone, Indicates the first A top-level action expert. by The set of experts that are activated for input.

[0039] The aforementioned robot manipulation method based on a visual language model, wherein in step 7, the rolling execution involves organizing the entire long-term manipulation process into a macro-based structure during the policy deployment and reasoning phase. The closed-loop rolling iterative process, which uses stage judgment, action generation, local execution, and feedback update as the basic unit, includes the following steps:

[0040] S1: In the At the start of each closed-loop macrostep, an internal state representation is constructed based on the latest environmental feedback, observations, language, and states. Based on this, the task-level expert layer first re-evaluates the current task stage, and its stage distribution... It can be represented as:

[0041]

[0042] The stage decision for the current macrostep is further derived from the stage distribution:

[0043]

[0044] in, express Corresponding stage variables, Indicates the first The current stage corresponding to each macro step This indicates a task-level hybrid expert network;

[0045] S2: After the stage discrimination is completed, based on the current observation Language instructions as well as The diffusion strategy backbone and action-level expert layer of the layered hybrid expert-enhanced diffusion strategy model jointly generate a length of [length missing]. Future action clips :

[0046]

[0047] in, This represents a phased conditional policy network; it employs a rolling time-domain policy, which is then issued and executed. The front of the middle Step action, the length of which is less than or equal to H, that is Once completed, it immediately retrieves environmental feedback and proceeds to the next macrostep to prevent the accumulation of early local action deviations.

[0048] Compared with existing technologies, this invention has significant advantages. Based on a diffusion-based robot strategy, and unifying visual observation, language commands, and robot state condition representation, it introduces a two-layer structure of task-level experts and action-level experts. This decouples the two questions of "what stage is currently in" and "how to generate actions in the current stage" into a single model, thereby reducing mutual interference between action modes at different stages in a unified parameter space. Furthermore, addressing the issue of error propagation during long-duration task execution, this paper introduces closed-loop execution and local replanning mechanisms in the inference phase. Through short-time domain action rolling execution and feedback correction, it improves the continuous task progression capability and execution robustness.

[0049] In summary, the present invention has the following advantages:

[0050] (1) To address the issues of strong stage coupling and easy overlap of action patterns in long-term manipulation tasks, a layered hybrid expert-enhanced diffusion-type manipulation framework is proposed. The decoupled modeling of stage discrimination and action generation is achieved through a two-layer structure of task-level experts and action-level experts.

[0051] (2) To address the issues of easy spread of local deviations and insufficient ability to continuously advance tasks during execution, a closed-loop execution and local replanning mechanism is designed. The robustness of execution is enhanced through phase re-judgment, short-term action rolling execution, and feedback correction.

[0052] The following specific embodiments further illustrate the beneficial effects of the present invention. Attached Figure Description

[0053] Figure 1 This is a flowchart of the present invention;

[0054] Figure 2 This is a diagram of the hierarchical hybrid expert-enhanced diffusion strategy model architecture in an embodiment of the present invention;

[0055] Figure 3 This is a flowchart of the rolling execution and replanning process in an embodiment of the present invention. Detailed Implementation

[0056] The following detailed description, in conjunction with the accompanying drawings and preferred embodiments, outlines the specific implementation methods, features, and effects of a robot manipulation method based on a visual language model proposed in accordance with the present invention.

[0057] See Figure 1 This invention discloses a robot manipulation method based on a visual language model, wherein the method includes the following steps:

[0058] Step 1: At time Real-time acquisition of multimodal system input information for the robot, including scene visual observation. Task language instructions and the robot's body state .

[0059] Step 2: Perform unified conditional modeling on the input information of the multimodal system to construct the time-space model. Multimodal conditional input vector :

[0060]

[0061] Among them, the robot's body state This includes joint angles, joint velocities, end effector pose, and gripper opening / closing states; the action output of the strategy is modeled from time [time]. Starting to look towards the future continuous action segments of steps This is used to explicitly maintain the temporal consistency of local motion trends.

[0062]

[0063] in, This is the output of continuous actions at the corresponding time points. This represents the length of the action segment.

[0064] Step 3: On the diffusion robot policy backbone network, construct a top-down two-layer hierarchical hybrid expert (MoE) structure, including a task-level expert layer for explicitly modeling the current task stage, and an action-level expert layer for adjusting the action generation feature transformation under stage condition constraints (e.g., ...). Figure 2 The hierarchical hybrid expert structure employs conditional routing computation, and the sparse activation form corresponding to any input feature x is:

[0065]

[0066] in Indicates the first A network of experts, This indicates the corresponding route weight. The set of experts selected for the top-(k); the routing weights are determined through a linear mapping and Function calculation:

[0067]

[0068] in, This is the routing weight matrix. Temperature parameters used to adjust the sharpness of route distribution;

[0069] To suppress expert collapse and balance expert calls, a load balancing auxiliary loss term is introduced during model training. :

[0070]

[0071] in The table represents the first Average route importance of an expert in a batch This indicates the percentage of the expert actually selected in the batch. The weights are for the load balancing items. The specific construction and optimization steps of the hierarchical hybrid expert-enhanced diffusion strategy model include:

[0072] Step 3.1: Utilize the unified conditional representation obtained from multimodal input encoding As input to the task-level expert layer, it is adaptively processed through the task-level expert network.

[0073] High-level stage characteristics should be aggregated and output. :

[0074]

[0075] in, Indicates the first A task-level expert, This indicates the corresponding route weight. The set of experts that are activated within the task-level experts;

[0076] Will Input the stage classification head and calculate the posterior probability distribution of the semantic stage the task is in at the current time. :

[0077]

[0078] in, Indicates time Corresponding stage variables, The parameters for the stage classification header are used; the cross-entropy loss function is employed. Supervised training based on the stage prediction results:

[0079]

[0080] in, The label represents the real stage and is used to transform implicit manipulation processes into stable explicit semantic boundaries.

[0081] Step 3.2: The action-level expert layer specifies the variables in the stage. Under high-level semantic constraints, parameterized specialized modeling is performed on heterogeneous action modes such as approach, grasp, transfer, alignment, or placement; the action-level expert layer is embedded through one of the following two equivalent implementation methods:

[0082] Method 1: Introduce an action-level expert into the feedforward neural network (FFN) module of the diffuse denoising network. The enhanced feedforward mapping is represented as follows:

[0083]

[0084] in It is stage embedding. This represents the routing input after concatenating or fusing stage conditions with the current features. Indicates the first A top-level action expert;

[0085] Method 2: Introduce an action-level expert at the final action output head of the diffusion-based policy backbone network. The action fragment output under stage conditions is represented as follows:

[0086]

[0087] in, The final output representation of the denoising network backbone, Indicates the first A top-tier action expert.

[0088] Step 4: See Figure 3 In the strategy deployment reasoning phase, the entire long-term manipulation process is organized into a macro-based approach. The basic unit is a closed-loop rolling iterative process of "stage judgment - action generation - local execution - feedback update":

[0089] Step 4.1: In the first... At the start of each closed-loop macrostep, an internal state representation is constructed based on the latest environmental feedback, observations, language, and states. Based on this, the task-level experts first re-evaluate the current task phase, and its phase distribution can be represented as follows:

[0090]

[0091] The stage decision for the current macrostep is further derived from the stage distribution:

[0092]

[0093] in, Indicates the first Each macrostep corresponds to the current stage.

[0094] Step 4.2: After the stage discrimination is completed, the system bases its decisions on the current observations. Language instructions and stage conditions The length is generated jointly by the diffusion strategy backbone and action-level experts. Future action clips:

[0095]

[0096] A rolling time-domain strategy is adopted, which only issues and executes the data. The front of the middle Step movement ( After execution, it immediately retrieves environmental feedback and proceeds to the next macrostep to prevent the accumulation of early local action deviations.

[0097] Step 4.3: After each round of local action execution, comprehensively evaluate the duration of the current stage, the magnitude of target state changes, and observed anomalies; when a local failure or stagnation state caused by grasping offset, target loss of contact, etc. is detected, the task is not terminated, but a local replanning mechanism is triggered, which includes: re-performing online stage discrimination to correct stage switching errors, rerouting action-level experts, and resampling the alternative control trajectory using multimodal distribution of diffusion strategy under the current stage conditions.

[0098] Step 5: Drive the robotic arm to execute the optimal motion segment to achieve long-term continuous manipulation of the robotic arm.

[0099] Performance Analysis:

[0100] To verify the effectiveness of the present invention, an experiment was first conducted on the RoboTwin simulation platform to compare the offline action quality of the original strategy RDT and the method H-MOE-RDT provided in this application. Under the same number of training steps, the experimental results are shown in Table 1.

[0101] Table 1. Comparison of Offline Motion Quality between RDT and H-MOE-RDT

[0102] RDT-1B 0.0433 0.1921 H-MOE-RDT 0.0015 0.1041

[0103] As can be seen from Table 1, the mean square error of H-MOE-RDT decreased from 0.0433 to 0.0015, and the Euclidean distance error decreased from 0.1921 to 0.1041, indicating that the proposed method can significantly improve the model's ability to fit the target action distribution.

[0104] Meanwhile, in order to further verify the online task evaluation effect of the hierarchical hybrid expert enhanced diffusion strategy (H-MOE-RDT) and the access closed-loop replanning mechanism (ReAct-lite) described in this application, an online comparative test was conducted on the material grabbing and placement task of the RoboTwin platform. The evaluation results of each method are shown in Table 2.

[0105] Table 2 Comparison of Online Evaluation and Closed-Loop Mechanism Results

[0106] RDT-1B 11% 255 H-MOE-RDT 17% 217 - H-MOE-RDT + ReAct-lite 20% 232 13

[0107] As shown in Table 2, compared to the original RDT baseline model, the H-MOE-RDT proposed in this invention improves the task success rate from 11% to 17%, while reducing the average number of steps from 255 to 217, a reduction of approximately 14.9%. This indicates that the hierarchical hybrid expert structure can effectively improve the stability and execution efficiency of the strategy in long-term tasks. Furthermore, after incorporating the ReAct-lite closed-loop execution mechanism, the task success rate further increases from 17% to 20%, with an average of 13 replanning iterations. This demonstrates the superior robustness of the closed-loop feedback correction mechanism in handling online accumulated errors in long-term tasks.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments without departing from the technical essence of the present invention shall still fall within the scope of the present invention.

Claims

1. A robot manipulation method based on a visual language model, characterized in that, Includes the following steps: Step 1, Multimodal Information Acquisition: The multimodal information includes task language instructions, scene visual observations, and robot body state; the robot is a robotic arm; Step 2, Visual Language Conditional Understanding: Based on the visual language model, cross-modal understanding is performed on the task language instructions and scene visual observations to obtain visual language semantic representations that characterize the task objectives, operation objects, and scene states; Step 3, Ontology State Fusion: The visual language semantic representation is fused with the robot ontology state to represent a multimodal conditional representation of the current task semantics, environmental state, and robotic arm motion state; Step 4, Task Stage Determination: Based on the multimodal conditional representation, determine the current task stage of the manipulation task and obtain the semantic information of the current stage of the manipulation task. Step 5, hierarchical expert strategy generation: Input the multimodal conditional representation and the stage semantic information into the hierarchical hybrid expert-enhanced diffusion strategy model, and perform expert selection or expert weighting through the task-level expert layer and action-level expert layer in the hierarchical hybrid expert-enhanced diffusion strategy model to generate strategy features that match the current manipulation task stage and the current manipulation requirements. Step 6, motion segment generation: Generate continuous motion segments oriented towards future time windows based on the strategy features. The continuous motion segments are used to control the execution actions in the end effector, joints, and end effector of the robotic arm. Step 7, Rolling Execution and Replanning: Execute the actions in the continuous action segments in a rolling time-domain manner, and update the multimodal condition representation based on the scene visual observation and robot body state after execution; when a change in task stage, execution deviation, target state change or action stagnation is detected, regenerate the continuous action segments based on the updated multimodal condition representation to drive the robotic arm to complete the manipulation task guided by the task language instructions.

2. The robot manipulation method based on a visual language model as described in claim 1, characterized in that, In step 5, the layered hybrid expert-enhanced diffusion strategy model is constructed on the diffusion robot strategy backbone network, including a task-level expert layer for explicitly modeling the current manipulation task stage and an action-level expert layer for adjusting the action generation feature transformation under stage condition constraints.

3. The robot manipulation method based on a visual language model as described in claim 2, characterized in that, The hierarchical hybrid expert structure of the diffusion strategy model enhanced by the hierarchical hybrid expert adopts conditional routing computation, and the sparse activation form corresponding to any input feature x is: in, Indicates the first A network of experts, This indicates the corresponding route weight. The set of activated experts is given as input to feature x; the routing weights are obtained through a linear mapping and Function calculation: in, This is the routing weight matrix. Temperature parameters used to adjust the sharpness of route distribution.

4. The robot manipulation method based on a visual language model as described in claim 3, characterized in that, To suppress expert collapse and balance expert calls during the training of the hierarchical hybrid expert-enhanced diffusion strategy model, a load balancing auxiliary loss term is introduced during model training. : in, Indicates the first Average route importance of an expert in a batch This indicates the percentage of the expert actually selected in the batch. The weights for load balancing items.

5. A robot manipulation method based on a visual language model as described in any one of claims 1-4, characterized in that, The specific construction and optimization steps of the hierarchical hybrid expert-enhanced diffusion strategy model include: S1: The multimodal conditional representation obtained by inputting multimodal information into the encoding. As input to the task-level expert layer, high-level stage features are output through adaptive aggregation by the task-level expert layer. : in, Indicates the first A task-level expert, Indicates the corresponding first Routing weights for task-level experts For The set of activated experts is the input; Will Input the stage classification head and calculate the posterior probability distribution of the semantic stage the task is in at the current time. : in, Indicates time Corresponding stage variables, The parameters for the stage classification header are used; the cross-entropy loss function is employed. Supervised training is performed on the stage prediction results, using the cross-entropy loss function. for: in, Indicates the actual stage label; S2: The action-level expert layer in the stage variables Under high-level semantic constraints, parameterized specialized modeling is performed on heterogeneous action modes such as approach, grasp, transfer, alignment or placement; the action-level expert layer is embedded through an equivalent implementation method.

6. The robot manipulation method based on a visual language model as described in claim 5, characterized in that, The action-level expert layer is embedded through an equivalent implementation: action-level experts are introduced into the feedforward neural network module of the diffusion denoising network in the hierarchical hybrid expert-enhanced diffusion strategy model, and the enhanced feedforward mapping... Represented as: in, It is stage embedding. This represents the routing input after concatenating or fusing stage conditions with the current features. Indicates the first A top-level action expert, Indicates the corresponding first Routing weights for each action-level expert Indicates The set of activated experts is the input.

7. The robot manipulation method based on a visual language model as described in claim 5, characterized in that, The action-level expert layer is embedded through an equivalent implementation: an action-level expert is introduced at the final action output head position of the diffusion policy backbone network in the hierarchical hybrid expert-enhanced diffusion policy model, and the action fragment output under stage conditions is... Represented as: in, The final output representation of the denoising network backbone, Indicates the first A top-level action expert. by The set of experts that are activated for input.

8. The robot manipulation method based on a visual language model as described in claim 1, characterized in that, In step 7, the rolling execution involves organizing the entire long-term manipulation process into macro-level components during the strategy deployment inference phase. The closed-loop rolling iterative process, which uses stage judgment, action generation, local execution, and feedback update as the basic unit, includes the following steps: S1: In the At the start of each closed-loop macrostep, an internal state representation is constructed based on the latest environmental feedback, observations, language, and states. Based on this, the task-level expert layer first re-evaluates the current task stage, and its stage distribution... It can be represented as: The stage decision for the current macrostep is further derived from the stage distribution: in, express Corresponding stage variables, Indicates the first The current stage corresponding to each macro step This indicates a task-level hybrid expert network; S2: After the stage discrimination is completed, based on the current observation Language instructions as well as The diffusion strategy backbone and action-level expert layer of the layered hybrid expert-enhanced diffusion strategy model jointly generate a length of [length missing]. Future action clips : in, This represents a phased conditional policy network; it employs a rolling time-domain policy, which is then issued and executed. The front of the middle Step action, the length of which is less than or equal to H, that is Once completed, it immediately retrieves environmental feedback and proceeds to the next macrostep to prevent the accumulation of early local action deviations.

Images