Lightweight visual language action system for robot operation

Abstract

The invention discloses a robot operation-oriented lightweight visual language action system, which comprises a training-free lightweight architecture, which is used for realizing a mobile-level CPU (Central Processing Unit) calculation requirement through the modular integration of FastSAM (Secure Assisted Module) segmentation and GPT-4V reasoning; the action step parameterization control framework decomposes a complex task into six basic action primitives. Through intelligent modular integration of pre-training components, the robot operation process is decomposed into a plurality of mutually decoupled functional modules such as visual perception, visual language reasoning, action planning and robot execution. In the visual perception stage, a lightweight zero sample segmentation model FastSAM is used to carry out target segmentation on a scene image, and mask information of a plurality of target objects in a scene is obtained; and in combination with depth image data, performing post-processing and screening on a target area obtained through segmentation, and extracting a space center position and attitude features of a target object, thereby realizing three-dimensional pose estimation of a scene object, and providing a reliable space perception basis for subsequent operation.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence in robots, specifically to the design of a lightweight Vision-Language-Action (VLA) robot system that can achieve efficient VLA deployment without model training or fine-tuning, enabling the robot to understand natural language commands and complete corresponding physical operations based on visual perception. Background Technology

[0002] With the continuous development of artificial intelligence, robotics, and multimodal perception and reasoning technologies, robot systems with natural language understanding and autonomous decision-making capabilities are gradually becoming an important development direction for performing complex tasks. Among them, the Vision-Language-Action (VLA) system, by combining visual perception, language understanding, and robot control, enables robots to autonomously complete target recognition, task planning, and action execution based on natural language instructions given by humans. It is considered a key technological path to improve the versatility and intelligence of robots.

[0003] However, existing VLA robot systems still face significant engineering and deployment bottlenecks in practical applications. One mainstream approach employs an end-to-end deep learning framework, unifying the modeling of visual input, linguistic input, and control strategies, and training on large-scale real or simulated data to achieve task generalization capabilities. Although this approach demonstrates strong performance in controlled experimental environments, it heavily relies on massive amounts of labeled data and high-performance graphics processors for training and inference, resulting in high computational resource consumption, high energy consumption, and high deployment costs, making it difficult to adapt to resource-constrained scenarios such as industrial sites, laboratories, and edge computing.

[0004] Furthermore, end-to-end VLA models typically tightly couple perception, reasoning, and control processes, leading to poor system interpretability and debugging difficulties. Once perception errors, target deviations, or action failures occur during execution, there is often a lack of effective error detection and recovery mechanisms, making it difficult to support robots in completing complex, long-term, multi-step operations. In real physical environments, environmental changes, sensing errors, and execution uncertainties are prevalent, further limiting the stability and practicality of existing VLA systems. Therefore, how to construct a VLA robot operation method with a clear structure, strong interpretability, robust execution, and error recovery capabilities without relying on large-scale training data and high-performance computing hardware has become a pressing technical problem in this field.

[0005] Traditional VLA systems primarily employ a task-centric design paradigm, devising specialized solutions for specific task types. However, complex robotic maneuvering tasks are essentially sequences of multiple atomic-level basic motor skills. For example, the complex task of "tidying a table" can be decomposed into an ordered combination of basic actions such as grasping, placing, and pushing. Therefore, shifting from a task-centric to a skill-centric design paradigm, decomposing complex tasks into parameterized atomic operations, not only improves the system's generalization ability but also achieves higher execution efficiency through skill reuse. This paradigm shift enables the system to handle an infinitely diverse range of task types using a finite combination of basic skills, significantly enhancing the system's adaptability and scalability.

[0006] In real-world applications, robot operations typically face challenges such as execution errors, limited computational resources, and a lack of adaptability during execution. Therefore, developing a vision-language-motion system capable of performing multi-step robot operations based on natural language commands, exhibiting high robustness and deployability in resource-constrained environments, without relying on large-scale training data and high-performance computing resources, remains a pressing technical challenge in this field. Summary of the Invention

[0007] To address the common problems of existing VLA robot systems, such as reliance on large-scale model training, high computational resource consumption, insufficient system reliability and error recovery mechanisms, and difficulty in deployment in resource-constrained environments, this invention proposes a training-free, lightweight VLA robot operation method and system. This method enables the robot to complete multi-step operation tasks based on natural language commands without requiring model training or fine-tuning, or configuring a graphics processor on the robot. It balances the system's versatility, robustness, and deployability. The entire system architecture includes: (1) a training-free lightweight architecture, achieving mobile-level CPU computing requirements through modular integration of FastSAM segmentation and GPT-4V inference; (2) a parameterized action step control framework, decomposing complex tasks into six basic action primitives to ensure reliable multi-step execution and independent error recovery mechanisms; and (3) a multi-layer coordinate transformation and deep visual fusion perception system for accurate three-dimensional spatial understanding and operational positioning.

[0008] This invention decomposes the complex robot operation process into multiple decoupled functional modules, such as visual perception, visual language reasoning, motion planning, and robot execution, through the intelligent modular integration of pre-trained components. Specifically, in the visual perception stage, this invention utilizes the lightweight zero-shot segmentation model FastSAM to segment scene images, obtaining mask information for multiple target objects in the scene. Combined with depth image data, the segmented target regions are post-processed and filtered to extract the spatial center position and pose features of the target objects, thereby achieving 3D pose estimation of scene objects and providing a reliable spatial perception foundation for subsequent operations.

[0009] To address the challenge of traditional visual models directly achieving semantic understanding, this invention introduces a visual language model to semantically parse natural language instructions and semantically binds the target objects mentioned in the language with target masks obtained during the visual perception stage. By inputting scene images with target labels into the visual language model, it maps the correspondence between language descriptions and visual targets, thereby achieving accurate localization of language instructions to specific objects without relying on target category training data. While visual language models are relatively weak in precise pixel localization, they exhibit relatively strong capabilities in angle reasoning and correction. Therefore, in object grasping posture planning, this invention employs a hybrid strategy of coarse estimation of the object's posture and refined inference of the grasping posture by the visual language model to plan the appropriate operating posture for the robotic arm. This approach is more accurate and stable than directly requiring the visual language model to output the appropriate grasping posture.

[0010] In terms of motion planning, this invention proposes a parameterized motion primitive planning module that decomposes complex robot operation tasks into multiple reusable atomic motion units. This module operates at the semantic level, converting natural language instructions through a two-stage decomposition process: (1) decomposing complex tasks into atomic subtasks, and (2) decomposing subtasks into parameterized motion primitives. The main feature of this module is that it utilizes the reasoning ability of a pre-trained model without fine-tuning, achieving zero-sample adaptation through structured prompting engineering. Each motion primitive contains target object information, spatial position parameters, end effector posture parameters, and interaction parameters. By combining and sequentially arranging atomic actions, robot operation flows of arbitrary complexity can be constructed. This approach avoids the complexity of traditional continuous trajectory planning, improves the versatility and interpretability of motion planning, and facilitates migration and application across different robot platforms.

[0011] To address the issue that robots are susceptible to sensor errors, environmental changes, and execution failures when performing multi-step tasks in real-world environments, this invention introduces a hierarchical error detection and recovery mechanism during the robot's execution phase. When a certain action primitive fails to execute, the system can automatically trigger local retries, parameter recalculation, scene re-perception, or overall action sequence replanning based on the failure type, thereby significantly improving the success rate and robustness of long-sequence, multi-step tasks.

[0012] The training-free vision-language-action robot operation method proposed in this invention can operate under CPU-only hardware conditions, significantly reducing the system's computing resource requirements and deployment costs. Compared with existing solutions that rely on end-to-end training or high-performance computing equipment, this invention maintains a high task success rate while improving the system's practicality and promotional value in industrial settings, edge computing, and resource-constrained environments. Attached Figure Description

[0013] Figure 1 This is the overall architecture diagram of the invention and the relationships between each module;

[0014] Figure 2 This is the process of planning action primitives that are parameterized from high to low levels in this invention;

[0015] Figure 3 This invention presents the task decomposition process of the lightweight VLA system and a live robot demonstration.

[0016] Figure 4 This is a demonstration of the VLA system proposed in this invention performing complex multi-step tasks. Detailed Implementation

[0017] Figure 1 This is an overall architecture diagram of a training-free vision-language-motion robot operating system provided by the present invention. The system includes a visual perception module, a visual language reasoning module, a motion planning module, and a robot execution module. The design focus of each module is computational efficiency and decoupling. The modules interact with each other through structured information. Under the condition that there is no need to train or fine-tune the model and the robot end only relies on the central processing unit, the robot can complete multi-step operation tasks based on natural language commands.

[0018] In the visual perception stage, the present invention first acquires a color image of the current scene using an RGB-D camera. With depth images Subsequently, on Median filtering is performed to remove noise (see Equation (1)), and combined with Alignment:

[0019]

[0020] In the formula This represents the filtered depth map, where k is the filter kernel; H and W represent the height and width pixels of the image, respectively. These steps ensure the quality of the input data and reduce the computational cost of subsequent processing. In the implementation, this corresponds to the warm-up frame and alignment filter in the function, minimizing the impact of noise.

[0021] The color image is input into the zero-shot object segmentation model FastSAM to perform classless segmentation of all separable objects in the scene, resulting in two-dimensional mask regions for multiple candidate objects. See equation (2).

[0022]

[0023] in These are pre-trained parameters. After segmentation, the mask is optimized by: 1) applying an area threshold filter to the mask region to remove noise regions with an area smaller than a preset threshold; and 2) combining the depth image to judge the depth consistency of pixels within the mask region and removing invalid masks with excessive depth dispersion. This achieves accurate object boundary extraction. The FastSAM method can achieve real-time performance on a CPU and assigns a unique target identifier number to each mask, preparing for subsequent pose calculation and visual language processing.

[0024] Then, the filtered scene images labeled with target numbers, along with the user's input natural language commands, are input into the visual language model. Based on the semantic content of the natural language, the visual language model performs semantic understanding of the targets in the scene and outputs the mapping relationship between the target objects mentioned in the language and their corresponding target numbers, thus completing a robust semantic binding between the natural language description and the visual perception results.

[0025] After completing the target semantic binding, this invention further estimates the 3D pose of the target object based on RGB-D information. Specifically, the depth information within the target mask region is back-projected into point cloud data, and principal component analysis (PCA) is performed on the point cloud to calculate the principal axis direction and spatial center position of the target object in the camera coordinate system. Combining the hand-eye calibration results between the camera and the robot base, the pose information of the target object is transformed into the robot base coordinate system to obtain the 3D pose representation of the target object in the robot coordinate system.

[0026]

[0027] in It is the pose of the target object relative to the robot base. This represents the conversion relationship between the camera and the base. Let be the coordinates of the target relative to the camera. In this invention, the coarse estimation of the object's posture in equation (3) and the optional grasping angle correction provided by VLM are used to optimize the posture of the robotic arm grasping the target object.

[0028] During the action planning phase, the VLM module performs semantic planning by breaking down complex tasks into executable action primitives. This module operates at the semantic level and is implemented through a two-phase process of atomic task decomposition and parameterized action primitive decomposition.

[0029] Firstly, during the high-level atomic task decomposition process, this invention employs a structured cue engineering strategy to guide VLM in semantic subtask reasoning. Equations (4) and (5) are cue structures that integrate multimodal inputs:

[0030]

[0031]

[0032] in For natural language input, For the target to be operated, The result of task decomposition. , These consist of action primitive rules and task templates. The suggested project framework has the following characteristics:

[0033] (1) Structured Semantic Decomposition: The system defines six semantic action primitives (APPROACH TARGET, INTERACT WITH TARGET, MODIFY TCP ORIENTATION, SET GRIPPER STATE, WAITDURATION, VERTICAL LIFT) as atomic building blocks for complex operation tasks. Each primitive is parameterized through spatial, kinematic, and interaction parameters, thereby achieving systematic task decomposition.

[0034] (2) Adaptive parameterization: The framework automatically determines interaction parameters using depth-based height calculation. Four interaction height strategies (ABOVE TABLE SURFACE, TARGET OBJECT MID HEIGHT, TARGET OBJECTTOP SURFACE, ABOVE TARGET OBJECT SURFACE) are defined to handle diverse operation scenarios and achieve millimeter-level fine-tuning through height offset parameters.

[0035] (3) Template guidance: Common operation patterns (grabbing and placing, twisting, pouring, opening containers, pressing) are encoded as template sequences, providing VLM with structured task decomposition guidance. This pattern-based approach ensures consistent and reliable motion planning across different task types.

[0036] (4) Constraint-aware planning: This prompt combines physical constraints and system limitations, including collision avoidance strategies, grasping state management, and motion planning considerations. This constraint-aware design ensures that the generated action sequence is semantically correct and physically feasible.

[0037] Figure 2 This demonstrates how advanced users can decompose commands into parameterized action primitives. The invention further maps the task decomposition results output by the visual language model into a sequence of parameterized action primitives. This module operates at the execution level, performing three key functions: (1) mapping semantic primitives to specific robot actions, (2) inserting collision avoidance actions, and (3) embedding error correction and reset actions. The system predefines several general-purpose atomic action units:

[0038]

[0039] in, : AT (APPROACH TARGET, INTERACT WITH TARGET, MODIFY TCP

[0040] The original action type in ORIENTATION, SET GRIPPER STATE, WAIT DURATION, VERTICAL LIFT; Define the target specifications of the interaction object (target coordinates, target object ID); : Precise spatial parameters for positioning (target pixel offset, interaction height type, height offset); : Kinematic parameters used for tool center point (TCP) orientation control (gripper rz angle); : Interaction parameters of the end effector behavior (fixture target state). In Equation (6), a can parameterize the action primitives, where each primitive represents a complete and independent execution unit with all the necessary spatial, kinematic and interaction specifications.

[0041] During the robot execution phase, the system sequentially parses the sequence of motion primitives and drives the robotic arm to complete the corresponding actions according to the set parameters. The mapping function operates on the structured JSON output of the VLM, parsing and validating each parameter to construct a complete motion specification. :

[0042]

[0043] The specific mapping process involves three key transformations: 1) Parameter parsing and verification. The JSON parameters of each action primitive are parsed and verified according to predefined constraints. The system extracts the target specification, spatial parameters, kinematic parameters, and interaction parameters to ensure that all required fields exist and are within the valid range; 2) Coordinate transformation. The image coordinates are converted into robot base coordinates through the hand-eye calibration matrix; 3) Height calculation. The interaction height is calculated based on the interaction height setting parameters, including object geometry and depth information. The final height can be obtained through equation (8):

[0044]

[0045] In the formula This indicates the interaction height setting parameter, which is derived from the depth map. This indicates fine-tuning the height; the parameters come from... Regularized parameter settings, This is the final operation height value.

[0046] During execution, the system monitors the end effector status and execution results of the robotic arm in real time. When gripping failure, target displacement, or abnormal pose is detected, the corresponding error handling strategy is automatically triggered. This invention designs a hierarchical error recovery mechanism to improve the overall success rate of multi-step tasks. This mechanism includes: 1) Action-level retry: When a single action primitive fails to execute, the action is re-executed without changing the task structure; 2) Parameter-level recalculation: When a change in target pose is detected, the target pose is recalculated and the action parameters are updated; 3) Perception-level refresh: When multiple retries fail, the scene image is reacquired and the complete visual perception and semantic binding process is executed; 4) Task-level replanning: When the above strategies fail to recover, the robot returns to a safe initial state and the task process is replanned.

[0047] In multi-step operation tasks, the robot can sequentially complete the grasping, placement, or interaction operations of multiple target objects based on the structured task planning results generated by the visual language model. After each action primitive is completed, the system can call the visual perception module again as needed to update the scene state, thereby avoiding task failure due to environmental changes or error accumulation.

[0048] This invention has been fully tested on a robotic platform and a standard CPU (Intel i7-12700K, 32GB RAM), with a response time of less than 10 seconds for each task. Notably, the VLA model of this invention runs without a GPU. Specifically, as... Figure 3As shown, several basic tasks validated the system's ability to handle different operational scenarios. In the tasks of "basic pick-up and place," "spatial relationship understanding," "obstacle avoidance during operation," "tool use for specific tasks," and "container operations for object placement," the system achieved success rates of 96.0%, 84.0%, 76.0%, 84.0%, and 88.0%, respectively, with an average success rate of 85.6%. Meanwhile, the CPU utilization rate was less than 15%, significantly lower than the 40% CPU utilization rate of mainstream models with GPU participation.

[0049] Figure 4 Three long task chains were demonstrated: (1) drawer opening, object placement, and closing operations; (2) bottle cap removal and liquid pouring, demonstrating precise manipulation and spatial control; and (3) surface cleaning using tools, illustrating tool manipulation and area coverage. Each sequence validated the system's ability to break down complex instructions into atomic operations and execute them in a high-precision sequence.

[0050] During the execution of the system described in this invention, the robot only needs to perform operations such as image processing, point cloud computing, motion parsing, and control command issuance, all of which can be run in real time on the central processing unit. The reasoning process of the visual language model can be completed through remote invocation, thereby significantly reducing the local hardware resource requirements.

[0051] Through the above embodiments, the present invention enables robots to accurately understand and execute natural language commands without requiring robot operation data training or local high-performance computing devices. This method maintains a high success rate and system robustness even in complex scenarios, multi-step tasks, and situations with execution uncertainties, making it suitable for intelligent robot operation applications in industrial settings and edge computing environments.

Claims

1. A lightweight visual language action system for robot operation, characterized in that, Through the intelligent modular integration of pre-trained components, the complex robot operation process is decomposed into functional modules that are decoupled from each other: visual perception module, visual language reasoning module, action primitive planning module, and robot execution module. The visual perception module uses the lightweight zero-shot segmentation model FastSAM to segment scene images and obtain mask information of multiple target objects in the scene. By combining depth image data, the segmented target region is post-processed and filtered to extract the spatial center position and pose features of the target object, thereby realizing the three-dimensional pose estimation of the scene object and providing a reliable spatial perception basis for subsequent operations. The visual language reasoning module performs semantic parsing on natural language instructions and semantically binds the target objects involved in the language with the target masks obtained in the visual perception stage. By inputting scene images with target labels into a visual language model, the model completes the mapping between language descriptions and visual targets, thereby achieving accurate localization of language instructions to specific objects without relying on target category training data. In terms of object grasping posture planning, a hybrid strategy of coarse estimation of the object's posture and refined inference of the grasping posture by the visual language model is adopted to plan the appropriate operating posture of the robot. The motion primitive planning module decomposes complex robot operation tasks into multiple reusable atomic motion units. The action primitive planning module operates at the semantic level, converting natural language instructions through a two-stage decomposition process: complex tasks are decomposed into atomic subtasks, and subtasks are decomposed into parameterized action primitives; each action primitive contains target object information, spatial position parameters, end effector posture parameters, and interaction parameters. By combining and arranging atomic actions in sequence, robot operation processes of any complexity can be constructed. The robot execution module introduces a hierarchical error detection and recovery mechanism; when a certain action primitive fails to execute, it can automatically trigger local retry, parameter recalculation, scene re-perception, or overall action sequence replanning according to the failure type.

2. The lightweight visual language action system for robot operation according to claim 1, characterized in that, The visual perception module first acquires a color image of the current scene from an RGB-D camera. With depth images ; then on Median filtering is used to remove noise, and... Alignment: ； In the formula This represents the filtered depth map, where k is the filter kernel; H and W represent the height and width pixels of the image, respectively. The color image is input into the zero-shot object segmentation model FastSAM to perform classless segmentation of all separable objects in the scene, resulting in two-dimensional mask regions for multiple candidate objects. ; ； in These are pre-trained parameters; after segmentation, the mask region is filtered by area threshold to remove noise regions with an area smaller than the preset threshold; combined with the depth image, the depth consistency of pixels in the mask region is judged, and invalid masks with excessive depth dispersion are removed to optimize the mask and achieve accurate object boundary extraction. Then, the scene images that have been filtered and labeled with target numbers, along with the natural language instructions input by the user, are input into the visual language model. The visual language model performs semantic understanding of the targets in the scene based on the semantic content of the natural language and outputs the mapping relationship between the target objects involved in the language and the corresponding target numbers, thereby completing the target semantic binding between the natural language description and the visual perception results.

3. A lightweight visual language action system for robot operation according to claim 2, characterized in that, After completing the target semantic binding, the 3D pose of the target object is estimated based on RGB-D information; the depth information within the target mask region is back-projected into point cloud data, and principal component analysis (PCA) is performed on the point cloud to calculate the principal axis direction and spatial center position of the target object in the camera coordinate system; combined with the hand-eye calibration results between the camera and the robot base, the pose information of the target object is transformed into the robot base coordinate system to obtain the 3D pose representation of the target object in the robot coordinate system. ； in It is the pose of the target object relative to the robot base. This represents the conversion relationship between the camera and the base. The coordinates of the target relative to the camera.

4. A lightweight visual language action system for robot operation according to claim 3, characterized in that, The action primitive planning module decomposes complex tasks into executable action primitives to perform semantic planning. Firstly, during the high-level atomic task decomposition process, a structured hint engineering strategy is adopted for semantic subtask reasoning: ； ； in For natural language input, For the target to be operated, The result of task decomposition. , These are action primitive rules and task templates, respectively.

5. A lightweight visual language action system for robot operation according to claim 4, characterized in that, It has the following characteristics: (1) Structured semantic decomposition: Six semantic action primitives are defined: APPROACHTARGET, INTERACTWITHTARGET, MODIFYTCPORIENTATION, SETGRIPPERSTATE, WAITDURATION, VERTICALLIFT, as atomic building blocks for complex operation tasks; each primitive is parameterized through spatial, kinematic and interaction parameters to achieve systematic task decomposition. (2) Adaptive parameterization: Automatically determine interaction parameters using depth-based height calculation; Define four interaction height strategies: ABOVETABLESURFACE, TARGETOBJECTMIDHEIGHT, TARGETOBJECTTOPSURFACE, ABOVETABLESARGETOBJECTSURFACE. (3) Template guidance: Common operation modes include grabbing and placing, twisting, pouring, opening containers, and pressing, which are encoded as template sequences to provide structured task decomposition guidance; (4) Constraint-aware planning: This prompt combines physical constraints and system limitations, including collision avoidance strategies, grasping state management and motion planning considerations.

6. A lightweight visual language action system for robot operation according to claim 5, characterized in that, The task decomposition results output by the visual language reasoning module are mapped to parameterized sequences of action primitives and run at the execution level: ； in, : in (APPROACHTARGET,INTERACTWITHTARGET,MODIFYTCP The primitive types of actions in ORIENTATION, SETGRIPPERSTATE, WAITDURATION, VERTICALLIFT; Define the target specifications of the interaction object, namely the target coordinates and the target object ID; The precise spatial parameters for positioning include target pixel offset, interaction height type, and height offset. : Kinematic parameters used for TCP direction control of the tool center point, i.e., gripper rz angle; : The interaction parameters of the end effector behavior, i.e. the fixture target state; a can parameterize the action primitives.

7. A lightweight visual language action system for robot operation according to claim 6, characterized in that, In the robot execution module, the sequence of action primitives is parsed sequentially, and the robotic arm is driven to complete the corresponding actions according to the set parameters; The mapping function operates on the structured JSON output, parsing and validating each parameter to build a complete action specification. : ； The specific mapping process involves three key transformations: 1) Parameter parsing and validation; the JSON parameters of each action primitive are parsed and validated according to predefined constraints; the system extracts target specifications, spatial parameters, kinematic parameters, and interaction parameters, ensuring that all required fields exist and are within valid ranges; 2) Coordinate transformation; image coordinates are converted to robot base coordinates using a hand-eye calibration matrix; 3) Height calculation; the interaction height is calculated based on the interaction height setting parameters, including object geometry and depth information; the final height is obtained using the following formula: ； In the formula This indicates the interaction height setting parameter, which is derived from the depth map. This indicates fine-tuning the height; the parameters come from... Regularized parameter settings, This is the final operation height value; During execution, the status of the robotic arm's end effector and the execution results are monitored in real time. When gripping failure, target offset, or abnormal pose is detected, the corresponding error handling strategy is automatically triggered. A hierarchical error recovery mechanism is designed to improve the overall success rate of multi-step tasks.

8. A lightweight visual language action system for robot operation according to claim 7, characterized in that, The graded error recovery mechanism includes: 1) Action-level retry: when a single action primitive fails to execute, the action is re-executed without changing the task structure; 2) Parameter-level recalculation: when a change in target pose is detected, the target pose is recalculated and the action parameters are updated; 3) Perception-level refresh: when multiple retries fail, the scene image is reacquired and the complete visual perception and semantic binding process is executed; 4) Task-level replanning: when the above strategies fail to recover, the robot returns to a safe initial state and the task process is replanned.

9. A lightweight visual language motion system for robot operation according to claim 7, characterized in that, In multi-step operation tasks, the robot sequentially completes the grasping, placement, or interaction operations of multiple target objects based on the structured task planning results generated by the visual language model. After each action primitive is completed, the visual perception module is called again to update the scene state as needed, thereby avoiding task failure due to environmental changes or error accumulation.

Images