Robot fine-grained skill learning and generalization method based on visual imitation learning

Abstract

The application provides a robot fine-grained skill learning and generalization (VLMimic) method based on visual imitation learning, which is a novel visual imitation learning framework driven by VLMs to learn generalizable robot skills from human demonstration videos; the VLMimic has a skill learner for knowledge extraction and a skill adapter for iterative skill refinement, and efficient skill acquisition and adaptation are realized; the application constructs an effective human-object interaction positioning algorithm, enhances the fine-grained action recognition capability, and proposes a hierarchical constraint representation for VLM reasoning to reduce information redundancy and promote comprehensive action understanding; the performance of the method of the application on RLBench exceeds that of other methods by more than 27%. In real-world operation tasks, VLMimic realizes more than 21% improvement in known environments and more than 34% improvement in unknown environments. In addition, VLMimic shows more than 37% improvement in long-term tasks.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a method for fine-grained skill learning and generalization in robots based on visual imitation learning. Background Technology

[0002] Current mainstream methods for visual imitation learning often face challenges when generalizing to unknown environments. Therefore, efficiently acquiring generalizable skills from limited video remains a significant challenge. An attractive prospect for addressing this challenge is leveraging large-scale pre-trained models that encapsulate prior knowledge from extensive data. Visual language models offer particularly promising tools in this regard, with emerging and rapidly growing conceptual understanding, common-sense knowledge, and reasoning abilities. However, current methods typically rely simply on visual language models to learn high-level plans and often depend on a set of predefined motion primitives. This reliance on individual skill acquisition is often considered a major bottleneck in the system due to the lack of large-scale robotic data. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method for fine-grained skill learning and generalization of robots based on visual imitation learning.

[0004] A fine-grained skill learning and generalization method for robots based on visual imitation learning includes:

[0005] Step 1: Video Analysis

[0006] S1.1 Overall task recognition, including: periodically extracting a series of image frames from human operation demonstration videos, and using VLMs to construct task descriptions and object information within the image frames;

[0007] S1.2 Video analysis, including: extracting interaction markers from human operation demonstration videos, capturing the interaction between two types of entities: hands and operation-related objects, and dividing the video into multiple sub-task segments based on the time period of the interaction between the two entities;

[0008] S1.3 Subtask identification, including: each subtask segment is divided into a grasping phase and a manipulation phase based on the interacting entities; VLMs analyze each subtask segment, generate subtask text descriptions, and identify the master object and slave objects; in each subtask, moving objects are defined as slave objects, and stationary target objects are defined as master objects; in the grasping phase, the robot performs a reaching and grasping action with the master object as the target, where the hand is defined as the slave object and the grasped object is designated as the master object; in the manipulation phase, the robot interacts with the master object using slave objects, where the master object is represented as the object being touched, and the slave object is represented as the object actively touching.

[0009] S1.4 Object-centric interaction extraction, including: obtaining the pose trajectory of the generated hand and the pose trajectory of the slave object in the current coordinate system relative to the main object;

[0010] Step 2: Skills learning methods with hierarchical representation, including:

[0011] S2.1 Extract the constraints of the crawling task;

[0012] S2.2 Extracting operational constraints;

[0013] S2.3, Construction of the knowledge base:

[0014] Establish a knowledge base to archive high-level planning and low-level skill insights, and use key-value pairs (k... i ,v i ) Store knowledge;

[0015] Among them, high-level planning knowledge uses the overall task description T t The action sequence T is indexed as a key. τ Pairing is done as values; for low-level skill knowledge, the key consists of the object image and the subtask description T. τi It consists of an object mesh model Ω and semantic constraints and code representing the learned skills;

[0016] Step 3: Adapt skills through iterative comparison strategies, including:

[0017] S3.1, High-rise Planning

[0018] Based on task guidance, high-level planning knowledge is retrieved from the knowledge base as contextual examples of VLMs, along with scene observations; VLMs act as task planners, generating a series of executable steps and descriptions of task-related objects. o ;

[0019] S3.2 Iterative Comparison

[0020] Low-level skill knowledge is retrieved from the knowledge base and updated through iterative comparison; in each iteration, VLMs adapt the interaction I and the interaction retrieved from the knowledge base. Perform comparative analysis between them, and then update the grab constraints and operation constraints.

[0021] Preferably, the method for extracting grasping task constraints in S2.1 includes: using VLMs to annotate each grasping pose on the object image, and each grasping pose has a specific index symbol; clustering these poses to form several bounded regions, that is, obtaining the position of the corresponding semantic constraint.

[0022] Subsequently, the range of parameters is calculated by pose and associated with object properties to obtain geometric constraints represented as bounded regions.

[0023] Preferably, in S2.1, VLMs use their inherent knowledge and visual understanding capabilities to summarize semantic constraints to cluster these poses, or use the DBSCAN method to cluster these grasping poses.

[0024] Preferably, in step S2.1, within the geometrically constrained region, the grasping quality score is estimated, and then the grasping pose corresponding to the best score is selected to perform the action; during the execution process, candidate objects are uniformly extracted within the learned constraints, the grasping quality score is evaluated, and the best pose is selected.

[0025] Preferably, the method for extracting operational constraints in S2.2 includes:

[0026] In the main object O m The trajectory of the robotic arm gripper is depicted above, and key points sampled from the trajectory are included in the text prompts. This is first based on the motion image and the subtask description T. τi The content is determined by the semantic constraints of the operation identified by VLMs, and then the geometric constraints of the operation are formulated based on the semantic constraints, key point values ​​and object attributes, resulting in the trajectory of the object centered on the object.

[0027] Preferably, the iterative comparison in S3.2 includes grasping constraint adaptation, specifically:

[0028] The visualized grasping constraint space is discretized into a grid, and T is annotated with text. n Annotate each grid to obtain an adaptive interaction I v VLMs interact with the reference and semantic constraints Perform a comparison to update the crawling semantic constraints Φ s And according to the adaptive interaction I v The updated grasping semantic constraints are sampled from the mesh of the grasping constraint space, and the sampled mesh forms the updated grasping geometric constraints Φ. g Then, for the updated grab geometry constraint Φ g The visualization will be used in the next iteration.

[0029] Preferably, the iterative comparison in S3.2 includes operational constraint adaptation, specifically: VLMimic generates an array that conforms to the operational geometric constraint Φ. g The trajectories, which are displayed on the main object, guide VLMs in analyzing and adapting interactive elements. v Interact with reference To revise semantic constraints Φ based on deviations s This leads to the updating of the geometric constraints Φ of the operation.g and for the updated Φ s and trajectory key point value V k and object property P o Make minor adjustments.

[0030] Furthermore, this also includes S3.3 and failure reasoning:

[0031] FoundationPose is used to track the poses of objects and robot end effectors throughout the entire task execution process. The pose trajectories of objects and end effectors during task execution are used as perception results for task failure reasoning and replanning. After the action is completed, the real-time position of the object is used to determine whether the task is successful until manual confirmation or a preset time is reached.

[0032] The present invention has the following beneficial effects:

[0033] This invention provides a Visual Imitation Learning-Based (VLMimic) method for robot fine-grained skill learning and generalization. It is a novel visual imitation learning framework driven by Visual Learning Models (VLMs) that learns generalizable robot skills from human demonstration videos. VLMimic features a knowledge-extracting skill learner and an iterative skill refinement adapter, achieving efficient skill acquisition and adaptation. This invention constructs an effective human-object interaction localization algorithm, enhancing fine-grained action recognition capabilities, and proposes a hierarchical constraint representation for VLM inference to reduce information redundancy and promote comprehensive action understanding. The method outperforms other methods by more than 27% on RLBench. In real-world operational tasks, VLMimic achieves an improvement of over 21% in known environments and 34% in unknown environments. Furthermore, VLMimic demonstrates an improvement of over 37% in long-term tasks. Attached Figure Description

[0034] Figure 1 The diagram below illustrates the VLMimic of the present invention, wherein (a) the basic module for human-object interaction parses the video into multiple segments and captures the object-centered motion; then, (b) the skill learner extracts knowledge from the motion and derives skills; in a novel scenario, (c) the skill adapter updates the learned skills to facilitate adaptation.

[0035] Figure 2 This is a schematic diagram of a video parsing method. (a) shows how it identifies the task and related objects from human videos; (b) shows how it parses the video into multiple segments based on this information; and (c) shows how it identifies object-centric interactions within each segment.

[0036] Figure 3 Demonstrates the overall framework for video analysis to humans;

[0037] Figure 4 This refers to the video parsing and task decomposition process;

[0038] Figure 5 A skill learning method based on hierarchical constraints;

[0039] Figure 6 Extracting constraints for hierarchical constraint representation;

[0040] Figure 7 Extraction of operational constraints for hierarchical constraint representation;

[0041] Figure 8 Build a knowledge base process;

[0042] Figure 9 This involves performing failure detection and replanning for the robot. Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0044] This invention provides a method for fine-grained skill learning and generalization in robots based on visual imitation learning, such as... Figure 1 As shown, the overall idea is as follows: Consider a video demonstration V, in which one person uses an RGB-D camera to record the execution of an operation task. The overall flow of VLMimic is as follows: Figure 2 As shown, the method of this invention first grounds human videos, subdividing them into sub-task intervals and capturing object-centric interactions. Then, a skill learner with hierarchical representations extracts knowledge from the acquired interactions, extracting skills with fine-grained actions. In an unknown environment, the skill adapter employs an iterative comparison strategy to refine and update the learned skills based on observations and task guidance.

[0045] Step 1: Video Analysis Method

[0046] Visual imitation learning aims to understand the content of human demonstration videos and thus mimic human actions. The recent rapid growth in conceptual understanding, common-sense knowledge, and reasoning abilities demonstrated by visual learning machines (VLMs) provides powerful tools for common-sense understanding and reasoning. However, VLMs still struggle to recognize fine-grained actions in videos. To address this issue, this invention first analyzes human demonstration videos using a method that analyzes interactions between human hands and objects. The video is broken down into segments, and each segment undergoes object-centric action evaluation. These actions are then provided to VLMs for analysis, transforming the complex problem of understanding low-level actions into a pattern reasoning problem that VLMs excel at solving. This method allows robots to learn and acquire new skills.

[0047] The overall framework for human demonstration video analysis is as follows: Figure 3 As shown, in order to obtain object-centered motion for skill learning, the three-stage process specifically includes video parsing, task recognition, object reconstruction, and object-centered motion estimation.

[0048] This invention integrates sensing results from VLMs, as follows: Figure 4 As shown, VLMs are used to transcribe human demonstration videos into task instructions and descriptions of objects related to the manipulation. By capturing the interaction between the hand and point clouds of objects in the video, the video is parsed into a series of subtask segments. Each segment is translated into a subtask description and classified by VLMs into a grasping or manipulating phase, during which VLMs identify master and slave objects. To facilitate robot understanding and analysis, this invention defines moving objects in each subtask as slave objects and stationary target objects as master objects. In the grasping phase, the robot performs a reaching and grasping action targeting the master object, where the hand is designated as the slave object; in the manipulating phase, the robot interacts with the master object using slave objects.

[0049] S1.1 Overall Task Identification

[0050] In the task recognition process, the TAP (TokenizeAnything via Prompting) model, centered on visual perception, is first used to obtain fine-grained scene understanding. For example, for an operation scene in a human demonstration video, the TAP model provides a detailed description such as "a bright red pizza is on a plate, and a red brush is in a bowl." Then, VLMs are used to integrate and filter the detailed descriptions to generate task descriptions and object information.

[0051] As above Figure 4 As shown in the left half, the method proposed in this invention periodically extracts a series of image frames from a human-operated demonstration video, using Visual Management Models (VLMs) to describe the event and object information within these image frames. This information is combined with each image and input into the VLMs. The VLMs transcribe the video into task instructions, filter out objects related to the operation, and compile them into text information, i.e., task descriptions, including a list of object names and spatial relationships between objects in the operation scene.

[0052] S1.2, Video Analysis

[0053] As above Figure 4As shown in the middle section, interactive markers are extracted from the human demonstration video, capturing the interactions between two types of entities: hands and objects related to the operation. These markers allow the human demonstration video to be segmented into multiple object-centric segments. Specifically, object descriptions are used as text cues, and SAM-Track is applied to generate entity masks and point cloud information for each frame. Subsequently, when the distance between any two entities meets a set condition, the two entities are considered to be in an interactive relationship; otherwise, they are considered non-interactive. The video is then divided into multiple sub-task segments based on the duration of the interaction.

[0054] S1.3 Subtask Identification

[0055] As above Figure 4 As shown in the right half, each subtask segment is divided into a grasping phase and a manipulation phase based on the interacting entities. VLMs analyze each subtask segment, generating a text description of the subtask and identifying the master and slave objects. To facilitate robot understanding and analysis, this invention defines moving objects in each subtask as slave objects and stationary target objects as master objects. In the grasping phase, the robot reaches out and grasps the master object, with the hand designated as the slave object and the grasped object designated as the master object. In the manipulation phase, the robot interacts with the master object using slave objects, where the master object is represented by the object being contacted, and the slave object is represented by the object actively contacting it.

[0056] S1.4 Object-Centered Interaction Extraction

[0057] This invention uses the MANO model to infer hand pose parameters and employs the Iterative Closest Point (ICP) method to improve the accuracy of motion estimation, resulting in a precise hand pose trajectory. Therefore, the hand pose trajectory needs to be transformed into the robotic arm end effector pose trajectory. Furthermore, by comparing the object mesh model with observations, the FoundationPose model is used for object pose estimation. In this case, only the pose estimation of the derived objects is performed; the stationary master object is not subject to pose estimation. The resulting hand pose trajectory and the derived object's pose trajectory are represented in the current coordinate system relative to the master object. This object-centric paradigm facilitates effective skill acquisition and allows the method of this invention to be adapted for demonstrations across different perspectives.

[0058] Step Two: Learning Techniques with Hierarchical Representation

[0059] In hierarchical constraint representation learning, the constrained representation consists of action forms or trajectories on an object image and text symbols. The hierarchical constraint representation learning method guides Visual Learning Models (VLMs) to focus on object attributes and related keypoint values, utilizes their visual understanding to analyze semantic constraints, and integrates keypoint values ​​through numerical analysis to derive geometric constraints, thereby promoting the visual and linguistic reasoning of VLMs, as follows: Figure 5 As shown.

[0060] S2.1, Extracting constraints

[0061] Inspired by task space regions (TSRs), the grasping constraint space M can be approximated as a series of bounded regions. In the constraint learning phase, this invention allows VLMs to autonomously select region N. C The quantity. As follows. Figure 6 As shown, each grasping pose is labeled on the object image, and each grasping pose has a specific index symbol. These images are presented to VLMs, which use their inherent knowledge and visual understanding to summarize semantic constraints to cluster these poses. Alternatively, the DBSCAN method can be used to cluster these grasping poses. The figure shows that the brush grasping poses can be divided into two main categories, forming two bounded regions, which correspond to the two positions of the semantic constraints.

[0062] Subsequently, by calculating the range of parameters through pose and associating them with object attributes, geometric constraints represented as bounded regions can be obtained. These object attributes specifically include 3D bounding box information. Within the geometrically constrained region, a grasping quality score is estimated, and the grasping pose corresponding to the best score is selected for action execution. This method transforms the complex task of generating constrained regions into a series of choices, which is generally easier for existing VLMs to handle. Furthermore, representing constraints through object attributes enhances generalization across different objects. During execution, this invention uniformly extracts candidate objects within the learned constraints, evaluates the grasping quality score, and selects the optimal pose.

[0063] S2.2 Operational Constraint Extraction

[0064] To facilitate robot understanding and analysis, this invention defines the moving object in each subtask as originating from object O. s The stationary target object is defined as the main object O. m In the main object O m The trajectory of the robotic arm gripper is depicted above, and key points sampled from the trajectory are included in the text prompts. In the experiment, 10 key points were uniformly sampled. For example... Figure 7 As shown, firstly, based on the action image and subtask description T... τi The content is generated by VLMs identifying the semantic constraints of the operation, then formulating the geometric constraints of the operation based on the semantic constraints, keypoint values, and object attributes, and obtaining the trajectory of the object centered on the object. To improve reusability, the code is divided into parameter estimation functions f. p and shape generation function f sThe former obtains shape parameters from the perception results, while the latter uses these parameters to generate a series of end-effector poses, thereby facilitating efficient generalization across various objects and spatial configurations. During execution, these trajectories centered on the main object are transformed into the world coordinate system through object pose estimation.

[0065] S2.3, Construction of the Knowledge Base

[0066] as follows Figure 8 As shown, a knowledge base B is established to archive high-level planning and low-level skill insights, using key-value pairs (k... i ,v i ) Store knowledge. High-level planning knowledge uses the overall task description T. t Used as a key for indexing, followed by the action sequence T τ Pairing is done using the object image and the subtask description T. For low-level skill knowledge, the key consists of the object image and the subtask description T. τi It consists of an object mesh model Ω and semantic constraints and codes representing learning skills.

[0067] Step 3: Adapt skills through iterative comparison strategies

[0068] While skill learners demonstrate efficient skill acquisition, the demonstration and execution scenarios may differ in objects and tasks, hindering the direct transfer of skills to unseen environments. To mitigate these challenges, guided VLMs adapt skills through iterative comparison strategies, such as... Figure 2 As shown, this method updates learned skills by iteratively comparing them with demonstrated knowledge, thereby enabling effective adaptation to new scenarios.

[0069] S3.1, High-rise Planning

[0070] Based on task guidance, high-level planning knowledge is retrieved from a knowledge base, serving as contextual examples for VLMs and scene observations. VLMs, acting as physically-based task planners, generate a series of executable steps and descriptions T of task-related objects. o .

[0071] S3.2 Iterative Comparison

[0072] Low-level skill knowledge is retrieved from the knowledge base and updated through iterative comparison. In each iteration, VLMs adapt the interaction I and the interaction retrieved from the knowledge base. Perform comparative analysis between them, and then update the skill constraint Φ. s and Φ g This iterative process continues until convergence is achieved or the maximum number of iterations N is reached. IThis approach facilitates reasoning in VLMs by directing attention to the differences and enables VLMs to determine the best available solution through an iterative process.

[0073] (1) Adaptation of Grasping Constraints. Since the grasping direction can usually be derived from the positional constraints using the grasping model, this invention focuses on the transfer of positional constraints. The visualized grasping constraint space is discretized into an m×n grid (m,n∈Z) and annotated with text T. n Annotate each grid to obtain an adaptive interaction I v VLMs interact with the reference and semantic constraints Perform a comparison to update the crawling semantic constraints Φ s And according to the adaptive interaction I v The updated grasping semantic constraints are sampled from the mesh of the grasping constraint space, and the sampled mesh forms the updated grasping geometric constraints Φ. g Then, the updated grab geometry constraint Φ is applied. g The visualization is then used for the next iteration. The 3D location region is represented using two perspectives, and the consistency of the overlapping areas of the selected regions validates the VLM output. The obtained constraints are represented through object properties to enhance generalization.

[0074] (2) Operational Constraint Adaptation. Guided VLMs to iteratively summarize and update operational constraints based on task guidance and scenario differences. VLMimic generation adheres to operational geometric constraints Φ. g The trajectories, which are displayed on the main object, guide VLMs in analyzing and adapting interactive elements. v Interact with reference To revise semantic constraints Φ based on deviations s This leads to the updating of the geometric constraints Φ of the operation. g and for the updated Φ s and trajectory key point value V k and object property P o Make minor adjustments.

[0075] S3.3, Failed Reasoning

[0076] While VLMs can generate effective constraints, various environmental noises, including trajectory estimation errors, can hinder successful task execution. Therefore, this invention utilizes VLMs to detect and resolve faults during execution, providing them with sensing results such as object pose and robot end effector trajectory, thereby enabling autonomous fault identification and replanning for correction.

[0077] Specifically, FoundationPose is used to track the poses of objects and robot end effectors throughout the entire task execution. (See below) Figure 9 As shown, the pose trajectories of objects and end effectors during task execution are used as perception results for task failure inference and replanning. After the action is completed, the real-time position of the object is used to determine whether the task is successful, until manual confirmation or a preset time is reached. Table 1 shows the success rate on RLbench. "Obs-act", "Template", and "Video" represent paired observation-action sequences, code templates, and videos of the execution of subtasks.

[0078]

[0079] Table 2 shows the success rate in real-world operational experiments. “Obs-act,” “Template,” and “Video” represent paired observation-action sequences, code templates, and videos of the execution of subtasks. “SE” and “UE” represent the observed environment.

[0080]

[0081]

[0082] Table 3 shows the success rate on long-term tasks. “Obs-act,” “Template,” and “Video” represent the observation-action sequence, code template, and video of the task execution.

[0083]

[0084] 1. Comparison Methods

[0085] VLMimic was compared with five representative methods: (1) R3M-DP, which uses pre-trained R3M visual representations and a state-of-the-art (SOTA) diffusion policy; (2) Diffusion Policy (DP), a SOTA end-to-end policy approach; (3) GraphIRL, a method that uses graph abstraction and learns reward functions for reinforcement learning (RL); (4) Policy as Code (CaP), a large language model-driven approach that reassembles API calls to generate new policy codes; and (5) Demo2code, a large language model-driven planning approach that converts demonstrations into task codes. This invention modifies Demo2code to integrate the analysis results of GPT-4V for Robotics, enabling it to transcribe videos into code. R3M-DP and DP are trained using paired observation and action sequences with robot demonstrations. GraphIRL is trained in a simulator using paired robot videos, while Demo2code and the method of this invention learn skills using human videos in real-world experiments and robot videos in simulated experiments.

[0086] 2. Simulated operation task

[0087] Experimental Setup: To evaluate the performance of the method of this invention on challenging robotic manipulation tasks, the RLBench benchmark was used to simulate the task. Since human video was lacking in the simulation environment, Demo2code and the method of this invention used robot video captured from a single camera viewpoint, which included the trajectory of the robot gripper.

[0088] Results: This invention investigated VLMimic's ability to learn skills from limited video demonstrations without additional training. The evaluation covered 12 operational tasks, as shown in Table 1, demonstrating that the method of this invention outperformed all other methods in 11 of these tasks. The method of this invention, learning from only 5 human videos, significantly outperformed R3M-DP and DP overall, although the latter two were trained on 100 robot demonstrations. Compared to CaP and Demo2code, the method of this invention showed a performance improvement of over 27%, highlighting the significant performance gains brought by the VLMimic framework.

[0089] 3. Real-world operational tasks

[0090] Experimental Setup: The real-world test environment (E) was divided into two categories: "Known" (SE) and "Unknown" (UE). The "Known" category allowed testing in the environment where demos were collected, while the "Unknown" category involved testing in different environments with different objects and layouts. Success criteria were evaluated manually, and the success rate was calculated from 10 randomized object positions and orientations.

[0091] Results: To validate the effectiveness of VLMimic in real-world environments, experiments were conducted involving 14 challenging real-world operational tasks selected from recent robotics research. Quantitative results, as shown in Table 2, demonstrate that VLMimic significantly outperforms other methods across all tasks, particularly in "unknown" environments (UEs). VLMimic achieves a performance improvement of over 21% in SEs and over 34% in UEs. These results reveal VLMimic's superior ability to learn skills from human videos and adapt them to unknown environments.

[0092] 4. Real-world long-term missions

[0093] Experimental setup: Because the baseline method is difficult to perform long-term tasks in the UE setup, the experiment was conducted in the SE setup. All other experimental settings were consistent with the real-world operational task.

[0094] Results: The performance of VLMimic on long-term tasks was quantitatively evaluated by successfully completing six different tasks, each containing at least five sub-tasks. The experimental results, as shown in Table 3, clearly demonstrate the significant improvement of the proposed method compared to baseline methods. These results indicate that the proposed method can develop powerful skills and thus achieve satisfactory performance in long-term tasks.

[0095] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for robot fine-grained skill learning and generalization based on visual imitation learning, characterized in that, Comprising: Step one, video analysis: S1.1, total task identification, including: periodically extracting a series of picture frames from human operation demonstration video, using VLMs to construct task description and object information within the picture frame; S1.2, video analysis, including: extracting interaction markers from human operation demonstration video, capturing the interaction between the two entities of hand and operation-related objects, and dividing the video into multiple sub-task segments according to the time period of the interaction between the two entities; S1.3, sub-task identification, including: each sub-task segment is divided into a grasping stage and an operation stage according to the interacting entities; VLMs analyze each sub-task segment to generate a sub-task text description and identify the main object and the from object; in each sub-task, the moving object is defined as the from object, and the stationary target object is defined as the main object; in the grasping stage, the robot performs the hand and grasping actions with the main object as the target, wherein the hand is defined as the from object and the object being grabbed is defined as the main object; in the operation stage, the robot interacts with the main object using the from object, wherein the main object is used to represent the object being touched, and the from object is used to represent the object actively contacting; S1.4, object-centered interaction extraction, including: obtaining the pose trajectory of the generated hand and the pose trajectory of the from object in the current coordinate system relative to the main object; Step two, skill learning method with hierarchical representation, comprising: S2.1, extracting grasping task constraints; S2.2, extracting operation constraints; S2.3, construction of knowledge base: Establishing a knowledge base to archive high-level planning and low-level skill insights, and storing knowledge using key-value pairs; where high-level planning knowledge uses total task descriptions indexed by keys, action sequences paired with values; for low-level skill knowledge, keys are composed of object images and subtask descriptions while values are composed of object mesh models and semantic constraints and code representing learned skills; Step three, adapting skills through iterative comparison strategy, comprising: S3.1, high-level planning Retrieving high-level planning knowledge from a knowledge base based on task guidance, as contextual examples for VLMs, and scenario observations; VLMs as task planners, generating a series of executable steps and descriptions of task-related objects ; S3.2, iterative comparison retrieving low-level skill knowledge from the knowledge base and updating it through iterative comparisons; in each iteration, the VLMs perform a comparative analysis between the adapted interactions and the knowledge base retrieval and subsequently update the grasping constraints and manipulation constraints.

2. The robot fine-grained skill learning and generalization method based on visual imitation learning according to claim 1, wherein, The method of extracting grasping task constraints in S2.1 includes: using VLMs to label each grasping pose on the object image, and each grasping pose has a specific index symbol; clustering these poses to form a number of bounded regions, i.e., obtaining the position of the corresponding semantic constraints; Then, calculate the parameter range of the pose and associate it with the object attribute to obtain the geometric constraints represented as bounded regions.

3. The robot fine-grained skill learning and generalization method based on visual imitation learning of claim 2, wherein, In S2.1, VLMs use inherent knowledge and visual understanding ability to summarize semantic constraints to cluster these poses, or use the DBSCAN method to cluster these grasping poses.

4. The robot fine-grained skill learning and generalization method based on visual imitation learning of claim 2, wherein, In the geometric constraint region of S2.1, the grasping quality score is estimated, and then the grasping pose corresponding to the best score is selected to perform the action; during execution, the candidate objects are uniformly extracted within the learned constraints, the grasping quality score is evaluated, and the best pose is selected.

5. The robot fine-grained skill learning and generalization method based on visual imitation learning of claim 1, wherein, The method of extracting operation constraints in S2.2 includes: In the main object The trajectory of the robotic arm gripper is depicted above, and key points sampled from the trajectory are included in the text prompts, based first on motion images and subtask descriptions. The content is determined by the semantic constraints of the operation identified by VLMs, and then the geometric constraints of the operation are formulated based on the semantic constraints, key point values ​​and object attributes, resulting in the trajectory of the object centered on the object.

6. The robot fine-grained skill learning and generalization method based on visual imitation learning of claim 1, wherein, The iterative comparison in S3.2 includes grasping constraint adaptation, specifically: The visualized grasp constraint space is discretized into a grid and annotated with text Each grid is annotated to obtain a visualized adapted interaction ; VLMs are updated by comparing with the reference interaction and semantic constraints The grasp semantic constraints are updated and the grid of the grasp constraint space is sampled according to the adapted interaction and the updated grasp semantic constraints, the sampled grid forms updated grasp geometric constraints ; then, the updated grasp geometric constraints are visualized and used for the next iteration.

7. The robot fine-grained skill learning and generalization method based on visual imitation learning according to claim 1 or 6, wherein, The iterative comparison in S3.2 includes operational constraint adaptation, specifically: VLMimic generates data that conforms to operational geometric constraints. Trajectory; guiding VLMs analysis visualization of adaptive interactions Interact with reference To revise semantic constraints based on deviations This leads to updating the geometric constraints of the operation. and for the updated and trajectory key point values and object properties Make minor adjustments.

8. The robot fine-grained skill learning and generalization method based on visual imitation learning of claim 1, wherein, Further comprising S3.3, failure reasoning: FoundationPose is used to track the pose of the object and the robot end-effector throughout the task execution; the object and end-effector pose trajectories during task execution are used as perception results for task failure reasoning and replanning, and after the action is completed, the real-time position of the object is used to determine whether the task is successful until manual confirmation or a preset time is reached.

Images