A language-driven robot morphable body operation generalization learning method

Abstract

This invention discloses a language-conditional generalization learning method for deformable robot operations. Addressing the challenges of high-dimensional state space, dynamic complexity, and difficulties in cross-task transfer in deformable robot operations, it constructs a generalization framework integrating language understanding, skill decomposition, and cross-modal learning. The method collects multimodal demonstration data such as image sequences and natural language commands. It utilizes a large language model to perform semantic parsing and hierarchical decomposition of tasks, forming reusable basic skill units. Then, through a Transformer-based cross-modal skill learning model, it combines images and skill descriptions to predict operation target points and execution parameters, establishing a general skill library. When faced with a new task, the large language model generates skill sequences and calls upon the skill library for step-by-step execution, thereby achieving generalized operation of unseen tasks in both simulation and real robots, significantly improving the system's generalization, stability, and scalability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and robot manipulation technology, and in particular to a language-driven generalization learning method for manipulating deformable robots. This method, by introducing natural language parsing, hierarchical task decomposition, and cross-modal skill learning and combined execution, achieves adaptive optimization for complex deformable robot manipulation tasks, significantly improving the robot's generalization ability, operational stability, and task execution scalability in dynamic environments.

[0002] Deformable body manipulation, a typical challenge in robotics, has significant applications in smart home chores, flexible manufacturing, healthcare, and human-robot collaboration. However, due to the high-dimensional state space, complex morphological changes, and high dynamic instability of deformable bodies, any improper action can lead to irreversible results (such as excessive wrinkling, structural misalignment, or task failure). This poses a significant challenge to robots performing multi-step, complex operations such as folding clothes, spreading towels, and tidying bed sheets.

[0003] Existing imitation learning methods typically employ end-to-end task-level strategies, learning the overall mapping from the initial state to the target state through a large amount of demonstration data. While these methods may perform well for specific tasks, they often lack generalization ability when faced with unfamiliar new tasks, making it difficult to transfer or adapt to new operational requirements.

[0004] In recent years, researchers have attempted to combine visual representation learning with imitation learning to enhance robots' understanding and execution of deformable body manipulations. However, these methods still have significant shortcomings:

[0005] 1) The lack of an effective task decomposition mechanism and the direct learning of policies on long-term complex operations result in excessive training data requirements and difficulty in capturing reusable local skills.

[0006] 2) Strategy learning is task-oriented and lacks cross-task reusability, making it difficult for robots to transfer their learning of "grasping corner folding" to new tasks such as "grasping cuff folding".

[0007] 3) There is a lack of flexible connection between the planning layer and the execution layer. Action splicing usually relies on manual rules or fixed templates, which makes it difficult to cope with real-world application scenarios with changing task objectives and complex state spaces.

[0008] With the development of Large Language Models (LLMs), robot task planning and semantic parsing have achieved new breakthroughs. LLMs can automatically decompose, semantically reorganize, and generalize reasoning for complex tasks based on natural language instructions, bringing new ideas to deformable object manipulation: through language condition parsing, common basic skills (such as "grasp," "place," "unfold," and "fold") are automatically discovered and defined, and skill-level parameterization is performed using cross-modal deep learning models, enabling robots to achieve generalized operations through language-driven skill combinations when performing new tasks.

[0009] Therefore, there is an urgent need for a new language-driven generalization learning method for manipulating deformable robots. This method combines language understanding, task decomposition, and skill base construction mechanisms to achieve cross-task generalization while ensuring operational stability. It breaks through the dependence of traditional methods on task-specific strategies, thereby significantly improving the adaptability, versatility, and scalability of robots in real-world environments. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of existing methods for manipulating deformable robots, such as insufficient generalization ability, lack of task decomposition, and limited skill reusability. This invention proposes a language-driven generalization learning method for manipulating deformable robots. This method introduces a large language model for semantic parsing and hierarchical decomposition to construct a general skill library. It then combines this with a cross-modal skill learning model to achieve skill-level learning and combined application, thereby achieving efficient, stable, and scalable execution in complex deformable robot manipulation. Traditional imitation learning methods often rely on end-to-end task-level strategies, making them difficult to transfer to new tasks. This invention, through language-driven task decomposition and dynamic skill scheduling, effectively overcomes this limitation.

[0011] The specific technical solution for achieving the objective of this invention is as follows:

[0012] A language-driven generalization learning method for manipulating deformable robot bodies, comprising the following steps:

[0013] 1) Collect long-term demonstration data of robot material operation, obtain the corresponding image sequence and language command or description, and perform time alignment on the image sequence and language command or description;

[0014] 2) Utilize a large language model to perform semantic parsing and task decomposition on the language instructions, breaking down the overall operation into multiple general basic skills;

[0015] 3) Construct a basic skills dataset for material manipulation based on the decomposition results, and pair language descriptions, image states, and corresponding action labels;

[0016] 4) A Transformer-based skill learning model is adopted. The input is a material image obtained from image sequence recognition and a language instruction or description. The output is the grasping or placement operation target point or key action parameters, and the model is trained to form a single-step skill model.

[0017] 5) When performing a new task, the large language model acts as a high-level planner, automatically generating a basic skill sequence based on the task description and current observations, and calling existing single-step skill models in the public skill library to complete the operation step by step;

[0018] 6) Execute the language instructions or descriptions on the simulation and real robot platforms to achieve generalized execution of new material manipulation tasks.

[0019] Furthermore, step 2) specifically includes:

[0020] 2-1 Parse natural language instructions into structured skill scripts, including skill name, parameters, preconditions and termination conditions;

[0021] 2-2 Automatically discover unlabeled but reusable basic skills using 2 to 8 sample hints or chained reasoning hints;

[0022] 2-3 Segment the long-duration demonstration and align it to the corresponding skill tags to obtain the skill-level segment boundaries and sequence relationships;

[0023] 2-4 Standardize the representation of skill parameters by combining environmental condition terms; among which, environmental condition terms include left sleeve, lower edge, and center line.

[0024] Furthermore, step 3) specifically includes:

[0025] 3-1 Extract material morphology features from image sequences, including edges and corners, wrinkle direction fields, and key points;

[0026] 3-2 Establish language-image-action triples, where the action label includes at least the grab point, the placement point and its pose and orientation, and the folding vector or target trajectory;

[0027] 3-3 Perform time alignment and interpolation on skill-level clips, and perform data augmentation, including viewpoint, lighting, texture, and material perturbations;

[0028] 3-4 Organize the language-image-action triplet and skill-level segments into a supervised training sample set for single-step skill learning.

[0029] Furthermore, step 4) specifically includes:

[0030] 4-1 A cross-modal fusion structure of visual encoder and language encoder is adopted, and cross-attention is used to align image features and language features;

[0031] 4-2 Model output includes capture or placement heatmaps, key point sets, and / or end effector pose parameters;

[0032] 4-3 With the goal of position and pose regression and key point matching, L1 and L2 norms and cross-entropy loss are jointly used, and executability constraints are introduced to ensure the reachability of the gripper and the safe distance of the material edge.

[0033] 4-4 Fine-tuning Transformer-based single-step skill models to improve robustness through behavioral cloning or DAgger strategies.

[0034] Furthermore, step 5) specifically includes:

[0035] 5-1 The high-level planner generates a basic skill sequence based on the task description and current visual observations, and performs a closed-loop update based on the new observation image after each step is executed;

[0036] 5-2 Set up failure detection and rollback rules, and call "re-lay" or "expand" recovery skills to correct errors when deviations are detected;

[0037] 5-3 During the skill splicing process, the preconditions or termination conditions and parameter passing strategies are followed, and the required parameters are filled and constrained by the large language model or the underlying perception module.

[0038] Furthermore, step 6) specifically includes:

[0039] 6-1 Perform domain randomization training on the simulation platform, including texture, lighting, friction coefficient and gravity perturbation;

[0040] 6-2 By using camera calibration and pose transformation, the grasping or placement strategies learned in simulation are transferred to real robots;

[0041] 6-3 Perform online threshold and gripper opening / closing parameter calibration on a real platform to adapt to different materials and thicknesses for different operations.

[0042] This invention extracts semantic-action mapping relationships from natural language instructions and operational demonstrations to automatically discover and learn reusable basic skills, thereby achieving cross-task generalization. Specifically: First, long-term demonstration data of the robot operating on deformable bodies is collected, including image sequences and accompanying language instructions. Second, a large language model is used to perform semantic parsing and task decomposition of the language descriptions, breaking down complex operations into multiple general skill units (such as "grasping edges," "folding along the centerline," and "aligning placement"). Then, a language-image-action triplet dataset is constructed, and a Transformer-based cross-modal skill learning model is used. The input is a deformable body image and skill-level language description, and the output is the corresponding operation target point or pose parameters, thus training a generalizable skill model. Subsequently, these skill models are stored in a skill library, and when executing a new task, the large language model acts as a high-level planner, automatically generating skill sequences based on the task description and calling the corresponding models for step-by-step execution. Finally, verification and execution are completed in a simulation environment and on a real robot platform, thereby achieving generalized learning and stable control of unseen new tasks.

[0043] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0044] 1) This invention proposes a language-driven generalization learning method for robot deformable body operations. By introducing semantic parsing, task decomposition and high-level planning mechanisms of a large language model, it can automatically decompose complex long-term deformable body operations into a series of reusable basic skill units, thereby significantly improving the execution stability and efficiency of the robot in multi-step operation processes.

[0045] 2) This invention constructs a skill-level dataset in the form of language-image-action triples and achieves skill learning and parameterization based on a cross-modal Transformer model, breaking through the limitations of traditional methods that directly rely on task-level end-to-end policies. This design enables robots to achieve cross-task generalization in unfamiliar tasks, effectively avoiding overfitting problems in single-task scenarios.

[0046] 3) This invention combines high-level planning of a large language model with low-level execution of a skill base, forming a closed-loop optimization mechanism of "task understanding—skill generation—execution feedback". During the execution of a new task, the system can dynamically select and combine skill models based on environmental observations and task objectives, significantly enhancing the robot's adaptability and versatility under different task objectives and complex operational scenarios.

[0047] 4) This invention performs learning and combination at the skill level rather than the task level, avoiding reliance on large-scale task-specific demonstration data and manual rule design, thus significantly reducing data collection and annotation costs. Simultaneously, this method exhibits higher robustness and scalability in real-world environments, enabling robots to achieve stable execution and efficient migration in diverse deformable body manipulation tasks. Attached Figure Description

[0048] Figure 1 This is a flowchart of the present invention;

[0049] Figure 2 is a schematic diagram of the specific operation of Example 1. Detailed Implementation

[0050] This invention aims to address the shortcomings of existing methods for manipulating deformable robots, particularly their insufficient generalization ability, lack of task decomposition mechanisms, and limited skill reusability. To this end, a language-driven generalization learning method for manipulating deformable robots is proposed, combining language parsing, skill-level learning, and high-level planning mechanisms to improve the stability, versatility, and scalability of robots in complex operations.

[0051] First, this invention acquires image sequences and corresponding language commands from long-term demonstration data of robot deformable body manipulation. Image data is used to characterize the morphological features of the deformable body (such as edges, wrinkles, corners, etc.), while language data provides operational semantics (such as "grasp the left corner" or "fold along the center line"). Through time alignment and preprocessing, a multimodal input set is formed that can be used for subsequent training.

[0052] Secondly, the Large Language Model (LLM) is used to perform semantic parsing and task decomposition of natural language instructions, automatically breaking down complex, long-running operations into a series of reusable basic skill units. For example, "fold a towel three times and place it on the table" can be decomposed into four skills: "grasp the edge—fold inward—repeatedly fold—place on the table." This mechanism can not only identify common skills (grasp, place, unfold, fold), but also mine implicit operation patterns through few-shot reasoning, thereby improving the versatility of the skill base.

[0053] Then, a skill dataset in the form of language-image-action triples is constructed. Specifically, image states (morphological features of deformable objects), language skill descriptions (such as "grab the bottom left corner") are paired with action labels (such as end effector pose, grasp point coordinates, and placement direction vector), and rich skill-level training samples are generated through temporal alignment and data augmentation.

[0054] Building upon this foundation, a Transformer-based cross-modal skill learning model is employed. Inputting a deformable image and skill description, the model outputs the target point or key action parameters. The model extracts multimodal features through visual and language encoders and fuses them in a cross-attention module, ensuring the robot can correctly interpret the environmental state and generate executable actions based on linguistic conditions. During training, position regression loss, action executability constraints, and behavior cloning strategies are combined to guarantee the robustness and transferability of the single-step skill model.

[0055] Subsequently, when executing a new task, the large language model acts as a high-level planner, generating a complete skill sequence based on the task description and progressively calling skill models from the skill library for execution. After each execution, the system dynamically updates subsequent plans based on the latest visual observations, achieving adaptive closed-loop control. Furthermore, a failure detection and recovery mechanism is implemented. When a grasping deviation or folding failure occurs, it automatically invokes error-correcting skills such as "expand" and "re-tilt" to prevent overall task failure.

[0056] Finally, the effectiveness of the method was verified on simulation and real robot platforms. In the simulation phase, domain randomization (lighting, texture, friction coefficient, gravity perturbation, etc.) was used to enhance the model's generalization ability. In the real system, camera calibration, end effector calibration, and gripper parameter adjustment were used to adapt the model to deformable bodies of different materials and thicknesses. Experiments show that this method can achieve cross-task generalization in various tasks such as folding towels, tidying T-shirts, and making bed sheets, and maintains strong execution stability under new, unseen instructions.

[0057] Through the above steps, this invention effectively solves the problem of traditional methods relying on task-specific strategies through the synergistic effect of language parsing, skill learning and task execution closed loop, avoids overfitting to a single task, and significantly improves the accuracy, stability and versatility of the robot in deformable body operation through skill reuse and high-level planning.

[0058] See Figure 1 This invention discloses a language-driven generalization learning method for manipulating deformable robot bodies, which specifically includes the following steps:

[0059] 1) Collect long-term demonstration data of robot material manipulation, obtain the corresponding image sequences and language commands / descriptions, and align the two in time;

[0060] 2) Utilize a large language model to perform semantic parsing and task decomposition on the language instructions, breaking down the overall operation into multiple general basic skills;

[0061] 2.1 Parse natural language instructions into structured skill scripts, including skill name, parameters, preconditions, and termination conditions;

[0062] 2.2 Automatically discover unlabeled but reusable basic skills through few-sample hints or chain reasoning hints;

[0063] 2.3 Segment the long-duration demonstration and align it to the corresponding skill tags to obtain the skill-level segment boundaries and sequence relationships;

[0064] 2.4 Standardize the representation of skill parameters by incorporating environmental condition terminology (such as "left sleeve", "lower edge", "center line").

[0065] 3) Construct a basic skills dataset for material manipulation based on the decomposition results, and pair language descriptions, image states, and corresponding action labels;

[0066] 3.1 Extract material morphology features from image sequences, including but not limited to edges / corners, wrinkle direction fields, and key points;

[0067] 3.2 Establish a language-image-action triplet, where the action label includes at least the grab point, the placement point and its pose / orientation, the folding vector or the target trajectory;

[0068] 3.3 Perform time alignment and interpolation on skill-level clips, and perform data augmentation, including viewpoint, lighting, texture, and material perturbations;

[0069] 3.4 Organize the above samples into a supervised training sample set for single-step skill learning.

[0070] 4) Employ a Transformer-based skill learning model, inputting material images and skill-level language instructions, outputting target points or key action parameters for operations such as grasping / placing, and training to form a single-step skill model;

[0071] 4.1 A cross-modal fusion structure of visual encoder and language encoder is adopted, and cross-attention is used to align image features and language features;

[0072] 4.2 The model output includes a capture / placement heatmap, a set of key points, and / or end effector pose parameters;

[0073] 4.3 With position / pose regression and keypoint matching as the goal, L1 / L2 and cross-entropy loss are jointly adopted, and executability constraints are introduced to ensure the reachability of the gripper and the safe distance from the material edge;

[0074] 4.4 Fine-tune the single-step skill model to improve robustness through strategies such as behavioral cloning and / or DAgger.

[0075] 5) When performing a new task, the large language model acts as a high-level planner, automatically generating a basic skill sequence based on the task description and current observations, and calling the single-step skill model in the skill library to complete the operation step by step;

[0076] 5.1 The high-level planner generates a basic skill sequence based on the task description and current visual observations, and performs a closed-loop update based on new observations after each step.

[0077] 5.2 Set failure detection and rollback rules, and call recovery skills such as "re-tilt" and "expand" to correct errors when deviations are detected;

[0078] 5.3 During the skill splicing process, the preconditions / termination conditions and parameter passing strategies are followed, and the required parameters are filled and constrained by the large language model or the underlying perception module.

[0079] 6) Execute the skill sequence on both simulated and real robot platforms to achieve generalized execution of tasks involving unseen new materials.

[0080] 6.1 Perform domain randomization training on the simulation platform, including texture, lighting, friction coefficient and gravity perturbation;

[0081] 6.2 By using camera calibration and pose transformation, the grasping / placement strategies learned in the simulation are transferred to the real robot;

[0082] 6.3 Perform online threshold and gripper opening / closing parameter calibration on a real platform to adapt to different materials and thicknesses.

[0083] Example 1

[0084] See Figure 2This embodiment describes a robot material manipulation generalization learning task. First, a dataset is constructed containing robot demonstration videos, image sequences, and natural language commands. The tasks cover various material manipulations such as folding towels, tidying T-shirts, and making bed sheets. A large language model is used to perform semantic parsing and task decomposition of the language commands. By inputting three labeled samples as prompts, or using chained reasoning prompts, the large language model is guided to automatically discover unlabeled but reusable basic skills, and complex operations (such as "fold a towel into thirds and place it on the table") are decomposed into basic skills such as "grasp the edge—fold—repeatedly fold—place on the table". Subsequently, image states, language skill descriptions, and corresponding action parameters are combined to form language-image-action triples, constructing a skill-level dataset. A Transformer-based cross-modal model takes images and skill descriptions as input and outputs the coordinates of the grasping or placing point, training a single-step skill model. Executability constraints ensure the results are applicable to the robot. When executing a new task, the large language model acts as a high-level planner, generating a skill sequence based on the task description and progressively calling upon the skill library for execution. After each operation, the system updates subsequent actions using the latest observations and invokes recovery skills such as "unfold" and "re-tilt" when gripping deviations or material slippage occur. In the simulation environment, domain randomization is performed through texture, lighting, and physical perturbations to improve the model's generalization ability; on a real robot, camera calibration and gripper parameter adjustment are used to adapt to different materials. Experiments show that this method can generate reasonable skill sequences and stably complete operations in unseen tasks, achieving generalized execution of tasks with complex materials.

[0085] The above description is only a preferred embodiment of the present invention. Modifications may be made within the scope defined by the claims of the present invention, but all such modifications shall fall within the protection scope of the present invention.

Claims

1. A language-driven generalization learning method for manipulating deformable robots, characterized in that, The method includes the following steps: 1) Collect long-term demonstration data of robot material operation, obtain the corresponding image sequence and language command or description, and perform time alignment on the image sequence and language command or description; 2) Utilize a large language model to perform semantic parsing and task decomposition on the language instructions, breaking down the overall operation into multiple general basic skills; 3) Construct a basic skills dataset for material manipulation based on the decomposition results, and pair language descriptions, image states, and corresponding action labels; 4) A Transformer-based skill learning model is adopted. The input is a material image obtained from image sequence recognition and a language instruction or description. The output is the grasping or placement operation target point or key action parameters, and the model is trained to form a single-step skill model. 5) When performing a new task, the large language model acts as a high-level planner, automatically generating a basic skill sequence based on the task description and current observations, and calling existing single-step skill models in the public skill library to complete the operation step by step; 6) Execute the language instructions or descriptions on the simulation and real robot platforms to achieve generalized execution of new material manipulation tasks.

2. The robot deformable body manipulation generalization learning method according to claim 1, characterized in that, Step 2) specifically includes: 2-1 Parse natural language instructions into structured skill scripts, including skill name, parameters, preconditions and termination conditions; 2-2 Automatically discover unlabeled but reusable basic skills using 2 to 8 sample hints or chained reasoning hints; 2-3 Segment the long-duration demonstration and align it to the corresponding skill tags to obtain the skill-level segment boundaries and sequence relationships; 2-4 Standardize the representation of skill parameters by combining environmental condition terms; among which, environmental condition terms include left sleeve, lower edge, and center line.

3. The robot deformable body manipulation generalization learning method according to claim 1, characterized in that, Step 3) specifically includes: 3-1 Extract material morphology features from image sequences, including edges and corners, wrinkle direction fields, and key points; 3-2 Establish language-image-action triples, where the action label includes at least the grab point, the placement point and its pose and orientation, and the folding vector or target trajectory; 3-3 Perform time alignment and interpolation on skill-level clips, and perform data augmentation, including viewpoint, lighting, texture, and material perturbations; 3-4 Organize the language-image-action triplet and skill-level segments into a supervised training sample set for single-step skill learning.

4. The robot deformable body manipulation generalization learning method according to claim 1, characterized in that, Step 4) specifically includes: 4-1 A cross-modal fusion structure of visual encoder and language encoder is adopted, and cross-attention is used to align image features and language features; 4-2 Model output includes capture or placement heatmaps, key point sets, and / or end effector pose parameters; 4-3 With the goal of position and pose regression and key point matching, L1 and L2 norms and cross-entropy loss are jointly used, and executability constraints are introduced to ensure the reachability of the gripper and the safe distance of the material edge. 4-4 Fine-tuning Transformer-based single-step skill models to improve robustness through behavioral cloning or DAgger strategies.

5. The robot deformable body manipulation generalization learning method according to claim 1, characterized in that, Step 5) specifically includes: 5-1 The high-level planner generates a basic skill sequence based on the task description and current visual observations, and performs a closed-loop update based on the new observation image after each step is executed; 5-2 Set up failure detection and rollback rules to call over-lay or expand recovery skills to correct errors when deviations are detected; 5-3 During the skill splicing process, the preconditions or termination conditions and parameter passing strategies are followed, and the required parameters are filled and constrained by the large language model or the underlying perception module.

6. The robot deformable body manipulation generalization learning method according to claim 1, characterized in that, Step 6) specifically includes: 6-1 Perform domain randomization training on the simulation platform, including texture, lighting, friction coefficient and gravity perturbation; 6-2 By using camera calibration and pose transformation, the grasping or placement strategies learned in simulation are transferred to real robots; 6-3 Perform online threshold and gripper opening / closing parameter calibration on a real platform to adapt to different materials and thicknesses for different operations.

Images