Robot operation generalization method and device based on self-evolution skill library

Abstract

This invention proposes a robot operation generalization method based on a self-evolving skill library, comprising: the robot receiving task instructions and observations; the system reading and determining, based on the robot's basic skill library and the observations, whether the robot's basic skills cover the behavior chain required by the task instruction; if so, executing a generation step; otherwise, executing an extension step; the generation step generating policy code corresponding to the behavior chain for the robot to execute; and the extension step generating extended skills based on a robot skill demonstration video, supplementing the basic skill library with these extended skills, and providing the robot with the corresponding policy code based on the extended skills and basic skills. This invention can achieve stronger manipulation generalization capabilities and higher task execution accuracy in complex and unseen environments, and possesses a clear engineering implementation path and resource controllability.

Description

Technical Field

[0001] This invention relates to the fields of robot learning, image recognition, embodied intelligence and visual language action model technology, and particularly to a robot operation generalization method, device, electronic device, computer-readable storage medium and computer program product based on a self-evolving skill library. Background Technology

[0002] With the development of vision-language-action models, code planning models, and foundational models, enabling robots to complete long-term, multi-step tasks according to natural language instructions has become an important research direction in embodied intelligence. Compared to task-centered methods that directly output actions end-to-end, skill-centered methods typically break down complex tasks into several atomic skills, which are then executed by the planner calling the skill interfaces. Therefore, they have advantages in reusability, interpretability, and combinatorial generalization. However, most existing skill-centered methods are built on a fixed skill library. When deployed, robots can usually only call pre-defined skill interfaces, such as grasping, placing, and moving. Once user instructions involve new skills not explicitly included, such as "wiping the table," "folding clothes," or "stirring blocks," the system often cannot complete the task, or requires manual demonstration, manual writing of new interfaces, and manual annotation of trajectories before it can continue to be used. This results in a significant bottleneck in the adaptability of existing solutions to new tasks.

[0003] Currently, representative research routes for humanoid robot skill planning and expansion mainly include:

[0004] 1) Code generation methods based on fixed skill APIs. This type of method uses a large language model to decompose natural language tasks into skill call sequences in code form. Its advantages are clear reasoning chains and easy integration with existing robot control systems; however, its planning ability is limited by a predefined set of skills. When the task exceeds the capability boundaries of the library, the generated plan, although formally correct, cannot be implemented.

[0005] 2) Trajectory generation methods based on visual cues or key point inference. These methods typically rely on large visual models to directly infer contact points, critical paths, or operation trajectories in the current scene, which is effective for trajectory-related tasks. However, due to the lack of structured skill knowledge and hierarchical example retrieval, they are still insufficient in long-term task decomposition, complex spatial constraint modeling, and generalization of unseen skills.

[0006] 3) New skill infusion methods based on human instruction. This type of method allows users to inform the system of the meaning of new skills through dialogue, demonstration, or human feedback, and then the robot learns the new interface. Although it can overcome the limitations of a fixed skill set to some extent, the deployment phase still heavily relies on human intervention, resulting in high costs, low efficiency, and difficulty in sustained large-scale expansion.

[0007] The inventors discovered in their research that massive amounts of robot videos naturally contain rich skill demonstration data. However, most of this data is unstructured, lacking a one-to-one correspondence with specific skills and a unified hierarchical semantic tagging system. Therefore, it cannot be efficiently used by existing systems for "missing skill completion" and "automatic implementation of new skills." Thus, how to automatically determine whether the robot's existing skills are sufficient when receiving a new task, how to automatically generate and implement new skill descriptions, and how to continuously expand the skill library using unstructured videos have become key issues restricting the improvement of the robot's generalization capabilities. Summary of the Invention

[0008] To address the problems of existing skill-centered robot operation methods that rely on fixed skill libraries, cannot automatically expand when faced with unseen tasks or skills, still require manual teaching or manual annotation during deployment, and are difficult to convert large-scale unstructured robot videos into reusable skill knowledge, this paper proposes a self-evolving skill library construction method for robot generalization operations. This method enables robots to adaptively plan for long-term natural language tasks, automatically complete missing skills, and continuously grow the skill library.

[0009] To address the aforementioned issues, this application constructs a unified framework of "skill perception planning + automatic skill evolution + hierarchical skill library retrieval." In the planning phase, the system first determines whether the basic skills are sufficient to complete the target task; if insufficient, it automatically generates supplementary skill descriptions and incorporates these supplementary skills into the current task's set of available extended skills. In the execution phase, the system retrieves sample fragments related to the extended skill from a hierarchical skill library constructed from large-scale unstructured robot videos, extracts semantic constraints and trajectory references, and then generates executable operation trajectories and pose sequences, thereby enabling the deployment of new skills without the need for manual instruction during the deployment period.

[0010] Specifically, such as Figure 4 As shown, this invention proposes a robot operation generalization method based on a self-evolving skill base, which includes:

[0011] In the initial steps, the robot receives task instructions and observations; the system reads these and, based on the robot's basic skill library and the observations, determines whether the robot's basic skills cover the behavior chain required by the task instructions. If so, the generation step is executed; otherwise, the extension step is executed.

[0012] The generation step generates the strategy code corresponding to the behavior chain for the robot to execute.

[0013] The extension step involves generating extended skills based on the robot's skill demonstration video, adding these extended skills to the basic skill library, and then providing the robot with the corresponding strategy code based on the extended skills and basic skills.

[0014] The robot operation generalization method based on a self-evolving skill base, wherein the extension step supplements the basic skill base with the extended skills, specifically includes:

[0015] Each skill s in this basic skill set is represented by H(s) = → → , Indicates the type of sport. Indicates semantic category, Represents a skill instance;

[0016] Calculate the mechanical attributes of the extended skill and the similarity between each movement category, select the movement category with the highest similarity as the movement category of the extended skill, assign a semantic category to the extended skill based on the action intention and role relationship of the extended skill, and finally use the object, tool and trajectory features of the extended skill as skill instances.

[0017] The robot operation generalization method based on a self-evolving skill library, wherein the extension step generates extended skills based on robot skill demonstration videos, specifically includes:

[0018] Get robot skill demonstration video ;in, This represents the visual observation in frame t. Indicates the state of the robot itself. Indicates the pose of the end effector or gripper. This indicates the corresponding action signal.

[0019] Demonstration video of using robot skills The robot locates skill switching points by analyzing state changes during execution; Gaussian smoothing is applied to the velocity sequence, and local minima of the smoothed velocity are selected as keyframes, provided that minimum interval constraints and maximum number constraints are applied to the keyframes.

[0020] Input the sequence of these keyframes into the visual language model, and require the model to output a step-by-step process description in chronological order:

[0021]

[0022] in, The process description represents the i-th action step. Based on the process description and the existing basic skill library, the visual language model determines whether the process description is covered by the basic skill library. If it is, the process description is replaced with the skill semantic description of the basic skill library; otherwise, a new skill semantic description is generated for the process description.

[0023] Each keyframe is labeled with an integer index and the number is displayed in the image; a dense description is generated for each keyframe; given a skill semantic description s and a set of keyframe descriptions, the model predicts the start and end frame intervals corresponding to the skill, extracts visual segments and trajectory segments corresponding to the skill from the original video, and adds them to the basic skill library.

[0024] like Figure 5 As shown, the present invention also proposes a robot operation generalization device based on a self-evolving skill base, comprising:

[0025] In the initial module, the robot receives task instructions and observations. The system reads these and, based on the robot's basic skill library and the observations, determines whether the robot's basic skills cover the behavior chain required by the task instructions. If so, the generation module is executed; otherwise, the extension module is executed.

[0026] The generation module generates the strategy code corresponding to the behavior chain for the robot to execute.

[0027] The extension module generates extended skills based on robot skill demonstration videos, adds these extended skills to the basic skill library, and provides the robot with the corresponding strategy code based on the extended skills and basic skills.

[0028] The robot operation generalization method based on a self-evolving skill library, wherein the extension module supplements the basic skill library with the extended skills, specifically includes:

[0029] Each skill s in this basic skill set is represented by H(s) = → → , Indicates the type of sport. Indicates semantic category, Represents a skill instance;

[0030] Calculate the mechanical attributes of the extended skill and the similarity between each movement category, select the movement category with the highest similarity as the movement category of the extended skill, assign a semantic category to the extended skill based on the action intention and role relationship of the extended skill, and finally use the object, tool and trajectory features of the extended skill as skill instances.

[0031] The robot operation generalization device based on a self-evolving skill library, wherein the extended module generates extended skills based on robot skill demonstration videos, specifically including:

[0032] Get robot skill demonstration video ;in, This represents the visual observation in frame t. Indicates the state of the robot itself. Indicates the pose of the end effector or gripper. This indicates the corresponding action signal.

[0033] Demonstration video of using robot skills The robot locates skill switching points by analyzing state changes during execution; Gaussian smoothing is applied to the velocity sequence, and local minima of the smoothed velocity are selected as keyframes, provided that minimum interval constraints and maximum number constraints are applied to the keyframes.

[0034] Input the sequence of these keyframes into the visual language model, and require the model to output a step-by-step process description in chronological order:

[0035]

[0036] in, The process description of the i-th action module is represented. Based on the process description and the existing basic skill library, the visual language model determines whether the process description is covered by the basic skill library. If it is, the process description is replaced with the skill semantic description of the basic skill library. Otherwise, a new skill semantic description is generated for the process description.

[0037] Each keyframe is labeled with an integer index and the number is displayed in the image; a dense description is generated for each keyframe; given a skill semantic description s and a set of keyframe descriptions, the model predicts the start and end frame intervals corresponding to the skill, extracts visual segments and trajectory segments corresponding to the skill from the original video, and adds them to the basic skill library.

[0038] The present invention also proposes a client for implementing any of the robot operation generalization devices based on a self-evolving skill library.

[0039] The present invention also proposes an electronic device, including the aforementioned robot operation generalization device based on a self-evolving skill library, wherein the electronic device may be connected to an information display device, which is used to display the extended skill with user-set display parameters, attributes or through an artificial intelligence model.

[0040] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the robot operation generalization method based on the self-evolving skill base.

[0041] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the robot operation generalization method based on the self-evolving skill library.

[0042] As can be seen from the above solutions, the advantages of the present invention are:

[0043] 1) Breaking the limitations of a fixed skill library. Through a closed-loop mechanism of "discrimination-generation-planning," the system can automatically supplement skill descriptions for missing abilities in a task and immediately incorporate them into the planning process, enabling the robot to have self-expanding capabilities for unseen tasks. The skill library is not a static list, but a hierarchical knowledge structure that can be automatically added as new video examples are added and continuously expanded as new skills are invoked, making it more suitable for long-term online deployment and cross-scene migration of the robot.

[0044] 2) Significantly reduces the cost of acquiring new skills. By automatically segmenting, describing, and aligning large-scale unstructured robot videos, this invention can accumulate a large number of skill examples offline, reducing the reliance on manual teaching and annotation during the deployment phase. The retrieval time is optimized from O(N) complexity to O(1), greatly reducing the time used for matching and improving the semantic relevance between examples and actual executed skills, modeling an efficient retrieval mechanism centered on dynamic mechanisms and semantic features.

[0045] 3) Enhance the generalization ability of long-term tasks and trajectory-based tasks. Since the implementation of extended skills does not rely solely on single visual inference, but simultaneously utilizes hierarchical semantic retrieval results organized by behavior category and semantic category, it efficiently captures semantic constraints and trajectory reference information corresponding to skill examples. Therefore, it has stronger spatial reasoning ability and execution stability in tasks such as revolving doors, drawers, folding, wiping, and stirring.

[0046] 4) The experimental results are significant. The experiments in this paper show that in RLBench, for 10 types of tasks that exceed the basic skill set, the average success rate of the method corresponding to this invention reaches 0.41, which is significantly better than the 0.01 of the CaP method and the 0.10 of the MOKA method; on 8 types of real robot tasks, the average success rate reaches about 0.73, which is higher than the approximately 0.29 of CaP and the 0.39 of MOKA, indicating that this invention has good zero-shot generalization ability in both simulation and real-world scenarios.

[0047] In summary, this application takes the hierarchical manipulation skill framework based on dynamics-semantic integrated modeling as the system innovation point, and builds an scalable foundation for embodied manipulation tasks around the three goals of "openness, continuity and efficiency". It can achieve stronger manipulation generalization ability and higher task execution accuracy in complex and unseen environments, and has a clear engineering implementation path and resource controllability. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the research scheme of the present invention;

[0049] Figure 2 A schematic diagram of a hierarchical skill framework for behavior-semantic encoding;

[0050] Figure 3Schematic diagram of the automated skill semantic annotation module

[0051] Figure 4 This is a flowchart of the method of the present invention;

[0052] Figure 5 This is a block diagram of the device of the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0054] Figure 7 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0055] Figure 8 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0056] Figure label:

[0057] A - First electronic device;

[0058] B- A robot operation generalization device based on a self-evolving skill library;

[0059] C-Data acquisition equipment;

[0060] D-Information display device;

[0061] 1000 - Second electronic device;

[0062] Ⅰ-Computational Unit;

[0063] II-ROM;

[0064] III-RAM;

[0065] N-bus;

[0066] V-Interface;

[0067] VI - Input Unit;

[0068] VII - Output Unit;

[0069] VIII - Storage medium;

[0070] IX - Communication Unit. Detailed Implementation

[0071] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0072] Without further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0073] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0074] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0075] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0076] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0077] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0078] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0079] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0080] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0081] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0082] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] Based on the above understanding, this invention proposes a unified skill center framework for robot generalization operations, addressing the closed-loop requirements of "determinable skill deficiencies—generable new skills—sustainable evolution of the skill library," and coordinating skill completion in the task planning phase with skill implementation in the execution phase:

[0087] Skill perception and planning module: Starting from the existing basic skill library, and combining the task understanding and code planning capabilities of the vision-language big model, the module decomposes the natural language instructions received by the robot and determines whether the skills required to complete the current task are fully covered by the existing skill library. When a skill gap is detected, the system automatically generates a corresponding new skill description, including skill function, applicable objects, input parameters, and expected output form. This allows the new skills to be added into the original skill library in a structured manner, thereby breaking through the limitation of fixed skill sets on task generalization ability and providing a unified semantic anchor for subsequent skill retrieval and implementation.

[0088] A hierarchical framework for dynamic modeling: The transferability between skills needs to be expressed in a structured parameter space. Its hierarchical skill modeling utilizes motion codes to characterize mechanical properties, the VerbNet framework to characterize semantic roles, and skill instances as fine-grained implementation nodes, thus forming a skill hierarchy that can be used for retrieval, planning, and policy conditionation. Using a skill hierarchy with equivalence between dynamics and semantics can better characterize the similarities and differences between skills, reducing category overlap caused by modeling only semantics, thereby achieving a more relevant and interpretable manipulation skill hierarchy in terms of dynamic characteristics.

[0089] The automatic skill evolution module utilizes skill demonstration information from massive amounts of unstructured robot videos to construct a hierarchical semantic organization skill library structure. This structure maps skill categories, skill descriptions, and demonstration fragments at different abstraction levels. Based on this, for new skill descriptions proposed by the planning module, it automatically retrieves semantically relevant demonstration samples and trajectory fragments from the skill library. This provides a reference for inferring the skill's action pattern and generating its spatial trajectory, thereby enabling automatic implementation and continuous completion of new skills with limited samples. This closed-loop mechanism eliminates the need for additional manual teaching for each new skill during the deployment phase, allowing the robot to continuously expand its skill boundaries and improve its generalization capabilities in new tasks and scenarios.

[0090] Combining a self-evolutionary planning framework based on skill base planning with a hierarchical skill base modeled by dynamics mechanisms can simultaneously solve the problems of fixed skill bases, low example retrieval efficiency, difficulty in implementing skills outside the skill base, and difficulty in reusing similar skill parameters in policy models.

[0091] In summary, the present invention proposes the following key technical points:

[0092] Key Point 1: A skill-aware planning mechanism is proposed. After receiving visual observations and natural language instructions, the system first uses a skill sufficiency assessment module to determine whether the basic skill set meets the task requirements. When the assessment result is insufficient, the skill generation module automatically generates a new skill function description, including the skill name, functional description, input parameters, and return form. Subsequently, the planning module generates a coded operation plan based on a self-enhancing skill set of "basic skills + extended skills." Through this mechanism, the robot can break through the boundaries of a fixed skill set and achieve adaptive task decomposition and planning for unseen tasks.

[0093] Key Point 2 proposes an automatic skill annotation workflow for unstructured videos. This workflow utilizes a visual language model to sequentially complete three types of processing: process extraction, skill description generation, and time alignment. First, it extracts step-by-step operation descriptions from the original robot video. Then, it determines whether new skill descriptions are needed based on a basic skill set. Finally, it automatically aligns the skill descriptions with the video's time interval to form a demonstration segment with skill tags. This workflow can transform unstructured video data into searchable samples for skill retrieval, significantly reducing the cost of manual processing and annotation.

[0094] Key Point 3 proposes a hierarchical skill library construction method that balances dynamic modeling mechanisms and semantic categories. The skill library includes dynamic categories clustered using six action encoding methods, semantic categories centered on VerbNet, and a skill instance layer containing a large number of execution trajectories extracted from real-world execution videos. The higher-level dynamic categories encode more coarse-grained dynamic features, while the lower-level semantic categories are filtered VerbNet categories used to organize skill instances relevant to specific targets and scenarios. Through top-down matching and dynamic expansion mechanisms, the system can continuously incorporate newly labeled video samples into the skill library, forming a skill knowledge base that grows with data accumulation.

[0095] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0096] like Figure 1 As shown, this implementation focuses on the efficient organization and expansion of a skill library for machine manipulation tasks, emphasizing "openness, continuity, and efficiency." The goal is to achieve stronger manipulation generalization capabilities and higher task execution accuracy in complex and unseen environments. The specific solution revolves around three modules: the overall system flow, hierarchical skill library modeling combining behavior and semantics, and an efficient skill semantic annotation module.

[0097] (1) Overall system flow

[0098] like Figure 1 As shown, when performing a specific task, the robot receives task instructions τ and observations O. Observations O may include multi-view images, depth maps, target detection boxes, robot joint states, and end effector states. The end effector refers to the actuator installed at the end of the robotic arm and directly interacting with the outside world, such as a gripper, wiping tool, or other manipulating tool. The system reads the basic skill library. Each skill is described in a unified format, including skill name, function description, parameter list, return value format, and invocation constraints. The parameter list specifies the information required to invoke the skill, such as target detection boxes and tool object detection boxes. The return value format specifies the data type of the output after skill planning, such as target object pose, waypoints, and six-DOF end-effector pose sequences, enabling different skills to be combined using a unified interface. Constraints include, for example, contact constraints and waypoint constraints. Next, the skill library sufficiency discriminator E combines task instructions, visual observations, and... Alignment is performed by segmenting the visual observation (video) into different segments according to the semantics of the task instructions, with each segment corresponding to a single skill, and then aligning it with the basic skill library. Align / correspond to the basic skills in the text, and make judgments. Can the basic skills in the system cover the skill sequence required for the task? If they can, the planner P directly generates strategy code composed of basic skills; if they cannot, the skill generator G generates extended skills. And for each extended skill The visual language model generates natural language descriptions, input parameters, and a return trajectory format based on the instructions and image sequence information of the skill segment. The planner then... , Generate call sequence , :

[0099]

[0100] Among them, skill call sequence It is the structured expression of the task execution program, and the strategy code is the executable code form of the call sequence. The two are in a one-to-one correspondence or can be converted into each other. Indicates the number of skill calls required to complete the task; This indicates the i-th skill interface that is invoked, which can come from the basic skill library. Or expand the skill set ; This represents the input parameters when calling the i-th skill.

[0101] System construction and Both are tree-structured hierarchical skill libraries, where each skill s in the skill library is represented as a hierarchical index path H(s) = → → The first layer Indicates the motion category, used to describe the motion attributes of a skill, such as translation, rotation, etc.; second layer Represents semantic categories, that is, the specific categories of skills within a semantic classification framework; the third layer. This represents a skill instance, i.e., the skill or operation corresponding to skill s itself. For example, for the skills push and pull, the skill instances in the third layer are "push" and "pull" respectively, but the first two layers are the same. This hierarchical structure serves as both an index for new skill retrieval and a conditional input for the strategy model. The strategy model is used to retrieve relevant skills from the skill library or create new skills, taking skill trajectories, relevant constraints, target scenarios, etc., as input, and outputting operational information for the entire skill execution process, including trajectory, gripper pose, etc. For extended skills, the system searches for nodes in the hierarchical skill library that are similar to its mechanical attributes and semantic roles, and returns similar demonstration clips. If a similar skill cannot be matched in the hierarchical skill library, a new skill instance node is created, waiting for examples to be automatically supplemented from robot videos and trajectories later. That is, if the hierarchical skill library lacks a corresponding demonstration video, it needs to wait until the videos in the hierarchical skill library are supplemented before continuing. Finally, the execution trajectory of the new skill is generated. The strategy model extracts contact constraints, waypoint constraints, and rotation patterns from the retrieved examples, and combines this with the object positions, depths, and reachability in the target scene to generate a sequence of contact points, waypoints, and 6-DOF poses. This sequence is encapsulated as the return result of the extended skill API for the planner to use. Online embodied perception module.

[0102] (2) Hierarchical skill set for behavior-semantic modeling (self-evolving skill set)

[0103] like Figure 2 As shown, the hierarchical skill base is constructed using a behavior-semantic hybrid skill modeling approach. Motion codes are used to characterize the mechanical attributes of skills. A motion code can consist of multiple discrete dimensions, including contact time, whether deformation occurs, whether the robotic arm rotates around a fixed axis, whether the object rotates around a fixed axis, whether a tool is used, and the object's translation method. Different dimensions can be assigned different weights. The system uses weighted Hamming distance to measure the mechanical similarity between skills and obtains several representative motion categories through spectral clustering. .

[0104] Semantic categories are constructed using the VerbNet framework. The system parses skill verb phrases and maps them to VerbNet categories and semantic role structures, such as agent, recipient, tool, target location, material, or outcome state. This semantic category... It is used to express the action intention and role relationship of skills, making up for the inability of motion codes to express task semantics.

[0105] At the skill instance level The system stores specific skill descriptions, target object constraints, parameter templates, example fragment indexes, and trajectory statistics. A directed acyclic structure is formed between motion categories, semantic categories, and skill instances. A motion category can connect to multiple semantic categories, and a semantic category can connect to multiple skill instances; the same skill instance can also be connected to multiple higher-level nodes based on polysemous verbs or compound actions.

[0106] When a new labeled fragment enters the system, it first matches the motion category based on the motion code, then matches the semantic category based on the VerbNet parsing results, and finally matches the skill instance based on object, tool, and trajectory features. If no suitable node is available at any layer, a new node is created at that layer and the fragment is attached to the new node. Through this mechanism, the skill library can be continuously expanded as robot videos and execution feedback occur.

[0107] (3) Hierarchical skill set for behavior-semantic modeling

[0108] like Figure 3 As shown, to reduce the reliance on manual annotation and teaching when building a skill library, this invention sets up an efficient skill semantic annotation module. This module automatically extracts semantic segments related to skills from unstructured robot demonstration videos and generates corresponding skill descriptions, time intervals, and example trajectories for each skill. The time interval is used to mark the start and end of the current skill in the entire video, accurately dividing the corresponding segments. The example trajectories of the skills can be used by the policy model to learn the corresponding operation methods. This module includes five steps: keyframe extraction, process description generation, skill semantic generation, time alignment, and database update. Let an original demonstration sequence be: .in, This represents the visual observation in frame t. Indicates the state of the robot itself. Indicates the pose of the end effector or gripper. This indicates the corresponding action signal. This demonstration sequence typically contains multiple consecutive sub-actions, such as approaching the target, grabbing the tool, moving the tool, performing a wipe, etc.

[0109] Keyframes are extracted based on changes in physical state. Unlike uniform sampling, this invention utilizes speed changes during robot execution to locate skill switching points, thereby segmenting skill fragments. Specifically, the speed change is determined by Gaussian smoothing of the speed sequence, selecting extreme points as keyframes, and a constraint on the total number of keyframes. Skill switching points refer to significant transitions between different skills; for example, when the action "push" switches to "pick," the gripper moves from the previous object to the next, which is a clear switching point. Keyframes serve as candidates for switching points, and further filtering yields the desired skill switching points. To suppress sensor jitter and trajectory noise, the speed sequence at the robot arm's end effector is Gaussian smoothed, and local minima of the smoothed speed are selected as candidate keyframes. Since switching robot skills or sub-steps is often accompanied by short pauses, changes in contact state, or changes in motion direction, local minima of speed effectively reflect the boundaries of action stages. Furthermore, to adapt to the context length limitations of the visual language model, minimum interval constraints and maximum number constraints are applied to the candidate keyframes. The maximum number of candidate frames can be set to 16 to ensure that the number of keyframes does not exceed the context capacity of a single inference by the visual language model.

[0110] After extracting keyframes, a process-level action description needs to be generated. The extracted keyframe sequence is input into the visual language model, which is required to output a step-by-step process description in chronological order.

[0111]

[0112] in, The process description represents the i-th action step, such as "approach the towel," "grab the towel," "move to the table," or "wipe the table." This process description is used to transform the original long video into a structured action flow, avoiding semantic confusion caused by directly retrieving skills from the complete video. Based on the process description and the existing basic skill library, the visual language model determines whether the current action step can be covered by existing basic skills; if not, it generates a new skill semantic description and a unified format.

[0113] After generating additional skill descriptions, time alignment is required. For example, when the skills contained in the video start and end frame segments correspond only to the current skill, the time alignment of the current skill is complete. To enhance the model's understanding of keyframe temporal relationships, this invention employs an explicit numbering strategy, labeling each keyframe with an integer index and displaying this number in the image. This number can also be used for rapid keyframe location and extraction. For example, reading keyframe number 3 allows for quick indexing to frame number 339, thus locating the video frame for that frame. Simultaneously, a dense description is generated for each keyframe. Subsequently, given the skill semantic description *s* and the set of keyframe descriptions, the visual language model predicts the start and end frame time intervals corresponding to the skill for time alignment. This allows for the extraction of visual segments corresponding to the skill from the original video, and further, the acquisition of the end effector trajectory segments through the visual segments of the current skill. This time alignment process can break down multiple interwoven actions in long-range demonstrations into skill-level samples, avoiding the incorrect addition of irrelevant action segments to the skill library. The skill library has a tree-like index structure. It first indexes the desired skill by the first level (motion category), then by the second level (semantic category), and finally by the third level (skill instance). Each skill stores visual information (visual fragments, trajectory fragments) and strategy code (skill name, function description, parameter list, return value format, and invocation constraints). The purpose of the visual information is to generate the strategy code that can be invoked during execution.

[0114] Finally, the obtained skill fragments undergo consistency checks, including fragment length, action completeness, target object consistency, trajectory validity, and semantic confidence. Skill fragments that pass the screening are added to a hierarchical skill library. Through this process, the system can automatically convert unstructured robot videos into skill-level semantic samples, providing high-quality data for subsequent skill retrieval, new skill implementation, and policy model updates. The technical effects of this module are: First, by utilizing local minima of velocity instead of uniform sampling to extract keyframes, the efficiency of skill boundary capture is improved; second, by enhancing the temporal understanding capability of the visual language model through explicit numbering and dense description, the alignment accuracy between skill descriptions and video fragments is improved; third, by synchronously storing skill semantics, time intervals, and execution trajectories in the library, the skill library can be continuously expanded without large-scale manual annotation.

[0115] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0116] like Figure 5 As shown, the present invention also proposes a robot operation generalization device based on a self-evolving skill base, comprising:

[0117] In the initial module, the robot receives task instructions and observations. The system reads these and, based on the robot's basic skill library and the observations, determines whether the robot's basic skills cover the behavior chain required by the task instructions. If so, the generation module is executed; otherwise, the extension module is executed.

[0118] The generation module generates the strategy code corresponding to the behavior chain for the robot to execute.

[0119] The extension module generates extended skills based on robot skill demonstration videos, adds these extended skills to the basic skill library, and provides the robot with the corresponding strategy code based on the extended skills and basic skills.

[0120] The robot operation generalization method based on a self-evolving skill library, wherein the extension module supplements the basic skill library with the extended skills, specifically includes:

[0121] Each skill s in this basic skill set is represented by H(s) = → → , Indicates the type of sport. Indicates semantic category, Represents a skill instance;

[0122] Calculate the mechanical attributes of the extended skill and the similarity between each movement category, select the movement category with the highest similarity as the movement category of the extended skill, assign a semantic category to the extended skill based on the action intention and role relationship of the extended skill, and finally use the object, tool and trajectory features of the extended skill as skill instances.

[0123] The robot operation generalization device based on a self-evolving skill library, wherein the extended module generates extended skills based on robot skill demonstration videos, specifically including:

[0124] Get robot skill demonstration video ;in, This represents the visual observation in frame t. Indicates the state of the robot itself. Indicates the pose of the end effector or gripper. This indicates the corresponding action signal.

[0125] Demonstration video of using robot skills The robot locates skill switching points by analyzing state changes during execution; Gaussian smoothing is applied to the velocity sequence, and local minima of the smoothed velocity are selected as keyframes, provided that minimum interval constraints and maximum number constraints are applied to the keyframes.

[0126] Input the sequence of these keyframes into the visual language model, and require the model to output a step-by-step process description in chronological order:

[0127]

[0128] in, The process description of the i-th action module is represented. Based on the process description and the existing basic skill library, the visual language model determines whether the process description is covered by the basic skill library. If it is, the process description is replaced with the skill semantic description of the basic skill library. Otherwise, a new skill semantic description is generated for the process description.

[0129] Each keyframe is labeled with an integer index and the number is displayed in the image; a dense description is generated for each keyframe; given a skill semantic description s and a set of keyframe descriptions, the model predicts the start and end frame intervals corresponding to the skill, extracts visual segments and trajectory segments corresponding to the skill from the original video, and adds them to the basic skill library.

[0130] The present invention also proposes a client for implementing any of the robot operation generalization devices based on a self-evolving skill library.

[0131] like Figure 6 As shown, in another embodiment of the present invention, a first electronic device A is proposed, which includes the aforementioned robot operation generalization device B based on a self-evolving skill library.

[0132] like Figure 7 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through wired or wireless information transmission schemes. The data acquisition device C is used to collect task instructions and observations, and the information display device D is used to display the robot execution results or extended skills obtained by the present invention.

[0133] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with the specified key information based on user-defined display parameters and / or attributes, such as the data range and font, color, and scrolling options. Users can access this information more quickly without needing to navigate to secondary pages or scroll through pages, saving them time and effort. Alternatively, the information display mechanism can be an artificial intelligence (AI) display model that learns the user's key information interests based on past usage habits, such as viewing time, number of clicks, and number of edits, and automatically presents the user with rich and necessary key information.

[0134] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the robot operation generalization method based on the self-evolving skill library provided by the above methods.

[0135] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that executes the robot operation generalization method based on the self-evolving skill library. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0136] Figure 8A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0137] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0138] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] Computing unit I can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit I performs the various methods and processes described above, such as method steps S1-S3. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0140] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A robot operation generalization method based on self-evolution skill library, characterized in that, include: In the initial steps, the robot receives task instructions and observations; the system reads these and, based on the robot's basic skill library and the observations, determines whether the robot's basic skills cover the behavior chain required by the task instructions. If so, the generation step is executed; otherwise, the extension step is executed. The generation step generates the strategy code corresponding to the behavior chain for the robot to execute. The extension step involves generating extended skills based on the robot's skill demonstration video, adding these extended skills to the basic skill library, and then providing the robot with the corresponding strategy code based on the extended skills and basic skills.

2. The robot operation generalization method based on self-evolving skill base of claim 1, wherein, This expansion step adds the expanded skill to the basic skill library, specifically including: The representation of each skill s in the base skill library is H(s) = (s, c(s), i(s)) → → , represents a motion category, represents a semantic category, represents a skill instance; Calculate the mechanical attributes of the extended skill and the similarity between each movement category, select the movement category with the highest similarity as the movement category of the extended skill, assign a semantic category to the extended skill based on the action intention and role relationship of the extended skill, and finally use the object, tool and trajectory features of the extended skill as skill instances.

3. The robot operation generalization method based on a self-evolving skill base as described in claim 1, characterized in that, This extended step generates extended skills based on the robot's skill demonstration video, specifically including: Get robot skill demonstration video ;in, This represents the visual observation in frame t. Indicates the state of the robot itself. Indicates the pose of the end effector or gripper. This indicates the corresponding action signal. Demonstration video of using robot skills The robot locates skill switching points by analyzing state changes during execution; Gaussian smoothing is applied to the velocity sequence, and local minima of the smoothed velocity are selected as keyframes, provided that minimum interval constraints and maximum number constraints are applied to the keyframes. Input the sequence of these keyframes into the visual language model, and require the model to output a step-by-step process description in chronological order: in, The process description represents the i-th action step. Based on the process description and the existing basic skill library, the visual language model determines whether the process description is covered by the basic skill library. If it is, the process description is replaced with the skill semantic description of the basic skill library; otherwise, a new skill semantic description is generated for the process description. Each keyframe is labeled with an integer index and the number is displayed in the image; a dense description is generated for each keyframe; given a skill semantic description s and a set of keyframe descriptions, the model predicts the start and end frame intervals corresponding to the skill, extracts visual segments and trajectory segments corresponding to the skill from the original video, and adds them to the basic skill library.

4. A robot operation generalization device based on a self-evolving skill base, characterized in that, include: In the initial module, the robot receives task instructions and observations. The system reads these and, based on the robot's basic skill library and the observations, determines whether the robot's basic skills cover the behavior chain required by the task instructions. If so, the generation module is executed; otherwise, the extension module is executed. The generation module generates the strategy code corresponding to the behavior chain for the robot to execute. The extension module generates extended skills based on robot skill demonstration videos, adds these extended skills to the basic skill library, and provides the robot with the corresponding strategy code based on the extended skills and basic skills.

5. The robot operation generalization method based on a self-evolving skill base as described in claim 5, characterized in that, This extension module adds the extended skill to the basic skill library, specifically including: Each skill s in this basic skill set is represented by H(s) = → → , Indicates the type of sport. Indicates semantic category, Represents a skill instance; Calculate the mechanical attributes of the extended skill and the similarity between each movement category, select the movement category with the highest similarity as the movement category of the extended skill, assign a semantic category to the extended skill based on the action intention and role relationship of the extended skill, and finally use the object, tool and trajectory features of the extended skill as skill instances.

6. The robot operation generalization device based on a self-evolving skill base as described in claim 5, characterized in that, This extension module generates extended skills based on robot skill demonstration videos, specifically including: Get robot skill demonstration video ;in, This represents the visual observation in frame t. Indicates the state of the robot itself. Indicates the pose of the end effector or gripper. This indicates the corresponding action signal. Demonstration video of using robot skills The robot locates skill switching points by analyzing state changes during execution; Gaussian smoothing is applied to the velocity sequence, and local minima of the smoothed velocity are selected as keyframes, provided that minimum interval constraints and maximum number constraints are applied to the keyframes. Input the sequence of these keyframes into the visual language model, and require the model to output a step-by-step process description in chronological order: in, The process description of the i-th action module is represented. Based on the process description and the existing basic skill library, the visual language model determines whether the process description is covered by the basic skill library. If it is, the process description is replaced with the skill semantic description of the basic skill library. Otherwise, a new skill semantic description is generated for the process description. Each keyframe is labeled with an integer index and the number is displayed in the image; a dense description is generated for each keyframe; given a skill semantic description s and a set of keyframe descriptions, the model predicts the start and end frame intervals corresponding to the skill, extracts visual segments and trajectory segments corresponding to the skill from the original video, and adds them to the basic skill library.

7. A client for implementing any of the robot operation generalization devices based on a self-evolving skill library as described in claims 4-6.

8. An electronic device, characterized in that, The invention includes a robot operation generalization device based on a self-evolving skill library as described in claims 4-6, wherein the electronic device is connected to an information display device, which is used to display the extended skill using user-set display parameters, attributes, or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the robot operation generalization method based on a self-evolving skill library as described in any one of claims 1-3.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the robot operation generalization method based on the self-evolving skill library as described in any of claims 1-3.

Images