Codeless teaching and skill transfer method and system for humanoid robot

Abstract

The application relates to the technical field of robot teaching, in particular to a code-free teaching and skill migration method and system for a humanoid robot, wherein the method comprises the following steps: collecting multi-modal teaching data in a teaching process; extracting multi-modal features of the multi-modal teaching data and fusing the multi-modal features to generate fused features; based on the fused features, dividing the teaching process into operation sub-stages through a pre-training model, and adding corresponding operation semantic labels to the operation sub-stages; constructing a framework of a skill learning model, inputting multi-modal teaching data of the operation sub-stages into the skill learning model for training; inputting operation object information and environmental obstacle information of a target scene into the trained learning model to generate a robot operation trajectory and a force control strategy, and generating a control instruction sequence based on the robot operation trajectory and the force control strategy; and issuing the control instruction sequence to a humanoid robot controller to drive the robot to perform an operation task and complete skill migration.

Description

Technical Field

[0001] This application relates to the field of robot teaching technology, and in particular to a method and system for no-code teaching and skill transfer of humanoid robots. Background Technology

[0002] The core bottleneck to the large-scale deployment of humanoid robots in industrial production and service scenarios lies in the rapid deployment of complex tasks and cross-scenario adaptability. Robot teaching is a crucial step in achieving rapid task deployment. Related robot teaching technologies mainly fall into two major directions: The first type is the traditional code programming and teach pendant teaching solution. This solution involves professional technicians writing robot control code or using a teach pendant to debug trajectory points one by one to complete the teaching deployment of the task. The core drawbacks of this type of solution are: the teaching threshold is extremely high, relying on professionals with robot programming and debugging capabilities; the teaching cycle is extremely long, with a single task teaching cycle of 3-7 days for complex multi-step operation tasks, which cannot adapt to the needs of rapid production changeover and small-batch non-standardized operations in industrial settings, and the implementation cost is extremely high.

[0003] The second type is the imitation learning teaching scheme based on single-modal data. This scheme uses a visual camera to collect the operator's motion trajectory and uses a behavior cloning algorithm to replicate the robot's trajectory without the need for complex code programming. The core flaws of this approach are as follows: First, its skill generalization ability is extremely poor. Most of these approaches merely replicate motion trajectories, utilizing only single-modal joint angle / visual trajectory data without integrating multimodal information such as vision, force, and audio. This results in insufficient semantic understanding of the operational task, and the taught skill becomes ineffective when the position, size, or posture of objects in the target scene changes, making cross-scene transfer impossible. Second, the quality of the taught data is low. Operations such as starting and stopping the process and segment marking during teaching require the operator to interrupt their hand work and control the process through the teach pendant or touchscreen. This leads to discontinuous motion trajectories that deviate significantly from actual operational behavior, severely impacting the robot's skill learning effectiveness. Third, the teaching and training processes are completely separated. Teaching data acquisition, model training, and robot execution are independent of each other, failing to achieve a real-time closed loop of "acquisition-training-execution-feedback." This results in low utilization of teaching data, poor efficiency in model iteration and optimization, and an inability to continuously improve the robot's operational capabilities through multiple teaching sessions. Summary of the Invention

[0004] The main objective of this application is to propose a no-code teaching and skills transfer method and system for humanoid robots, aiming to solve the problems in the background art.

[0005] To achieve the above objectives, one aspect of this application proposes a no-code teaching and skills transfer method for humanoid robots, the method comprising: Collect multimodal teaching data during the teaching process; The multimodal features of the multimodal teaching data are extracted and fused to generate fused features. Based on the fused features, the teaching process is divided into operation sub-stages through a pre-trained model, and corresponding operation semantic labels are added to the operation sub-stages. A skill learning model framework is constructed, and the multimodal teaching data of the operation sub-stage is input into the skill learning model. The model is then trained using an imitation learning algorithm to generate a trained skill learning model. The system acquires information about the target object and environmental obstacles, and inputs it into the trained skill learning model. It then adaptively generates a robot operation trajectory and force control strategy that are adapted to the target scene, and generates a sequence of control commands based on the robot operation trajectory and force control strategy. The control command sequence is sent to the humanoid robot controller to drive the robot to perform operation tasks and complete skill transfer.

[0006] In some embodiments, after the drive robot performs the operation task and completes the skill transfer, it further includes: Collect execution status data of the robot during the execution of operation tasks, and calculate the comprehensive execution error based on the key status indicators in the execution status data; If the execution error exceeds a preset threshold, the operation task will be automatically stopped and supplementary teaching data will be provided.

[0007] If the execution error is less than a preset threshold, the operation task continues to be executed.

[0008] In some embodiments, after the drive robot performs the operation task and completes the skill transfer, it further includes: Acquire successful execution status data and supplementary teaching data, and add them to the multimodal teaching data to form an incremental training set; Based on the incremental training set, the skill learning model is updated using a parameter-freezing incremental learning algorithm.

[0009] In some embodiments, dividing the teaching process into operation sub-stages using a pre-trained model and adding corresponding operation semantic labels to the operation sub-stages specifically includes: By using a pre-trained model trained on robot operation data, the teaching process is automatically divided into multiple consecutive operation sub-stages. For each operation sub-stage, based on preset indicators, the start frame, key operation frame and end frame of each operation sub-stage are automatically identified and labeled, and the corresponding operation semantic label is labeled for each key operation frame.

[0010] In some embodiments, obtaining the operation object information and environmental obstacle information of the target scene specifically includes: The robot uses an RGB-D visual perception model to obtain information about the objects to be manipulated and environmental obstacles in the target scene, combined with target detection and pose estimation algorithms.

[0011] In some embodiments, the expression for the comprehensive execution error is: ; in Overall execution error For pose error, For force perception error, For timing error, These are the weighting coefficients for each error term.

[0012] In some embodiments, after adaptively generating the robot operation trajectory and force control strategy adapted to the target scene, the method further includes: optimizing the robot operation trajectory by combining a motion constraint optimization function, wherein the optimized robot operation trajectory consists of optimization variables and a trajectory mapping matrix, and the expression of the motion constraint optimization function is: ; ; ; in Optimize variables, Trajectory mapping matrix, The optimized robot operating trajectory, The target trajectory reference sequence generated by the skill learning model; The regularization coefficient is . These represent the lower and upper limits of the angular velocity of each joint of the robot. These are the coefficient matrix and threshold vector for collision avoidance constraints and kinematic constraints, respectively.

[0013] To achieve the above objectives, one aspect of this application proposes a no-code teaching and skills transfer system for humanoid robots, the system comprising: The data acquisition module is used to collect multimodal teaching data during the teaching process; The data analysis module is used to extract and fuse the multimodal features of the multimodal teaching data to generate fused features. Based on the fused features, the teaching process is divided into operation sub-stages through a pre-trained model, and corresponding operation semantic labels are added to the operation sub-stages. The skills learning module is used to construct a skills learning model framework. The multimodal teaching data of the operation sub-stage is input into the skills learning model, and the model is trained by imitation learning algorithm to generate a trained skills learning model. The trajectory generation module is used to acquire the operation object information and environmental obstacle information of the target scene, and input them into the trained skill learning model to adaptively generate robot operation trajectory and force control strategy adapted to the target scene, and generate control command sequence based on robot operation trajectory and force control strategy. The skill transfer module is used to send the control command sequence to the humanoid robot controller to drive the robot to perform operation tasks and complete skill transfer.

[0014] In some embodiments, the multimodal teaching data includes visual data, tactile force data, posture data, and audio data.

[0015] In some embodiments, the skill learning model includes: A visual feature encoder is used to extract scene and target object features from the visual data; A tactile feature encoder is used to extract the temporal features of the tactile force data; A posture feature encoder is used to extract the motion features of joint angles and hand poses from the posture data. An audio feature encoder is used to extract the acoustic features of the audio data; The cross-modal attention fusion unit is used to fuse scene features, target object features, temporal features, motion features, and acoustic features to generate fused features; The strategy decoding unit is used to decode the fused features and output the robot's operation trajectory and force control strategy.

[0016] The embodiments of this application include at least the following beneficial effects: This application provides a no-code teaching and skills transfer method and system for humanoid robots. This application eliminates the need for teaching via code, lowering the teaching threshold and adapting to the needs of rapid production changeover and non-standardized operations in industrial settings, thus reducing the barriers to the deployment of humanoid robots. The skill learning and control instruction sequence of this application can be directly deployed on mainstream humanoid robot and industrial robotic arm platforms without additional model conversion and code adaptation. It can be applied to various scenarios such as industrial assembly, logistics handling, precision operation and home service, and has a strong ability to be deployed on a large scale. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a no-code teaching and skills transfer method for a humanoid robot provided in this application embodiment; Figure 2 This is a schematic diagram of a no-code teaching and skills transfer method for a humanoid robot provided in an embodiment of this application; Figure 3 A closed-loop logic diagram of the no-code teaching and skill transfer method for humanoid robots provided in the embodiments of this application; Figure 4 A block diagram of a no-code teaching and skills transfer system for a humanoid robot provided in an embodiment of this application; Figure 5 This is a structural block diagram of the skills learning model provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0023] First-person perspective multimodal embodied data refers to multi-dimensional data such as vision, touch, posture, and audio collected by operators from a first-person perspective through wearable devices. This data directly maps the operator's work behavior and the interaction information between the work environment, and is the core data foundation for robot imitation and learning. Skill learning model: refers to a neural network model that is trained through multimodal feature fusion and imitation learning, and can represent the deep semantics, action logic and constraints of the operation task. It is the core carrier for robots to achieve skill adaptation and cross-scene transfer. Non-intrusive teaching control: refers to a teaching process control method that does not interrupt the operator's hand movements or change the normal work process, ensuring the consistency between teaching data and actual work behavior.

[0024] refer to Figures 1-3 As shown, one aspect of this application proposes a no-code teaching and skills transfer method for humanoid robots, including: S101: Collect multimodal teaching data during the teaching process; S102: Extract the multimodal features of the multimodal teaching data and fuse them to generate fused features. Based on the fused features, divide the teaching process into operation sub-stages through a pre-trained model and add corresponding operation semantic labels to the operation sub-stages. S103: Construct a framework for a skills learning model, input multimodal teaching data from the operation sub-stage into the skills learning model, train it using an imitation learning algorithm, and generate a trained skills learning model. S104: Obtain the information of the target object and the environmental obstacles in the target scene, and input them into the trained skill learning model to adaptively generate the robot operation trajectory and force control strategy adapted to the target scene, and generate a sequence of control instructions based on the robot operation trajectory and force control strategy. S105: Send the control command sequence to the humanoid robot controller to drive the robot to perform operation tasks and complete skill transfer.

[0025] Specifically, in S101, operators use wearable multimodal data acquisition devices to demonstrate the entire process of the target task in a real work scenario. During the operation, the wirelessly connected foot pedal control terminal can start / pause / stop the data acquisition and mark the teaching segments. Each button corresponds to a single function and is triggered by foot pressing. The hand operation is not interrupted throughout the process, and multimodal teaching data in the teaching process is collected simultaneously.

[0026] Furthermore, in some embodiments, the wearable multimodal acquisition device in step S101 includes at least a tactile glove, a wrist camera, a head camera, a microphone (array), and a computing vest.

[0027] Specifically, the multimodal teaching data in step S101 adopts first-person perspective multimodal embodied data, which includes visual data, tactile force data, posture data, and audio data.

[0028] Furthermore, after synchronously collecting multimodal teaching data in the teaching process, spatiotemporal synchronization is completed through hardware-level high-precision timestamp calibration, with time synchronization accuracy ≤1ms, and the unified data format is a structured format containing timestamp, device ID, data type, and data body.

[0029] Specifically, the pre-trained model in step S102 is a temporal convolutional network model trained on industrial / service robot operation data, and is configured as follows: The teaching process is divided into operation sub-stages using a pre-trained model, and corresponding operation semantic labels are added to each operation sub-stage, including: By using a pre-trained model trained on robot operation data, the teaching process is automatically divided into multiple consecutive operation sub-stages. For each operation sub-stage, based on preset indicators, the start frame, key operation frame and end frame of each operation sub-stage are automatically identified and labeled, and the corresponding operation semantic label is labeled for each key operation frame.

[0030] The operation sub-stages include grasping, handling, assembling, and placing, and the preset indicators include force perception abrupt change points, motion trajectory inflection points, and changes in the position of visual targets. The operation semantic labels include key parameters such as force perception threshold, target object pose, and joint angles.

[0031] Specifically, in step S103, an imitation learning algorithm based on behavior cloning and supplemented by a diffusion strategy is used for training. The model parameters are optimized using teaching data as a supervision signal to learn a skill learning model for the operation task. The skill learning model of this application simultaneously learns the operation action trajectory, force constraints in the operation process, visual target association, and operation temporal logic to achieve a deep semantic understanding of the task, rather than a simple trajectory replication.

[0032] Specifically, in step S1O4, the information on the target object and environmental obstacles in the target scene is obtained, including: By using the RGBD visual perception model onboard the robot, combined with YOLOv8 object detection and EPnP pose estimation algorithms, the robot obtains the pose, size, orientation of the object being manipulated, and information about environmental obstacles in the target scene. Furthermore, after adaptively generating the robot's operational trajectory and force control strategy adapted to the target scene, the process also includes: optimizing the robot's operational trajectory using a motion constraint optimization function. The optimized robot operational trajectory consists of optimization variables and a trajectory mapping matrix. The expression for the motion constraint optimization function is: ; ; ; in The optimization variables are the angular velocity sequences of each joint of the robot, which are the core parameters to be solved. The trajectory mapping matrix is ​​derived from the robot's forward kinematics model and is used to map joint angular velocities to the Cartesian space trajectory of the end effector. The optimized robot operating trajectory, Target trajectory reference sequence generated for skill learning model; This is a regularization coefficient used to control the smoothness of the trajectory and avoid sudden changes in joint angular velocity. These represent the lower and upper limits of the angular velocity for each joint of the robot, corresponding to the joint limit constraints. These are the coefficient matrix and threshold vector for collision avoidance constraints and kinematic constraints, respectively, used to ensure that the trajectory is collision-free and conforms to the robot's kinematic characteristics.

[0033] Specifically, motion constraints include robot kinematic constraints, collision avoidance constraints, and joint limit constraints, and trajectory smoothing optimization is achieved through a quadratic programming algorithm.

[0034] Specifically, the expression for the overall execution error in S105 is as follows: ; in Overall execution error For pose error, For force perception error, For timing error, These are the weighting coefficients for each error term.

[0035] Specifically, after driving the robot to perform operational tasks in S105 and completing skill transfer, it also includes: The system collects execution status data of the robot during the execution of its tasks, and calculates the comprehensive execution error based on the key status indicators in the execution status data. The execution status data includes the robot's visual perception data, end effector force data, and joint angle data.

[0036] If the execution error exceeds the preset threshold, the operation task will be automatically stopped and supplementary teaching data will be provided.

[0037] If the execution error is less than the preset threshold, the operation task will continue to be executed.

[0038] After the skill transfer is completed, the execution status data of successful execution and supplementary teaching data are obtained and added to the multimodal teaching data to form an incremental training set; The skill learning model is updated based on the incremental training set using a parameter-freezing incremental learning algorithm.

[0039] Furthermore, in some other embodiments, the preset threshold can be manually adjusted or the system can adaptively optimize it according to the work scenario.

[0040] A parameter-freezing incremental learning algorithm is used to fine-tune and update the skill learning model. The learning rate for fine-tuning is set to 5e-4, and the number of iterations is adaptively set to 50-100 rounds based on the amount of incremental data. While retaining the original bolt tightening skill, it adapts to new workstation scenarios, bolt specifications, and operational requirements. As the number of teaching and execution times increases, the model's operational accuracy, anti-interference ability, and cross-scenario generalization ability continue to improve, forming a complete process loop.

[0041] The following specific examples illustrate this application: The optimized control command sequence is sent to the humanoid robot controller via ROS2 to drive the robot to complete the entire bolt tightening operation.

[0042] During execution, the robot captures real-time operation footage via a wrist camera, collects tightening force data via a six-dimensional force sensor at the end effector, and acquires actual joint angles via a joint encoder. The system compares the real-time collected execution status data with key status indicators (tightening force threshold, bolt alignment accuracy, and action timing) in the teaching data, and calculates the comprehensive execution error using the formula for comprehensive execution error, where the weighting coefficients are set to... =0.5、 =0.4、 =0.1; If the alignment accuracy error exceeds 0.5mm, the tightening force threshold error exceeds 10%, or the overall execution error exceeds the preset threshold, the task will be automatically paused. The operator will be prompted by voice to perform supplementary teaching or manual remote operation correction. The supplementary teaching data and the manually corrected operation data will be directly synchronized to the multimodal teaching data for subsequent model incremental updates.

[0043] refer to Figures 4-5 As shown, to achieve the above objectives, another aspect of this application embodiment proposes a code-free teaching and skills transfer system for humanoid robots, wherein the system includes: The data acquisition module is used to collect multimodal teaching data during the teaching process; The data analysis module is used to extract and fuse the multimodal features of the multimodal teaching data to generate fused features. Based on the fused features, the teaching process is divided into operation sub-stages through a pre-trained model, and corresponding operation semantic labels are added to the operation sub-stages. The skills learning module is used to build a skills learning model framework. It inputs multimodal teaching data from the operation sub-stage into the skills learning model, trains it through imitation learning algorithms, and generates a trained skills learning model. The trajectory generation module is used to acquire information about the target object and environmental obstacles in the target scene, and input them into the trained skill learning model to adaptively generate robot operation trajectories and force control strategies that are adapted to the target scene. Based on the robot operation trajectory and force control strategies, a sequence of control instructions is generated. The skill transfer module is used to send control command sequences to the humanoid robot controller, drive the robot to perform operation tasks, and complete skill transfer.

[0044] Specifically, skill learning models include: The visual feature encoder, using a ResNet50 backbone network, is used to extract scene and target object features from visual data. The tactile feature encoder uses a 1D convolutional network to extract temporal features from tactile force data; The posture feature encoder uses an MLP (Multilayer Perceptron) network to extract the motion features of joint angles and hand poses from the posture data. An audio feature encoder, employing a CNN convolutional neural network, is used to extract acoustic features from audio data. The cross-modal attention fusion unit achieves the association and fusion of multimodal features through a multi-head attention mechanism with 8-16 attention heads. It is used to fuse scene and target object features, temporal features, motion features and acoustic features to generate fused features. The strategy decoding unit, using a Transformer decoder, is used to decode the fused features and output the robot's operating trajectory and force control strategy.

[0045] The core calculation formula for the cross-modal attention fusion unit is: ; The meanings of each factor in the formula are as follows: The query matrix is ​​obtained by mapping visual feature encoding vectors, with dimensions of [missing information]. , The length of the feature sequence. For single-head attention, the feature dimension is... The key matrix is ​​obtained by concatenating and mapping tactile, gesture, and audio feature encoding vectors, with dimensions equal to... Consistent; Value matrix, and The homologous mapping is obtained, with dimension . , The characteristic dimension of the value matrix; : Scaling factor, used to prevent the gradient of the Softmax function from vanishing due to excessively large inner product values; : Normalized exponential function, used to calculate the attention weight distribution; The output cross-modal fusion attention features enable the modeling of the intrinsic relationship between visual targets, force constraints, and motion trajectories.

[0046] Furthermore, in this application, the feature encoder backbone network of the parameter-frozen incremental learning algorithm only fine-tunes the cross-modal attention unit and policy decoding unit. The fine-tuning learning rate is set to 1e-4 to 1e-3, and the number of iterations is adaptively adjusted according to the amount of incremental data. While retaining the original skills, the model can be incrementally updated and continuously optimized.

[0047] In summary, this application enables no-code, low-barrier teaching of humanoid robots, eliminating the need for professional programming and trajectory debugging, and shortening the teaching cycle of complex tasks from several days to minutes, thus adapting to the needs of rapid production changeover and non-standardized operations in industrial settings. By achieving deep semantic understanding of operational tasks through multimodal feature fusion, the robot's cross-scene generalization ability is greatly improved, adapting to unstructured scenarios with changes in object pose, size, and environment; through non-intrusive teaching control, the continuity and authenticity of teaching action trajectories are ensured, the quality of teaching data is improved, and the learning effect of robot skills is optimized. The constructed closed-loop process enables continuous iterative optimization of the model through incremental learning, continuously improving the model's operational accuracy and generalization ability as the number of teaching sessions increases. The skill learning and control instructions generated in this application can be directly deployed on mainstream humanoid robot and industrial robotic arm platforms that support ROS2, without the need for additional model conversion and code adaptation.

[0048] The teaching and skills transfer method provided in this application relates to the field of information technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing the method, but is not limited to the above forms.

[0049] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0050] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0051] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0052] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0053] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0054] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0055] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0056] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0057] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0058] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0059] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0060] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0061] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for no-code teaching and skill transfer of a humanoid robot, characterized in that, The method includes: Collect multimodal teaching data during the teaching process; The multimodal features of the multimodal teaching data are extracted and fused to generate fused features. Based on the fused features, the teaching process is divided into operation sub-stages through a pre-trained model, and corresponding operation semantic labels are added to the operation sub-stages. A skill learning model framework is constructed, and the multimodal teaching data of the operation sub-stage is input into the skill learning model. The model is then trained using an imitation learning algorithm to generate a trained skill learning model. The system acquires information about the target object and environmental obstacles, and inputs it into the trained skill learning model. It then adaptively generates a robot operation trajectory and force control strategy that are adapted to the target scene, and generates a sequence of control commands based on the robot operation trajectory and force control strategy. The control command sequence is sent to the humanoid robot controller to drive the robot to perform operation tasks and complete skill transfer.

2. The method for no-code teaching and skill transfer of a humanoid robot according to claim 1, characterized in that, After the drive robot performs the operation task and completes the skill transfer, it also includes: Collect execution status data of the robot during the execution of operation tasks, and calculate the comprehensive execution error based on the key status indicators in the execution status data; If the execution error exceeds a preset threshold, the operation task will be automatically stopped and supplementary teaching data will be provided. If the execution error is less than a preset threshold, the operation task continues to be executed.

3. The method for no-code teaching and skill transfer of a humanoid robot according to claim 2, characterized in that, After the drive robot performs the operation task and completes the skill transfer, it also includes: Acquire successful execution status data and supplementary teaching data, and add them to the multimodal teaching data to form an incremental training set; Based on the incremental training set, the skill learning model is updated using a parameter-freezing incremental learning algorithm.

4. The method for no-code teaching and skill transfer of a humanoid robot according to claim 1, characterized in that, The step of dividing the teaching process into operation sub-stages using a pre-trained model and adding corresponding operation semantic labels to each operation sub-stage specifically includes: By using a pre-trained model trained on robot operation data, the teaching process is automatically divided into multiple consecutive operation sub-stages. For each operation sub-stage, based on preset indicators, the start frame, key operation frame and end frame of each operation sub-stage are automatically identified and labeled, and the corresponding operation semantic label is labeled for each key operation frame.

5. The method for no-code teaching and skill transfer of a humanoid robot according to claim 1, characterized in that, The acquisition of the target scene's object information and environmental obstacle information specifically includes: The robot uses an RGB-D visual perception model to obtain information about the objects to be manipulated and environmental obstacles in the target scene, combined with target detection and pose estimation algorithms.

6. The method for no-code teaching and skill transfer of a humanoid robot according to claim 2, characterized in that, The expression for the overall execution error is: ； in To account for execution errors, For pose error, For force perception error, For timing error, These are the weighting coefficients for each error term.

7. The method for no-code teaching and skill transfer of a humanoid robot according to claim 1, characterized in that, After adaptively generating the robot's operating trajectory and force control strategy adapted to the target scene, the method further includes: optimizing the robot's operating trajectory using a motion constraint optimization function, wherein the optimized robot operating trajectory consists of optimization variables and a trajectory mapping matrix, and the expression of the motion constraint optimization function is: ； ； ； in To optimize variables, The trajectory mapping matrix, The optimized robot operating trajectory, A target trajectory reference sequence generated for the skills learning model; The regularization coefficient is . These represent the lower and upper limits of the angular velocity of each joint of the robot. These are the coefficient matrix and threshold vector for collision avoidance constraints and kinematic constraints, respectively.

8. A no-code teaching and skill transfer system for a humanoid robot, characterized in that, The system includes: The data acquisition module is used to collect multimodal teaching data during the teaching process; The data analysis module is used to extract and fuse the multimodal features of the multimodal teaching data to generate fused features. Based on the fused features, the teaching process is divided into operation sub-stages through a pre-trained model, and corresponding operation semantic labels are added to the operation sub-stages. The skills learning module is used to construct a skills learning model framework. The multimodal teaching data of the operation sub-stage is input into the skills learning model, and the model is trained by imitation learning algorithm to generate a trained skills learning model. The trajectory generation module is used to acquire the operation object information and environmental obstacle information of the target scene, and input them into the trained skill learning model to adaptively generate robot operation trajectory and force control strategy adapted to the target scene, and generate control command sequence based on robot operation trajectory and force control strategy. The skill transfer module is used to send the control command sequence to the humanoid robot controller to drive the robot to perform operation tasks and complete skill transfer.

9. The code-free teaching and skill transfer system for humanoid robots according to claim 8, characterized in that, The multimodal teaching data includes visual data, tactile and force data, posture data, and audio data.

10. The code-free teaching and skill transfer system for humanoid robots according to claim 9, characterized in that, The skill learning model includes: A visual feature encoder is used to extract scene and target object features from the visual data; A tactile feature encoder is used to extract the temporal features of the tactile force data; A posture feature encoder is used to extract the motion features of joint angles and hand poses from the posture data. An audio feature encoder is used to extract the acoustic features of the audio data; The cross-modal attention fusion unit is used to fuse scene features, target object features, temporal features, motion features, and acoustic features to generate fused features; The strategy decoding unit is used to decode the fused features and output the robot's operation trajectory and force control strategy.

Images