Humanoid robot operation skill virtual-real transfer learning method based on knowledge distillation and meta-skill learning

Abstract

The invention relates to the technical field of robot operation skill learning and deployment, in particular to a humanoid robot operation skill virtual-real transfer learning method based on knowledge distillation and meta-skill learning. The method comprises the following steps: firstly, constructing a multi-task simulation training environment containing typical operation tasks, and training a teacher strategy model through a reinforcement learning algorithm to obtain operation track data of each task; mapping the multi-task operation track data to a low-dimensional skill space by using a track coding function, and identifying and generating a plurality of universal meta-skill sets in the low-dimensional skill space through a clustering algorithm; compressing and migrating a strategy model summarized by the teacher strategy model into a lightweight student strategy model to form a meta-skill condition strategy model capable of being reused across tasks; and optimizing the meta-skill condition strategy model through an adversarial domain discriminator and a skill embedding comparison mechanism. According to the method, the deployment reliability and the execution robustness of the strategy in a real physical environment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of robot operation skill learning and deployment technology, specifically to a virtual-real transfer learning method for humanoid robot operation skills based on knowledge distillation and meta-skill learning. Background Technology

[0002] As research into robotic systems' multi-task operation, complex scenario adaptation, and autonomous decision-making deepens, existing methods are increasingly revealing technical bottlenecks when addressing generalization needs, deployment requirements, and transfer challenges in practical applications. These bottlenecks include low policy learning efficiency, weak cross-task capabilities, and a severe disconnect between virtual and real-world applications. In typical operational tasks, robots often need to identify multiple target classes, select skills, and execute actions under limited data conditions. This places higher demands on their operational skill learning paradigms, policy model structures, and deployment reliability.

[0003] Current mainstream operational strategy learning methods mostly rely on end-to-end reinforcement learning or imitation learning to train high-performance teacher policy networks in specific task scenarios. However, these methods often face three problems: First, operational skills cannot be abstracted and transferred between tasks, requiring training from scratch for each new task class, resulting in low learning efficiency; second, policy models are usually computationally complex and structurally large, making them unsuitable for deployment on humanoid robot platforms with limited computing power, affecting the system's real-time response and deployment feasibility; third, due to significant differences between simulated training environments and real physical environments in terms of perceptual noise, dynamic parameters, and interactive feedback, policies that perform well in virtual environments may experience a severe drop in accuracy in real deployments, making it difficult to guarantee transfer robustness.

[0004] To alleviate these problems, some studies have attempted to introduce mechanisms such as multi-task learning, modular skill modeling, and policy compression to improve the generalization ability of operational behaviors and the lightweight nature of policy structures. However, these methods generally lack a unified skill representation system and an efficient knowledge transfer mechanism, and still have limitations in skill combination reuse and rapid task adaptation. At the same time, most existing virtual-real transfer strategies focus on image style transformation or random perturbation enhancement of the input perception modality, but lack systematic modeling of the consistency between state distribution shift and skill representation at the policy level, making it difficult to achieve highly reliable policy deployment.

[0005] In summary, existing technologies still have significant shortcomings in the following aspects: (1) a lack of general operational skill modeling methods that support task abstraction and skill reuse; (2) a trade-off between policy compression and generalization ability, and a lack of student policy models with lightweight structures that can quickly adapt to task changes; and (3) a lack of effective domain alignment mechanisms and skill-level representation comparison strategies during virtual-real policy transfer, making it difficult to guarantee the stability and robustness of policies in actual deployment. Therefore, it is urgent to construct a unified policy learning framework that integrates meta-skill modeling, knowledge distillation and compression, and virtual-real domain alignment mechanisms to improve the learning efficiency, transfer ability, and deployment performance of humanoid robots in complex operational tasks. Summary of the Invention

[0006] The present invention aims to at least partially solve one of the technical problems in the related art.

[0007] Therefore, the first objective of this invention is to propose a virtual-real transfer learning method for humanoid robot operation skills based on knowledge distillation and meta-skill learning, comprising: S1. Construct a multi-task simulation training environment containing typical operation tasks, and train the teacher policy network through reinforcement learning algorithm to obtain operation trajectory data for each task. S2, using a trajectory encoding function to map the multi-task operation trajectory data to a low-dimensional skill space, and using a clustering algorithm to identify and generate multiple sets of general meta-skills in the low-dimensional skill space; S3. Based on the general meta-skill set, construct a conditional policy network, and introduce meta-skill vectors as conditional input parameters into the structural design of the student policy network to form a meta-skill conditional policy model that can be reused across tasks. S4. The meta-skill conditional policy model is optimized by an adversarial domain discriminator and a skill embedding comparison mechanism, so that the model can achieve state distribution alignment and skill representation consistency between the simulation environment and the real physical environment.

[0008] In one embodiment of the present invention, S1 further includes, S11, each task is defined as an independent Markov decision process, and the task is represented as follows: , in Representing the state space, Represents action space, Represents the state transition probability, Represents the reward function; S12 uses the policy gradient algorithm to train the teacher policy network, maximizing discount-type rewards, and obtaining at least one task trajectory.

[0009] In one embodiment of the present invention, S12 further includes that the teacher policy network training optimization objective is represented as: ; in, To be in discrete time steps The environmental state in which the humanoid robot is located. In the state The actions performed by the following humanoid robots For time steps The discount weight of instant rewards For the first The policy network parameter vector corresponding to each task.

[0010] In one embodiment of the present invention, S2 further includes: S21, the multiple task trajectories generated during training are mapped to a low-dimensional skill space through a trajectory encoding function to obtain a skill representation vector. S22 uses clustering methods to identify and establish multiple sets of general meta-skills in the skill space to represent the general skill representations of each typical operational task.

[0011] In one embodiment of the present invention, S3 further includes: S31, define the teacher model strategy output action distribution as follows: The student model outputs the action distribution as follows: Where 'a' represents the state of the humanoid robot. The executable actions or control commands are given below, and s is a state vector describing the current state information of the humanoid robot itself and its operating environment; S32, Construct a distillation loss function using the knowledge distillation method, wherein the distillation loss function includes the consistency loss of action distribution. Consistency loss with state-value function ; S33, define the optimization objective of the student strategy model as: Furthermore, meta-skills are introduced into the student strategy model, and conditional strategies are defined as follows: This leads to the formation of a meta-skill conditional strategy model that can be reused across tasks.

[0012] In one embodiment of the present invention, the consistency loss of the action distribution is defined as the KL divergence between two policy distributions, expressed by the formula: .

[0013] In one embodiment of the present invention, the consistency of the state value function is constrained by the mean square error loss, and the formula for the consistency loss of the state value function is: ; in, For the teacher model in state The state value function outputs the following. For the student model in state The output of the state value function is D, which is the state dataset used for knowledge distillation training. This state dataset is a set of states sampled from the interaction between the humanoid robot and the simulation environment and / or the real environment.

[0014] In one embodiment of the present invention, S4 further includes: S41, construct the state distribution of the simulated domain and the state distribution of the real domain, design a domain adversarial training mechanism, and introduce a domain discriminator to distinguish between the simulated domain state and the real domain state. Define the discriminator loss as: ; in, The output of the region discriminator. To construct the state distribution of the simulation domain, The distribution of states in the real domain; S42 optimizes the student policy model by maximizing the confusion of the discriminator through an adversarial approach, thereby aligning the state distributions of the two domains. S43, Construct a contrastive loss function in the skill space and define the simulated skill trajectory. The encoding is Real skill trajectory The encoding is This results in contrast loss. minimize.

[0015] To achieve the above objectives, a second aspect of the present invention proposes a virtual-real transfer system for humanoid robot operation skills based on knowledge distillation and meta-skill learning, comprising: The multi-task simulation training and meta-skill modeling module is used to construct a multi-task simulation training environment containing a variety of typical operation tasks. It trains the teacher policy network through reinforcement learning algorithm to obtain operation trajectory data of each task, and uses trajectory encoding function to map the multi-task operation trajectory data to a low-dimensional skill space. It also uses clustering algorithm to identify and generate multiple general meta-skill sets in the low-dimensional skill space. The strategy compression and student strategy construction module compresses and migrates the strategy model summarized by the teacher's strategy model to a lightweight student strategy model, forming a meta-skill conditional strategy model that can be reused across tasks. The strategy transfer and virtual-real domain alignment module is used to optimize the meta-skill conditional strategy model through an adversarial domain discriminator and a skill embedding comparison mechanism, so that the model can achieve state distribution alignment and skill representation consistency between the simulation environment and the real physical environment.

[0016] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0017] The methods, systems, and storage media of this invention effectively solve the problem of the transfer stability of humanoid robot operation skills between simulation and real environments. By aligning state distribution and optimizing the consistency of skill representation, the deployment reliability and execution robustness of strategies in real physical environments are significantly improved.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a virtual-real transfer learning method for humanoid robot operation skills based on knowledge distillation and meta-skill learning according to an embodiment of the present invention; Figure 2 This is a framework diagram of a multi-task simulation training and meta-skill modeling module for virtual-real transfer learning of humanoid robot operation skills based on knowledge distillation and meta-skill learning, according to an embodiment of the present invention. Figure 3 This is a framework diagram of a strategy compression and student strategy construction module for virtual-real transfer learning of humanoid robot operation skills based on knowledge distillation and meta-skill learning according to an embodiment of the present invention. Figure 4 This is a framework diagram of a strategy transfer and virtual-real domain alignment module for virtual-real domain transfer learning of humanoid robot operation skills based on knowledge distillation and meta-skill learning, according to an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] The following description, with reference to the accompanying drawings, describes a virtual-real transfer learning method and system for humanoid robot operation skills based on knowledge distillation and meta-skill learning, according to an embodiment of the present invention.

[0023] Example 1 Figure 1 This is a flowchart of a virtual-real transfer learning method for humanoid robot operation skills based on knowledge distillation and meta-skill learning, according to an embodiment of the present invention.

[0024] like Figure 1 As shown, the virtual-real transfer learning of humanoid robot operation skills based on knowledge distillation and meta-skill learning includes the following steps: S1. Construct a multi-task simulation training environment containing typical operation tasks, and train the teacher's strategy model through reinforcement learning algorithm to obtain operation trajectory data for each task.

[0025] Furthermore, S1 includes: S11, each task is defined as an independent Markov decision process, and the task is represented as follows: , in Representing the state space, Represents action space, Represents the state transition probability, Represents the reward function; S12 uses the policy gradient algorithm to train the teacher policy network, maximizing discount-type rewards to obtain the task trajectory for each task.

[0026] Specifically, in step S1 of this invention, a multi-task simulation training environment containing typical operational tasks is constructed, and a teacher policy model is trained using a reinforcement learning algorithm to obtain operational trajectory data for each task. This is a crucial starting point for realizing virtual-real transfer learning of humanoid robot operational skills. The core of this step lies in providing a high-quality training data foundation for subsequent meta-skill extraction and policy compression through structured modeling and efficient policy training.

[0027] At the technical implementation level, in some implementations, this step first models multiple typical operational tasks (such as grasping, carrying, inserting, stacking, etc.) as independent Markov Decision Processes (MDPs), each task... It can be represented as: ; in, For state space, For the action space, Let be the state transition probability function. A task-specific reward function. In the simulation environment, the state space... This typically includes the position and orientation of the robot's end effector, the geometry of the target object, contact force feedback, and the motion space. This is composed of control commands such as joint torque or end-effector velocity. Reinforcement learning algorithms (such as policy gradient methods) are used to train the teacher policy network. Its optimization objective is to maximize the cumulative discount reward: ; in, This is a discount factor used to balance the importance of current and future rewards. Indicates at discrete time step The state of humanoid robots and their operating environment; This indicates the state of a humanoid robot. The teacher's policy network executes actions or control commands. During training, the teacher's policy network generates task trajectories through interaction with the environment. These trajectory data will serve as input for subsequent meta-skill modeling and knowledge distillation.

[0028] In terms of application scenarios, this step is widely used in the pre-training stage of multi-task skills for humanoid robots in virtual environments, such as simulating scenarios like home service, industrial assembly, and medical assistance in digital twin systems. By training teacher strategies for multiple tasks in simulation, the high-cost and high-risk trial-and-error learning in real-world environments can be avoided, while providing structured and reusable skill data for subsequent strategy transfer.

[0029] In terms of technical effectiveness: By constructing a multi-task simulation training environment and training teacher strategies, this invention achieves structured modeling and efficient acquisition of operational skills, providing a unified representation basis for meta-skill extraction. This step not only improves the efficiency of strategy training but also ensures the performance of teacher strategies in complex tasks through reinforcement learning algorithms, laying a solid data and model foundation for subsequent strategy compression and virtual-real transfer.

[0030] S2, the multi-task operation trajectory data is mapped to a low-dimensional skill space using a trajectory encoding function, and multiple general meta-skill sets are generated in the low-dimensional skill space by a clustering algorithm.

[0031] Furthermore, S2 includes: S21, the multiple task trajectories generated during training are mapped to a low-dimensional skill space through a trajectory encoding function to obtain a skill representation vector. S22 uses clustering methods to identify and establish multiple sets of general meta-skills in the skill space to represent the general skill representations of each typical operational task.

[0032] Specifically, such as Figure 2 As shown, in step S2, this invention maps multi-task operation trajectory data to a low-dimensional skill space using a trajectory encoding function, and uses a clustering algorithm to identify and generate multiple sets of general meta-skills, thereby achieving structured abstraction of operational behavior and skill generalization. This step is a key link in the entire technical solution to realize modular skill modeling and cross-task reuse.

[0033] At the technical implementation level, trajectory encoding function It is a neural network-based embedding mapper whose input is the multi-task operation trajectory obtained by training the teacher policy network in a simulation environment. The output is a low-dimensional skill representation vector. The encoding function typically employs temporal modeling structures, such as LSTM, Transformer, or GRU, to capture dynamic behavioral patterns within the operational trajectory. The encoded skill vector... Typically compressed to a dimension of low-dimensional space (e.g.) or This reduces representation complexity and enhances generalization ability. During the encoding process, attention mechanisms or adaptive normalization layers can be optionally introduced to improve sensitivity and stability for critical operation stages.

[0034] Furthermore, in the low-dimensional skill space, clustering algorithms (such as Gaussian Mixture Model, K-Means, or DBSCAN) are used to perform cluster analysis on the skill representation vectors of all tasks, identifying skill patterns with similar operational characteristics and forming multiple sets of general meta-skills. The parameter settings of the clustering algorithm need to be adjusted according to the task complexity and data distribution characteristics. For example, in K-Means, the number of cluster centers... It can be set according to the number of tasks and the similarity threshold. to This ensures that each meta-skill set covers common operational behaviors across multiple tasks. The clustering results can serve as conditional inputs for subsequent student policy models, guiding rapid policy adaptation and task transfer.

[0035] In application scenarios, this step is suitable for skill abstraction and modeling of humanoid robots in typical operational tasks such as grasping, carrying, inserting, and stacking. By uniformly mapping the trajectories of different tasks to a low-dimensional space, the robot can quickly generate feasible operational strategies in new tasks by retrieving the nearest set of meta-skills, thereby significantly improving learning efficiency and deployment adaptability.

[0036] The technical benefits of this step lie in achieving modular abstraction of operational behaviors through structured skill representation and cluster analysis, providing a unified skill semantic foundation for subsequent policy compression and cross-domain transfer. Simultaneously, the construction of a low-dimensional skill space effectively reduces the complexity of the policy model, improves its interpretability and transferability, and provides solid support for the efficient operation of humanoid robots in multiple tasks and environments.

[0037] S3 compresses and transfers the teacher strategy model to a lightweight student strategy model, forming a meta-skill conditional strategy model with meta-skill vectors as conditional inputs.

[0038] Furthermore, in S31, the distribution of the teacher model's strategy output actions is defined as follows: The student model outputs the action distribution as follows: Where 'a' represents the state of the humanoid robot. The executable action or control command; s is a state vector describing the current state information of the humanoid robot itself and its operating environment; S32, Construct a distillation loss function using the knowledge distillation method, wherein the distillation loss function includes the consistency loss of action distribution. Consistency loss with state-value function ; S33, define the optimization objective of the student strategy model as: Furthermore, meta-skills are introduced into the student strategy model, and conditional strategies are defined as follows: This leads to the formation of a meta-skill conditional strategy model that can be reused across tasks.

[0039] Specifically, such as Figure 3 As shown, in the strategy compression and student strategy construction module, this invention uses knowledge distillation technology to transfer operational strategy knowledge from the teacher's strategy model to a lightweight student strategy model, thereby forming a meta-skill conditional strategy model with meta-skill vectors as conditional inputs. This step is a key link in realizing the efficient deployment and rapid transfer of humanoid robot operational skills.

[0040] At the technical implementation level, teacher strategy model Typically, these are deep reinforcement learning networks with a large number of parameters and high computational complexity. In simulation environments, they are optimized using the policy gradient algorithm, with the objective function being: ; in, Indicates at time step State observation; Indicates the state The following actions are output by the teacher's strategy model; This is a discount factor used to balance the importance of current rewards and future rewards; For the first The parameter vector of the teacher strategy network corresponding to each task is used to determine the strategy. The specific weights and structure; For humanoid robots in state Execute action Instant rewards obtained at that time; Indicating in strategy Calculation of the expected cumulative reward for discounts under the induced state-action distribution; In strategy The objective function for the cumulative reward of the corresponding operation task is then applied. Through this process, the teacher strategy model accumulates rich operation trajectory data across multiple tasks, providing high-quality "soft labels" for knowledge distillation.

[0041] In the knowledge distillation process, the teacher's strategy output is defined as an action distribution. The student's strategy output is To ensure that students' strategies accurately mimic the behavioral decisions of teachers' strategies, two key loss terms were constructed: action distribution consistency loss. Consistency loss with state-value function The consistency loss of action distribution is measured using KL divergence, with the following formula: ; The consistency loss of the state-value function is constrained by the mean squared error (MSE), as shown in the formula: ; The overall optimization objective of the student strategy model is the weighted sum of the two, i.e. Building on this, meta-skill vectors are further introduced into the student strategy model. Use it as a conditional input to construct a conditional strategy. This leads to the formation of a meta-skill conditional strategy model. This model has cross-task reuse capabilities and can quickly adapt to new task requirements by switching meta-skill vectors in different tasks.

[0042] This step is applicable in practical applications to scenarios where humanoid robots, after being trained in a simulation environment, need to be deployed to real-world platforms with limited computing resources (such as embedded controllers or edge devices). Through policy compression, the number of parameters in the student model can be reduced by more than 50%, and inference latency reduced to milliseconds, while maintaining performance comparable to the teacher's policy. Combined with a meta-skill conditional input mechanism, the student policy can quickly adapt to new tasks with only a small number of samples via meta-skill vectors, significantly improving the policy's generalization ability and deployment efficiency.

[0043] In summary, this step, through the fusion design of knowledge distillation and meta-skill conditional input, realizes the knowledge transfer from high-dimensional and complex teacher strategies to lightweight student strategies, laying a solid foundation for subsequent virtual-real domain alignment and practical deployment, and has significant engineering practical value and technological innovation.

[0044] S4. The meta-skill conditional policy model is optimized by an adversarial domain discriminator and a skill embedding comparison mechanism, so that the model can achieve state distribution alignment and skill representation consistency between the simulation environment and the real physical environment.

[0045] Furthermore, S4 includes, S41, construct the state distribution of the simulated domain and the state distribution of the real domain, design a domain adversarial training mechanism, and introduce a domain discriminator to distinguish between the simulated domain state and the real domain state. Define the discriminator loss as: ; in, The output of the region discriminator. To construct the state distribution of the simulation domain, The distribution of states in the real domain; S42 optimizes the student policy model by maximizing the confusion of the discriminator through an adversarial approach, thereby aligning the state distributions of the two domains. S43, Construct a contrastive loss function in the skill space and define the simulated skill trajectory. The encoding is Real skill trajectory The encoding is This results in contrast loss. minimize.

[0046] Specifically, such as Figure 4 As shown, in step S4, this invention optimizes the meta-skill conditional policy model through an adversarial domain discriminator and a skill embedding comparison mechanism. This aims to address the state distribution shift between the simulation environment and the real physical environment, and ensure the consistency of skill representation between the two domains. This step is crucial for achieving stable policy migration from simulation to reality, and has significant engineering value and theoretical implications.

[0047] At the technical implementation level, this invention employs a Domain Adversarial Training (DAT) mechanism, which introduces a domain discriminator. To distinguish simulation domain states With the real domain state The training objective of the domain discriminator is to maximize its ability to distinguish between two domain states, and its loss function is defined as: ; Meanwhile, the student policy model minimizes the discriminator's discriminative power, i.e., maximizes its confusion, through adversarial training, thereby causing the state distribution in the simulation domain to converge with that in the real domain. This adversarial training process typically employs a gradient reversal layer (GRL), which reverses the gradient direction of the discriminator during backpropagation, guiding the policy model to learn domain-invariant features.

[0048] Furthermore, at the skill representation level, this invention constructs a contrastive loss function. Used to constrain the trajectory of simulation skills With real skill trajectory Representation consistency in low-dimensional skill space. Specifically, trajectory is mapped to skill representation vectors through a trajectory encoding function. and And define the contrast loss as: ; This loss function encourages simulated and real trajectories to have similar embedding representations in the skill space, thereby improving the generalization ability and execution stability of the policy in real-world environments.

[0049] This step is widely used in the policy transfer deployment of humanoid robots in tasks such as grasping, carrying, and inserting. Through state distribution alignment and skill representation consistency constraints, the student policy model can more accurately perceive the state and generate reasonable actions in the real environment, significantly improving the robustness and adaptability of the policy and ensuring the stable operation of the robot in complex real-world scenarios.

[0050] The virtual-to-real transfer method for humanoid robot operation skills in this invention improves the cross-task generalization ability and virtual-to-real transfer robustness of humanoid robot operation skills. It achieves policy compression and lightweight deployment through meta-skill modeling and knowledge distillation, and effectively aligns the state distribution and skill representation of simulation and real environments by combining domain adversarial and skill embedding comparison mechanisms.

[0051] Example 2 A virtual-to-real transfer system for humanoid robot manipulation skills based on knowledge distillation and meta-skill learning includes: The multi-task simulation training and meta-skill modeling module is used to construct a multi-task simulation training environment containing a variety of typical operation tasks. It trains the teacher policy network through reinforcement learning algorithm to obtain operation trajectory data of each task, and uses trajectory encoding function to map the multi-task operation trajectory data to a low-dimensional skill space. It also uses clustering algorithm to identify and generate multiple general meta-skill sets in the low-dimensional skill space. The strategy compression and student strategy construction module compresses and migrates the strategy model summarized by the teacher's strategy model to a lightweight student strategy model, forming a meta-skill conditional strategy model that can be reused across tasks. The strategy transfer and virtual-real domain alignment module is used to optimize the meta-skill conditional strategy model through an adversarial domain discriminator and a skill embedding comparison mechanism, so that the model can achieve state distribution alignment and skill representation consistency between the simulation environment and the real physical environment.

[0052] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for virtual-real transfer of humanoid robot operation skills based on knowledge distillation and meta-skill learning.

[0053] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0054] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A virtual-real transfer learning method for humanoid robot operation skills based on knowledge distillation and meta-skill learning, characterized in that, include: S1. Construct a multi-task simulation training environment containing typical operation tasks, and train the teacher's strategy model through reinforcement learning algorithm to obtain the operation trajectory data of each task. S2, using a trajectory encoding function to map the multi-task operation trajectory data to a low-dimensional skill space, and using a clustering algorithm to identify and generate multiple sets of general meta-skills in the low-dimensional skill space; S3 compresses and transfers the strategy model summarized by the teacher strategy model to a lightweight student strategy model, forming a meta-skill conditional strategy model that can be reused across tasks. S4. The meta-skill conditional policy model is optimized by an adversarial domain discriminator and a skill embedding comparison mechanism, so that the model can achieve state distribution alignment and skill representation consistency between the simulation environment and the real physical environment.

2. The method as described in claim 1, characterized in that, S1 also includes, S11, each task is defined as an independent Markov decision process, and the task is represented as follows: ， in Representing the state space, Represents action space, Represents the state transition probability, Represents the reward function; S12 uses the policy gradient algorithm to train the teacher policy network, maximizing discount-type rewards to obtain the task trajectory for each task.

3. The method as described in claim 2, characterized in that, S12 further includes that the teacher strategy network training optimization objective is expressed as: ； in, To be in discrete time steps The environmental state in which the humanoid robot is located. In the state The actions performed by the humanoid robot For time steps The discount weight of instant rewards For the first The policy network parameter vector corresponding to each task.

4. The method as described in claim 1, characterized in that, S2 further includes: S21, the multiple task trajectories generated during training are mapped to a low-dimensional skill space through a trajectory encoding function to obtain a skill representation vector. S22 uses clustering methods to identify and establish multiple sets of general meta-skills in the skill space to represent the general skill representations of each typical operational task.

5. The method as described in claim 1, characterized in that, S3 further includes: S31, define the teacher model strategy output action distribution as follows: The student model outputs the action distribution as follows: Where 'a' represents the state of the humanoid robot. The executable actions or control commands are given below, and s is a state vector describing the current state information of the humanoid robot itself and its operating environment; S32, Construct a distillation loss function using the knowledge distillation method, wherein the distillation loss function includes the consistency loss of action distribution. Consistency loss with state-value function ; S33, define the optimization objective of the student strategy model as: Furthermore, meta-skills are introduced into the student strategy model, and conditional strategies are defined as follows: This leads to the formation of a meta-skill conditional strategy model that can be reused across tasks.

6. The method as described in claim 5, characterized in that, The consistency loss of the action distribution is defined as the KL divergence between the two policy distributions, expressed by the formula: 。 7. The method as described in claim 5, characterized in that, The consistency of the state-value function is constrained by the mean squared error loss. The formula for the consistency loss of the state-value function is as follows: ； in, For the teacher model in state The state value function outputs the following. For the student model in state The output of the state value function is D, which is the state dataset used for knowledge distillation training. This state dataset is a set of states sampled from the interaction between the humanoid robot and the simulation environment and / or the real environment.

8. The method as described in claim 1, characterized in that, S4 further includes: S41, construct the state distribution of the simulated domain and the state distribution of the real domain, design a domain adversarial training mechanism, and introduce a domain discriminator to distinguish between the simulated domain state and the real domain state. Define the discriminator loss as: ； in, The output of the region discriminator. To construct the state distribution of the simulation domain, The distribution of states in the real domain; S42 optimizes the student policy model by maximizing the confusion of the discriminator through an adversarial approach, thereby aligning the state distributions of the two domains. S43, Construct a contrastive loss function in the skill space and define the simulated skill trajectory. The encoding is Real skill trajectory The encoding is This results in contrast loss. minimize.

9. A virtual-real transfer system for humanoid robot operation skills based on knowledge distillation and meta-skill learning, characterized in that, include: The multi-task simulation training and meta-skill modeling module is used to construct a multi-task simulation training environment containing a variety of typical operation tasks. It trains the teacher policy network through reinforcement learning algorithm to obtain operation trajectory data of each task, and uses trajectory encoding function to map the multi-task operation trajectory data to a low-dimensional skill space. It also uses clustering algorithm to identify and generate multiple general meta-skill sets in the low-dimensional skill space. The strategy compression and student strategy construction module compresses and migrates the strategy model summarized by the teacher's strategy model to a lightweight student strategy model, forming a meta-skill conditional strategy model that can be reused across tasks. The strategy transfer and virtual-real domain alignment module is used to optimize the meta-skill conditional strategy model through an adversarial domain discriminator and a skill embedding comparison mechanism, so that the model can achieve state distribution alignment and skill representation consistency between the simulation environment and the real physical environment.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as claimed in any one of claims 1-8.

Images