Humanoid robot hierarchical skill fusion control method and system based on attention mechanism

Abstract

The invention discloses a humanoid robot hierarchical skill fusion control method and system based on an attention mechanism, and relates to the field of humanoid robot control. The problems that an existing SkillBlender method is poor in skill fusion weight adaptability, unstable in control and high in energy consumption are solved, and the method comprises the steps that a plurality of basic skill strategies of the human-shaped robot are obtained through pre-training, and parameters of all basic skills are frozen after training is completed; a high-level controller is constructed, state information and task target information of the humanoid robot at the current moment are collected, feature coding is carried out through a state coding module, and high-level state representation is obtained; a learnable skill embedding vector is generated for each basic skill through a skill embedding module, and an attention fusion weight corresponding to each basic skill is generated through an attention weight calculation module; and generating a corresponding sub-target for each basic skill based on the high-level state representation through a sub-target generation module, inputting the current state and the sub-target into a corresponding basic skill strategy, and obtaining action output of each basic skill.

Description

Technical Field

[0001] This invention relates to the field of humanoid robot control and reinforcement learning technology, specifically to a hierarchical skill fusion control method and system for humanoid robots based on an attention mechanism. Background Technology

[0002] With the development of simulation technology and computing power, reinforcement learning has become a core tool for solving complex robot control problems, showing great potential in scenarios such as humanoid robot walking and operation. However, humanoid robots have dozens to hundreds of degrees of freedom, highly nonlinear dynamics, and frequent contact, and end-to-end reinforcement learning methods face core challenges such as low sample efficiency, unstable training, and poor policy generalization.

[0003] Hierarchical reinforcement learning effectively reduces the difficulty of policy learning by decomposing complex tasks into reusable sub-skills. The SkillBlender method, for example, achieves the ability to adapt to new tasks without retraining low-level policies through the linear fusion of pre-trained basic skills. However, this method still has significant drawbacks: first, the static or weakly conditional weight allocation cannot adapt to the dynamic control requirements of multi-stage tasks; second, simple weight mapping makes it difficult to extract key decision features from high-dimensional states, resulting in limited feature selection capabilities; and third, the lack of structural constraints on weight changes leads to frequent switching, resulting in unsmooth control, increased energy consumption, and limited performance in complex multi-stage humanoid robot tasks. Summary of the Invention

[0004] This invention addresses the problems of poor adaptability of skill fusion weights, unstable control, and high energy consumption inherent in the existing SkillBlender method. To this end, this invention proposes a hierarchical skill fusion control method based on an attention mechanism, aiming to achieve state-adaptive dynamic skill fusion. This significantly improves the success rate, stability, and energy efficiency of multi-task control for humanoid robots without retraining basic skills.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: Option 1: This invention proposes a hierarchical skill fusion control method for humanoid robots based on an attention mechanism, comprising the following steps: Step 1: Pre-train the basic skills to obtain multiple basic skill strategies for humanoid robots. After training is completed, freeze the parameters of all basic skills. Step 2: Based on the parameters of all basic skills frozen after training in Step 1, construct a high-level controller. The high-level controller includes a state encoding module, a skill embedding module, an attention weight calculation module, a sub-target generation module, and an action fusion module. The PPO algorithm is used to train the high-level controller. Step 3: Collect the current state information and task target information of the humanoid robot, and perform feature encoding through the state encoding module described in Step 2 to obtain a high-level state representation; Step 4: Generate learnable skill embedding vectors for each basic skill through the skill embedding module. Combined with the high-level state representation obtained in Step 3, generate attention fusion weights for each basic skill through the attention weight calculation module. Step 5: Using the sub-target generation module described in Step 2, generate corresponding sub-targets for each basic skill based on the high-level state representation. Input the current state and sub-targets into the corresponding basic skill strategy to obtain the action output of each basic skill. Step 6: Using the motion fusion module described in Step 2, the motion outputs of each basic skill are weighted and fused based on the attention fusion weight to generate the final control command for the humanoid robot.

[0006] Furthermore, a preferred embodiment is provided, wherein the pre-training in step one yields multiple basic humanoid robot skill strategies, including walking, reaching, squatting, and stepping. Each basic skill strategy is independently trained using objective conditional reinforcement learning.

[0007] Furthermore, a preferred embodiment is provided, wherein the method of training the high-level controller using the PPO algorithm in step two is as follows: during the training process, only the parameters of the high-level controller are optimized, while the parameters of all basic skill strategies remain frozen.

[0008] Furthermore, a preferred embodiment is provided, wherein the method for the state encoding module in step three to perform feature encoding to obtain a high-level state representation is as follows: using an encoding function composed of a multilayer sensing mechanism, features are extracted from the spliced ​​robot state information and task target information, and a high-level state representation with consistent dimensions is output.

[0009] Furthermore, a preferred implementation is provided, wherein the method for generating the attention fusion weights corresponding to each basic skill in step four through the attention weight calculation module is as follows: the matching degree between the current high-level state representation and the embedding vector of each skill is calculated using the scaled dot product attention mechanism, and the attention fusion weights that satisfy non-negativity and sum to 1 are obtained after Softmax normalization.

[0010] Furthermore, a preferred embodiment is provided in which the sub-target generation module described in step five configures an independent sub-target generation network for each basic skill, and all sub-target generation networks share the high-level state representation output by the state encoding module as input.

[0011] Furthermore, a preferred embodiment is provided, wherein the final control command described in step six is ​​the target position command of each joint of the humanoid robot, which is converted into joint torque by a proportional-derivative (PD) controller and then drives the robot to execute.

[0012] Option 2: A hierarchical skill fusion control system for humanoid robots based on an attention mechanism, the system comprising: The pre-training module is used to pre-train basic skills to obtain multiple basic skill strategies for humanoid robots. After training is completed, the parameters of all basic skills are frozen. The high-level controller construction module is used to construct a high-level controller based on the parameters of all basic skills frozen after the pre-training module is trained. The high-level controller includes a state encoding module, a skill embedding module, an attention weight calculation module, a sub-target generation module, and an action fusion module, and the PPO algorithm is used to train the high-level controller. The feature encoding module is used to collect the current state information and task target information of the humanoid robot, and perform feature encoding through the state encoding module in the high-level controller construction module to obtain the high-level state representation. The attention fusion weight module is used to generate learnable skill embedding vectors for each basic skill through the skill embedding module, and combine them with the high-level state representations obtained in the feature encoding module to generate attention fusion weights for each basic skill through the attention weight calculation module. The basic skill action output module is used to generate corresponding sub-targets for each basic skill based on the high-level state representation of the sub-target generation module constructed by the high-level controller module. The current state and sub-targets are input into the corresponding basic skill strategy to obtain the action output of each basic skill. The control command generation module is used to construct the motion fusion module described in the high-level controller module, and to perform weighted fusion of the motion outputs of each basic skill based on attention fusion weights to generate the final control command for the humanoid robot.

[0013] Option 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.

[0014] Option 4: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.

[0015] The advantages of this invention are: The humanoid robot hierarchical skill fusion control method and system based on attention mechanism described in this invention do not require retraining of basic skills. They can quickly adapt to new tasks simply by high-level attention fusion, which greatly reduces the workload of reward engineering and improves the development efficiency of new tasks.

[0016] The humanoid robot hierarchical skill fusion control method and system based on attention mechanism described in this invention realizes dynamic skill weight allocation with state adaptation based on attention mechanism, which can accurately match the control requirements of different stages of the task, avoid ineffective competition between skills, and significantly improve the execution success rate of multi-stage complex tasks.

[0017] The attention mechanism-based humanoid robot hierarchical skill fusion control method and system attention structure described in this invention apply regularization constraints at the implicit level, effectively suppressing high-frequency oscillations of skill weights, significantly reducing control energy consumption and motion fluctuations, improving the physical feasibility, execution security and engineering deployability of the strategy, possessing excellent cross-embodiment generalization ability, adapting to humanoid robot platforms of different configurations, and applicable to a wide range of scenarios.

[0018] This invention is also applicable to multi-task control scenarios for the whole-body motion and operation of high-degree-of-freedom humanoid robots. Attached Figure Description

[0019] Figure 1 This is a schematic diagram comparing the original SkillBlender described in Implementation Method 1 with the hierarchical skill fusion described in this invention.

[0020] Figure 2 This is a schematic diagram of the hierarchical structure of SkillBlenderPro as described in Implementation Method 1.

[0021] Figure 3 This is a schematic diagram of the four basic skills employed as described in Implementation Method 1.

[0022] Among them, (a) is a diagram of the Walking skill, (b) is a diagram of the Reaching skill, (c) is a diagram of the Squatting skill, and (d) is a diagram of the Stepping skill.

[0023] Figure 4 This is a schematic diagram illustrating the completion of one of the eight high-level tasks described in Implementation Method 1.

[0024] Among them, (a) is a schematic diagram of the completion of the FarReach task, (b) is a schematic diagram of the completion of the ButtonPress task, (c) is a schematic diagram of the completion of the CabinetClose task, (d) is a schematic diagram of the completion of the FootballShoot task, (e) is a schematic diagram of the completion of the BoxPush task, (f) is a schematic diagram of the completion of the PackageLift task, (g) is a schematic diagram of the completion of the BoxTransfer task, and (h) is a schematic diagram of the completion of the PackageCarry task. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0026] Implementation Method 1, see Figures 1 to 4 This embodiment describes a humanoid robot hierarchical skill fusion control method and system based on an attention mechanism, specifically including: Basic Skills Pre-training and Freezing: Target-conditional reinforcement learning was used to independently pre-train four atomized basic skills: Walking, Reaching, Squatting, and Stepping. Each skill corresponds to a basic motor / operational ability. After training, the network parameters of all basic skills were frozen.

[0027] High-level controller construction: The high-level controller comprises a state encoding module, a skill embedding module, an attention weight calculation module, a sub-target generation module, and an action fusion module. The state encoding module uses a multilayer perceptron to encode features from the stitched robot state and task target information, outputting a high-level state representation. The skill embedding module configures a learnable embedding vector for each basic skill, representing the skill's functional semantic attributes. The attention weight calculation module uses scaled dot product attention to calculate the matching degree between the state representation and the skill embedding, and generates attention fusion weights after Softmax normalization. The sub-target generation module configures an independent network for each basic skill, sharing the high-level state representation to generate corresponding sub-targets. The action fusion module uses attention weights to weight and fuse the action outputs of each basic skill to generate the final control command.

[0028] The training process consists of two stages: the first stage involves independent pre-training of basic skills, and the second stage uses the PPO algorithm to train the high-level controller. During the training process, the basic skill parameters are frozen throughout, and only 1-2 task-related sparse reward items are used to complete the training.

[0029] Physical feasibility evaluation system: Construct a comprehensive evaluation system that includes mission success rate, average energy consumption, control smoothness, balance stability, joint torque and power to achieve a comprehensive quantitative assessment of strategy performance.

[0030] Example 1: Implementation Environment and Platform Setup This embodiment is built on the NVIDIA Isaac Gym physics simulation platform, using the PhysX physics engine. The experimental object is the Unitree H1 19-DOF humanoid robot, and the generalization experiment uses the Unitree G1 21-DOF humanoid robot. The simulation utilizes GPU parallel acceleration, with 4096 parallel environments enabled during the training phase. The control frequency is set to 50Hz, and the motion space is the target positions of all joints on the robot's body, converted into joint torques by a PD controller to drive the robot's execution. All experiments use a fixed random seed, and environmental noise and random perturbations are disabled during the evaluation phase to ensure experimental reproducibility.

[0031] Basic Skills Pre-training Implementation: This example pre-trains four basic skills. The definitions and training methods for each skill are as follows: Walking skill: Control a humanoid robot to walk and maintain balance in the XY plane according to a given velocity command. The sub-objective includes the desired linear velocity (v). x ,v y The yaw rate ωz is used for training with objective-conditional reinforcement learning. The reward function includes velocity tracking reward, balance reward and joint regularization reward. Reaching skill: Control the robot's two wrists to reach a specified three-dimensional target position while the base remains stationary. The sub-target is the relative distance between the current position of the wrists and the target position. The reward function includes position tracking reward and attitude maintenance reward. Squatting skill: Control the robot to squat / stand up to a specified root height. The sub-target is the relative difference between the current root height and the target height. The reward function includes height tracking reward and upper body posture stability reward. Stepping skill: Control the robot to step on the ground sampling point with both feet to achieve non-grabbing operation. The sub-target is the relative distance between the current position of the two feet and the target point on the ground. The reward function includes position tracking reward and balance reward.

[0032] Once all basic skills training is complete, freeze all network parameters and stop updating them.

[0033] The high-level controller is implemented as follows: State encoding module: The input is a concatenated robot state vector and task target vector, where the robot state includes joint angles and angular velocities, base position and attitude, and contact information, and the task target is defined according to the specific task; the state encoding module adopts a 2-layer fully connected neural network with ReLU activation function, and outputs a high-level state representation with a dimension of 256.

[0034] Skill embedding module: Configure 256-dimensional learnable embedding vectors for each of the four basic skills, and optimize them synchronously during the training of the high-level controller to represent the functional semantics and applicable scenarios of each skill.

[0035] Attention weight calculation module: The matching degree is calculated using scaled dot product attention, and the calculation formula is as follows: ,in This represents the state at a higher level. For the first Embedding vectors of each skill, The vector dimension is used; attention weights are obtained after Softmax normalization. ,satisfy ≥0 and Σ =1.

[0036] Sub-target generation module: Each basic skill is configured with an independent 2-layer fully connected sub-target generation network. All networks share a 256-dimensional high-level state representation as input and output the sub-target vector of the corresponding skill.

[0037] Motion fusion module: Inputs the current state and each skill sub-target into the corresponding basic skill to obtain the motion output of each skill. Then, it performs a weighted summation using attention weights to obtain the final joint target position control command. The calculation formula is as follows: ,in For the first Basic skill strategies.

[0038] High-level controller training implementation: The high-level controller is trained using the PPO algorithm, with a discount factor γ=0.99, a clip range of 0.2, 1000 iterations, a batch size of 8192, and a learning rate of 0.0003. During training, basic skill parameters are frozen throughout; only the network parameters of the high-level controller are optimized. For each task, only 1-2 intuitive task-related reward items are set; for example, the reward function for the FarReach task is... ,in Current wrist position The target location requires no complex reward system.

[0039] Task Implementation and Performance Evaluation: This embodiment was validated across eight high-level tasks, which were categorized into three difficulty levels: Simple tasks: FarReach, ButtonPress, CabinetClose, involving short-duration interactions and minimal contact; Medium-level tasks: FootballShoot, BoxPush, PackageLift, involving a wide range of interactive elements; Difficult tasks: BoxTransfer and PackageCarry, involving multi-stage and long-term whole-body coordination and control.

[0040] During the evaluation phase, 20 independent rollouts were performed for each task. Evaluation metrics included task success rate, average energy consumption (E), and control smoothness (S), supplemented by feasibility metrics such as tilt angle, average joint torque, and average joint power. Experimental results show that this invention significantly outperforms the original SkillBlender and the end-to-end PPO baseline in all tasks. Specifically, it improves the success rate of the complex task PackageCarry by 8.2%, reduces energy consumption by 21.6%, and improves control smoothness by 9.2%. Furthermore, it achieves excellent cross-body generalization performance on the G1 robot.

[0041] In summary, the method described in this embodiment does not require retraining of basic skills and can quickly adapt to new tasks through high-level attention fusion, significantly reducing the workload of reward engineering and improving the development efficiency of new tasks. Furthermore, based on the attention mechanism, it achieves state-adaptive dynamic skill weight allocation, which can accurately match the control requirements of different stages of the task, avoid ineffective competition between skills, and significantly improve the success rate of multi-stage complex tasks. Regularization constraints are applied implicitly to the attention structure, effectively suppressing high-frequency oscillations of skill weights, significantly reducing control energy consumption and motion fluctuations, and improving the physical feasibility, execution safety, and engineering deployability of the strategy. It possesses excellent cross-embodiment generalization capabilities, can adapt to humanoid robot platforms of different configurations, and has a wide range of applicable scenarios.

[0042] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0043] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A hierarchical skill fusion control method for humanoid robots based on an attention mechanism, characterized in that, Includes the following steps: Step 1: Pre-train the basic skills to obtain multiple basic skill strategies for humanoid robots. After training is completed, freeze the parameters of all basic skills. Step 2: Based on the parameters of all basic skills frozen after training in Step 1, construct a high-level controller. The high-level controller includes a state encoding module, a skill embedding module, an attention weight calculation module, a sub-target generation module, and an action fusion module. The PPO algorithm is used to train the high-level controller. Step 3: Collect the current state information and task target information of the humanoid robot, and perform feature encoding through the state encoding module described in Step 2 to obtain a high-level state representation; Step 4: Generate learnable skill embedding vectors for each basic skill through the skill embedding module. Combined with the high-level state representation obtained in Step 3, generate attention fusion weights for each basic skill through the attention weight calculation module. Step 5: Using the sub-target generation module described in Step 2, generate corresponding sub-targets for each basic skill based on the high-level state representation. Input the current state and sub-targets into the corresponding basic skill strategy to obtain the action output of each basic skill. Step 6: Using the motion fusion module described in Step 2, the motion outputs of each basic skill are weighted and fused based on the attention fusion weight to generate the final control command for the humanoid robot.

2. The humanoid robot hierarchical skill fusion control method based on attention mechanism according to claim 1, characterized in that, The pre-training described in step one yields several basic humanoid robot skill strategies, including Walking, Reaching, Squatting, and Stepping. Each basic skill strategy is trained independently using objective-conditional reinforcement learning.

3. The humanoid robot hierarchical skill fusion control method based on attention mechanism according to claim 1, characterized in that, The method for training the high-level controller using the PPO algorithm described in step two is as follows: during the training process, only the parameters of the high-level controller are optimized, while the parameters of all basic skill strategies remain frozen.

4. The humanoid robot hierarchical skill fusion control method based on attention mechanism according to claim 1, characterized in that, The method for obtaining a high-level state representation by feature encoding in step three is as follows: using an encoding function composed of a multilayer sensing mechanism, features are extracted from the spliced ​​robot state information and task target information, and a high-level state representation with consistent dimensions is output.

5. The humanoid robot hierarchical skill fusion control method based on attention mechanism according to claim 1, characterized in that, In step four, the method for generating the attention fusion weights corresponding to each basic skill through the attention weight calculation module is as follows: the matching degree between the current high-level state representation and the embedding vector of each skill is calculated using the scaled dot product attention mechanism, and the attention fusion weights that satisfy non-negativity and sum to 1 are obtained after Softmax normalization.

6. The humanoid robot hierarchical skill fusion control method based on attention mechanism according to claim 1, characterized in that, The sub-target generation module described in step five configures an independent sub-target generation network for each basic skill, and all sub-target generation networks share the high-level state representation output by the state encoding module as input.

7. The humanoid robot hierarchical skill fusion control method based on attention mechanism according to claim 1, characterized in that, The final control command mentioned in step six is ​​the target position command for each joint of the humanoid robot, which is converted into joint torque by the proportional-derivative (PD) controller and then drives the robot to execute.

8. A hierarchical skill fusion control system for humanoid robots based on an attention mechanism, characterized in that, The system includes: The pre-training module is used to pre-train basic skills to obtain multiple basic skill strategies for humanoid robots. After training is completed, the parameters of all basic skills are frozen. The high-level controller construction module is used to construct a high-level controller based on the parameters of all basic skills frozen after the pre-training module is trained. The high-level controller includes a state encoding module, a skill embedding module, an attention weight calculation module, a sub-target generation module, and an action fusion module, and the PPO algorithm is used to train the high-level controller. The feature encoding module is used to collect the current state information and task target information of the humanoid robot, and perform feature encoding through the state encoding module in the high-level controller construction module to obtain the high-level state representation. The attention fusion weight module is used to generate learnable skill embedding vectors for each basic skill through the skill embedding module, and combine them with the high-level state representations obtained in the feature encoding module to generate attention fusion weights for each basic skill through the attention weight calculation module. The basic skill action output module is used to generate corresponding sub-targets for each basic skill based on the high-level state representation of the sub-target generation module constructed by the high-level controller module. The current state and sub-targets are input into the corresponding basic skill strategy to obtain the action output of each basic skill. The control command generation module is used to construct the motion fusion module described in the high-level controller module, and to perform weighted fusion of the motion outputs of each basic skill based on attention fusion weights to generate the final control command for the humanoid robot.

9. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

10. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-7.

Images