Humanoid robot control method and device, robot and storage medium

Abstract

The invention discloses a humanoid robot control method and device, a robot and a storage medium, and the method comprises the steps: firstly generating an upper body operation instruction and a lower body motion instruction which meet task requirements and are high-level semantics through a pre-trained VLA model, thereby guaranteeing the continuity and consistency of the upper body and lower body motions of the humanoid robot; then control signals of all joints of the lower limbs of the humanoid robot are generated through a pre-trained reinforcement learning model, accurate lower limb joint control over the lower body of the humanoid robot is achieved, and it is ensured that the posture of the humanoid robot is balanced and the action is stable in the instruction execution process; the technical problems that in the prior art, motion and operation of a humanoid robot are separated, the motion is discontinuous, and control is unstable are solved.

Description

Technical Field

[0001] This application relates to the field of intelligent device control technology, and in particular to a humanoid robot control method, device, robot and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, humanoid robots, possessing a body structure similar to humans and flexible maneuvering capabilities, have become an important direction in intelligent agent research and application. Compared with traditional wheeled or quadrupedal robots, humanoid robots can interact with the environment in a more natural manner, and have broad application prospects in scenarios such as home service, logistics handling, medical assistance, and industrial collaboration.

[0003] In recent years, with the advancement of visual and language understanding technologies, researchers have begun to explore how to enable humanoid robots to possess autonomous cognition and task execution capabilities through a unified perception, understanding, and control system. In particular, a unified control framework integrating vision, language, and motion has become an important research direction for promoting the realization of general intelligence in humanoid robots.

[0004] While existing modular architectures enhance flexibility through layering and skill decoupling, their fundamental flaw lies in their rigid, block-based structure, lacking a unified underlying action representation and a comprehensive collaborative optimization mechanism. Specifically, the cognitive, perceptual, and executive layers rely on explicit interface communication, leading to semantic gaps and information loss between high-level language reasoning and low-level dynamic control. Each skill module is trained independently for specific tasks, failing to achieve parameter sharing and joint optimization within a unified framework. This results in poor coordination and abrupt switching when combining skills, making it difficult to meet the needs of continuous dynamic adjustment in large-scale, long-term tasks. Although frameworks like R2S2 attempt to construct a latent skill space, this mapping remains a shallow patchwork, failing to achieve end-to-end differentiable optimization from perception to action. The transfer from simulation to reality relies on manual calibration rather than adaptive alignment. Such methods can complete pre-set multi-step tasks in laboratory environments, but they lack robustness to disturbances in open environments, and computational latency increases linearly with module complexity, exposing scalability bottlenecks. Therefore, relying solely on the modular serial paradigm is no longer sufficient to support the agility and generalization requirements of humanoid robots' full-body coordination. There is an urgent need to shift towards unified latent space representation and end-to-end learning to achieve true integration of perception, decision-making, and action. Summary of the Invention

[0005] This application provides a humanoid robot control method, device, robot, and storage medium to solve the aforementioned technical problems existing in the prior art.

[0006] In view of the above, the first aspect of this application provides a humanoid robot control method, the method comprising: Acquire sensory information; The perceived information is processed by a pre-trained VLA model to generate upper body operation commands and lower body movement commands. The upper body of the humanoid robot is controlled according to the upper body operation instructions, and the lower body movement instructions are used as the high-level semantic input of the pre-trained reinforcement learning model. The pre-trained reinforcement learning model processes the lower body movement commands to generate control signals for each joint of the humanoid robot's lower limbs. The movement of each joint of the lower limbs of the humanoid robot is controlled according to the control signal.

[0007] Optionally, the pre-training steps of the VLA model specifically include: Acquire visual and linguistic features; The visual and linguistic features are fused using the cross-modal Transformer structure of the VLA model to generate a multimodal representation vector. The multimodal representation vector is then decoded into a first latent action vector using an action decoder. The first latent action vector includes a first motion latent action vector and a first operation latent action vector. The visual features are processed by a preset potential action model to predict and generate potential motion actions and potential operation actions, and quantified into corresponding second potential motion action vectors and second potential operation action vectors. Align the first motion potential action vector and the first operation potential action vector with the second motion potential action vector and the second operation potential action vector.

[0008] Optionally, the pre-training step of the preset latent action model specifically includes: Obtain a first action video training set, which includes motion videos and operation videos; A preset latent action model is constructed, which is based on a discrete variational autoencoder framework, wherein the encoder adopts a DINOv2 feature extraction network and a temporal Transformer structure; The preset latent action model is trained using the first action video training set, so that the decoder of the preset latent action model reconstructs the operation action of the next frame based on the current frame image and the second latent action vector, wherein the second latent action vector includes a second motion latent action vector and a second operation latent action vector. The second potential action vector is quantized into an action dictionary, which includes discrete indices of potential motion actions and potential operation actions.

[0009] Optionally, the pre-training step of the reinforcement learning model specifically includes: Construct a reward function based on the position and orientation accuracy of the humanoid robot; The reinforcement learning model is trained using the reward function, with constraints including minimizing positional deviation, ensuring orientation alignment, and maintaining center of gravity stability.

[0010] Optionally, the pre-training step of the reinforcement learning model further includes: Obtain a second action video training set, which contains operation trajectory sequences; The upper body motion trajectory is sampled from the second action video training set; The upper body motion trajectory is combined with the periodic angular perturbation and torque load change applied to the humanoid robot as an upper body perturbation signal to train the reinforcement learning model.

[0011] Optionally, the upper body operation command is specifically the target joint angle control signal of the humanoid robot; The lower body movement commands are specifically four-dimensional movement commands, including forward and backward movement speed commands, left and right movement speed commands, turning angular velocity commands, and target body height commands.

[0012] Optionally, the lower body movement command is a discrete action command; The specific steps of processing the lower body movement commands through the pre-trained reinforcement learning model to generate control signals for each joint of the humanoid robot's lower limbs are as follows: The discrete motion commands are processed by the pre-trained reinforcement learning model, and combined with the proximal policy optimization algorithm, control signals for each joint of the humanoid robot's lower limbs are generated.

[0013] A second aspect of this application provides a humanoid robot control device, the device comprising: Acquisition unit, used to acquire perceived information; The first processing unit is used to process the perceived information through a pre-trained VLA model to generate upper body operation instructions and lower body movement instructions; and to perform motion control on the upper body of the humanoid robot according to the upper body operation instructions, and to use the lower body movement instructions as the high-level semantic input of the pre-trained reinforcement learning model. The second processing unit is used to process the lower body movement commands through the pre-trained reinforcement learning model, generate control signals for each joint of the lower limbs of the humanoid robot, and perform motion control on each joint of the lower limbs of the humanoid robot according to the control signals.

[0014] A third aspect of this application provides a robot, the device including a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the steps of the humanoid robot control method as described in the first aspect above, according to the instructions in the program code.

[0015] A fourth aspect of this application provides a computer-readable storage medium for storing program code for executing the humanoid robot control method described in the first aspect above.

[0016] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: This application provides a humanoid robot control method, device, robot, and storage medium. First, a pre-trained VLA model is used to generate upper body operation commands and lower body movement commands that meet task requirements and are of high-level semantics, thereby ensuring the continuity and consistency of the upper and lower body movements of the humanoid robot. Then, a pre-trained reinforcement learning model is used to generate control signals for each joint of the lower limbs of the humanoid robot, realizing precise lower limb joint control of the humanoid robot's lower body, ensuring the posture balance and movement stability of the humanoid robot during command execution, and solving the technical problems of separation of motion and operation, discontinuous movement, and unstable control in humanoid robots in the prior art. Attached Figure Description

[0017] Figure 1 This is a flowchart of the humanoid robot control method in the embodiments of this application; Figure 2 This is a schematic diagram of the model architecture for implementing the humanoid robot control method in the embodiments of this application; Figure 3 This is a flowchart illustrating the VLA model training method in the embodiments of this application. Figure 4 This is a schematic diagram of the humanoid robot control device in the embodiments of this application; Figure 5 This is a schematic diagram of the humanoid robot structure in an embodiment of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0019] In recent years, academia has proposed various novel architectures combining large-scale models with robot control, among which systems based on modular design principles are particularly representative. These modular design-based systems generally employ a hierarchical structure for modeling, dividing robot control into cognitive, perceptual, and execution layers. The higher layers use language models for task understanding and reasoning, generating action plans; the middle layers use visual-language models for scene recognition and semantic parsing; and the lower layers consist of multiple independent skill modules that execute specific operations. Each skill module typically uses an independent model for individual training through imitation learning, such as stabilizing movement, posture adjustment, or grasping and placing. During execution, the system invokes the corresponding skill module based on language commands, with the middle layer handling the conversion from visual to action parameters. This approach offers good flexibility and interpretability in task execution, enabling multi-step tasks such as object handling, sorting, or assembly to be completed in a laboratory environment, demonstrating language-driven task planning and multi-skill invocation capabilities.

[0020] The inventors of this application have discovered that while such modular design methods are structurally clear and easy to implement, when motion and operation are implemented by different control modules or skill networks, the overall system lacks a collaborative optimization mechanism due to inconsistencies in the input characteristics, control objectives, and optimization methods of different modules. Because different actions employ different learning models, discontinuities and error accumulation easily occur during state transitions between skill modules, leading to problems such as unstable action transitions, response delays, and task failures during execution. Furthermore, the lack of unified action representations and dynamic constraints among modules results in significant discontinuities during the transition between motion and operation. When the robot needs to complete an operation during a movement or continue moving after an operation, the system struggles to maintain posture balance and action consistency. This fragmented modular architecture makes it difficult to form a complete closed-loop feedback mechanism and achieve cross-stage collaborative optimization, resulting in insufficient stability for the robot when performing long-duration, multi-stage tasks. This limitation is particularly pronounced in large-scale, long-distance, and continuous motion tasks, severely restricting the actual performance capabilities of humanoid robots in complex environments.

[0021] Another modular design approach centers on skill space learning. It pre-trains multiple basic skills (including movement, posture adjustment, and hand reach) in simulated and real-world environments, then maps high-level strategies to skills through a latent skill space. The system integrates skills within this latent space to achieve full-body tasks such as "touching a target" and "picking up an object." Technically, this design proposes a hierarchical mapping mechanism from basic skills to latent space and then to task control, achieving skill reuse and transfer to a certain extent. Its feasibility from simulation to reality has been verified in various experiments.

[0022] However, the inventors of this application discovered that, in this approach, due to differences in training objectives, range of motion, and control cycles of various basic skills, it is difficult to accurately model the coordination relationships between different skills. High-level planning strategies can only switch at discrete skill levels, rather than being optimized in a continuous control space, thus failing to achieve true whole-body coordinated control. When tasks involve complex behaviors such as continuous motion, posture adjustment, and dynamic manipulation, robots often exhibit problems such as fragmented actions, delayed responses, or control instability.

[0023] Therefore, overall, while the aforementioned methods of hierarchical modeling or skill integration enable robots to perform complex tasks to a certain extent, their overall control remains fragmented, lacking a unified motion representation and a whole-body collaborative optimization mechanism. Furthermore, existing systems generally lack unified training objectives and state representations; motion and manipulation are independent in terms of data acquisition methods, input features, optimization functions, and reward design, making joint training difficult to converge and causing the model to degrade easily during cross-task transfer. Moreover, the lack of a unified working representation leads to low information transfer efficiency between different modules, limiting the overall controllability and robustness of the system. Additionally, some control systems rely on cloud-based inference or large language models for task planning, resulting in response latency and making them unable to operate under high-frequency closed-loop control.

[0024] As research progressed, the inventors of this application gradually realized that relying solely on modular serialization was insufficient to meet the execution needs of humanoid robots in large spaces, long durations, and continuous tasks. Future technological directions are shifting from discrete control to unified latent space representation and end-to-end learning, aiming to achieve the integration of perception, decision-making, and action generation.

[0025] In view of this, embodiments of this application propose a humanoid robot control method, device, robot, and computer-readable storage medium to solve the aforementioned technical problems existing in the prior art. Embodiments of this application construct an intelligent humanoid robot control framework with unified representation, a whole-body system, and high stability. It fuses visual, linguistic, and action information through a pre-trained Vision-Language-Action (VLA) model. The VLA model establishes an end-to-end closed-loop control from perception to execution, generating operational actions for the upper body of the humanoid robot and lower body movement commands with high-level semantics for the lower body. These lower body movement commands are then input into a reinforcement learning model, which generates specific operational actions for each joint of the lower body. This application's embodiments integrate the VLA model with a reinforcement learning model, achieving a balance between high-level semantic interpretability and low-level dynamic stability at the algorithmic level. On one hand, the VLA model generates upper-body maneuvering actions and high-level semantic lower-body movement commands that meet task requirements, coordinating upper and lower-body movements to ensure their continuity and consistency, thus achieving a unified representation of motion and manipulation. On the other hand, the reinforcement learning model enables precise lower limb joint control, forming stable whole-body coordination and achieving precise movement and stable manipulation.

[0026] Furthermore, in this embodiment, a unified latent action learning framework is employed during the training of the VLA model. This framework maps both motion and manipulation actions to the same latent motion space for joint modeling and shared representation, achieving a unified action representation for upper and lower body manipulation actions. Simultaneously, through learning and quantization in the latent space, the VLA model in this embodiment can extract transferable whole-body motion features under low-cost, limited-data conditions, significantly reducing the dependence on whole-body motion-manipulation data collection and improving data utilization and learning efficiency.

[0027] Furthermore, in this embodiment, discretized command design and precision-guided training are used between the VLA model and the reinforcement learning model to enable the humanoid robot to adjust its position and orientation more accurately when performing operational tasks. This improves the generalization ability of the humanoid robot, avoids motion drift and posture error accumulation problems, and significantly enhances the predictability and stability of whole-body coordinated control, providing a solid foundation for the task execution of general-purpose humanoid robots in practical application scenarios.

[0028] It should be noted that the humanoid robot control method proposed in this application is not only applicable to bipedal humanoid robots, but can also be extended to mobile platforms with robotic arms, quadrupedal robotic arm composites, exoskeletons, or other multi-degree-of-freedom systems. In addition to tasks such as handling, loading, and pushing / pulling, it can also be used in various scenarios such as opening doors, assembly, collaborative operation, emergency rescue, inspection, and home assistance.

[0029] Specifically, please refer to Figures 1 to 3 , Figure 1 This is a flowchart of the humanoid robot control method in the embodiments of this application; Figure 2 This is a schematic diagram of the model architecture for implementing the humanoid robot control method in the embodiments of this application; Figure 3 This is a flowchart illustrating the VLA model training method in an embodiment of this application; as follows: Figures 1 to 3 As shown, specifically: S101. Acquire sensory information; It should be noted that perceptual information can be divided into visual observation and task instructions. In terms of visual observation, humanoid robots can use monocular, binocular or multi-view vision, depth cameras, inertial measurement units, tactile or force sensors, or combinations thereof. In terms of task instructions, humanoid robots can use language, images, gestures, buttons, example trajectories or other symbolic input methods.

[0030] S102. Process the perceived information using a pre-trained VLA model to generate upper body operation commands and lower body movement commands. Before analyzing the perceived information using the VLA model, the VLA model needs to be trained so that it can generate corresponding motion control outputs based on the acquired perceived information.

[0031] In the training process of the VLA model, this application proposes a unified modeling mechanism based on latent action representation. Latent action refers to a high-level action vector that discretizes the motion changes between adjacent image frames in the latent space, describing the dynamic trend of body movements rather than specific joint parameters. In this application, latent actions are divided into two types: motion and manipulation. The specific training process for the VLA model is as follows: Figure 2 and Figure 3 As shown, it includes: S201. Obtain visual and linguistic features; It should be noted that the VLA model constructed in this application embodiment is based on the EVA-CLIP visual encoder and the LLaVA-1.6 language encoder, combined with a cross-modal Transformer structure. The example is to use the EVA-CLIP visual encoder to extract visual features (such as images and videos) and the LLaVA language encoder to extract language features (such as task descriptions).

[0032] The EVA-CLIP visual encoder is an advanced visual feature extraction model that improves and optimizes upon the CLIP (Contrastive Language-Image Pre-training) architecture, specifically designed for processing and understanding visual information. Building upon CLIP, EVA-CLIP further enhances the model's visual encoding capabilities, particularly in handling complex scenes and fine-grained visual features. It improves the accuracy of recognizing objects, scenes, and actions in images by introducing a deeper convolutional neural network (CNN) structure and attention mechanisms. EVA-CLIP can generate high-dimensional visual feature vectors that not only contain static information in images but also capture dynamic changes and spatiotemporal relationships.

[0033] The LLaVA (Large Language Model with Vision Alignment) series of models aims to learn aligned representations between language and vision through joint training on large-scale multilingual text and image data, thereby achieving excellent performance in various language understanding and generation tasks. It employs a Transformer architecture and processes sequential data through a self-attention mechanism. During training, the model learns deep associations between language and visual modalities by predicting occluded words or image regions.

[0034] In this embodiment, visual and linguistic features are extracted using the EVA-CLIP visual encoder and the LLaVA language encoder, which improves the feature extraction capability of the VLA model and enhances the training effect.

[0035] S202. Visual features and language features are fused through the cross-modal Transformer structure of the VLA model to generate a multimodal expression vector. The multimodal expression vector is then decoded into a first latent action vector through an action decoder. The first latent action vector includes a first motion latent action vector and a first operation latent action vector. It should be noted that after feature extraction, the visual and linguistic features are first aligned using a cross-modal attention mechanism, enabling the model to focus on relevant parts of the image and text and establish a connection between them.

[0036] Then, the aligned visual and linguistic features are fused using a cross-modal Transformer structure. The cross-modal Transformer structure interacts and integrates features from both modalities at multiple levels through self-attention and cross-attention mechanisms. In each Transformer layer, visual and linguistic features first integrate internal information through a self-attention layer, and then interact between modalities through a cross-attention layer. This process is repeated, allowing visual and linguistic features to be further fused and refined at each layer.

[0037] Ultimately, the cross-modal Transformer generates a unified multimodal representation vector. This multimodal representation vector not only contains the original information of the image and text but also integrates the correlation and interaction between them, enabling a more comprehensive representation of the input data. This multimodal representation vector can then be used for downstream tasks such as image caption generation, video question answering, and visual reasoning, generating corresponding outputs through a decoder.

[0038] S203. Visual features are processed by a preset latent action model to predict and generate motion latent actions and operation latent actions, and quantified into corresponding second motion latent action vectors and second operation latent action vectors. It should be noted that, in the embodiments of this application, a latent action model is introduced when training the VLA model. The latent action model outputs a high-level action vector by discretizing the motion changes between adjacent image frames in the latent space, which is used to describe the dynamic trend of body movements rather than specific joint parameters.

[0039] Before predicting the potential action output using the latent action model, the latent action model first needs to be trained, specifically as follows: Obtain the first action video training set, which includes motion videos and operation videos; It should be noted that when pre-training the potential action model, it is first necessary to obtain an action video training set, which includes motion videos and operation videos. Among them, motion videos can be human first-person perspective motion videos, which capture human movements and posture changes during natural movement; operation videos can be Agibot-World open-source dual-arm operation videos, which contain about 250 hours of complex operation videos, such as tasks such as picking up objects, packing boxes, pushing carts, carrying, and stacking. These video materials provide rich samples of dual-arm operation actions.

[0040] A pre-defined latent action model is constructed, which is based on a discrete variational autoencoder framework, wherein the encoder adopts a DINOv2 feature extraction network and a temporal Transformer structure. It should be noted that the preset latent action model can be divided into two categories: motion latent action model and operation latent action model. Both can adopt the discrete variational autoencoder (VQ-VAE) framework, and the encoder adopts the DINOv2 feature extraction network and temporal Transformer structure.

[0041] The preset latent action model is trained by the first action video training set, so that the decoder of the preset latent action model can reconstruct the operation action of the next frame based on the current frame image and the second latent action vector. The second latent action vector includes the second motion latent action vector and the second operation latent action vector. It should be noted that when training a preset latent motion model based on motion videos in the acquired motion video training set, temporal modeling is performed on consecutive video frames. The decoder reconstructs the next frame based on the current frame image and the motion latent motion vector, thereby learning the changing patterns of motion in the latent space.

[0042] When training a preset latent action model on operation videos in the acquired action video training set, the model learns the changing patterns of operation actions. The decoder then reconstructs the operation action of the next frame based on the current frame image and the operation latent action vector.

[0043] The second potential action vector is quantized into an action dictionary, which includes discrete indices of potential motion actions and potential operational actions.

[0044] It should be noted that the aforementioned potential motion vectors and potential operation vectors are quantized into 1024 discrete word vectors to form an action dictionary. Each word vector is 8-dimensional and is used to distinguish different types of body movements, such as gait changes, arm extensions, and grasping trajectories.

[0045] In this embodiment, by mapping two types of actions, motion and operation, to the same latent space for joint modeling and shared representation, and through the learning and quantification of the latent space, transferable whole-body motion features can be extracted under low-cost and limited data conditions, which greatly reduces the dependence on whole-body motion-operation data collection and improves data utilization and learning efficiency.

[0046] When predicting output latent motion actions and operation latent actions through the trained latent action model, the output head of the VLA model predicts the discrete index of the output latent motion action and the discrete index of the operation latent action based on the multimodal representation vector. Then, it decodes the discrete index of the operation latent action and the discrete index of the motion latent action through the action dictionary to generate the operation latent action vector and the motion latent action vector.

[0047] S204. Align the first motion potential action vector and the first operation potential action vector with the second motion potential action vector and the second operation potential action vector.

[0048] It should be noted that the VLA training phase achieves the fusion and unified supervision of the motion latent action model and the operation latent action model. Through the abstract representation of the latent space, the model's generalization ability and data utilization efficiency in cross-task scenarios are improved, the dependence on motion capture labels is reduced, and stable full-body coordinated behavior can be learned using only video data.

[0049] Through the above steps, this embodiment of the application utilizes motion and manipulation data to learn latent action models during the training phase, and achieves fusion representation in a unified latent space, thereby optimizing the motion and manipulation collaborative reinforcement learning strategy. Experimental results show that this method can stably learn coordinated whole-body control strategies while significantly reducing the amount of whole-body motion and manipulation training data. Compared with traditional modular methods, this embodiment of the application shows significant improvements in both training convergence speed and policy stability.

[0050] S103. Perform motion control on the upper body of the humanoid robot according to the upper body operation instructions, and use the lower body motion instructions as the high-level semantic input of the pre-trained reinforcement learning model. It should be noted that the VLA model's Action Decoder generates upper-body manipulation actions and lower-body movement commands separately. The upper-body manipulation actions can be specific movements targeting the upper limbs. These actions are then converted by the decoding network into angle control signals for the 14 joints of both arms, directly controlling the upper body of the humanoid robot. The lower-body movement commands are high-level semantics, mapped to four-dimensional motion commands, specifically forward / backward speed, left / right translation speed, turning angular velocity, and target height control values, which are finally passed to the reinforcement learning model.

[0051] In this embodiment, the inventors consider that the upper body movements of humanoid robots (such as grasping and carrying with the arms and hands) are relatively simple. Target joint angle control signals can be directly generated through visual and verbal commands. The VLA model, through cross-modal feature fusion, can directly output control signals for the upper body, achieving precise operation. However, the lower body movements of humanoid robots (such as movement, turning, and balance) are more complex, involving dynamic balance, posture adjustment, and ground contact. This embodiment uses a reinforcement learning model to control the lower body. Through interactive learning with the environment, it can generate continuous control signals adapted to different terrains and load conditions, ensuring the stability and accuracy of movement.

[0052] Meanwhile, to achieve full-body coordinated control while "walking and operating," the lower body needs to dynamically coordinate with the upper body movements. In this embodiment, the VLA model simultaneously outputs specific operational actions for the upper body and high-level semantic operational instructions for the lower body. This enables the reinforcement learning model to adjust its motion strategy in real time based on the perturbations and load changes generated by the upper body operations, maintaining overall balance and stability. The VLA model is primarily responsible for understanding and executing specific operational tasks, while the reinforcement learning model focuses on motion and posture control. The combination of the two can achieve more efficient and stable full-body control.

[0053] Therefore, in this step, the upper body is directly controlled by the VLA model, while the lower body is controlled by the reinforcement learning model. This combines the advantages of both and achieves stable and coordinated full-body control of the humanoid robot in complex environments.

[0054] S104. Process the lower body movement commands through a pre-trained reinforcement learning model to generate control signals for each joint of the lower limbs of the humanoid robot. It should be noted that, in this embodiment of the application, in order to improve the generalization ability of the reinforcement learning model and avoid problems such as posture deviation and discontinuous movement caused by velocity tracking errors, the lower limb control interface of the VLA model is redesigned to be based on discrete motion intentions, so that the output lower body movement commands are discrete motion commands. The discrete motion commands use three directional dimensions (forward and backward, left and right, and turning) to represent start, stop, and reverse with "1 / 0 / -1" respectively, rather than continuous velocity values, and an additional target body height channel is added to control standing up and squatting down.

[0055] Upon receiving four-dimensional motion commands from the VLA model, the reinforcement learning model in this embodiment utilizes Proximal Policy Optimization (PPO) to generate control signals for 12 lower limb joints, including target joint angles and motor torques. The reinforcement learning model consists of two fully connected perceptron layers with hidden layer dimensions of 512 and 256, respectively, and a control frequency of 50Hz. It is synchronized in real-time with the simulation physics engine to generate low-level joint control signals, i.e., lower body movements, achieving posture stability and coordinated control.

[0056] Among them, the proximal policy optimization algorithm is a widely used policy optimization algorithm in the field of deep reinforcement learning. It aims to address the problem of excessively large or small update step sizes in traditional policy gradient methods (such as the REINFORCE algorithm) during training. By introducing a "proximal" constraint, it limits the magnitude of policy updates, thereby ensuring the stability of policy updates. In the embodiments of this application, other algorithms can also be used to generate specific low-level joint control signals.

[0057] It should be noted that, in addition to using proximal policy optimization algorithms to generate lower body movements, reinforcement learning models can also employ soft actor-critic, distributed or offline reinforcement learning, and can be combined with imitation learning, self-supervised contrastive learning or curriculum training.

[0058] In this embodiment of the application, in order to achieve precise control of the reinforcement learning model, it is necessary to pre-train the reinforcement learning model. In this embodiment of the application, a two-stage course learning mechanism is used to train the reinforcement learning model, specifically: Construct a reward function based on the position and orientation accuracy of a humanoid robot; The reinforcement learning model is trained using a reward function, with constraints including minimizing positional deviation, ensuring orientation alignment, and maintaining center of gravity stability.

[0059] It should be noted that the first stage is the basic gait learning stage, with the main objective of achieving posture stability and precise displacement control. The reward function in this stage, based on the posture balance and energy consumption constraints design in HOMIE (RSS 2025), adds a reward function tailored to the humanoid robot's position and orientation accuracy. During training, the reinforcement learning model no longer pursues continuous velocity tracking, but instead encourages the robot to accurately reach the target pose, including minimizing positional deviation, orientation alignment, and center of gravity stability.

[0060] Specifically, the reward function considers the verticality of the upper body, the balance of the center of mass projection, the continuity of foot contact, the gait symmetry, and the target pose error, so that the robot can reach the target with high precision and maintain balance in various terrains, thus "walking accurately" rather than just "walking steadily".

[0061] in: Upper body verticality: The reward robot maintains an upright upper body and reduces tilting; Center of mass projection balance: Rewards the robot to keep its center of mass within the support surface, reducing the risk of falling; Foot contact continuity: The robot is rewarded for maintaining continuous contact between its feet and the ground, avoiding jumping or sliding; Gait symmetry: Reward the robot to maintain gait symmetry and reduce swaying during movement; Target pose error: reward the robot for accurately reaching the target position and orientation, reducing deviation.

[0062] The humanoid robot is trained in various terrains and, guided by the aforementioned rewards, learns how to maintain posture stability and precise control during movement.

[0063] Obtain the second action video training set, which contains operation videos; Sampling of upper body movement trajectories was performed on the second set of action video training data; The upper body motion trajectory is combined with the periodic angular perturbations and torque load changes applied to the humanoid robot as upper body perturbation signals to train the reinforcement learning model.

[0064] It should be noted that the second stage is the collaborative behavior optimization stage. Based on the basic gait stabilization, the upper limb motion perturbation signal generated by the VLA model is introduced, and the upper body motion trajectory is sampled from the Agibot-World dual-arm operation dataset. Periodic angle perturbation and torque load changes are applied to the humanoid robot. The reinforcement learning model is trained based on the upper body motion trajectory, periodic angle perturbation and torque load changes to realistically simulate the inertial coupling and reaction force effects in actual operation.

[0065] Reinforcement learning models learn how to maintain body balance and stable movement direction when performing upper limb operations by continuously optimizing posture adjustment and center of gravity compensation strategies under perturbed environments, thereby achieving natural coordination between operation and movement. The optimization focus at this stage is no longer limited to maintaining stability, but also maximizes task execution accuracy and posture recovery capability, enabling the robot to accurately control its position and direction when performing pushing, pulling, carrying, or bi-handed operations, thus completing complex "walking and operating" tasks.

[0066] In the above embodiments, the reinforcement learning model trained using the aforementioned training method not only possesses higher posture robustness and anti-interference capability, but also achieves breakthroughs in the interpretability and execution accuracy of motion commands. Based on semantic-level motion commands generated by VLA, the reinforcement learning model dynamically adjusts its posture and outputs coordinated lower limb movements in a perturbed environment, thereby achieving full-body coordinated control while walking. The reinforcement learning model is driven at a high level by visual-language input, effectively connecting the semantic and physical layers through latent action learning, and then implementing low-level execution control through reinforcement learning strategies. Experiments show that this method can stably complete multi-stage tasks, including continuous actions such as "picking up an object—turning—carrying—placing," without relying on motion capture data. The action coherence is significantly improved, the task success rate is increased by more than 30%, and posture stability and behavioral consistency are maintained under different load and environmental perturbation conditions.

[0067] S105. Perform motion control on each joint of the lower limbs of the humanoid robot according to the control signal.

[0068] Please see Figure 4 , Figure 4 This is a schematic diagram of the humanoid robot control device in an embodiment of this application, as shown below. Figure 4 As shown, specifically: Acquisition unit 401 is used to acquire perceived information; The first processing unit 402 is used to process the perceived information through a pre-trained VLA model to generate upper body operation instructions and lower body movement instructions; and to perform motion control on the upper body of the humanoid robot according to the upper body operation instructions, and to use the lower body movement instructions as the high-level semantic input of the pre-trained reinforcement learning model. The second processing unit 403 is used to process the lower body movement commands through a pre-trained reinforcement learning model, generate control signals for each joint of the lower limbs of the humanoid robot, and perform motion control on each joint of the lower limbs of the humanoid robot according to the control signals.

[0069] Another embodiment of the present invention provides a humanoid robot, such as Figure 5 As shown, device 10 includes: One or more processors 110 and memory 120, Figure 5 The following description uses a processor 110 as an example. The processor 110 and the memory 120 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0070] Processor 110 is used to perform various control logics of device 10, and can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0071] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the method for constructing the multilingual phoneme representation model in this embodiment of the invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, thereby implementing the method for constructing the multilingual phoneme representation model in the above-described method embodiment.

[0072] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0073] One or more units are stored in memory 120, and when executed by one or more processors 110, perform the following steps: Acquire sensory information; The pre-trained VLA model processes the perceived information to generate upper body operation commands and lower body movement commands. The upper body of the humanoid robot is controlled by upper body operation commands, and the lower body movement commands are used as high-level semantic input to the pre-trained reinforcement learning model. The lower body movement commands are processed by a pre-trained reinforcement learning model to generate control signals for each joint of the humanoid robot's lower limbs. The movement of each joint of the humanoid robot's lower limbs is controlled according to the control signals.

[0074] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by one or more processors, they implement any one of the embodiments of the humanoid robot control method described above.

[0075] As examples, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0076] This application provides a humanoid robot control method, device, robot, and storage medium. First, a pre-trained VLA model is used to generate upper body operation commands and lower body movement commands that meet task requirements and are of high-level semantics, thereby ensuring the continuity and consistency of the upper and lower body movements of the humanoid robot. Then, a pre-trained reinforcement learning model is used to generate control signals for each joint of the lower limbs of the humanoid robot, realizing precise lower limb joint control of the humanoid robot's lower body, ensuring the posture balance and movement stability of the humanoid robot during command execution, and solving the technical problems of separation of motion and operation, discontinuous movement, and unstable control of humanoid robots in the prior art.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0078] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0079] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0084] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0085] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A method for controlling a humanoid robot, characterized in that, include: Acquire sensory information; The perceived information is processed by a pre-trained VLA model to generate upper body operation commands and lower body movement commands. The upper body of the humanoid robot is controlled according to the upper body operation instructions, and the lower body movement instructions are used as the high-level semantic input of the pre-trained reinforcement learning model. The pre-trained reinforcement learning model processes the lower body movement commands to generate control signals for each joint of the humanoid robot's lower limbs. The movement of each joint of the lower limbs of the humanoid robot is controlled according to the control signal.

2. The humanoid robot control method according to claim 1, characterized in that, The pre-training steps of the VLA model are as follows: Acquire visual and linguistic features; The visual and linguistic features are fused using the cross-modal Transformer structure of the VLA model to generate a multimodal representation vector. The multimodal representation vector is then decoded into a first latent action vector using an action decoder. The first latent action vector includes a first motion latent action vector and a first operation latent action vector. The visual features are processed by a preset potential action model to predict and generate potential motion actions and potential operation actions, and quantified into corresponding second potential motion action vectors and second potential operation action vectors. Align the first motion potential action vector and the first operation potential action vector with the second motion potential action vector and the second operation potential action vector.

3. The humanoid robot control method according to claim 2, characterized in that, The pre-training steps of the preset latent action model are as follows: Obtain a first action video training set, which includes motion videos and operation videos; A preset latent action model is constructed, which is based on a discrete variational autoencoder framework, wherein the encoder adopts a DINOv2 feature extraction network and a temporal Transformer structure; The preset latent action model is trained using the first action video training set, so that the decoder of the preset latent action model reconstructs the operation action of the next frame based on the current frame image and the second latent action vector, wherein the second latent action vector includes a second motion latent action vector and a second operation latent action vector. The second potential action vector is quantized into an action dictionary, which includes discrete indices of potential motion actions and potential operation actions.

4. The humanoid robot control method according to claim 1, characterized in that, The pre-training steps of the reinforcement learning model are as follows: Construct a reward function based on the position and orientation accuracy of the humanoid robot; The reinforcement learning model is trained using the reward function, with constraints including minimizing positional deviation, ensuring orientation alignment, and maintaining center of gravity stability.

5. The humanoid robot control method according to claim 4, characterized in that, The pre-training steps of the reinforcement learning model also include: Obtain a second action video training set, which contains operation trajectory sequences; The upper body motion trajectory is sampled from the second action video training set; The upper body motion trajectory is combined with the periodic angular perturbation and torque load change applied to the humanoid robot as an upper body perturbation signal to train the reinforcement learning model.

6. The humanoid robot control method according to claim 1, characterized in that, The upper body operation command is specifically the target joint angle control signal of the humanoid robot; The lower body movement commands are specifically four-dimensional movement commands, including forward and backward movement speed commands, left and right movement speed commands, turning angular velocity commands, and target body height commands.

7. The humanoid robot control method according to claim 1, characterized in that, The lower body movement commands are discrete action commands; The specific steps of processing the lower body movement commands through the pre-trained reinforcement learning model to generate control signals for each joint of the humanoid robot's lower limbs are as follows: The discrete motion commands are processed by the pre-trained reinforcement learning model, and combined with the proximal policy optimization algorithm, control signals for each joint of the humanoid robot's lower limbs are generated.

8. A humanoid robot control device, characterized in that, include: Acquisition unit, used to acquire perceived information; The first processing unit is used to process the perceived information through a pre-trained VLA model to generate upper body operation instructions and lower body movement instructions; and to perform motion control on the upper body of the humanoid robot according to the upper body operation instructions, and to use the lower body movement instructions as the high-level semantic input of the pre-trained reinforcement learning model. The second processing unit is used to process the lower body movement commands through the pre-trained reinforcement learning model, generate control signals for each joint of the lower limbs of the humanoid robot, and perform motion control on each joint of the lower limbs of the humanoid robot according to the control signals.

9. A humanoid robot, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the humanoid robot control method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the humanoid robot control method according to any one of claims 1-7.

Images