Robot control method, model training method, device and electronic equipment

Abstract

This application discloses a control method, model training method, device, and electronic device for an embodied robot, relating to the field of robotics. The control method includes: acquiring instruction text, robot pose information, and environmental images captured by the robot. The environmental images include a first two-dimensional environmental image and a first environmental depth image captured by the robot's left arm, a second two-dimensional environmental image and a second environmental depth image captured by the robot's right arm, and a third two-dimensional environmental image and a third environmental depth image captured by the robot's head. The method processes the instruction text, robot pose information, and environmental images based on a motion prediction model to obtain the robot's action at the next moment, thereby controlling the robot's movement based on the robot's action at the next moment. This method enhances the environmental perception capability of the motion prediction model and improves its generalization performance and motion execution stability in scenarios with few samples.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a control method, model training method, device, and electronic equipment for an embodied robot. Background Technology

[0002] In embodied robot control scenarios, a model is typically trained to predict the robot's actions to execute commands, thereby controlling the robot to perform those actions and complete the commands. However, most current models are trained based on open-source data of embodied robots, resulting in a lack of scenario generalization in embodied robot control schemes. The models have low accuracy in training scenarios with few or no samples. Furthermore, the visual feature information acquired by the models in the aforementioned existing technical solutions is broad, resulting in poor precision in predicting the position in the action space, thus leading to low accuracy in robot action control. Summary of the Invention

[0003] In view of this, embodiments of this application provide a android control method, a model training method, an apparatus, and an electronic device.

[0004] According to a first aspect of this application, embodiments of this application provide a method for controlling a hymenoidae robot, including:

[0005] The robot obtains command text, robot pose information, and environmental images collected by the robot. The environmental images include a first two-dimensional environmental image and a first environmental depth image collected by the robot's left arm, a second two-dimensional environmental image and a second environmental depth image collected by the robot's right arm, and a third two-dimensional environmental image and a third environmental depth image collected by the robot's head.

[0006] The motion prediction model processes instruction text, robot pose information, and environmental images to obtain the robot's next action, thereby controlling the robot's movement based on the next action. The training process of the motion prediction model includes a pre-training process and a fine-tuning training process. The training data in the pre-training process includes a first training instruction text, a first training environment image captured when the human body executes the instruction in the first training instruction text, and human pose information. The first training environment image includes a first two-dimensional image and a first training environment depth image captured by the left arm, a second two-dimensional image and a second training environment depth image captured by the right arm, and a third two-dimensional image and a third training environment depth image captured by the head. The training data in the fine-tuning training process includes a second training instruction text, a second training environment image captured when the training robot executes the instruction in the second training instruction text, and training robot pose information. The second training environment image includes a fourth two-dimensional image and a fourth training environment depth image captured by the left arm, a fifth two-dimensional image and a fifth training environment depth image captured by the right arm, and a sixth two-dimensional image and a sixth training environment depth image captured by the head.

[0007] Optionally, the action prediction model includes a visual feature extraction module, a text feature extraction module, a pose feature extraction module, a feature fusion module, and a decision module;

[0008] The visual feature extraction module is used to determine the robot's visual features based on environmental images;

[0009] The text feature extraction module is used to determine the robot's command features based on the command text;

[0010] The pose feature extraction module is used to determine the joint pose features of the robot based on the robot pose information;

[0011] The feature fusion module is used to fuse visual features, command features, and joint pose features to obtain the first fused feature;

[0012] The decision-making module is used to predict the robot's next action based on the first fused features.

[0013] Optionally, the visual feature extraction module is used to determine first-view two-dimensional image features based on the first environmental two-dimensional image; determine first-view depth image features based on the first environmental depth image; and fuse the first-view two-dimensional image features and the first-view depth image features to obtain first-view image features.

[0014] Based on the second environment two-dimensional image, determine the second view two-dimensional image features; based on the second environment depth image, determine the second view depth image features; fuse the second view two-dimensional image features and the second view depth image features to obtain the second view image features;

[0015] Based on the three-dimensional image of the third environment, the features of the three-dimensional image of the third view are determined; based on the depth image of the third environment, the features of the depth image of the third view are determined; the features of the three-dimensional image of the third view and the features of the depth image of the third view are fused to obtain the features of the third view image.

[0016] The robot's visual features are obtained by fusing the features from the first-view image, the second-view image, and the third-view image.

[0017] Optionally, the pose feature extraction module is used to determine the joint pose features of the robot based on the robot pose information and the robot's historical pose information; the robot's historical pose information includes the robot's pose information at least at the previous moment.

[0018] Optionally, the decision module includes a cross-series cross-attention module and a hybrid expert model module, as well as an action head module;

[0019] The cross-attention module is used to fuse the first fusion feature to obtain the second fusion feature;

[0020] The hybrid expert model module is used to process the second fusion feature based on a multi-expert strategy to obtain multi-expert prediction results;

[0021] The motion head module is used to predict the robot's next action based on multi-expert prediction results.

[0022] Optionally, the hybrid expert model module includes a routing submodule, multiple expert submodules, a shared expert submodule, and an output submodule;

[0023] The routing submodule is used to route the second fusion feature to two expert submodules and a shared expert submodule based on the second fusion feature;

[0024] Two expert submodules are used to determine the expert prediction results based on the second fusion feature;

[0025] The shared expert submodule is used to determine the shared expert prediction results based on the second fusion feature;

[0026] The output submodule is used to fuse expert prediction results and shared expert prediction results to obtain multi-expert prediction results.

[0027] According to a second aspect of this application, embodiments of this application provide a model training method, including:

[0028] The system acquires a first training instruction text, a first training environment image captured when the human body executes the instructions in the first training instruction text, and human body pose information. The first training environment image includes a first training environment two-dimensional image and a first training environment depth image captured by the human body's left arm, a second training environment two-dimensional image and a second training environment depth image captured by the right arm, and a third training environment two-dimensional image and a third training environment depth image captured by the head.

[0029] Based on the first training instruction text, the first training environment image, and human pose information, the neural network is trained to obtain an initial action prediction model.

[0030] The system acquires a second training instruction text, a second training environment image captured when the robot executes the instructions in the second training instruction text, and pose information of the training robot. The second training environment image includes a fourth training environment two-dimensional image and a fourth training environment depth image captured by the left arm of the training robot, a fifth training environment two-dimensional image and a fifth training environment depth image captured by the right arm, and a sixth training environment two-dimensional image and a sixth training environment depth image captured by the head.

[0031] Based on the second training instruction text, the second training environment image, and the pose information of the training robot, the initial motion prediction model is trained to obtain the motion prediction model.

[0032] Optionally, obtaining human pose information includes:

[0033] Obtain the human image corresponding to the human body executing the instructions in the first training instruction text;

[0034] Human images are processed based on a human pose estimation model to obtain the position information of human joints;

[0035] Human pose information is obtained by fusing the positional information of human joints with the depth information of human joints.

[0036] According to a third aspect of this application, embodiments of this application provide a control device for an android, comprising:

[0037] The acquisition unit is used to acquire instruction text, robot pose information and environmental images collected by the robot. The environmental images include a first two-dimensional environmental image and a first environmental depth image collected by the robot's left arm, a second two-dimensional environmental image and a second environmental depth image collected by the robot's right arm, and a third two-dimensional environmental image and a third environmental depth image collected by the robot's head.

[0038] The processing unit is used to process the instruction text, robot pose information, and environmental images based on the action prediction model to obtain the robot's action at the next moment, so as to control the robot's movement based on the robot's action at the next moment. The training process of the action prediction model includes a pre-training process and a fine-tuning training process. The training data in the pre-training process includes a first training instruction text, a first training environment image collected when the human body executes the instruction in the first training instruction text, and human body pose information. The first training environment image includes a first training environment two-dimensional image and a first training environment depth image collected by the human body's left arm, a second training environment two-dimensional image and a second training environment depth image collected by the right arm, and a third training environment two-dimensional image and a third training environment depth image collected by the head. The training data in the fine-tuning training process includes a second training instruction text, a second training environment image collected when the training robot executes the instruction in the second training instruction text, and training robot pose information. The second training environment image includes a fourth training environment two-dimensional image and a fourth training environment depth image collected by the training robot's left arm, a fifth training environment two-dimensional image and a fifth training environment depth image collected by the right arm, and a sixth training environment two-dimensional image and a sixth training environment depth image collected by the head.

[0039] According to a fourth aspect of this application, embodiments of this application provide an electronic device, including:

[0040] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform a embodied robot control method as in the first aspect or any embodiment of the first aspect, or a model training method as in the second aspect or any embodiment of the second aspect.

[0041] The embodied robot control method, model training method, device, and electronic equipment provided in this application process instruction text, robot pose information, and multi-view environmental images through motion prediction models. By combining pre-training and fine-tuning training processes, they effectively integrate three-dimensional scene information and depth priors, enhance the environmental perception capability of motion prediction models, improve generalization performance and motion execution stability in scenarios with few samples, and have the advantages of improving the robot's scene adaptability, environmental perception accuracy, and motion execution reliability.

[0042] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0043] Figure 1This is a flowchart illustrating a control method for an embodied robot according to an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of the action prediction model in the embodiments of this application;

[0045] Figure 3 This is a schematic diagram of the visual feature extraction module in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the posture feature extraction module in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the decision module in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of the hybrid expert model module in an embodiment of this application;

[0049] Figure 7 This is a flowchart illustrating a model training method in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the structure of a body robot control device according to an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] This application provides a method for controlling a android, such as... Figure 1 As shown, it includes:

[0054] S101, acquire the instruction text, robot pose information and environmental images collected by the robot. The environmental images include a first two-dimensional environmental image and a first environmental depth image collected by the robot's left arm, a second two-dimensional environmental image and a second environmental depth image collected by the robot's right arm, and a third two-dimensional environmental image and a third environmental depth image collected by the robot's head.

[0055] In this embodiment, an embodied robot refers to a robot capable of sensing its environment, understanding instructions, and performing physical actions. Such robots are typically equipped with sensors and actuators, enabling them to interact with their environment in the physical world.

[0056] Instruction text refers to textual information provided in natural language to instruct an embodied robot to perform tasks or actions. This textual information is parsed and understood by the robot, serving as the basis for its behavioral decisions.

[0057] Robot pose information refers to data describing the robot's position and orientation in three-dimensional space. This information typically includes the angles of the robot's joints, the position and orientation of the end effector, etc., reflecting the robot's current physical state.

[0058] Environmental images refer to image data of the surrounding environment collected by the robot's onboard vision sensors. This image data can include two-dimensional images and depth images, providing visual and spatial information about the environment. In this embodiment, the environmental images are divided into multiple perspectives from the robot's left arm, right arm, and head to obtain comprehensive environmental perception.

[0059] In this embodiment, the command text can be acquired in various ways, such as converting the user's voice commands into text through a speech recognition system, or directly receiving the user's input text through a text input interface. Robot pose information can be directly read through the robot's internal sensor system or measured through an external positioning system. Environmental images can be acquired by installing multiple cameras and depth sensors on the robot. For example, a standard camera and a depth camera can be installed on the robot's left arm, right arm, and head to respectively acquire a first 2D environmental image, a first environmental depth image, a second 2D environmental image, a second environmental depth image, a third 2D environmental image, and a third environmental depth image. This image data can be directly transmitted to the processor.

[0060] S102, based on the action prediction model, the instruction text, robot pose information, and environmental images are processed to obtain the robot's action at the next moment, so as to control the robot's movement based on the robot's action at the next moment; the training process of the action prediction model includes a pre-training process and a fine-tuning training process; the training data in the pre-training process includes a first training instruction text, a first training environment image collected when the human body executes the instruction in the first training instruction text, and human body pose information; the first training environment image includes a first training environment two-dimensional image and a first training environment depth image collected by the human body's left arm, a second training environment two-dimensional image and a second training environment depth image collected by the right arm, and a third training environment two-dimensional image and a third training environment depth image collected by the head; the training data in the fine-tuning training process includes a second training instruction text, a second training environment image collected when the training robot executes the instruction in the second training instruction text, and training robot pose information; the second training environment image includes a fourth training environment two-dimensional image and a fourth training environment depth image collected by the training robot's left arm, a fifth training environment two-dimensional image and a fifth training environment depth image collected by the right arm, and a sixth training environment two-dimensional image and a sixth training environment depth image collected by the head.

[0061] In this embodiment, the motion prediction model is a machine learning model that receives multimodal input and outputs the sequence of actions or motion parameters that the robot should execute in the next moment. This model establishes a mapping relationship between input and output through training data, enabling the robot to make autonomous decisions and move.

[0062] In this embodiment, the robot's movement is controlled based on the predicted action of the robot in the next moment. This control can be achieved by directly sending the predicted action to the robot's motion controller. After receiving the predicted action, the motion controller generates instructions to drive the robot's joints or actuators to move according to the instructions, thereby enabling the robot to complete the specified task. For example, if the predicted action is "move forward," the motion controller will drive the robot's chassis to move forward; if the predicted action is "grasp an object," the motion controller will drive the robotic arm to extend, grasp, and retract. In some implementations, the motion controller can determine the final predicted action based on the predicted action through filtering and data interpolation, and then generate instructions to ensure the smoothness and continuity of the robotic arm's movement.

[0063] In this embodiment, the pre-training process refers to the initial training stage of the action prediction model on an open-source human dataset. This stage aims to enable the model to learn feature representations and knowledge, laying the foundation for training subsequent tasks. During this pre-training process, the training data may include a first training instruction text, a first training environment image captured when the human body executes the instructions in the first training instruction text, and human pose information. The acquisition method of the first training environment image can be similar to the acquisition method of robot environment images, for example, by wearing or placing sensors on the left arm, right arm, and head of the human body to acquire a first two-dimensional image of the training environment, a first environmental depth image, a second two-dimensional image of the training environment, a second environmental depth image, a third two-dimensional image of the training environment, and a third environmental depth image. Human pose information can be acquired through wearable sensors or manual annotation. In some embodiments, acquiring human pose information includes: acquiring the human image corresponding to the human body executing the instructions in the first training instruction text; processing the human image based on a human pose estimation model to obtain the position information of the human joints; and fusing the position information of the human joints with the depth information of the human joints to obtain human pose information.

[0064] The fine-tuning training process aims to adapt the pre-trained model to the robot platform and task. During this process, training data may include a second training instruction text, second training environment images captured when the robot executes the instructions in the second training instruction text, and the robot's pose information. The second training environment images are acquired in the same way as the environmental images acquired during actual robot operation; for example, fourth, fifth, and sixth 2D and depth images of the training environment are acquired using sensors mounted on the robot's left arm, right arm, and head. The robot's pose information can be obtained through the robot's own sensor system or external measurement devices. Through these two stages of training, the motion prediction model can learn skills from human behavior and transfer them to the robot platform, while simultaneously optimizing for the robot's specific characteristics.

[0065] In this embodiment, the mean squared error (MSE) function can be used as the loss function during the training of the action prediction model.

[0066] The android control method provided in this application has the following advantages:

[0067] 1. Compared to existing methods that directly train models on limited robot data, this method improves the model's generalization ability by introducing a two-stage training strategy: pre-training and fine-tuning training. In the pre-training stage, the model learns using human manipulation data, enabling it to acquire skills and environmental understanding from human experience. For example, in the "placing a water cup" task, the model learns how humans identify the cup, plan a path, grasp, and place it, gaining action knowledge. This strategy allows the model to exhibit better adaptability and accuracy when facing new tasks and scenarios with few or no samples.

[0068] 2. Some existing models may rely solely on 2D images from a single perspective for environmental perception, resulting in insufficient understanding of the environment's 3D structure and depth information. This method, however, acquires 2D and depth images from multiple perspectives of the robot's left arm, right arm, and head, providing comprehensive and accurate environmental perception information for the action prediction model. For example, using depth images, the model can determine the distance between the water cup and the robot, as well as the cup's position in 3D space. This is crucial for performing actions like "grabbing the water cup," thereby improving the accuracy of action prediction.

[0069] 3. This method also addresses the issue of limited training data in existing models. Traditional methods are often limited by the scale and diversity of data collected by the robot itself, resulting in insufficient model generalization and a tendency for information to be forgotten during fine-tuning. This method expands the model's learning scope by utilizing human manipulation data during the pre-training phase, overcoming the challenge of data scarcity. Subsequently, in the fine-tuning phase, the model is optimized using robot data to adapt to the robot's own physical characteristics, while avoiding the forgetting of knowledge learned in the pre-training phase, thereby improving the model's performance and accuracy on the task.

[0070] In an optional embodiment, the action prediction model is as follows: Figure 2 As shown, it includes a visual feature extraction module 21, a text feature extraction module 22, a pose feature extraction module 23, a feature fusion module 24, and a decision module 25.

[0071] The visual feature extraction module is used to determine the robot's visual features based on environmental images; the text feature extraction module is used to determine the robot's command features based on command text; the posture feature extraction module is used to determine the robot's joint posture features based on robot pose information; the feature fusion module is used to fuse visual features, command features, and joint posture features to obtain a first fused feature; and the decision module is used to predict the robot's action at the next moment based on the first fused feature.

[0072] In this embodiment, the visual feature extraction module is a component specifically designed to process raw visual data (i.e., environmental images) and extract meaningful, high-level visual features from it. These features are crucial for understanding the robot's environment and its task context. This module can be implemented in various ways, such as using a convolutional neural network (CNN) architecture, or it can utilize a deep learning model to process the image as a series of image patches and capture global dependencies through a self-attention mechanism to extract robust visual features. The text feature extraction module is responsible for converting natural language instruction text into a dense numerical representation, i.e., instruction features, which capture the semantic meaning and intent of the instruction. This module can be implemented using variants of recurrent neural networks (RNNs), such as long short-term memory networks (LSTM) or gated recurrent units (GRUs), which are suitable for processing sequential data. The pose feature extraction module is a component that processes the robot's current pose information (e.g., joint angles, end effector position) to derive a compact and information-rich representation, i.e., joint pose features, reflecting the robot's physical state. This module can be constructed using a multilayer perceptron (MLP) that maps the raw pose vectors to a higher-dimensional feature space. Alternatively, if the robot's kinematic chain is represented as a graph structure, a graph neural network (GNN) can be used to capture the relationships between different joints. The feature fusion module aims to combine different feature representations (visual features, command features, pose features) into a unified, comprehensive representation, the first fused feature, which integrates all relevant information for decision-making. This module can be simply implemented through a concatenation operation, where feature vectors are concatenated end-to-end and then passed through a fully connected layer to reduce dimensionality or learn the interactions between features. Another implementation is to use attention mechanisms (e.g., cross-attention), selectively weighting and combining features from different modalities based on their relevance to the current task. The decision module is a component that receives the integrated feature representation (first fused feature) and outputs the robot's predicted actions for the next moment, translating high-level understanding into executable commands. This module can be implemented as a feedforward neural network (MLP) that directly maps the fused features to a continuous action space (e.g., joint torque, end effector speed) or a discrete action space (e.g., predefined primitive actions). Furthermore, it can also integrate the policy network of a reinforcement learning agent, which learns to select actions that maximize reward signals based on the current state represented by the fused features.

[0073] In practice, the text feature extraction module may include a large language model, such as the Qwen2.5B large language model, which processes the instruction text to obtain instruction features.

[0074] In practice, the feature fusion module includes a multi-layer cross-attention module and a stitching module. Visual features and command features are first input into the multi-layer cross-attention module, with visual features acting as keys and values, and command features as queries. Cross-attention is continuously performed to ensure thorough fusion of the visual and command features. Then, the fused features are stitched together with the joint pose features in the stitching module to obtain the first fused feature.

[0075] This application embodiment refines the action prediction model into a visual feature extraction module, a text feature extraction module, a posture feature extraction module, a feature fusion module, and a decision module. This enables the action prediction model to employ specialized feature extraction strategies for different types of data, ensuring full utilization of information from each modality. The feature fusion module integrates these heterogeneous features into a unified first fused feature, providing the decision module with a comprehensive and high-level semantic representation. Based on this fused feature, the decision module predicts the robot's next action, significantly improving the robot's ability to understand complex instructions, perceive environmental changes, and accurately execute tasks. Through clear functional division and data flow design, this application embodiment enhances the model's interpretability, training efficiency, and final control accuracy, enabling the embodied robot to perform various operations more intelligently and robustly.

[0076] In an optional embodiment, the visual feature extraction module is used to determine first-view two-dimensional image features based on a first environmental two-dimensional image; determine first-view depth image features based on a first environmental depth image; fuse the first-view two-dimensional image features with the first-view depth image features to obtain first-view image features; determine second-view two-dimensional image features based on a second environmental two-dimensional image; determine second-view depth image features based on a second environmental depth image; fuse the second-view two-dimensional image features with the second-view depth image features to obtain second-view image features; determine third-view two-dimensional image features based on a third environmental two-dimensional image; determine third-view depth image features based on a third environmental depth image; fuse the third-view two-dimensional image features with the third-view depth image features to obtain third-view image features; and fuse the first-view image features, second-view image features, and third-view image features to obtain the robot's visual features.

[0077] In this embodiment, two-dimensional image features typically include information such as texture, color, edges, and shape. These are crucial for identifying object categories and distinguishing different surface materials. Depth image features provide three-dimensional geometric information about the scene, playing an irreplaceable role in understanding the spatial location and distance of objects, obstacle detection, and 3D reconstruction. The step of fusing first-view two-dimensional image features with first-view depth image features to obtain first-view image features aims to combine two-dimensional image features and depth image features from the same viewpoint (e.g., the left arm) to form a more comprehensive and robust viewpoint image feature. This fusion can compensate for the deficiencies of single-modal information; for example, two-dimensional images lack depth information, while depth images lack texture and color information.

[0078] The step of fusing first-view, second-view, and third-view image features to obtain the robot's visual features aims to integrate image features from different perspectives (left arm, right arm, head) to gain a global, multi-angle understanding of the robot's surrounding environment. Multi-view fusion provides a wider field of view, reduces occlusion, and captures key information from different angles, thus forming a comprehensive visual profile of the robot.

[0079] In practical implementation, the structure of the visual feature extraction module is as follows: Figure 3As shown. For the first environmental 2D image (e.g., an RGB image) acquired by the left-arm camera, it is input into the corresponding FG-CLIP (Fine Grained CLIP) image encoder and simultaneously input into the Dinov3 image feature encoder. The features output from both image encoders are then input into the first cross-attention module. To further enhance image feature fusion, the features output from the first cross-attention module are input into the channel attention mechanism module (SE module) and the first linear module to acquire information along the channel dimension. Simultaneously, these features are input into the second linear module and the convolutional block attention module (CBAM). The features output from both modules are then concatenated in the first stitching module to obtain the 2D image features from a single viewpoint, i.e., the first-view 2D image features. To allow the model to better perceive environmental information, this embodiment introduces depth information. Based on the acquired first environmental depth image from a single viewpoint, it is input into the first convolutional block module and the third linear module. The result is then input into the Unet++ module and subsequently into the second convolutional block module. The resulting features are then reshaped by the reshape module. After the operation, the first-view depth image features are obtained. These first-view depth image features are used as keys and values, and the previously obtained first-view two-dimensional image features are used as queries for fusion in the second cross-attention module to obtain the first-view image features. Subsequently, the second-environment two-dimensional image and second-environment depth image acquired by the right arm camera, as well as the third-environment two-dimensional image and third-environment depth image acquired by the head, are processed in the same way. Finally, the visual features of the three images are stitched and fused in the second stitching module to obtain the visual features of the feature robot.

[0080] In this embodiment, the visual feature extraction module fully utilizes environmental information collected by the robot's multiple sensors and from multiple perspectives. First, by fusing the two-dimensional image features and depth image features from each perspective, it effectively combines the object's appearance information and three-dimensional geometric information, making the perception from a single perspective more comprehensive and accurate, avoiding misjudgments caused by insufficient information from a single modality. Second, by fusing image features from three different perspectives—the left arm, right arm, and head—it greatly expands the robot's perception range, effectively solving the occlusion problem and field-of-view limitations that may exist under a single perspective. This multi-view, multi-modal visual feature extraction method enables the action prediction model to gain a richer and more robust understanding of the environment, thereby significantly improving the accuracy and adaptability of action prediction when the embodied robot performs tasks in complex and dynamic environments, allowing it to perceive the environment more accurately and make appropriate action decisions.

[0081] In an optional embodiment, the pose feature extraction module is used to determine the joint pose features of the robot based on the robot pose information and the robot's historical pose information; the robot's historical pose information includes the robot's pose information at least at the previous moment.

[0082] In this embodiment, robot historical pose information refers to the robot pose information recorded and stored by the robot system at different points in time before the current moment. This information may include state parameters such as the angle, position, velocity, and acceleration of each joint of the robot, as well as the position and orientation of the robot's end effector in three-dimensional space. Its purpose is to provide the motion prediction model with contextual information about the robot's past motion trajectory and dynamic state, enabling the model to understand the robot's current motion trend rather than just its static position.

[0083] In practical implementation, the structure of the pose feature extraction module is as follows: Figure 4 As shown, robot pose information can be input into the embedding module for embedding encoding, and then input into the first linear layer to obtain the variable (qpos) features of the robot joint's current position. Simultaneously, for historical pose information, the most recent pose information sequence of the robot can be extracted to obtain historical pose information, which is then input into a bidirectional long short-term memory (Bi-LSTM) network to obtain historical sequence feature information. This information is then input into the second linear layer and concatenated with the previous qpos features in the concatenation module to obtain the robot's joint pose features.

[0084] In this embodiment, by incorporating the robot's historical pose information into the pose feature extraction module, the embodied robot control method can more comprehensively capture the robot's dynamic motion state and trends. This allows the action prediction model to consider not only the current static pose but also the robot's past motion trajectory information when predicting the robot's next action, thereby significantly improving the accuracy and consistency of action prediction. The robot's actions will be smoother and more natural, avoiding the problems of discontinuous or uncoordinated actions that may result from relying solely on instantaneous pose information. This is particularly suitable for complex task scenarios that require continuous and smooth robot operations, improving the robot's overall motion performance and task completion efficiency.

[0085] In an optional embodiment, such as Figure 5 As shown, the decision-making module includes a cross-attention module 51 and a hybrid expert model module 52, as well as an action head module 53. The cross-attention module 51 is used to fuse the first fusion feature to obtain the second fusion feature. The hybrid expert model module 52 is used to process the second fusion feature based on a multi-expert strategy to obtain a multi-expert prediction result. The action head module 53 is used to predict the robot's action at the next moment based on the multi-expert prediction result.

[0086] In this embodiment, the decision-making module is a core component of the action prediction model. Its main function is to transform the information after feature fusion into specific robot action commands. The cross-attention module is a mechanism capable of processing multimodal or multi-sequence inputs and extracting their interrelated information. It identifies and reinforces the most critical information for the current task decision in the fused features by allowing different feature streams (e.g., visual, textual, and pose features) to pay attention to each other.

[0087] In some implementations, such as Figure 6 As shown, the hybrid expert model module includes a routing submodule, multiple expert submodules, a shared expert submodule, and an output submodule. The routing submodule is used to route the second fusion feature to two expert submodules and the shared expert submodule of the multiple expert submodules based on the second fusion feature. The two expert submodules are used to determine the expert prediction results based on the second fusion feature. The shared expert submodule is used to determine the shared expert prediction results based on the second fusion feature. The output submodule is used to fuse the expert prediction results and the shared expert prediction results to obtain the multi-expert prediction results.

[0088] The hybrid expert model module is a machine learning model architecture designed to improve performance by combining the predictions of multiple expert sub-models, each of which may excel at handling different aspects of the input data or different types of tasks. It typically includes a gating network (routing sub-module) to determine which experts should be activated and how their outputs should be combined. The shared expert sub-module is a special type of expert sub-module that works in parallel with multiple expert sub-modules, but its role is to handle general or common features that are not suitable for a particular expert sub-module, or to serve as a baseline expert to provide more stable predictions. It ensures that the model maintains a certain predictive ability even when a particular expert is not perfectly applicable. The output sub-module obtains the multi-expert prediction result by adding the predictions of two expert sub-models, multiplying by the activation weights obtained from the routing, and adding the prediction result of the shared expert.

[0089] In this embodiment, the action head module may consist of a linear layer, an activation layer, and a normalization layer.

[0090] In this embodiment, by introducing a cross-series cross-attention module, the fused features from vision, text, and pose are first deeply interacted and refined to generate a second fused feature that is more informative and context-aware. This refinement process helps the model better understand the complex relationship between the current task instructions and the environmental state, preventing key information from being overwhelmed. Subsequently, the hybrid expert model module uses a multi-expert strategy to process this refined feature. This means that the model no longer relies on a single, general decision path, but can dynamically select or combine multiple specialized "expert" networks to make decisions based on the current context; and it can reduce the parameters involved in model inference, improving inference efficiency while ensuring effectiveness. This modular and specialized decision-making mechanism significantly enhances the model's ability to handle diverse tasks and complex environments, improving the accuracy and adaptability of decisions.

[0091] This application also provides a model training method, such as... Figure 7 As shown, it includes:

[0092] S701, acquire the first training instruction text, the first training environment image captured when the human body executes the instruction in the first training instruction text, and the human body pose information; the first training environment image includes the first training environment two-dimensional image and the first training environment depth image captured by the left arm of the human body, the second training environment two-dimensional image and the second training environment depth image captured by the right arm, and the third training environment two-dimensional image and the third training environment depth image captured by the head.

[0093] S702, based on the first training instruction text, the first training environment image, and human pose information, train the neural network to obtain an initial action prediction model.

[0094] S703, acquire the second training instruction text, the second training environment image collected when the training robot executes the instructions in the second training instruction text, and the pose information of the training robot; the second training environment image includes the fourth training environment two-dimensional image and the fourth training environment depth image collected by the left arm of the training robot, the fifth training environment two-dimensional image and the fifth training environment depth image collected by the right arm, and the sixth training environment two-dimensional image and the sixth training environment depth image collected by the head.

[0095] S704, based on the second training instruction text, the second training environment image, and the pose information of the training robot, trains the initial motion prediction model to obtain the motion prediction model.

[0096] In this embodiment, the mean squared error (MSE) function can be used as the loss function during the training of the action prediction model.

[0097] In this embodiment, a two-stage training approach is adopted, combining pre-training with human operation data and fine-tuning with robot data. This fully utilizes massive amounts of open-source human data to enhance the model's feature extraction capabilities while avoiding information forgetting during downstream task training. This significantly improves the model's generalization and accuracy in predicting actions in low-sample scenarios. Specifically, because existing technologies limit model training data to known common structure data, generalization is insufficient and information forgetting easily occurs during fine-tuning. This embodiment first pre-trains the model based on multi-view environmental images and human pose information collected when the human executes commands, enabling the model to learn general action features from massive amounts of human operation data. Based on this, by retaining the pre-training weights and fine-tuning the training on environmental images and pose information actually collected by the robot, the model adapts to the characteristics of the robot platform, effectively solving the problem of poor accuracy in low-sample and zero-sample scenarios. Furthermore, the environmental depth images and pose information introduced during the model stage allow the model to acquire richer prior knowledge of the 3D environment, enhancing its ability to perceive depth features.

[0098] In an optional embodiment, obtaining human pose information includes:

[0099] Obtain the human image corresponding to the human body executing the instructions in the first training instruction text; process the human image based on the human pose estimation model to obtain the position information of the human joints; fuse the position information of the human joints with the depth information of the human joints to obtain the human pose information.

[0100] Specifically, acquiring human images corresponding to the execution of instructions in the first training instruction text aims to capture visual data of humans performing specific actions, providing raw input for subsequent posture analysis. RGB cameras or multi-view camera arrays can be used to record video in real time while the human executes instructions, thereby acquiring a continuous sequence of human images. Alternatively, sensor devices equipped with image acquisition capabilities, such as smartphones or wearable cameras, can be used to periodically take photos or record short videos during human activity to obtain discrete or continuous human images.

[0101] The human image is processed based on a human pose estimation model to obtain the position information of human joints, with the aim of extracting the two-dimensional or three-dimensional spatial coordinates of key human joints from the captured image.

[0102] The position information of the human joints is fused with the depth information of the human joints to obtain the human pose information. The aim is to combine the previously obtained two-dimensional / three-dimensional joint position information with the corresponding depth data to form comprehensive human pose data containing spatial coordinates and depth information.

[0103] In this embodiment, the model training method can acquire human pose data containing depth information in a systematic and accurate manner.

[0104] This application also provides a control device for a hymenoidae robot, such as... Figure 8 As shown, it includes:

[0105] The acquisition unit 81 is used to acquire instruction text, robot pose information and environmental images collected by the robot. The environmental images include a first two-dimensional environmental image and a first environmental depth image collected by the robot's left arm, a second two-dimensional environmental image and a second environmental depth image collected by the robot's right arm, and a third two-dimensional environmental image and a third environmental depth image collected by the robot's head.

[0106] The processing unit 82 is used to process the instruction text, robot pose information, and environmental images based on the action prediction model to obtain the robot's action at the next moment, so as to control the robot's movement based on the robot's action at the next moment. The training process of the action prediction model includes a pre-training process and a fine-tuning training process. The training data in the pre-training process includes a first training instruction text, a first training environment image collected when the human body executes the instruction in the first training instruction text, and human body pose information. The first training environment image includes a first training environment two-dimensional image and a first training environment depth image collected by the human body's left arm, a second training environment two-dimensional image and a second training environment depth image collected by the right arm, and a third training environment two-dimensional image and a third training environment depth image collected by the head. The training data in the fine-tuning training process includes a second training instruction text, a second training environment image collected when the training robot executes the instruction in the second training instruction text, and training robot pose information. The second training environment image includes a fourth training environment two-dimensional image and a fourth training environment depth image collected by the training robot's left arm, a fifth training environment two-dimensional image and a fifth training environment depth image collected by the right arm, and a sixth training environment two-dimensional image and a sixth training environment depth image collected by the head.

[0107] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0108] Figure 9A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0109] like Figure 9 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0110] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0111] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as embodied robot control methods or model training methods. For example, in some embodiments, the embodied robot control methods or model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the embodied robot control methods or model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform embodied robot control methods or model training methods by any other suitable means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0117] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0118] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0119] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0120] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A somatic robot control method characterized by, include: The robot obtains command text, robot pose information, and environmental images collected by the robot. The environmental images include a first two-dimensional environmental image and a first environmental depth image collected by the robot's left arm, a second two-dimensional environmental image and a second environmental depth image collected by the robot's right arm, and a third two-dimensional environmental image and a third environmental depth image collected by the robot's head. The command text, robot pose information, and environmental image are processed based on the motion prediction model to obtain the robot's action at the next moment, so as to control the robot's movement based on the robot's action at the next moment; the training process of the motion prediction model includes a pre-training process and a fine-tuning training process. The training data in the pre-training process includes a first training instruction text, a first training environment image collected when the human body executes the instructions in the first training instruction text, and human body pose information. The first training environment image includes a first two-dimensional image and a first training environment depth image captured by the left arm of the human body, a second two-dimensional image and a second training environment depth image captured by the right arm, and a third two-dimensional image and a third training environment depth image captured by the head. The training data in the fine-tuning training process includes a second training instruction text, a second training environment image collected when the training robot executes the instructions in the second training instruction text, and the pose information of the training robot; the second training environment image includes a fourth training environment two-dimensional image and a fourth training environment depth image collected by the left arm of the training robot, a fifth training environment two-dimensional image and a fifth training environment depth image collected by the right arm, and a sixth training environment two-dimensional image and a sixth training environment depth image collected by the head.

2. The embodied robot control method according to claim 1, characterized in that, The action prediction model includes a visual feature extraction module, a text feature extraction module, a pose feature extraction module, a feature fusion module, and a decision module; The visual feature extraction module is used to determine the visual features of the robot based on the environmental image; The text feature extraction module is used to determine the robot's instruction features based on the instruction text; The posture feature extraction module is used to determine the joint posture features of the robot based on the robot pose information; The feature fusion module is used to fuse the visual features, the command features, and the joint pose features to obtain a first fused feature; The decision module is used to predict the robot's next action based on the first fused features.

3. The embodied robot control method according to claim 2, characterized in that, The visual feature extraction module is used to determine first-view two-dimensional image features based on the first environmental two-dimensional image; determine first-view depth image features based on the first environmental depth image; and fuse the first-view two-dimensional image features with the first-view depth image features to obtain first-view image features. Based on the second environmental two-dimensional image, determine the features of the second viewpoint two-dimensional image; based on the second environmental depth image, determine the features of the second viewpoint depth image; The second-view two-dimensional image features and the second-view depth image features are fused together to obtain the second-view image features; Based on the aforementioned three-dimensional environmental image, the features of the three-dimensional image from the third perspective are determined; Based on the third environmental depth image, the features of the third-view depth image are determined; The third-view two-dimensional image features are fused with the third-view depth image features to obtain the third-view image features; The visual features of the robot are obtained by fusing the features of the first viewpoint image, the second viewpoint image, and the third viewpoint image.

4. The embodied robot control method according to claim 2, characterized in that, The posture feature extraction module is used to determine the joint posture features of the robot based on the robot pose information and the robot's historical pose information; the robot's historical pose information includes the robot's pose information at least at the previous moment.

5. The embodied robot control method according to claim 2, characterized in that, The decision-making module includes a cross-series cross-attention module and a hybrid expert model module, as well as an action head module; The cross-attention module is used to fuse the first fusion feature to obtain the second fusion feature; The hybrid expert model module is used to process the second fusion feature based on a multi-expert strategy to obtain multi-expert prediction results; The motion head module is used to predict the robot's next action based on the multi-expert prediction results.

6. The embodied robot control method according to claim 5, characterized in that, The hybrid expert model module includes a routing submodule, multiple expert submodules, a shared expert submodule, and an output submodule. The routing submodule is used to route the second fusion feature to two expert submodules and the shared expert submodule of the multiple expert submodules based on the second fusion feature; The two expert submodules are used to determine the expert prediction result based on the second fusion feature; The shared expert submodule is used to determine the shared expert prediction result based on the second fusion feature; The output submodule is used to fuse the expert prediction results and the shared expert prediction results to obtain multi-expert prediction results.

7. A model training method, characterized in that, include: Acquire the first training instruction text, the first training environment image captured when the human body executes the instructions in the first training instruction text, and the human body pose information; The first training environment image includes a first two-dimensional image and a first training environment depth image captured by the left arm of the human body, a second two-dimensional image and a second training environment depth image captured by the right arm, and a third two-dimensional image and a third training environment depth image captured by the head. Based on the first training instruction text, the first training environment image, and the human pose information, the neural network is trained to obtain an initial action prediction model. The system acquires a second training instruction text, a second training environment image captured when the training robot executes the instructions in the second training instruction text, and the pose information of the training robot. The second training environment image includes a fourth training environment two-dimensional image and a fourth training environment depth image captured by the left arm of the training robot, a fifth training environment two-dimensional image and a fifth training environment depth image captured by the right arm, and a sixth training environment two-dimensional image and a sixth training environment depth image captured by the head. Based on the second training instruction text, the second training environment image, and the pose information of the training robot, the initial motion prediction model is trained to obtain the motion prediction model.

8. The model training method according to claim 7, characterized in that, Obtaining human pose information includes: Obtain the human image corresponding to the human body executing the instructions in the first training instruction text; The human image is processed based on a human pose estimation model to obtain the position information of the human joints; The position information of the human joints is fused with the depth information of the human joints to obtain the human pose information.

9. A control device for an embodied robot, characterized in that, include: The acquisition unit is used to acquire instruction text, robot pose information and environmental images collected by the robot. The environmental images include a first two-dimensional environmental image and a first environmental depth image collected by the robot's left arm, a second two-dimensional environmental image and a second environmental depth image collected by the robot's right arm, and a third two-dimensional environmental image and a third environmental depth image collected by the robot's head. The processing unit is used to process the instruction text, the robot pose information, and the environmental image based on the action prediction model to obtain the robot's action at the next moment, so as to control the robot's movement based on the robot's action at the next moment; the training process of the action prediction model includes a pre-training process and a fine-tuning training process. The training data in the pre-training process includes a first training instruction text, a first training environment image collected when the human body executes the instructions in the first training instruction text, and human body pose information. The first training environment image includes a first two-dimensional image and a first training environment depth image captured by the left arm of the human body, a second two-dimensional image and a second training environment depth image captured by the right arm, and a third two-dimensional image and a third training environment depth image captured by the head. The training data in the fine-tuning training process includes a second training instruction text, a second training environment image collected when the training robot executes the instructions in the second training instruction text, and the pose information of the training robot; the second training environment image includes a fourth training environment two-dimensional image and a fourth training environment depth image collected by the left arm of the training robot, a fifth training environment two-dimensional image and a fifth training environment depth image collected by the right arm, and a sixth training environment two-dimensional image and a sixth training environment depth image collected by the head.

10. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the embodied robot control method as described in any one of claims 1-6 or the model training method as described in any one of claims 7-8.

Images