Humanoid robot control method, system and humanoid robot

Abstract

The present application provides a humanoid robot control method, system and humanoid robot, wherein the method includes: obtaining task instructions and field of view images; inputting the field of view images and task instructions into a visual language perception model to obtain the three-dimensional coordinates of the target object; obtaining prompt words, and inputting the prompt words and three-dimensional coordinates into a task planner to determine the control information corresponding to the prompt words; controlling the arm movement of the humanoid robot through an arm control module, controlling the hand movement of the humanoid robot through a hand control module, and controlling the leg movement of the humanoid robot through a body control module. The present application can integrate multiple modules such as visual perception, language understanding, task planning, and motion control into a unified framework, realizing end-to-end control from perception to execution, improving the integration of the humanoid robot control process, enhancing the robot's adaptability in complex environments, and improving the transparency and credibility of the humanoid robot control process.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a humanoid robot control method, system, and humanoid robot. Background Art

[0002] Over the years, robotics has undergone significant development and research, achieving remarkable results in various forms. Humanoid robots, in particular, have garnered increasing attention in recent years due to their high similarity to humans. Because they can operate directly within human living and working spaces, they are expected to be able to perform more complex tasks.

[0003] Numerous studies have demonstrated that large language models (LLMs) and visual language models (VLMs) can empower robots with significant semantic planning and logical reasoning capabilities, enabling them to directly understand and execute complex instructions via natural language. However, these studies typically target common robots with relatively simple configurations and sophisticated controllers.

[0004] In contrast, there are very few studies on large models based on humanoid robots, although there are many models suitable for handling different tasks of humanoid robots, such as visual understanding, motion planning and control. 。 However, these models are often isolated and have not yet been successfully integrated and implemented on humanoid robots. Even if they can be integrated, it will take a lot of time. At the same time, it is difficult to ensure adaptability and interpretability in complex environments.

[0005] Therefore, the humanoid robot control method in the prior art has the problems of low integration, poor adaptability and poor interpretability. Summary of the Invention

[0006] The purpose of this application is to provide a humanoid robot control method, system and humanoid robot to address the deficiencies in the above-mentioned prior art, so as to solve the problems of low integration, poor adaptability and poor interpretability of the humanoid robot control method in the prior art.

[0007] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0008] In a first aspect, an embodiment of the present application provides a method for controlling a humanoid robot, the method comprising:

[0009] Acquiring a task instruction and a field of view image, wherein the task instruction is used to instruct the humanoid robot to perform an action on a target object;

[0010] Inputting the visual field image and the task instruction into a pre-trained visual language perception model, and having the visual language perception model perceive and process the visual field image and the task instruction to obtain the three-dimensional coordinates of the target object in the visual field image;

[0011] Obtaining a prompt word, and inputting the prompt word and the three-dimensional coordinates into a pre-trained task planner based on a large language model, wherein the task planner performs task planning based on the prompt word and the three-dimensional coordinates, and determines control information corresponding to the prompt word, wherein the control information includes first position information, second position information, and hand state information, and the prompt word includes at least the task instruction;

[0012] The arm movement of the humanoid robot is controlled by the arm control module in the motion control system according to the first position information, and the hand movement of the humanoid robot is controlled by the hand control module in the motion control system according to the hand state information, and the leg movement of the humanoid robot is controlled by the body control module in the motion control system according to a pre-deployed reinforcement learning strategy and the second position information, and the reinforcement learning strategy is obtained through pre-training.

[0013] In a possible implementation, the visual language perception model includes an encoding module and a decoding module connected in sequence;

[0014] The visual language perception model performs perception processing on the field of view image and the task instruction to obtain the three-dimensional coordinates of the target object in the field of view image, including:

[0015] Inputting the visual field image and the task instruction into the encoding module, and encoding the visual field image and the task instruction by the encoding module to obtain encoded features;

[0016] The encoded features are input into the decoding module, and the decoding module decodes the encoded features to obtain the three-dimensional coordinates.

[0017] In a possible implementation, the encoding module includes: an image encoding module, a text encoding module, and an encoding fusion module;

[0018] The inputting the visual field image and the task instruction into the encoding module, and the encoding module encoding the visual field image and the task instruction to obtain encoded features, includes:

[0019] Inputting the visual field image into the image coding module, and having the image coding module perform image perception coding processing on the visual field image to obtain image features;

[0020] Inputting the task instruction into the text encoding module, and performing text-aware encoding processing on the task instruction by the text encoding module to obtain text features;

[0021] The image features and the text features are input into the encoding fusion module, and the encoding fusion module performs feature fusion on the image features and the text features to obtain the encoded features.

[0022] In a possible implementation, the prompt word further includes: pre-configured configuration information, the configuration information including: skill configuration information, operation space constraint configuration information, safety constraint configuration information, and kinematic configuration information;

[0023] The skill configuration information is used to indicate the executable actions of the arms, hands and legs of the humanoid robot, the operation space constraint configuration information is used to indicate the operable space of the humanoid robot, and the kinematic configuration information is used to indicate the prior kinematic information of the humanoid robot.

[0024] In a possible implementation, the task planner performs task planning based on the prompt word and the three-dimensional coordinates, and determines the control information corresponding to the prompt word, including:

[0025] Obtaining the current height of the humanoid robot;

[0026] The task planner performs task planning based on the prompt word, the current height, and the three-dimensional coordinates, and determines control information corresponding to the prompt word.

[0027] In one possible implementation, controlling the leg movement of the humanoid robot according to the reinforcement learning strategy and the second position information includes:

[0028] generating a target action sequence according to the reinforcement learning strategy and the second position information;

[0029] The legs of the humanoid robot are controlled to move according to the target action sequence, and when the legs move to the target position indicated by the second position information, a first start instruction is sent to the arm control module.

[0030] In a possible implementation, controlling the movement of the arm of the humanoid robot according to the first position information includes:

[0031] In response to the first start-up instruction, based on the first position information, a pre-built inverse kinematics model is used to predict the joint rotation angle of the arm of the humanoid robot, and the arm of the humanoid robot is controlled to rotate according to the joint rotation angle, and after the rotation is completed, a second start-up instruction is sent to the hand control module.

[0032] In a possible implementation, controlling the hand movement of the humanoid robot according to the hand state information includes:

[0033] In response to the second start instruction, obtaining a current state of the hand of the humanoid robot;

[0034] Determine whether the current state is consistent with the hand state information; if not, adjust the state of the hand of the humanoid robot to be consistent with the hand state information.

[0035] In the second aspect, another embodiment of the present application provides a humanoid robot control system, which includes: a visual language perception model, a task planner based on a large language model, and a motion control system, and the motion control system includes: an arm control module, a hand control module, and a body control module; a reinforcement learning strategy is deployed in the body control module, and the reinforcement learning strategy is obtained through pre-training. The humanoid robot control system is used to execute the steps of the humanoid robot control method described in the first aspect.

[0036] In a third aspect, another embodiment of the present application provides a humanoid robot, comprising: the humanoid robot control system described in the second aspect.

[0037] The beneficial effects of the present application are: using a visual language perception model to perceive and process the field of view image and task instructions, obtaining the three-dimensional coordinates of the target object in the field of view image, and using a task planner based on a large language model to perform task planning based on the prompt words and the three-dimensional coordinates, and determining the control information corresponding to the prompt words, so that the arm control module, the body control module, and the hand control module in the motion control system can respectively control the arm movement, the hand movement, and the leg movement of the humanoid robot. It can integrate multiple modules such as visual perception, language understanding, task planning, and motion control into a unified framework, realize end-to-end control from perception to execution, and improve the integration of the humanoid robot control process. At the same time, it improves the integration of the large language model, the visual language perception model, and the motion control system. It can also adapt to the changes of target objects in a dynamic environment through the visual language perception model and adapt to different task requirements through the task planner, thereby enhancing the robot's adaptability in complex environments. In addition, the visual language perception model provides intuitive perception results, and the task planner based on the large language model generates interpretable control information, which is easy for humans to understand and verify, thereby enhancing the interpretability of the humanoid robot control process and improving the transparency and credibility of the humanoid robot control process. In addition, by controlling the arm movement, hand movement and leg movement of the humanoid robot respectively through the arm control module, the body control module and the hand control module, the control accuracy and task execution capability of the robot can be improved, and the flexibility of the humanoid robot control process can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A schematic flow chart of a humanoid robot control method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of the structure of the visual language perception model provided in the embodiment of the present application;

[0041] Figure 3 A schematic diagram of a process for obtaining the three-dimensional coordinates of a target object in a field of view image in the humanoid robot control method provided in an embodiment of the present application;

[0042] Figure 4 Another structural diagram of the visual language perception model provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a flow chart for obtaining encoded features in the humanoid robot control method provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of a flow chart for determining control information corresponding to a prompt word in the humanoid robot control method provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a flow chart for controlling leg movement of a humanoid robot in a humanoid robot control method provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of a flow chart for controlling the hand movement of a humanoid robot in the humanoid robot control method provided in an embodiment of the present application;

[0047] Figure 9 A schematic diagram of a humanoid robot control system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0049] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0050] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0051] Robotics research and applications based on large language models are constantly emerging. Extensive research demonstrates that large language models (LLMs) and visual language models (VLMs) can empower robots with significant semantic planning and logical reasoning capabilities, enabling them to directly understand and execute complex instructions via natural language. However, these studies typically focus on standard robots with relatively simple configurations and sophisticated controllers.

[0052] In contrast, there are very few studies on large models based on humanoid robots, although there are many models suitable for handling different tasks of humanoid robots, such as visual understanding, motion planning and control. 。 However, these models are often isolated and have not yet been successfully integrated and implemented on humanoid robots. Even if they can be integrated, it will take a lot of time. At the same time, it is difficult to ensure adaptability and interpretability in complex environments.

[0053] Therefore, the humanoid robot control method in the prior art has the problems of low integration, poor adaptability and poor interpretability.

[0054] Based on the above-mentioned problems, the embodiment of the present application proposes a humanoid robot control method. The method uses a visual language perception model to perceive and process the field of view image and task instructions, obtains the three-dimensional coordinates of the target object in the field of view image, and uses a task planner based on a large language model to perform task planning based on prompt words and three-dimensional coordinates, and determines the control information corresponding to the prompt words. This method enables the arm control module, the body control module, and the hand control module in the motion control system to control the arm movement, hand movement, and leg movement of the humanoid robot respectively. It can integrate multiple modules such as visual perception, language understanding, task planning, and motion control into a unified framework, realize end-to-end control from perception to execution, improve the integration of the humanoid robot control process, and at the same time, enhance the robot's adaptability in complex environments. In addition, it also enhances the interpretability of the humanoid robot control process, and improves the transparency and credibility of the humanoid robot control process. In addition, it can also improve the robot's control accuracy and task execution capability, ensuring the flexibility of the humanoid robot control process.

[0055] Figure 1 A flow chart of a humanoid robot control method provided in an embodiment of the present application, referring to Figure 1 As shown, the execution subject of the method can be any electronic device with processing capabilities, such as a main control device of a humanoid robot, and the method includes:

[0056] S101: Obtain task instructions and visual field images.

[0057] The task instructions are used to instruct the humanoid robot to perform actions on the target object.

[0058] Alternatively, the target object can be understood as an object that needs to be operated by the humanoid robot, such as a valve waiting to be opened or closed, goods waiting to be moved, etc. It can be obtained through voice commands, text commands or gesture commands, and the task information to be performed by the humanoid robot can be determined through the task command.

[0059] Optionally, the field of view image can be understood as an image within the field of view of the humanoid robot in the current environment, which can be obtained by using an RGB camera or an RGB-D camera (such as Intel RealSense, Azure Kinect) to capture the environment image, or by using a depth sensor (such as LiDAR, ToF camera) to obtain depth information of the environment.

[0060] S102 , inputting the visual field image and the task instruction into a pre-trained visual language perception model, and having the visual language perception model perceive and process the visual field image and the task instruction to obtain the three-dimensional coordinates of the target object in the visual field image.

[0061] Optionally, the obtained task instructions and the visual field image are input into a pre-trained visual language perception model for perception processing to obtain the three-dimensional coordinates of the target object in the visual field image. The three-dimensional coordinates may be the three-dimensional coordinates of the target object in the reference coordinate system of the humanoid robot.

[0062] Exemplarily, the task instructions and the field of view image can be input into the visual language perception model, and then the visual language perception model performs preprocessing on the field of view image such as pixel value normalization, image enhancement, and resolution adjustment, and performs preprocessing on the task instructions such as word segmentation, padding, or truncation to obtain the preprocessed image and text sequence. The preprocessed image and text sequence are respectively input into the visual encoder and text encoder for encoding processing, and then feature fusion is performed, and target detection processing is performed on the fused features to obtain the bounding box and category of the target object, and then the target object is estimated and the coordinate system is converted to obtain the three-dimensional coordinates.

[0063] For example, the visual encoder can be implemented based on a convolutional neural network (CNN) or a visual Transformer, the language encoder can be implemented based on a Transformer or BERT, object detection can be implemented based on an object detection head (such as FasterR-CNN, YOLO), and coordinate estimation can be implemented based on depth information (such as the depth channel of an RGB-D image) or a monocular depth estimation model.

[0064] S103 , obtaining a prompt word, and inputting the prompt word and the three-dimensional coordinates into a pre-trained task planner based on a large language model. The task planner performs task planning based on the prompt word and the three-dimensional coordinates, and determines the control information corresponding to the prompt word.

[0065] The control information includes first position information, second position information and hand state information, and the prompt word includes at least task instructions.

[0066] Optionally, a prompt word is obtained, which is used to indicate relevant commands and constraint information when the humanoid robot performs a task. The prompt word and three-dimensional coordinates are input into a pre-trained task planner based on a large prediction model, and task planning is performed through the task planner to obtain control information corresponding to the prompt word.

[0067] For example, a prompt word and 3D coordinates are input into the mission planner, which then parses and normalizes them. For example, natural language processing techniques (such as word segmentation, part-of-speech tagging, and named entity recognition) can be used to pre-process the prompt word to facilitate understanding by the large language model. The input 3D coordinates can be validated to ensure they are valid and meet the workspace requirements of the humanoid robot. The 3D coordinates can then be converted into a format suitable for processing by the large language model, such as encoding the coordinate values ​​as strings or numeric sequences.

[0068] Exemplarily, the task planner combines standardized prompt words and three-dimensional coordinates into an input sequence, and then perceives and models the robot's environment in real time. For example, it uses sensor data (such as cameras and lidar) to identify objects, spatial relationships, and possible obstacles in the environment, constructing a map or model of the robot's surroundings. Based on the input sequence and the results of environmental perception, it then performs task planning, specifically determining the robot's motion path, arm trajectory, and hand status information. During the task planning process, the task planner can employ a variety of algorithms and strategies, such as the A* algorithm, Dijkstra's algorithm, genetic algorithms, and reinforcement learning. This allows it to quickly find optimal or near-optimal solutions in complex environments and generate corresponding control information.

[0069] For example, the task planner can plan the task of the humanoid robot into one or more subtasks, each subtask has corresponding control information. The task planner can interact with the motion control system of the humanoid robot for data, and send the control information corresponding to each subtask to the corresponding controller according to the execution order of each subtask and the execution feedback of the previous task.

[0070] The control information for each subtask may include first position information for controlling the humanoid robot's arm, hand state information for controlling the movement of the humanoid robot's hand, and second position information for controlling the humanoid robot's main body. The first position information may include the position coordinates of the humanoid robot's arm when performing the subtask, as well as the speed or time information when the arm reaches the position coordinates. The second position information may include the position coordinates of the humanoid robot's main body when performing the subtask, as well as the speed or time information when the main body reaches the position coordinates. The hand state information may include the opening and closing state of the hand, the size of the hand, etc. The position coordinates may be Cartesian space coordinates.

[0071] S104. Control the arm movement of the humanoid robot according to the first position information through the arm control module in the motion control system, control the hand movement of the humanoid robot according to the hand state information through the hand control module in the motion control system, and control the leg movement of the humanoid robot according to the pre-deployed reinforcement learning strategy and the second position information through the body control module in the motion control system.

[0072] Among them, the reinforcement learning strategy is obtained through pre-training.

[0073] Optionally, the task planner may transmit the first position information in the control information to the arm control module in the motion control system, transmit the second position information to the body control module in the motion control system, and transmit the hand state information to the hand control module in the motion control system.

[0074] Optionally, data can also be exchanged between the arm control module, the body control module and the hand control module in the motion control system to ensure that the motion process of the humanoid robot is more accurate, vivid and anthropomorphic.

[0075] Optionally, a mapping relationship between pre-obtained position coordinates and joint angles can be deployed in the arm control module. After obtaining the first position information, the arm control module can calculate the joint angle of the arm of the humanoid robot based on the position coordinates and the mapping relationship in the first position information, thereby controlling the movement of the arm of the humanoid robot according to the joint angle.

[0076] Optionally, a pre-trained reinforcement learning strategy can be deployed in the main control module. After obtaining the second position information, the main control module can perform reasoning and control based on the reinforcement learning strategy and the position coordinates in the second position information, so that the humanoid robot can move to the target position.

[0077] For example, the main control module can obtain the current state of the humanoid robot in real time through encoders, IMUs, plantar force sensors, etc., and encode the current state and the position coordinates in the second position information as input vectors for the reinforcement learning strategy. This strategy then generates control actions and decodes these control actions into specific leg joint control instructions (such as leg joint angles or torques). The leg joint controllers (such as motor drivers) in the main control module then adjust the joint angles, speeds, or torques based on the leg joint control instructions, generating an appropriate gait (such as walking or running), adjusting the leg motion trajectory, and using zero-moment point (ZMP) control or model predictive control (MPC) to ensure the robot maintains balance during movement. The new state of the humanoid robot is then measured in real time, and the control actions are dynamically adjusted based on the new state and the position coordinates in the second position information. This process is repeated until the humanoid robot reaches the position coordinates in the second position information.

[0078] Optionally, the hand control module may control hand movement according to hand state information.

[0079] In this embodiment, a visual language perception model is used to perceive and process the field of view image and task instructions to obtain the three-dimensional coordinates of the target object in the field of view image. A task planner based on a large language model then performs task planning based on the prompt words and the three-dimensional coordinates, and determines the control information corresponding to the prompt words. This enables the arm control module, the body control module, and the hand control module in the motion control system to respectively control the arm, hand, and leg movements of the humanoid robot. This integrates multiple modules such as visual perception, language understanding, task planning, and motion control into a unified framework, achieving end-to-end control from perception to execution and improving the integration of the humanoid robot control process. Furthermore, the integration of the large language model, the visual language perception model, and the motion control system is improved. Furthermore, the visual language perception model can adapt to changes in target objects in dynamic environments, and the task planner can adapt to different task requirements, thereby enhancing the robot's adaptability in complex environments. Furthermore, the visual language perception model provides intuitive perception results, and the large language model-based task planner generates interpretable control information, which facilitates human understanding and verification. This enhances the interpretability of the humanoid robot control process and improves the transparency and credibility of the humanoid robot control process. In addition, by controlling the arm movement, hand movement and leg movement of the humanoid robot respectively through the arm control module, the body control module and the hand control module, the control accuracy and task execution capability of the robot can be improved, and the flexibility of the humanoid robot control process can be ensured.

[0080] In one possible implementation, Figure 2 A schematic diagram of the structure of the visual language perception model provided in the embodiment of the present application is provided. Figure 3A schematic diagram of a flow chart for obtaining the three-dimensional coordinates of a target object in a field of view image in a humanoid robot control method provided in an embodiment of the present application, referring to Figure 2 as well as Figure 3 As shown, the visual language perception model includes an encoding module and a decoding module connected in sequence; in the above S102, the visual language perception model perceives and processes the visual field image and the task instruction to obtain the three-dimensional coordinates of the target object in the visual field image, including:

[0081] S301 , inputting the visual field image and the task instruction into a coding module, and the coding module coding the visual field image and the task instruction to obtain coded features.

[0082] Optionally, the field of view image and task instruction can be pre-processed and then input into the encoding module, which encodes the field of view image and task instruction to capture key information in the field of view image and task instruction and obtain encoded features.

[0083] Exemplarily, the encoding module can be implemented based on a deep learning framework (such as TensorFlow, PyTorch, etc.), for example, it can be implemented based on a convolutional neural network (CNN) and a recurrent neural network (RNN) or a Transformer.

[0084] S302: Input the encoded features into a decoding module, which decodes the encoded features to obtain three-dimensional coordinates.

[0085] Optionally, the encoded features are input into a decoding module, which decodes the encoded features to obtain three-dimensional coordinates.

[0086] Exemplarily, the decoding module can be implemented based on dense layers, upsampling layers or transposed convolutional layers.

[0087] By processing field of view images and task instructions through encoding and decoding, multimodal information fusion can be achieved, the comprehensiveness of perception can be improved, and features can be efficiently extracted and compressed, reducing computational complexity. At the same time, this modular design can enhance system scalability, suppress noise and improve robustness. In addition, it also supports end-to-end learning and improves adaptability.

[0088] In one possible implementation, Figure 4 Another structural diagram of the visual language perception model provided in the embodiment of the present application is shown below. Figure 5 A schematic diagram of a flow chart for obtaining encoded features in the humanoid robot control method provided in an embodiment of the present application, referring to Figure 4as well as Figure 5 As shown, the encoding module includes: an image encoding module, a text encoding module, and an encoding fusion module; the above S301 inputs the field of view image and task instruction into the encoding module, and the encoding module encodes the field of view image and task instruction to obtain encoded features, including:

[0089] S501: Input the field of view image into the image coding module, and the image coding module performs image perception coding processing on the field of view image to obtain image features.

[0090] Optionally, the image encoding module can preprocess the input field of view image, including denoising, enhancement, and cropping, to improve image quality and reduce the complexity of subsequent encoding. Key features of the image, such as edges, texture, color, and shape, are then extracted and encoded to obtain image features. Discrete encoding methods such as one-hot encoding and hash encoding, or continuous encoding methods such as floating-point representation, can be used.

[0091] S502: Input the task instruction into a text encoding module, and the text encoding module performs text-aware encoding on the task instruction to obtain text features.

[0092] Optionally, the text encoding module can first perform natural language processing on the input task instructions, such as word segmentation, part-of-speech tagging, named entity recognition, etc., to understand the semantics and structure of the task instructions, and then parse the processed task instructions into machine-understandable formats such as semantic trees and abstract syntax trees, and then encode the parsed instructions into a feature matrix of a preset type or format to obtain text features.

[0093] S503: Input the image features and text features into a coding fusion module, and the coding fusion module performs feature fusion on the image features and text features to obtain encoded features.

[0094] Optionally, the obtained image features and text features are input into a coding fusion module, which performs multimodal feature fusion on the image features and text features to obtain encoded features.

[0095] Among them, multimodal feature fusion can be implemented based on deep learning models, such as attention mechanism or neural network.

[0096] By encoding text and images separately and then performing multimodal fusion, we can fully utilize the characteristics of each modality, improve perception accuracy, suppress unimodal noise, increase robustness, and enhance information complementarity, which helps to fully understand the task and environment. Moreover, this modular design is easy to optimize and expand, supports complex task scenarios, and helps to improve the intelligence level of humanoid robots.

[0097] In a possible implementation, the prompt word further includes: pre-configured configuration information, the configuration information including: skill configuration information, operation space constraint configuration information, safety constraint configuration information, and kinematic configuration information.

[0098] The skill configuration information is used to indicate the executable actions of the arms, hands, and legs of the humanoid robot, the operation space constraint configuration information is used to indicate the operable space of the humanoid robot, and the kinematic configuration information is used to indicate the prior kinematic information of the humanoid robot.

[0099] Optionally, the prompt words include task instructions, skill configuration information, operation space constraint configuration information, safety constraint configuration information and kinematic configuration information.

[0100] Skill configuration information can include the humanoid robot's arm, hand, and leg skills. For example, arm skills include left and right arm movement, arm swinging, arm folding, and bimanual coordination. Hand skills include grasping, grasping, hand switching, and releasing. Leg skills include standing upright, bending, weight shifting, kicking, and leg lifting. This skill configuration information enables the humanoid robot to adapt to diverse task requirements and improve the efficiency and success rate of task execution.

[0101] The operating space constraint configuration information is used to define the spatial range of the robot's operation, such as the working area and obstacle locations. This information can prevent collisions between the humanoid robot and the environment, ensuring safe operation. It can also optimize path planning and improve the humanoid robot's movement efficiency.

[0102] Among them, the safety constraint configuration information is used to set safety restrictions, such as: the minimum safe distance between the humanoid robot and humans to prevent the robot from colliding with humans when moving or performing tasks. A list of fragile items, valuables, etc. that the humanoid robot must not touch or damage. Limiting the force used by the humanoid robot when performing tasks to prevent damage to humans or objects due to excessive force. The maximum movement speed of the humanoid robot in different scenarios to ensure that it can remain safe when interacting with humans or objects, and the emergency stop mechanism of the humanoid robot to prevent the humanoid robot from overloading or performing dangerous actions. The reliability and safety of the system can be enhanced through safety constraint configuration information.

[0103] Kinematic configuration information is used to indicate the kinematic characteristics of the humanoid robot, including motion parameters such as joint angles, angular velocities, accelerations, and the position and posture information of the humanoid robot. This ensures the feasibility and accuracy of motion planning, and also ensures the execution of complex movements, improving the quality of task completion.

[0104] In one possible implementation, Figure 6 A flow chart of determining the control information corresponding to the prompt word in the humanoid robot control method provided in the embodiment of the present application, referring to Figure 6 As shown, in the above S103, the task planner performs task planning based on the prompt word and the three-dimensional coordinates, and determines the control information corresponding to the prompt word, including:

[0105] S601: Obtain the current height of the humanoid robot.

[0106] Optionally, the current height of the humanoid robot can be obtained by performing calculations based on the environmental depth information obtained by a depth sensor (such as LiDAR or ToF camera).

[0107] For example, the environmental depth information can be filtered and segmented through the Open3D point cloud processing library, and the ground plane can be fitted through the RANSAC algorithm to obtain the plane equation. The point cloud data belonging to the ground is then extracted as a reference for height calculation, and the bottom point of the humanoid robot is detected. Then, according to the ground plane equation, the vertical distance from the bottom point of the humanoid robot to the ground is calculated to obtain the current height of the humanoid robot.

[0108] S602: The task planner performs task planning based on the prompt word, the current height, and the three-dimensional coordinates, and determines control information corresponding to the prompt word.

[0109] Exemplarily, the task planner can encode the prompt word, current height and three-dimensional coordinates respectively to obtain the prompt word encoding features, height encoding features and coordinate encoding features, and perform feature fusion on the prompt word encoding features, height encoding features and coordinate encoding features to obtain the fused features, and then perform task decomposition and task planning to obtain the control information corresponding to the prompt word.

[0110] By obtaining the current height of the humanoid robot and performing task planning based on the prompt word, current height, and three-dimensional coordinates, the control information corresponding to the prompt word is obtained. This can combine height and coordinate information to achieve precise motion planning and execution. It can also avoid collisions and dangerous actions through height and space constraints, thereby optimizing path and action planning and improving task execution efficiency.

[0111] In one possible implementation, Figure 7 A schematic diagram of a flow chart of controlling the leg movement of a humanoid robot in the humanoid robot control method provided in an embodiment of the present application, with reference to Figure 7 As shown, S104 controls the leg movement of the humanoid robot according to the reinforcement learning strategy and the second position information, including:

[0112] S701: Generate a target action sequence according to the reinforcement learning strategy and the second position information.

[0113] Optionally, the main control module can obtain the current state of the humanoid robot in real time through encoders, IMUs, plantar force sensors, etc., and encode the current state and the position coordinates in the second position information as input vectors of the reinforcement learning strategy, thereby generating a target action sequence through the reinforcement learning strategy.

[0114] The target action sequence includes at least one action that the subject needs to perform when the humanoid robot moves to the position corresponding to the second position information. For example, this can be the angle, velocity, or torque of a leg joint to achieve actions such as standing upright, bending, shifting center of gravity, kicking, and lifting the leg of the humanoid robot.

[0115] S702: Control the legs of the humanoid robot to move according to the target action sequence, and send a first start instruction to the arm control module when the legs move to the target position indicated by the second position information.

[0116] Optionally, after obtaining the target action sequence, the subject control module may traverse the target action sequence to implement execution of the target action sequence.

[0117] Optionally, during the traversal process, for the current action in the traversed target action sequence, the main control module obtains the state of the humanoid robot after the previous action is executed in real time through the encoder, IMU, plantar force sensor, etc., and combines the reinforcement learning strategy, the current action and the state after the previous action is executed to determine whether to adjust the current action. If so, the current action is adjusted through the reinforcement learning strategy and the state after the previous action is executed, and the current action is executed. If not, the current action is executed directly. Specifically, the execution of the current action can be achieved by controlling the leg joint controller of the leg until the traversal is completed.

[0118] Optionally, after the traversal is completed, the main control module can determine the current position of the humanoid robot again, and when it is determined that the humanoid robot has moved to the target position indicated by the second position information, send a first start instruction to the arm control module to make the arm control module start moving.

[0119] Exemplarily, the main body control module may also send a first start instruction to the arm control module when executing the last action in the target action sequence, so as to make the movement process of the humanoid robot more rich and anthropomorphic.

[0120] In a possible implementation, the above S104 controls the movement of the arm of the humanoid robot according to the first position information, including:

[0121] In response to the first start-up instruction, based on the first position information, a pre-built inverse kinematics model is used to predict the joint rotation angle of the arm of the humanoid robot, and the arm of the humanoid robot is controlled to rotate according to the joint rotation angle. After the rotation is completed, a second start-up instruction is sent to the hand control module.

[0122] Optionally, the arm control module responds to the first start instruction sent by the main body control module, calculates and predicts the joint rotation angle of the arm of the humanoid robot based on the first position information and a pre-built inverse kinematics model, and sends the joint rotation angle to the joint controller corresponding to the arm, so that the arm of the humanoid robot rotates according to the joint rotation angle, and sends a second start instruction to the controlled module after the rotation is completed.

[0123] In one possible implementation, Figure 8 A schematic diagram of a flow chart for controlling the hand movement of a humanoid robot in the humanoid robot control method provided in an embodiment of the present application, with reference to Figure 8 As shown, the above S104 controls the hand movement of the humanoid robot according to the hand state information, including:

[0124] S801. Respond to the second start instruction and obtain the current state of the hand of the humanoid robot.

[0125] Optionally, the hand control module may respond to the second start instruction, obtain the current state of the humanoid robot in real time through an encoder, an IMU, etc., and determine the current state of the hand of the humanoid robot.

[0126] S802: Determine whether the current state is consistent with the hand state information; if not, adjust the state of the hand of the humanoid robot to be consistent with the hand state information.

[0127] Optionally, after determining the current state of the hand of the humanoid robot, the hand control module may determine whether the current state of the hand is consistent with the hand state information, and if so, maintain the current state of the hand.

[0128] Optionally, if not, the state of the hand of the humanoid robot can be adjusted to be consistent with the hand state information by controlling the joint driver corresponding to the hand.

[0129] By controlling the main body to move first, then controlling the arms after the legs have completed their movements, and finally controlling the hands, this control method, which progresses from the lower limbs to the upper limbs and then to the extremities, ensures that the humanoid robot maintains balance and stability during movement, providing a good platform for subsequent arm and hand operations and allowing the humanoid robot to gradually enter a task state. It also breaks down complex control tasks into simpler subtasks, reducing the overall control difficulty. Furthermore, it ensures that the robot does not make sudden, unpredictable movements during operations, thus avoiding potential collisions or injuries. It also optimizes energy consumption.

[0130] Based on the same inventive concept, the embodiments of the present application also provide a humanoid robot control device corresponding to the humanoid robot control method. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the above-mentioned humanoid robot control method in the embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0131] Figure 9 A schematic diagram of a humanoid robot control system provided in an embodiment of the present application, referring to Figure 9 As shown, the humanoid robot control system includes: a visual language perception model, a task planner based on a large language model, and a motion control system. The motion control system includes: an arm control module, a hand control module, and a main body control module. A reinforcement learning strategy is deployed in the main body control module, and the reinforcement learning strategy is obtained through pre-training. The humanoid robot control system is used to execute the steps of the above-mentioned humanoid robot control method.

[0132] For the description of the visual language perception model, the task planner based on the large language model, and the processing flow of the motion control system in the humanoid robot control system, as well as the interaction flow between the visual language perception model, the task planner based on the large language model, and the motion control system, please refer to the relevant instructions in the above method embodiments and will not be described in detail here.

[0133] The present application also provides a humanoid robot, which includes the above-mentioned humanoid robot control system, and the humanoid robot can execute user instructions through interaction with the user.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0135] In addition, the functional units in the various embodiments of the present application can be integrated into a single processing unit, each unit can exist physically separately, or two or more units can be integrated into a single unit. If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0136] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A method for controlling a humanoid robot, characterized in that: include: Acquiring a task instruction and a field of view image, wherein the task instruction is used to instruct the humanoid robot to perform an action on a target object; Inputting the visual field image and the task instruction into a pre-trained visual language perception model, and having the visual language perception model perceive and process the visual field image and the task instruction to obtain the three-dimensional coordinates of the target object in the visual field image; Obtaining a prompt word, and inputting the prompt word and the three-dimensional coordinates into a pre-trained task planner based on a large language model, wherein the task planner performs task planning based on the prompt word and the three-dimensional coordinates, and determines control information corresponding to the prompt word, wherein the control information includes first position information, second position information, and hand state information, and the prompt word includes at least the task instruction; Controlling arm motion of the humanoid robot according to the first position information by an arm control module in a motion control system, controlling hand motion of the humanoid robot according to the hand state information by a hand control module in the motion control system, and controlling leg motion of the humanoid robot according to a pre-deployed reinforcement learning strategy and the second position information by a body control module in the motion control system, wherein the reinforcement learning strategy is obtained through pre-training and is deployed in the body control module; The controlling the leg movement of the humanoid robot according to the reinforcement learning strategy and the second position information includes: generating a target action sequence according to the reinforcement learning strategy and the second position information, the target action sequence including at least one action that the subject needs to perform when the humanoid robot moves to the position corresponding to the second position information; The legs of the humanoid robot are controlled to move according to the target action sequence, and when the legs move to the target position indicated by the second position information, a first start instruction is sent to the arm control module.

2. The humanoid robot control method according to claim 1, characterized in that: The visual language perception model includes an encoding module and a decoding module connected in sequence; The visual language perception model performs perception processing on the field of view image and the task instruction to obtain the three-dimensional coordinates of the target object in the field of view image, including: Inputting the visual field image and the task instruction into the encoding module, and the encoding module encoding the visual field image and the task instruction to obtain encoded features; The encoded features are input into the decoding module, and the decoding module decodes the encoded features to obtain the three-dimensional coordinates.

3. The humanoid robot control method according to claim 2, characterized in that: The encoding module includes: an image encoding module, a text encoding module and an encoding fusion module; The inputting the visual field image and the task instruction into the encoding module, and the encoding module encoding the visual field image and the task instruction to obtain encoded features, includes: Inputting the field of view image into the image coding module, and having the image coding module perform image perception coding processing on the field of view image to obtain image features; Inputting the task instruction into the text encoding module, and performing text-aware encoding processing on the task instruction by the text encoding module to obtain text features; The image features and the text features are input into the encoding fusion module, and the encoding fusion module performs feature fusion on the image features and the text features to obtain the encoded features.

4. The humanoid robot control method according to claim 1, characterized in that: The prompt word also includes: pre-configured configuration information, the configuration information including: skill configuration information, operation space constraint configuration information, safety constraint configuration information and kinematic configuration information; The skill configuration information is used to indicate the executable actions of the arms, hands and legs of the humanoid robot, the operation space constraint configuration information is used to indicate the operable space of the humanoid robot, and the kinematic configuration information is used to indicate the prior kinematic information of the humanoid robot.

5. The humanoid robot control method according to claim 1, characterized in that: The task planner performs task planning based on the prompt word and the three-dimensional coordinates, and determines control information corresponding to the prompt word, including: Obtaining the current height of the humanoid robot; The task planner performs task planning based on the prompt word, the current height, and the three-dimensional coordinates, and determines control information corresponding to the prompt word.

6. The humanoid robot control method according to claim 1, characterized in that: The controlling the movement of the arm of the humanoid robot according to the first position information includes: In response to the first start-up instruction, based on the first position information, a pre-built inverse kinematics model is used to predict the joint rotation angle of the arm of the humanoid robot, and the arm of the humanoid robot is controlled to rotate according to the joint rotation angle, and after the rotation is completed, a second start-up instruction is sent to the hand control module.

7. The humanoid robot control method according to claim 6, characterized in that: The controlling the hand movement of the humanoid robot according to the hand state information includes: In response to the second start instruction, obtaining a current state of the hand of the humanoid robot; Determine whether the current state is consistent with the hand state information; if not, adjust the state of the hand of the humanoid robot to be consistent with the hand state information.

8. A humanoid robot control system, characterized in that: include: A visual language perception model, a task planner based on a large language model, and a motion control system, wherein the motion control system includes an arm control module, a hand control module, and a body control module; A reinforcement learning strategy is deployed in the main control module, and the reinforcement learning strategy is obtained through pre-training. The humanoid robot control system is used to execute the humanoid robot control method according to any one of claims 1 to 7.

9. A humanoid robot, characterized in that: include: The humanoid robot control system according to claim 8.

Images