Humanoid robot control system, control method and related devices based on world knowledge augmentation

Abstract

This application provides a humanoid robot control system, control method, and related equipment based on world knowledge enhancement, belonging to the field of robotics technology. The system includes an execution module, a data acquisition module, and pre-trained encoding, knowledge enhancement, and planning modules. The data acquisition module collects real-time scene data around the humanoid robot; the encoding module encodes the real-time scene data to obtain scene features; the knowledge enhancement module processes the scene features based on preset world knowledge to obtain a task description; and the planning module solves the optimization objective function based on the task description, scene features, and the humanoid robot's body parameters to obtain motion control commands. This application can improve the robot's task execution accuracy and its cross-scene adaptability.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a humanoid robot control system, control method, and related equipment based on world knowledge enhancement. Background Technology

[0002] Bipedal humanoid robots have broad application prospects in industrial flexible manufacturing and home services due to their human-like movement capabilities. However, in related technologies, robots perform tasks based solely on environmental data, which can easily lead to insufficient task execution accuracy. When robots operate across different scenarios, human adjustments to robot parameters are required, resulting in rigid service processes and poor adaptability across different scenarios.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a humanoid robot control system, control method, and related equipment based on world knowledge enhancement, which can improve the robot's task execution accuracy and its adaptability across different scenarios.

[0005] To achieve the above objectives, one aspect of this application proposes a humanoid robot control system based on world knowledge enhancement, the system including an execution module, a data acquisition module, and a trained encoding module, a knowledge enhancement module, and a planning module;

[0006] The acquisition module is used to collect real-time scene data around the humanoid robot;

[0007] The encoding module is used to encode the real-time scene data to obtain scene features;

[0008] The knowledge enhancement module is used to process the scene features based on preset world knowledge to obtain a task description;

[0009] The planning module is used to solve the optimization objective function based on the task description, the scene features and the body parameters of the humanoid robot to obtain motion control commands;

[0010] The execution module is used to control the humanoid robot to move according to the motion control instructions.

[0011] In some embodiments, the trained encoding module, knowledge enhancement module, and planning module are obtained through the following steps:

[0012] Obtain a training dataset, which includes scene training data, world knowledge, and real task descriptions;

[0013] The scene training data is input into the encoding module to obtain the scene prediction code;

[0014] Input the scene prediction code and the world knowledge into the knowledge enhancement module to obtain the task prediction description;

[0015] Input the scene prediction code and the real task description into the planning module to obtain the motion prediction trajectory;

[0016] Errors for each module are calculated based on the scene prediction encoding, the task prediction description, the motion prediction trajectory, and the real data in the training dataset to obtain the error value for each module.

[0017] The total error value is obtained by weighted summation of the error values.

[0018] With the goal of reducing the total error value, the parameters of the encoding module, the knowledge enhancement module, and the planning module are adjusted respectively to obtain the trained encoding module, the knowledge enhancement module, and the planning module.

[0019] In some embodiments, the acquisition module includes a lidar, a camera, a force sensor, and a joint angle sensor;

[0020] The lidar and camera are used to collect real-time scene data around the humanoid robot;

[0021] The force sensor is used to collect the contact force at different parts of the humanoid robot;

[0022] The joint angle sensor is used to collect the real-time angles of different joints of the humanoid robot;

[0023] The body parameters include the contact force and the real-time angle.

[0024] In some embodiments, the planning module is further configured to:

[0025] Obtain feedback data on the humanoid robot's execution of the motion control commands;

[0026] Based on the feedback data, the motion control command is adjusted using a PID feedback control algorithm to obtain the corrected motion control command.

[0027] To achieve the above objectives, another aspect of this application proposes a humanoid robot control method based on world knowledge enhancement. This control method is applied to the humanoid robot control system based on world knowledge enhancement as described in the preceding embodiments. The method includes:

[0028] The control method includes the following steps:

[0029] Collect real-time scene data around the humanoid robot;

[0030] The real-time scene data is encoded to obtain scene features;

[0031] The scene features are processed based on pre-defined world knowledge to obtain a task description;

[0032] The objective function is solved based on the task description, the scene features, and the body parameters of the humanoid robot to obtain motion control commands;

[0033] The humanoid robot is controlled to move according to the motion control command.

[0034] In some embodiments, encoding the real-time scene data to obtain scene features includes:

[0035] Spatial features are extracted from the real-time scene data to obtain spatial features;

[0036] The real-time scene data is encoded with temporal features to obtain temporal features;

[0037] The spatial features and the temporal features are combined to obtain the scene features.

[0038] In some embodiments, processing the scene features based on preset world knowledge to obtain a task description includes:

[0039] The world knowledge is invoked to obtain a knowledge embedding vector;

[0040] The task description is obtained by performing knowledge fusion based on the knowledge embedding vector and the scene features.

[0041] In some embodiments, the optimization objective function is expressed as follows:

[0042] ;

[0043] in, , and These represent the weighting coefficients, This represents the joint angle vector at time t. This represents the reference joint angle vector at time t. This represents the joint torque vector at time t. This indicates the maximum torque threshold of the joint. Describing the normal norm, This represents the task execution error vector at time t. This represents the L2 norm.

[0044] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0045] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0046] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0047] The embodiments of this application include at least the following beneficial effects: This application provides a humanoid robot control system, control method, device, electronic device, storage medium, and program product based on world knowledge enhancement. This solution obtains a task description by fusing preset world knowledge and scene features, and solves the motion control command based on the task description, scene features, and the humanoid robot's ontological parameters, realizing a "knowledge-scene-motion" mapping link. No manual parameter configuration is required, enabling the robot to autonomously execute tasks across scenes, improving the robot's cross-scene adaptability, enhancing motion stability and natural interaction, and also improving the robot's task execution accuracy. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the structure of a humanoid robot control system based on world knowledge enhancement provided in an embodiment of this application;

[0049] Figure 2 This is a schematic diagram of the operation of the humanoid robot control system based on world knowledge enhancement provided in the embodiments of this application;

[0050] Figure 3 This is a flowchart of the humanoid robot control method based on world knowledge enhancement provided in the embodiments of this application;

[0051] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0054] Bipedal humanoid robots, possessing human-like locomotion capabilities, have broad application prospects in flexible industrial manufacturing and home services; however, the related technologies face the following technical challenges:

[0055] 1. Poor adaptability to different scenarios and rigid knowledge: In industrial scenarios, when faced with different workpiece weights and conveyor belt speeds, joint motion parameters (such as angle range and torque threshold) need to be manually reconfigured. It is impossible to dynamically adjust the parameters by combining the mechanical knowledge that "the center of gravity needs to be controlled when handling heavy objects". In household scenarios, it is difficult to understand the common sense of "handling glass products with care" and "preventing hot drinks from tipping over", making it difficult to adapt to new scenarios and resulting in rigid service processes.

[0056] 2. Disconnection and lack of closed loop between perception, planning and execution: The robot's perception module is based solely on environmental data and does not integrate world knowledge, resulting in the neglect of safety constraints and insufficient execution accuracy. During execution, there is a lack of clear feedback and adjustment logic. In industrial scenarios, the robot is prone to failure in picking up items due to changes in conveyor belt speed, while in home scenarios, the robot is prone to delivery deviations due to changes in the user's position.

[0057] 3. Difficulty in coordinating multiple degrees of freedom and non-reproducible parameters: Full-size bipedal robots typically have 20+ degrees of freedom. Existing motion planning methods are mostly optimized for single scenarios, resulting in large joint coordination errors when switching between scenarios. Furthermore, key parameters of the sensing hardware are missing, and there is no experimental basis for motion planning weights, making it impossible for technicians in the relevant technical field to reproduce the system.

[0058] 4. Training data relies on manual annotation and has weak generalization: Related training methods require manual annotation of a large amount of scene-task-trajectory data. In industrial scenarios, the annotation cost of "workpiece type-location-trajectory" is high. In home scenarios, the diversity of user habits leads to incomplete coverage of annotation data, poor model generalization ability, and difficulty in adapting to variables in real scenarios (such as workpiece weight fluctuations and user position changes).

[0059] In view of this, this application provides a humanoid robot control system, control method and related equipment based on world knowledge enhancement. This scheme obtains a task description by fusing preset world knowledge and scene features, and solves the motion control command based on the task description, scene features and humanoid robot body parameters, realizing the "knowledge-scene-motion" mapping link. No manual parameter configuration is required, enabling the robot to autonomously execute tasks across scenes, improving the robot's cross-scene adaptability, enhancing motion stability and interaction naturalness, and also improving the robot's task execution accuracy.

[0060] Figure 1 This is an optional structural diagram of a humanoid robot control system based on world knowledge enhancement provided in an embodiment of this application. The system includes, but is not limited to, an execution module, a data acquisition module, and a trained encoding module, a knowledge enhancement module, and a planning module.

[0061] The data acquisition module is used to collect real-time scene data around the humanoid robot;

[0062] The encoding module is used to encode real-time scene data to obtain scene features;

[0063] The knowledge enhancement module is used to process scene features based on preset world knowledge to obtain a task description;

[0064] The planning module is used to solve the optimization objective function based on the task description, scene characteristics and the body parameters of the humanoid robot to obtain motion control commands;

[0065] The execution module is used to control the humanoid robot to move according to motion control commands.

[0066] In some embodiments, the acquisition module is used to acquire real-time scene data around the humanoid robot;

[0067] Specifically, the acquisition module is configured to collect real-time scene data. For example, in an industrial scenario, the real-time scene data includes workpiece position, equipment parameters, and environmental obstacles; in a home scenario, the real-time scene data includes furniture layout, user position, and item type, which serve as the basis for the humanoid robot to perform subsequent tasks.

[0068] In some embodiments, the acquisition module includes a lidar, a camera, a force sensor, and a joint angle sensor;

[0069] LiDAR and cameras are used to collect real-time scene data around the humanoid robot;

[0070] Force sensors are used to collect contact forces at different parts of a humanoid robot;

[0071] Joint angle sensors are used to collect the real-time angles of different joints of a humanoid robot;

[0072] The body parameters include contact force and real-time angle.

[0073] Specifically, a lidar is installed on the head of the humanoid robot, with a detection range of 0.1-50m; two RGB-D cameras are installed on the chest of the humanoid robot; six six-axis force sensors are installed on the palms of both hands, the soles of both feet, and the elbows of both arms; and a joint angle sensor is installed on each joint of the humanoid robot. The lidar and RGB-D cameras collect three-dimensional environmental data, the force sensors collect contact force or torque, and the joint angle sensors collect real-time joint angles.

[0074] In some embodiments, the encoding module is used to encode real-time scene data to obtain scene features;

[0075] Specifically, the encoding module may include CNN and Transformer. Scene features are high-dimensional vectors that include both temporal and spatial dimensions, facilitating real-time mapping of spatiotemporal changes in the surrounding environment. Scene features can be represented as... Scene features are integrated with spatial and temporal characteristics, and the frame rates of laser scanning and camera shooting are matched with real-time planning requirements, enabling... It can be updated dynamically.

[0076] In some embodiments, the knowledge enhancement module is used to process scene features based on preset world knowledge to obtain a task description;

[0077] Specifically, the knowledge enhancement module operates based on a generative language model. World knowledge is stored in a world knowledge base, which is configured to store industrial mechanics knowledge, household common sense, and robot body parameters, and supports both cloud-based incremental updates and local manual updates. The generative language model calls upon the world knowledge base to... Process and output task description. For example, in an industrial setting, This can be characterized by taking a 3kg workpiece from a conveyor belt (speed 0.5m / s) and placing it on a 1.2m high shelf, with the left arm joints 1-3 at angles of 30°-60°, and a torque ≤50N. m, center of gravity offset ≤5cm; in home use scenarios This can represent the action of taking a glass of water from the kitchen countertop and handing it to the user (x=2.5m, y=1.8m), with the right arm joint angle at 4-6 degrees (45°-90°) and the contact force ≤10N. The generative language model will also output a scene explanation. Scene explanation It can characterize phenomena such as smooth conveyor belt operation, unobstructed shelves, and undamaged workpieces. The knowledge enhancement module is initialized based on massive amounts of data, enabling it to understand scene features and generate executable constraints, reducing manual intervention.

[0078] In some embodiments, the planning module is used to solve the optimization objective function based on the task description, scene features and the body parameters of the humanoid robot to obtain motion control commands;

[0079] Specifically, the planning module calculates motion control commands for k joints using either a model predictive control (MPC) algorithm or a trajectory optimization algorithm. , Including joint angle trajectory Torque ,and The constraints are satisfied: , Where q(t) is the k-dimensional joint angle vector at time t. Let be the k-dimensional joint moment vector at time t. The body parameters include: 14 degrees of freedom for both arms: 3 for the shoulder joint (pitch ±90°, rotation ±60°, lateral swing ±30°), 2 for the elbow joint (flexion / extension ±120°, rotation ±45°), and 2 for the wrist joint (pitch ±60°, rotation ±90°); 10 degrees of freedom for both legs: 3 for the hip joint (pitch ±60°, rotation ±30°, lateral swing ±20°), 1 for the knee joint (flexion / extension ±120°), and 1 for the ankle joint (pitch ±30°). The planning module is based on... and Solving motion control commands This enables the mapping link between "knowledge-scene-motion" and reduces the need for manual parameter configuration.

[0080] In some embodiments, the execution module is used to control the humanoid robot to move according to motion control instructions;

[0081] Specifically, the execution module executes the motion control commands. Joint angle trajectory Torque These mechanisms drive the joints of the humanoid robot, enabling it to perform related tasks.

[0082] In some embodiments, the system further includes a feedback module, which is used for:

[0083] Collect feedback data on the execution of motion control commands by the humanoid robot;

[0084] Specifically, the feedback module collects feedback data through force sensors and vision sensors. The data is then sent to the planning module for adjustments. The foundation.

[0085] In some embodiments, the planning module is further configured to:

[0086] Obtain feedback data on the execution of motion control commands by the humanoid robot;

[0087] Based on the feedback data, the motion control command is adjusted using a PID feedback control algorithm to obtain the corrected motion control command.

[0088] Specifically, the planning module is based on feedback data. Adjustment is achieved through PID feedback control algorithm If the force sensor provides feedback... Then adjust according to the coefficient. Reduce the torque of the corresponding joint ( If the visual sensor feedback position is off); Then adjust according to the coefficient. Correcting joint angles ( ), This indicates the original joint angle. This indicates the corrected joint angle until the task is completed.

[0089] In some embodiments, please refer to Figure 2 The humanoid robot control system includes a perception module (acquisition module + encoding module), a world knowledge base, a generative language model, a motion planning module (planning module), and a task execution module (execution module + feedback module).

[0090] The sensing module is configured to collect scene data (industrial scene: workpiece position, equipment parameters, environmental obstacles; home scene: furniture layout, user position, item type) and encode it into a scene representation. (512-dimensional vector, the first 256 dimensions are spatial features, and the last 256 dimensions are temporal features);

[0091] The world knowledge base is configured to store industrial mechanics knowledge (such as "the center of gravity offset of heavy object handling is ≤5cm based on the robot body weight of 60kg and the workpiece weight ≤5kg" and "torque = force × lever arm"), household common sense (such as "the contact force is ≤10N based on the impact resistance of glass products ≥50N and the safety redundancy coefficient of 0.2"), and robot body parameters (the joint limit angles of k degrees of freedom). Maximum torque k is a positive integer, k=28 in the example), and supports incremental updates in the cloud (receiving new scene knowledge text through the 5G network, importing it after verifying its effectiveness through a pre-trained generative language model) and manual updates in the local area (entering custom thresholds or common sense through the robot control panel).

[0092] The generative language model is configured to load a world knowledge base for scene representation. Process the data and output a task description in natural language. (Including task objectives and motion constraints) and scenario explanation ;

[0093] The motion planning module is configured to be based on scene representation. Task Description Given the robot's body parameters, motion control commands for k joints are calculated using model predictive control (MPC) or trajectory optimization algorithms. (including joint angle trajectory) Torque ),and The constraints are satisfied: q(t) is the k-dimensional joint angle vector at time t. Let k be the k-dimensional joint torque vector at time t;

[0094] The task execution module is configured to receive motion control commands. It drives k joints to perform tasks and collects feedback data F (contact force, position deviation) through sensors, and adjusts the system in real time based on a PID feedback control algorithm. .

[0095] The system also includes a training module for adjusting the parameters of the generative language model, motion planning module, and task execution module.

[0096] In some embodiments, the trained encoding module, knowledge enhancement module, and planning module are obtained through the following steps:

[0097] Obtain the training dataset, which includes scene training data, world knowledge, and real-world task descriptions;

[0098] Input the scene training data into the encoding module to obtain the scene prediction code;

[0099] Input the scene prediction code and world knowledge into the knowledge enhancement module to obtain the task prediction description;

[0100] Input the scene prediction code and the real task description into the planning module to obtain the motion prediction trajectory;

[0101] Errors for each module are calculated based on scene prediction coding, task prediction description, motion prediction trajectory, and real data in the training dataset to obtain the error values ​​for each module.

[0102] The total error value is obtained by weighted summation of the error values.

[0103] With the goal of reducing the total error, the parameters of the encoding module, knowledge enhancement module, and planning module are adjusted respectively to obtain the trained encoding module, knowledge enhancement module, and planning module.

[0104] Specifically, before performing a task, the humanoid robot needs to be trained. This embodiment mainly trains the coding module, knowledge enhancement module, and planning module of the humanoid robot control system. The training dataset includes multiple samples, each of which is ( ),in, For the training data of the i-th scene, In order to correspond to world knowledge, This is a description of a real task. This represents the actual motion trajectory.

[0105] Input scene training data into the encoding module To obtain scene prediction codes .

[0106] Input scene prediction codes into the knowledge enhancement module and world knowledge Obtain the task prediction description .

[0107] Input scene prediction codes into the planning module and the actual task description Obtain the predicted motion trajectory .

[0108] Error calculations are performed separately for the encoding module, knowledge enhancement module, and planning module. For the encoding module, the error is calculated... ,in, for In a real-world scenario, backpropagation adjusts the parameters of the encoding module, iterating for 100 rounds. The error decreased from 0.02 to 0.005; for the knowledge enhancement module, the cross-entropy error was calculated. After adjusting the parameters of the generative language model in the knowledge enhancement module and iterating for 200 rounds... The error decreased from 0.15 to 0.03; for the planning module, the calculation error... After adjusting the planning module parameters and iterating 150 times... It decreased from 0.05 to 0.01.

[0109] This embodiment also constructs a fine-tuning dataset. Execution data including real-world scenarios ( ), Encoding for real-world scenarios, This is a description of a task in a real-world scenario. These are motion control commands for real-world scenarios. To account for execution errors (e.g., industrial scenarios: torque exceeding 5N). m, positional deviation 0.03m; home use scenario: tilt angle 8°).

[0110] Finally, calculate the total error. ,in , , and In order to fine-tune the weights, in this embodiment, ,satisfy Backpropagation adjusts the parameters of all modules, ultimately yielding trained encoding, knowledge enhancement, and planning modules, which can improve the model's generalization ability.

[0111] In some embodiments, the pre-trained dataset In the industrial scenario samples, there are variable combinations of workpiece weight 1-5kg and conveyor belt speed 0.3-0.8m / s, while the household scenario samples include variable combinations of item type (glass, ceramic, plastic) and user location (x=1.5-3.0m, y=1.0-2.0m), enabling the model to adapt to multiple scenario variables.

[0112] In some embodiments, the training method for the control system includes the following steps:

[0113] T1: Constructing the pre-training dataset Each sample is ( ),in For the i-th scenario data (industrial scenario: 20,000 records of automotive parts handling, 30,000 records of electronic component assembly; household scenario: 20,000 records of housework cleaning, 20,000 records of item delivery), To correspond with world knowledge (industrial mechanics such as "F=mg", and common household knowledge such as "hot drinks should not be poured"), This is a description of a real task. The actual motion trajectory (k-dimensional joint angle sequence);

[0114] T2: Pre-trained perception module (acquisition module + encoding module): Input Output Calculation error for In a real-world scenario, 512-dimensional features were manually labeled. Backpropagation was used to adjust the parameters of the perception module, and after 100 iterations... It decreased from 0.02 to 0.005;

[0115] T3: Pre-trained generative language model: Input Calculate the cross-entropy error After adjusting the parameters of the generative language model and iterating for 200 rounds... It decreased from 0.15 to 0.03;

[0116] T4: Pre-training motion planning module (planning module): Input Calculation error After adjusting the motion planning module parameters and iterating for 150 rounds... It decreased from 0.05 to 0.01;

[0117] T5: Building a Fine-tuning Dataset Execution data including real-world scenarios ( ), For execution error (industrial scenario: torque exceeding 5N) m, positional deviation 0.03m; home use scenario: tilt angle 8°).

[0118] T6: Fine-tuning training: Calculate the total error ,in To fine-tune the weights (recommended values: ,satisfy This value was determined through 500 real-world scenario experiments. The execution error exceeds 0.1N. When the task description accuracy drops below 85%, backpropagation is used to adjust all module parameters, iterating for 50 rounds. The generalization ability of the model improved by 30% from 0.066 to 0.01.

[0119] Figure 3 This is an optional flowchart of the humanoid robot control method based on world knowledge enhancement provided in the embodiments of this application. Figure 3 The method may include, but is not limited to, steps S101 to S105.

[0120] Step S101: Collect real-time scene data around the humanoid robot;

[0121] Step S102: Encode the real-time scene data to obtain scene features;

[0122] Step S103: Process the scene features based on preset world knowledge to obtain the task description;

[0123] Step S104: Solve the optimization objective function based on the task description, scene features and the body parameters of the humanoid robot to obtain motion control commands;

[0124] Step S105: Control the humanoid robot to move according to the motion control command.

[0125] It is understood that the content of the above system embodiments is applicable to this method embodiment. The specific functions implemented in this method embodiment are the same as those in the above system embodiments, and the beneficial effects achieved are also the same as those achieved in the above system embodiments.

[0126] In some embodiments, step S102 may include, but is not limited to, steps S201 to S203:

[0127] Step S201: Extract spatial features from real-time scene data to obtain spatial features;

[0128] Step S202: Encode the real-time scene data using time-series features to obtain time-series features;

[0129] Step S203: Combine spatial features and temporal features to obtain scene features.

[0130] In steps S201 to S203 of some embodiments, spatial features are extracted using a convolutional neural network (CNN) to obtain spatial features, and temporal features are encoded using a Transformer to obtain temporal features. The spatial features and temporal features are then combined to obtain scene features. For example, the spatial features and temporal features are 256-dimensional vectors, and the combined scene features are 512-dimensional, with the first 256 dimensions being spatial features and the last 256 dimensions being temporal features.

[0131] In some embodiments, step S103 may include, but is not limited to, steps S301 to S302:

[0132] Step S301: Invoke world knowledge to obtain knowledge embedding vectors;

[0133] Step S302: Perform knowledge fusion based on knowledge embedding vectors and scene features to obtain a task description.

[0134] In steps S301 to S302 of some embodiments, the knowledge embedding vector can represent relevant knowledge data, which facilitates the fusion of knowledge and scene features to obtain an accurate task description, which is convenient for subsequent motion planning and improves the task execution accuracy of the humanoid robot.

[0135] In some embodiments, the optimization objective function is expressed as follows:

[0136] ;

[0137] in, This represents the total loss value. , and These represent the weighting coefficients, This represents the joint angle vector at time t. This represents the reference joint angle vector at time t. This represents the joint torque vector at time t. This indicates the maximum torque threshold of the joint. Describing the normal norm, This represents the task execution error vector at time t. This represents the L2 norm.

[0138] Specifically, Weighting coefficients This range was determined through 1000 sets of industrial / domestic scenario simulation experiments. The time trajectory smoothness error exceeds 0.1 rad. The task error exceeded 0.05m. The probability of torque exceeding the limit exceeds 10%. The time trajectory response delay exceeds 0.02s; The accuracy of the task was not up to standard. The robustness decreases over time, which is used to balance trajectory smoothness, torque constraints, and task errors; : k-dimensional joint angle vector at time t (in the example k=28, including 2-dimensional head, 2-dimensional torso, 14-dimensional arms, and 10-dimensional legs); Reference joint angle vector at time t (based on the preset task objective, such as the hand reaching the workpiece position at t=0.8s in an industrial part-picking scenario); : k-dimensional joint torque vector at time t (Example: left arm joint) ); : Maximum torque threshold of the joint; Otherwise, it is 0). : Task execution error vector at time t (industrial scenario: workpiece position deviation; household scenario: item placement deviation); L2 norm: measures the magnitude of vector error.

[0139] In some embodiments, a humanoid robot control method is as follows:

[0140] S1: The perception module collects scene data (LiDAR scans the environment, RGB-D camera captures dynamic targets), extracts spatial features through a convolutional neural network (CNN), and encodes temporal features using a Transformer to generate a scene representation (scene features). (512-dimensional vector);

[0141] S2: The generative language model calls upon the world knowledge base to... Process and output task description. and scene explanation ;

[0142] S3: Motion planning module based on Given the robot's body parameters, the objective function is solved using a model predictive control (MPC) algorithm, and motion control commands are output. ;

[0143] S4: Task execution module drives joint execution Feedback data F is collected through force sensors and vision sensors;

[0144] S5: The motion planning module adjusts based on F using a PID feedback control algorithm. If force sensor feedback Then adjust according to the coefficient. Reduce the torque of the corresponding joint ( If the visual sensor feedback position is off); Then adjust according to the coefficient. Correcting joint angles ( ), return to S4 until the task is completed.

[0145] The solutions of the embodiments of the present invention will be described in detail and explained below, in conjunction with specific home or industrial scenarios:

[0146] In system hardware implementation.

[0147] Robot body: Full-size design (1.7m tall, 60kg), 28 degrees of freedom:

[0148] Head: Pitch ±30° (nodding), Rotation ±45° (turning head);

[0149] Trunk: Pitch ±15° (forward / backward tilt), Lateral tilt ±10° (left / right tilt);

[0150] Both arms (7 in each arm, 14 in total): 3 shoulder joints (pitch ±90°, rotation ±60°, lateral swing ±30°), 2 elbow joints (flexion and extension ±120°, rotation ±45°), and 2 wrist joints (pitch ±60°, rotation ±90°).

[0151] Both legs (5 per leg, 10 in total): 3 hip joints (pitch ±60°, rotation ±30°, lateral swing ±20°), 1 knee joint (flexion ±120°), 1 ankle joint (pitch ±30°);

[0152] The joint drive uses a servo motor (position accuracy ±0.1°, response delay ≤0.005s) to meet the multi-degree-of-freedom coordination requirements.

[0153] Sensing hardware:

[0154] Head: LiDAR (detection distance 0.1-50m, accuracy ±2cm, scanning frequency 10Hz), the 10Hz scanning frequency matches the 0.01s calculation time of the motion planning module, thereby realizing real-time updates of environmental data;

[0155] Chest: One RGB-D camera on each side (1280×720 resolution, depth error ±1%, frame rate 30fps). The 30fps frame rate can capture dynamic targets (such as moving conveyor belt workpieces), and the depth error is ≤1%, which improves the positioning accuracy.

[0156] Force sensing: One six-axis force sensor each on the palms of both hands, the soles of both feet, and the elbows of both arms (range 0-500N, accuracy ±0.1N). The ±0.1N accuracy can identify excessive contact force on glass products (e.g., triggering adjustment when the force exceeds the limit).

[0157] Joint sensing: Each joint has one angle sensor (range ±180°, accuracy ±0.05°), and the ±0.05° accuracy ensures that the execution error of the joint angle trajectory q(t) is ≤0.1rad.

[0158] Computing hardware: Equipped with a GPU processor (200 TOPS computing power), supporting parallel processing.

[0159] Perceptual encoding: Generation time ≤ 0.05s;

[0160] Language Model: Output time ≤ 0.03s;

[0161] Exercise planning: Calculation time ≤ 0.01s;

[0162] Meets real-time control requirements (total time ≤ 0.09s < 0.1s control cycle).

[0163] In an embodiment of the control method, the handling of automotive parts in an industrial setting is taken as an example.

[0164] Step S1: Scene data acquisition and encoding.

[0165] The lidar (10Hz) acquires the position of a workpiece (model: automotive bearing, weight 3kg, size 80mm×50mm) on a conveyor belt (speed 0.5m / s): x=1.5m, y=0.8m, z=0.5m, attitude: rotation angle 15°;

[0166] An RGB-D camera (30fps) confirmed the workpiece was undamaged for three consecutive frames, and depth data verified the positional deviation to be ≤0.02m.

[0167] Force sensor initial reading: 0N (no contact with workpiece); joint angle sensor obtains initial angle q_0 (left arm joint 1=0°, joint 2=45°, joint 3=30°).

[0168] CNN extracts environmental spatial features (conveyor belt edges, shelf locations), Transformer encodes temporal features (conveyor belt speed changes of 0.5 m / s), and outputs a scene representation. (512 dimensions: the first 256 dimensions include the workpiece position / orientation, and the last 256 dimensions include the conveyor belt speed).

[0169] Step S2: Task description generation.

[0170] Generative language models call upon the world knowledge base:

[0171] Industrial mechanics knowledge: "Workpiece weight 3kg, body weight 60kg, center of gravity offset = (3×0.5) / (60×0.8) ≈ 0.031m ≤ 5cm" "Torque = 3kg × 9.8m / s" 2 ×0.3m≈8.82N m≤50N m (left arm joint 1τmax=60N·m)”;

[0172] Scenario rule: "Conveyor belt speed 0.5m / s, the speed of picking up the parts must be synchronized to avoid the workpiece slipping";

[0173] Output task description "Take 3 kg of automotive bearings (x=1.5, y=0.8, z=0.5) from the conveyor belt (speed 0.5 m / s) and place them on the shelf (x=3.0, y=0.8, z=1.2). Left arm joints 1-3 should have an angle of 30°-60° and a torque ≤50N." m, center of gravity offset ≤ 5cm;

[0174] Explanation of output scenarios "The conveyor belt is smooth (no speed fluctuations were detected by 10Hz scanning), the shelves are free of obstacles (no abnormalities were found in the depth data from the RGB-D camera), and the workpieces are undamaged."

[0175] Step S3: Calculate motion control commands.

[0176] The motion planning module substitutes the optimization objective function ( ):

[0177]

[0178] Reference trajectory Based on the conveyor belt speed, the picking time is 0.8s (when the hand reaches the workpiece position at t=0.8s, the synchronous conveyor belt displacement is 0.5m / s), and the placement time is 1.7s (when the workpiece arrives at the shelf at t=2.5s).

[0179] MPC algorithm for solving constraints:

[0180] Joint angle: Left arm joint 1 ≤ 60° ( ), left arm joint 2≤90°;

[0181] Torque: ;

[0182] Output motion control commands :

[0183] t=0.2s: Left arm joint 1=30°, joint 2=40°, joint 3=35°, ;

[0184] t=0.8s (time of pickup): left arm joint 1=45°, joint 2=55°, joint 3=45°, ;

[0185] t=2.5s (placement time): left arm joint 1=60°, joint 2=60°, joint 3=60°, .

[0186] Step S4: Task execution and feedback adjustment.

[0187] The task execution module drives the left arm joint. At t=0.8s, the hand contacts the workpiece, and the force sensor feedback is F=8N (≤10N, safe).

[0188] At t=1.0s, the speed of the conveyor belt detected by the lidar slightly increased to 0.55m / s (fluctuation of 10%), and the vision sensor reported the workpiece position deviation. ;

[0189] PID feedback adjustment: Press Adjust the joint angle: increase the left arm joint angle by 2° from 55° to 55° - 0.9 × 0.03 m × (180 / π) ≈ 55° - 1.54° = 53.46°, simultaneously pressing... Fine-tune the torque to 48 × 0.8 = 38.4 N. m;

[0190] At t=2.5s, the workpiece is placed on the shelf. The force sensor reports F=0N, and the vision sensor verifies the position deviation as 0.01m≤0.05m. The task is completed, and the total torque is ≤48N. m, center of gravity offset ≤3cm.

[0191] In an embodiment of the control method, the delivery of a glass water cup in a household setting is taken as an example.

[0192] Steps S1-S2: Data acquisition and task generation.

[0193] Sensing module: RGB-D camera (30fps) captures the position of a glass water glass (weight 0.3kg, size Φ80mm×150mm) on the kitchen countertop at x=2.0m, y=1.5m, z=0.9m, and the user's position at x=2.5m, y=1.8m, z=1.1m (sitting posture).

[0194] Knowledge base quotes: "Glass impact resistance ≥ 50N, safety redundancy coefficient 0.2, contact force ≤ 50 × 0.2 = 10N" and "Water cup height 150mm, tilt angle ≤ 5° to avoid leakage".

[0195] It means: "Take a glass of water (x=2.0, y=1.5, z=0.9) and hand it to the user (x=2.5, y=1.8, z=1.1), with the right arm joint angle 4-6 at 45°-90°, contact force ≤10N, and tilt angle ≤5°."

[0196] Steps S3-S5: Execution and Adjustment.

[0197] Exercise planning: (Prioritize trajectory smoothing) (Torque constraint) (Task accuracy). set up Take the cup. Pass the cup;

[0198] Execution: t=0.5s, hand touches water cup, force sensor feedback F=5N (normal), vision sensor detects tilt angle 3° (normal);

[0199] Feedback adjustment: t=0.8s, the user's hand moves to x=2.6m, y=1.8m, z=1.1m, position deviation e=0.1m, press... The angle of the right arm joint 4 was increased from 45° to 45° - 0.9 × 0.1 × (180 / π) ≈ 45° - 5.16° = 39.84°;

[0200] Cup handover complete: t=1.2s The user receives the cup, the force sensor feedback F=8N (user's grip force), then drops to 0N, the tilt angle is ≤4°, and there is no leakage.

[0201] In some embodiments of training a humanoid robot control system.

[0202] Step T1: Construction of the pre-training dataset.

[0203] Industrial scenario samples (50,000 entries):

[0204] Automotive parts handling (20,000 pieces): workpiece weight 1-5kg (step 0.5kg), conveyor belt speed 0.3-0.8m / s (step 0.1m / s), covering 20 types of workpieces;

[0205] Electronic component assembly (30,000 pieces): component size 5-50mm (5mm increment), assembly accuracy 0.01-0.05m (0.01m increment);

[0206] Home scenario samples (40,000):

[0207] Household cleaning (20,000 items): Cleaning tools weigh 0.5-2kg, cleaning area 1-5m²;

[0208] Item delivery (20,000 items): Item type (glass, ceramic, plastic), user location x=1.5-3.0m, y=1.0-2.0m;

[0209] World knowledge texts (50,000 entries): Industrial mechanics formulas (F=mg, torque = force × lever arm), household common sense (hot drinks ≥60℃ require protection against burns), safety regulations (joint torque should not exceed 80% of the rated value).

[0210] Steps T2-T4: Pre-training.

[0211] Sensing module: Input 10,000 data points from industrial / domestic scenarios, output With reality (Comparison of manually labeled 512-dimensional features) The value decreased from 0.02 to 0.005, making Spatial characteristic error ≤ 0.02m, temporal characteristic error ≤ 0.01m / s;

[0212] Generative language models: Input and Output Cross-entropy The accuracy of task description is reduced from 0.15 to 0.03, with a accuracy rate of ≥95% (e.g., "3kg workpiece" is not misjudged as "5kg").

[0213] Motion planning module: Input and Output , From 0.05 to 0.01, the joint angle trajectory error is ≤0.1 rad, and the torque error is ≤2 N. m.

[0214] Steps T5-T6: Fine-tuning.

[0215] Fine-tuning the dataset (8000 entries):

[0216] Industrial scenario error samples (5000): Torque exceeding 5N m (2000), positional deviation 0.03m (3000);

[0217] Error samples in home scenarios (3000): tilt angle 8° (1500), contact force exceeding 10N (1500);

[0218] Fine-tuning training:

[0219] Total error ,initial ;

[0220] Iterate 50 times: 200 samples are selected in each round, and parameters are adjusted through backpropagation. In the 50th round... ;

[0221] Verification results: The success rate of picking up items in industrial scenarios increased from 90% in pre-training to 98%+, and the delivery deviation in household scenarios decreased from 0.1m to below 0.05m.

[0222] Step T7: Knowledge base update example.

[0223] New scenario: Semiconductor component handling (contact force ≤ 2N);

[0224] Cloud update: 5G network pushes knowledge text "Semiconductor component impact resistance ≥10N, safety redundancy coefficient 0.2, contact force ≤2N";

[0225] Model Validation: The generative language model validates that this knowledge does not conflict with existing parameters (such as force sensor range 0-500N, accuracy ±0.1N), and the calculated torque corresponding to a contact force of 2N is ≤10N. m (consistent with joints) );

[0226] Import knowledge base: Automatically add the entry "contact force of semiconductor components ≤ 2N", without retraining the model, the system can directly perform semiconductor pick-and-place tasks.

[0227] This system embodiment has at least the following beneficial effects:

[0228] The perception module integrates spatial and temporal features, and uses laser scanning and camera frame rate matching to achieve real-time planning requirements. Dynamically updated.

[0229] The world knowledge base enables the derivation of knowledge thresholds (e.g., glass contact force = impact resistance strength × safety factor) and supports incremental updates to adapt to new scenarios.

[0230] Generative language models are initialized based on massive amounts of data, can understand scene features and generate executable constraints, reducing human intervention.

[0231] The motion planning module (planning module) can achieve multi-objective balance.

[0232] The task execution module is based on closed-loop adjustment logic, which can quantitatively correct for force over-limit or positional deviation, thereby improving execution stability.

[0233] This control method embodiment has at least the following beneficial effects:

[0234] Deep integration of knowledge, perception, and planning: Generative language models transform abstract world knowledge into executable Dt, and the motion planning module solves Cm based on Dt and Vs, opening up the "knowledge-scene-motion" mapping link without the need for manual parameter configuration.

[0235] Multi-degree-of-freedom dynamic optimization and closed-loop control: Motion planning is based on multi-objective functions and is adapted to 28 degrees of freedom coordination; during execution, motion control commands are corrected in real time through PID feedback, achieving a success rate of ≥98% in industrial scenarios and a delivery deviation of ≤0.05m in household scenarios.

[0236] Dynamic knowledge updates adapt to new scenarios: The knowledge base supports incremental updates in the cloud and manual updates locally. When new knowledge is added, the system automatically verifies and imports it without retraining the model, making it highly scalable.

[0237] This training method embodiment has at least the following beneficial effects:

[0238] Pre-training cost reduction and parameterized coverage: The model is initialized based on 100,000+ industrial / household pre-training samples (including multi-variable combinations), reducing manual annotation by 60%; the samples cover multiple variables in industrial / household scenarios, improving model adaptability.

[0239] Fine-tuning for precise adaptation to real-world scenarios: Fine-tuning based on real execution errors (such as torque overruns and position deviations). The total error Ltotal weight was determined through 500 sets of experiments. After iteration, Ltotal decreased from 0.066 to 0.01, achieving a 40% reduction in motion error in industrial scenarios and an interaction satisfaction rate of ≥90% in home scenarios.

[0240] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0241] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0242] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0243] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0244] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application.

[0245] The input / output interface 903 is used to implement information input and output;

[0246] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0247] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0248] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0249] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0250] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0251] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0252] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0253] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0254] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0255] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0256] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0257] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0258] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0259] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0260] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0261] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0262] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0263] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0264] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A humanoid robot control system based on enhanced world knowledge, characterized in that, The system includes an execution module, a data acquisition module, and pre-trained encoding, knowledge enhancement, and planning modules. The acquisition module is used to collect real-time scene data around the humanoid robot; The encoding module is used to encode the real-time scene data to obtain scene features; The knowledge enhancement module is used to process the scene features based on preset world knowledge to obtain a task description; The planning module is used to solve the optimization objective function based on the task description, the scene features and the body parameters of the humanoid robot to obtain motion control commands; The execution module is used to control the humanoid robot to move according to the motion control command; The optimization objective function is expressed as follows: ； in, This represents the total loss value. , and These represent the weighting coefficients, This represents the joint angle vector at time t. This represents the reference joint angle vector at time t. This represents the joint torque vector at time t. This indicates the maximum torque threshold of the joint. Describing the normal norm, This represents the task execution error vector at time t. Represents the L2 norm; The trained encoding module, knowledge enhancement module, and planning module are obtained through the following steps: Obtain a training dataset, which includes scene training data, world knowledge, and real task descriptions; The scene training data is input into the encoding module to obtain the scene prediction code; Input the scene prediction code and the world knowledge into the knowledge enhancement module to obtain the task prediction description; Input the scene prediction code and the real task description into the planning module to obtain the motion prediction trajectory; Errors for each module are calculated based on the scene prediction encoding, the task prediction description, the motion prediction trajectory, and the real data in the training dataset to obtain the error value for each module. The total error value is obtained by weighted summation of the error values. With the goal of reducing the total error value, the parameters of the encoding module, the knowledge enhancement module, and the planning module are adjusted respectively to obtain the trained encoding module, the knowledge enhancement module, and the planning module.

2. The system according to claim 1, characterized in that, The acquisition module includes a lidar, a camera, a force sensor, and a joint angle sensor; The lidar and camera are used to collect real-time scene data around the humanoid robot; The force sensor is used to collect the contact force at different parts of the humanoid robot; The joint angle sensor is used to collect the real-time angles of different joints of the humanoid robot; The body parameters include the contact force and the real-time angle.

3. The system according to claim 1, characterized in that, The planning module is also used for: Obtain feedback data on the humanoid robot's execution of the motion control commands; Based on the feedback data, the motion control command is adjusted using a PID feedback control algorithm to obtain the corrected motion control command.

4. A humanoid robot control method based on world knowledge enhancement, characterized in that, The control method is applied to the humanoid robot control system based on world knowledge enhancement as described in any one of claims 1 to 3, and the control method includes the following steps: Collect real-time scene data around the humanoid robot; The real-time scene data is encoded to obtain scene features; The scene features are processed based on pre-defined world knowledge to obtain a task description; The objective function is solved based on the task description, the scene features, and the body parameters of the humanoid robot to obtain motion control commands; The humanoid robot is controlled to move according to the motion control command; The optimization objective function is expressed as follows: ； in, This represents the total loss value. , and These represent the weighting coefficients, This represents the joint angle vector at time t. This represents the reference joint angle vector at time t. This represents the joint torque vector at time t. This indicates the maximum torque threshold of the joint. Describing the normal norm, This represents the task execution error vector at time t. This represents the L2 norm.

5. The humanoid robot control method based on world knowledge enhancement according to claim 4, characterized in that, The process of encoding the real-time scene data to obtain scene features includes: Spatial features are extracted from the real-time scene data to obtain spatial features; The real-time scene data is encoded with temporal features to obtain temporal features; The spatial features and the temporal features are combined to obtain the scene features.

6. The humanoid robot control method based on world knowledge enhancement according to claim 4, characterized in that, The process of processing the scene features based on preset world knowledge to obtain a task description includes: The world knowledge is invoked to obtain a knowledge embedding vector; The task description is obtained by performing knowledge fusion based on the knowledge embedding vector and the scene features.

7. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the control method as described in any one of claims 4 to 6.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the control method as described in any one of claims 4 to 6.

Images