A method and related equipment for humanoid robot task decision-making and execution based on embodied cognition enhancement

Abstract

This application discloses a method and related equipment for humanoid robot operation decision-making and execution based on embodied cognition enhancement. The method includes: acquiring sensor data of the current working environment through the humanoid robot's sensor array and performing two-level fusion to obtain working environment information; inputting this information into an embodied cognition data generation model to obtain simulated embodied cognition data; inputting the working environment information and simulated embodied cognition data into an end-to-end humanoid robot operation model for multi-dimensional query; performing cross-attention calculation based on the simulated embodied cognition data and multi-dimensional query results to generate humanoid operation trajectory planning results, converting them into control commands, and executing corresponding operation actions through the humanoid robot's actuators. The embodiments of this application can integrate simulated cognitive data and sensor data as the basis for operation decision-making, improving the flexibility and reliability of robot operation decision-making. This application can be widely applied in the field of robot control technology.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a method and related equipment for humanoid robot operation decision-making and execution based on embodied cognition enhancement. Background Technology

[0002] Humanoid robots, as a new generation of intelligent equipment, have been gradually applied in industrial manufacturing and home services. In industrial settings, existing robots mostly rely on preset programs to perform fixed tasks (such as assembling single-specification parts). When the working environment changes dynamically (such as part position shifts or tool wear), assembly errors are prone to occur. Assembly achieved solely through visual perception and preset trajectory planning has a low success rate in deep groove ball bearing assembly scenarios, and it cannot adaptively adjust when part position shifts exceed 1mm. In home settings, robots often fail to complete tasks when faced with complex layouts of objects because they cannot understand "object relationships" and "operational force control." Current methods for desktop organization using humanoid robots in home use often have organization strategies that are incompatible with the common operational logic of human workers, resulting in high damage rates and low accuracy when organizing fragile items such as eggs.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a humanoid robot operation decision-making and execution method and related equipment based on embodied cognition enhancement. By fusing acquired operation environment information and generating cognitively related physiological simulation signals corresponding to the environment, the two are input into a pre-trained end-to-end humanoid robot operation model to generate humanoid operation trajectory planning results and drive the execution mechanism to execute, thereby improving the accuracy and stability of operation decision-making and execution.

[0005] To achieve the above objectives, one aspect of this application proposes a humanoid robot task decision-making and execution method based on embodied cognition enhancement, applied to a humanoid robot. The humanoid robot task decision-making and execution method includes:

[0006] The humanoid robot acquires sensor data of the current working environment through its sensor array, and performs data-level fusion and feature-level fusion on the sensor data to obtain the working environment information of the current working environment.

[0007] The work environment information is input into a pre-trained embodied cognition data generation model to obtain simulated embodied cognition data corresponding to the current work environment;

[0008] The work environment information and the simulated embodied cognition data are input into a pre-trained end-to-end humanoid robot work model. The end-to-end humanoid robot work model performs a multi-dimensional query on the work environment information to obtain multi-dimensional query results. Based on the simulated embodied cognition data and the multi-dimensional query results, cross-attention calculation is performed to generate humanoid work trajectory planning results.

[0009] The humanoid work trajectory planning results are converted into control commands, and the humanoid robot's actuators execute the corresponding work actions according to the control commands.

[0010] In some embodiments, the sensor group includes a depth camera, a lidar, a force sensor, and a tactile sensor. The working environment information includes 3D point cloud information and environmental feature vectors. The process of acquiring sensor data of the current working environment through the humanoid robot's sensor group, and performing data-level fusion and feature-level fusion on the sensor data to obtain the working environment information of the current working environment includes:

[0011] The depth camera acquires color images and depth information of the current working environment, and the lidar acquires the original three-dimensional point cloud information of the current working environment.

[0012] The color image, depth information, and original 3D point cloud information are integrated into the environmental 3D point cloud information by Kalman filtering.

[0013] Visual features of the current working environment are extracted based on the color image and the depth information; force sensor features of the current working environment are obtained through a force sensor; and tactile features of the current working environment are obtained through a tactile sensor.

[0014] The visual features, force sensor features, and tactile features are fused into an environmental feature vector using a preset cross-attention algorithm.

[0015] In some embodiments, the embodied cognition data generation model is trained through the following steps:

[0016] Acquire samples of work environment information and real embodied cognition data;

[0017] The work environment information sample is input into the embodied cognition data generation model to obtain the embodied cognition data prediction value;

[0018] The first loss value of the embodied cognition data generation model is calculated based on the predicted value of the embodied cognition data and the real embodied cognition data sample using a preset loss function of the embodied cognition data generation model.

[0019] The parameters of the embodied cognition data generation model are updated based on the first loss value using the backpropagation algorithm;

[0020] The simulated embodied cognitive data and the real embodied cognitive data samples are both cognitive-related physiological simulation signals.

[0021] In some embodiments, the multidimensional query results include object query results, environmental layout query results, and task query results. The process of performing a multidimensional query on the work environment information using the end-to-end humanoid robot work model to obtain multidimensional query results, and then performing cross-attention calculation based on the simulated embodied cognition data and the multidimensional query results to generate humanoid work trajectory planning results, includes:

[0022] The end-to-end humanoid robot operation model is used to perform multi-dimensional queries on the operation environment information to obtain the object query results, the environment layout query results, and the operation task query results.

[0023] The end-to-end humanoid robot operation model performs cross-attention calculation based on the simulated embodied cognition data, the object query results, and the operation task query results to obtain the first interaction result.

[0024] The end-to-end humanoid robot operation model performs cross-attention calculations based on the simulated embodied cognition data, the object query results, and the environmental layout query results to obtain the second interaction result.

[0025] The first interaction result and the second interaction result are fused by the end-to-end humanoid robot operation model to obtain the humanoid operation trajectory planning result.

[0026] In some embodiments, the end-to-end humanoid robot operation model is trained through the following steps:

[0027] Acquire samples of work environment information, real-world embodied cognition data, and end-to-end work decision samples;

[0028] The embodied cognition data generation model generates simulated embodied cognition data samples based on the work environment information samples;

[0029] The work environment information sample and the simulated embodied cognition data sample are input into the end-to-end humanoid robot work model to obtain the work decision prediction value;

[0030] The second loss value of the end-to-end humanoid robot operation model is calculated based on the real embodied cognitive data sample, the end-to-end operation decision sample, the simulated embodied cognitive data sample, and the operation decision prediction value using a preset end-to-end humanoid robot operation model loss function.

[0031] The parameters of the end-to-end humanoid robot operation model are updated based on the second loss value using the backpropagation algorithm.

[0032] In some embodiments, the expression for the preset end-to-end humanoid robot task model loss function is as follows:

[0033]

[0034]

[0035] in, This is the second loss value. The first loss value is generated for the model based on the embodied cognition data. For querying loss of objects, For environmental layout query loss, To imitate learning loss, Loss due to scenario constraints As the first weight, As the second weight, As the third weight, As the fourth weight, As the fifth weight, The signal value of the j-th sensor electrode at the i-th sampling point in the simulated embodied cognition data sample. The signal value of the j-th sensor electrode at the i-th sampling point in the real embodied cognition data sample. This represents the total number of sampling points. This represents the total number of sensor electrodes.

[0036] In some embodiments, the actuator includes at least a robotic arm, a mobile chassis, and an end effector. The step of converting the humanoid work trajectory planning result into control commands, and then having the humanoid robot's actuator execute corresponding work actions according to the control commands, includes:

[0037] Based on the humanoid work trajectory planning results, control commands are generated to drive the motion of the actuator.

[0038] The control commands are sent to the actuator via the EtherCAT protocol;

[0039] The actuator performs the corresponding work actions according to the control instructions.

[0040] To achieve the above objectives, another aspect of this application proposes a humanoid robot task decision-making and execution device based on embodied cognition enhancement, the device comprising:

[0041] The environmental perception module is used to acquire sensor data of the current working environment through the sensor group of the humanoid robot, and to perform data-level fusion and feature-level fusion on the sensor data to obtain the working environment information of the current working environment.

[0042] An embodied cognition data generation module is used to input the work environment information into a pre-trained embodied cognition data generation model to obtain simulated embodied cognition data corresponding to the current work environment.

[0043] The task decision module is used to input the task environment information and the simulated embodied cognition data into a pre-trained end-to-end humanoid robot task model, perform multi-dimensional queries on the task environment information through the end-to-end humanoid robot task model to obtain multi-dimensional query results, and perform cross-attention calculation based on the simulated embodied cognition data and the multi-dimensional query results to generate humanoid task trajectory planning results.

[0044] The task execution module is used to convert the humanoid task trajectory planning results into control commands, and the humanoid robot's actuator executes the corresponding task actions according to the control commands.

[0045] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the methods described above.

[0046] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0047] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0048] The embodiments of this application include at least the following beneficial effects: This application provides a method and related equipment for humanoid robot operation decision-making and execution based on embodied cognition enhancement. This solution effectively improves the accuracy and completeness of environmental perception by performing two-level fusion processing on sensor data of the operation environment, providing a more reliable input basis for subsequent decision-making; by generating simulated embodied cognition data corresponding to the current operation environment based on the operation environment information, and inputting the operation environment information and simulated embodied cognition data together into the end-to-end humanoid robot operation model to complete the operation decision, the robot further introduces embodied cognition data that imitates the tendencies of human operators as auxiliary cues based on its understanding of the operation environment, so that the operation model can better take into account factors such as object, environment and task, and improve the rationality, robustness and human-likeness of trajectory planning; by converting the humanoid operation trajectory planning results into control commands and executing them by the actuator, the planning results can be implemented into actual actions in a controllable manner, improving the stability of operation action execution. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the steps of a humanoid robot task decision-making and execution method based on embodied cognition enhancement provided in an embodiment of this application.

[0050] Figure 2 This is a schematic diagram of the structure of a humanoid robot operation decision-making and execution device based on embodied cognition enhancement provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0054] The concept of the present invention will now be explained in conjunction with the background art.

[0055] Humanoid robots, as a new generation of intelligent equipment, have been gradually applied in industrial manufacturing and home services. In industrial settings, existing robots mostly rely on preset programs to perform fixed tasks (such as assembling single-specification parts). When the working environment changes dynamically (such as part position shifts or tool wear), assembly errors are prone to occur. Assembly achieved solely through visual perception and preset trajectory planning has a low success rate in deep groove ball bearing assembly scenarios, and it cannot adaptively adjust when part position shifts exceed 1mm. In home settings, robots often fail to complete tasks when faced with complex layouts of objects because they cannot understand the "relationships between objects" and "control the force of manipulation." Current methods for desktop organization using humanoid robots in home use do not integrate human electromyography / electroencephalography signals, resulting in high damage rates and low accuracy when organizing fragile items such as eggs.

[0056] While end-to-end robotic operation technology attempts to integrate the "perception-decision-execution" process, it suffers from two major flaws: First, it fails to incorporate the cognitive mechanisms of human operators—during human operation, prefrontal cortex EEG reflects attention allocation, and hand EMG reflects force control; these physiological signals are crucial for flexible decision-making. Second, the sensor fusion layer is too simple, and communication latency is not well controlled, resulting in insufficient environmental perception accuracy (3D point cloud error ≥ ±1mm) and trajectory execution deviation (robotic arm motion error ≥ ±0.05mm). These flaws lead to low intelligence and low success rate in current robotic operations, making it difficult to meet the needs of complex scenarios.

[0057] To address the shortcomings of existing humanoid robot technologies, such as low intelligence, low success rate, and insufficient trajectory accuracy, this application provides a humanoid robot operation decision-making and execution method and related equipment based on embodied cognition enhancement. The method overcomes existing bottlenecks through the following technological innovations: First, it introduces "simulated embodied cognitive data" to reproduce the correlation mechanism between human operator cognition and action; second, it designs a two-level sensor fusion system ("data-level + feature-level") to improve environmental perception accuracy; and third, it manages the "decision-execution" communication delay to ensure accurate trajectory landing. These three elements work together to make the humanoid robot suitable for industrial scenarios (such as precision parts assembly and equipment bolt maintenance, with operation accuracy requirements of ±0.02mm~±0.1mm) and household scenarios (such as fragile item organization and clothing folding, with operation force control accuracy ≤±1N). By integrating simulated human cognitive data with two-level sensor fusion technology, the flexibility of robot operation decision-making and the reliability of execution are improved.

[0058] This application provides a method for humanoid robot task decision-making and execution based on embodied cognition enhancement, relating to the field of information technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the humanoid robot task decision-making and execution method based on embodied cognition enhancement, but is not limited to the above forms.

[0059] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0060] Figure 1 This is an optional flowchart of a humanoid robot task decision-making and execution method based on embodied cognition enhancement provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S104.

[0061] S101. Obtain sensor data of the current working environment through the sensor group of the humanoid robot, and perform data-level fusion and feature-level fusion on the sensor data to obtain the working environment information of the current working environment.

[0062] In some embodiments, the sensor array includes a depth camera, a lidar, a force sensor, and a tactile sensor. The working environment information includes 3D point cloud information and environmental feature vectors. Sensor data of the current working environment is acquired through the humanoid robot's sensor array, and data-level fusion and feature-level fusion are performed on the sensor data to obtain the current working environment information, including:

[0063] S1011. Obtain color images and depth information of the current working environment through a depth camera, and obtain the original three-dimensional point cloud information of the current working environment through a lidar.

[0064] S1012. The color image, depth information and original 3D point cloud information are integrated into environmental 3D point cloud information by Kalman filtering;

[0065] S1013. Extract the visual features of the current working environment based on the color image and depth information, obtain the force sensor features of the current working environment through the force sensor, and obtain the tactile features of the current working environment through the tactile sensor.

[0066] S1014. Visual features, force sensor features, and tactile features are fused into an environmental feature vector using a preset cross-attention algorithm.

[0067] Specifically, in this embodiment, a sensor array performs multi-source perception and two-level fusion of the current working environment to form the working environment information required for subsequent decision-making. First, the sensor array performs complementary acquisition of the same working environment. A depth camera is used to acquire color images containing texture and appearance information, as well as depth information representing spatial distance relationships, while a lidar is used to acquire raw 3D point cloud information. Due to differences in measurement noise, sampling frequency, and coordinate systems among different sensors, this embodiment further performs data-level fusion of the above multi-source data: for example, Kalman filtering is used to estimate and integrate the color images, depth information, and raw 3D point cloud information, unifying them into the same spatial representation, thereby obtaining environmental 3D point cloud information that more stably reflects the spatial structure of the environment. Based on this, in order to simultaneously characterize visual appearance, spatial structure features, and physical state features during contact interaction, this embodiment also performs feature-level fusion of the multi-source information: on the one hand, visual features are extracted from the color images and depth information; on the other hand, force sensor features reflecting external force are acquired by a force sensor, and tactile features reflecting contact distribution, pressure, friction, etc., are acquired by a tactile sensor. Subsequently, a pre-defined cross-attention algorithm is used to establish correlations between different modal features, enabling visual features to align and complement force sensor features and tactile features, fusing them into a more compact environmental feature vector. Therefore, the final operational environment information generated in this embodiment may include: 3D point cloud information describing the spatial geometry, and environmental feature vectors describing the multimodal interaction states.

[0068] For example, Kalman filtering (filtering frequency ≥ 100 Hz) can be used to integrate the raw data from an RGB-D camera (depth accuracy ≤ ±2%) and a LiDAR (point cloud density ≥ 100 points / cm²) to output a 3D point cloud of the environment (accuracy ≤ ± 0.5 mm), solving the problems of "visual occlusion" and "laser noise" of a single sensor. An 8-head cross-attention algorithm can be used to fuse visual features (object edges, textures), force sensor features (assembly force, gripping force, accuracy ≤ ± 0.1 N), and tactile features (material hardness of objects) to generate a unified environmental feature vector with a dimension of 1024, ensuring the comprehensiveness and consistency of environmental information.

[0069] It can be recognized that this embodiment can improve the perception accuracy of environmental geometry by generating 3D point cloud information of the environment through data-level fusion; and can integrate visual, force and tactile information by generating environmental feature vectors through feature-level fusion, thereby more comprehensively representing the characteristics of the working environment and providing more reliable environmental input for subsequent work decisions.

[0070] S102. Input the work environment information into the pre-trained embodied cognition data generation model to obtain simulated embodied cognition data corresponding to the current work environment;

[0071] In some embodiments, the embodied cognition data generation model is trained through the following steps:

[0072] S201. Obtain samples of work environment information and samples of real embodied cognition data;

[0073] S202. Input the work environment information sample into the embodied cognition data generation model to obtain the embodied cognition data prediction value;

[0074] S203. Calculate the first loss value of the embodied cognition data generation model based on the predicted value of embodied cognition data and the real embodied cognition data sample using the preset loss function of the embodied cognition data generation model.

[0075] S204. Update the parameters of the embodied cognition data generation model based on the first loss value using the backpropagation algorithm;

[0076] Among them, both the simulated embodied cognition data and the real embodied cognition data samples are cognitive-related physiological simulation signals.

[0077] Specifically, in this embodiment, an embodied cognition data generation model is trained and used to generate simulated embodied cognitive data that matches the current work environment based on the current work environment information. During the inference phase of the applied model, this embodiment first inputs the work environment information into the pre-trained embodied cognitive data generation model, which then outputs simulated embodied cognitive data corresponding to the current work environment. This simulated embodied cognitive data consists of cognitive-related physiological simulation signals, such as electromyography (EMG) and electroencephalography (EEG) signals. These signals characterize the cognitive and physiological response characteristics that an individual may exhibit in a specific environment and task context, thereby providing auxiliary information that more closely aligns with the embodied cognitive mechanism for subsequent work decisions.

[0078] To obtain the aforementioned embodied cognition data generation model, this embodiment constructs sample pairs during the training phase, acquiring work environment information samples and their corresponding real embodied cognition data samples. Then, the work environment information samples are input into the embodied cognition data generation model to be trained, obtaining embodied cognition data prediction values. Further, a preset loss function is used to measure the difference between the embodied cognition data prediction values ​​and the real embodied cognition data samples, calculating a first loss value. Based on the first loss value, the backpropagation algorithm is used to update the model parameters, enabling the model to gradually learn the mapping relationship between environmental information and cognition-related physiological signals. It should be noted that the real embodied cognition data samples are also cognition-related physiological simulation signals, which can originate from the collection of real work processes or be obtained from existing datasets, serving as a supervised target for model training.

[0079] For example, the real embodied cognitive data samples are real physiological signals related to cognition collected by human operators in the corresponding work scenarios through sensors at designated locations, including but not limited to: hand / arm electromyography (EMG, sampling rate ≥800Hz), scalp prefrontal cortex electroencephalography (EEG, 32 channels or more, sampling rate ≥250Hz), eye movement signals (sampling rate ≥120Hz), and wrist heart rate signals (sampling rate ≥60Hz); and the EMG signal collection site must correspond to the movement joint of the robot execution module, and the EEG signal must cover the brain regions related to cognitive decision-making.

[0080] For example, the structure of the embodied cognition data generation model may include a visual encoding layer (ResNet50, ImageNet pre-trained weights), a projection layer (fully connected + ReLU), a spatiotemporal convolutional network (1 convolutional module + 3 residual modules), and an output layer (fully connected + Sigmoid). The training process may include data preprocessing, performing linear interpolation to align the work environment video (sampling rate 25~30fps) with human physiological signals (EEG 250Hz, EMG 800~1000Hz), with a timestamp deviation ≤ ±1ms; feature extraction, where ResNet50 encodes video frames to obtain a 2048-dimensional initial visual embedding vector, and the projection layer maps it to a 512-dimensional worker visual embedding vector; deep learning, where the spatiotemporal convolutional network extracts the "visual-cognitive" spatiotemporal correlation features and outputs the worker's cognitive embedding vector; and loss optimization, where the model parameters are optimized based on the loss value calculated by the loss function using backpropagation algorithms such as gradient descent until the error between the generated physiological simulation signal and the real signal is ≤5%.

[0081] It can be recognized that this embodiment, by learning the mapping relationship between work environment information and cognitively related physiological signals, can generate simulated embodied cognitive data that matches the environment in real time. This supplements embodied cognitive cues with a lower system burden, improving the richness and consistency of environmental understanding and subsequent decision inputs. During model training, after acquiring work environment information samples and real embodied cognitive data samples, the work environment information samples can be preprocessed and segmented, such as denoising and normalization, to obtain work environment training data. The sampling rate of the work environment information samples and the sampling rate of the real embodied cognitive data samples are segmented and aligned using a linear interpolation algorithm (interpolation step size = 1 / video sampling rate) to ensure that the timestamp deviation meets the accuracy requirements.

[0082] S103. Input the work environment information and simulated embodied cognition data into the pre-trained end-to-end humanoid robot work model. Perform multi-dimensional query on the work environment information through the end-to-end humanoid robot work model to obtain multi-dimensional query results. Perform cross-attention calculation based on the simulated embodied cognition data and multi-dimensional query results to generate humanoid work trajectory planning results.

[0083] In some embodiments, the multidimensional query results include object query results, environmental layout query results, and task query results. A multidimensional query is performed on the work environment information using an end-to-end humanoid robot work model to obtain the multidimensional query results. Cross-attention calculation is then performed based on simulated embodied cognition data and the multidimensional query results to generate humanoid work trajectory planning results, including:

[0084] S1031. Perform multi-dimensional queries on the work environment information through the end-to-end humanoid robot operation model to obtain object query results, environment layout query results, and work task query results.

[0085] S1032. The first interaction result is obtained by performing cross-attention calculation based on simulated embodied cognition data, object query results and task query results through the end-to-end humanoid robot operation model.

[0086] S1033. The second interaction result is obtained by performing cross-attention calculation based on simulated embodied cognition data, object query results and environmental layout query results through the end-to-end humanoid robot operation model.

[0087] S1034. The first and second interaction results are fused using an end-to-end humanoid robot operation model to obtain the humanoid operation trajectory planning result.

[0088] In some embodiments, an end-to-end humanoid robot operation model is trained through the following steps:

[0089] S301. Obtain samples of work environment information, samples of real embodied cognitive data, and samples of end-to-end work decisions;

[0090] S302. Generate simulated embodied cognitive data samples based on work environment information samples using an embodied cognitive data generation model;

[0091] S303. Input the work environment information sample and the simulated embodied cognition data sample into the end-to-end humanoid robot work model to obtain the work decision prediction value;

[0092] S304. Calculate the second loss value of the end-to-end humanoid robot operation model based on the real embodied cognitive data samples, end-to-end operation decision samples, simulated embodied cognitive data samples, and operation decision prediction values ​​using the preset end-to-end humanoid robot operation model loss function.

[0093] S305. Update the parameters of the end-to-end humanoid robot operation model based on the second loss value using the backpropagation algorithm.

[0094] In some embodiments, the expression for the preset end-to-end humanoid robot task model loss function is as follows:

[0095]

[0096]

[0097] in, This is the second loss value. The first loss value for the embodied cognition data generation model. For querying loss of objects, For environmental layout query loss, To imitate learning loss, Loss due to scenario constraints As the first weight, As the second weight, As the third weight, As the fourth weight, As the fifth weight, To simulate the signal value of the j-th sensor electrode at the i-th sampling point in the embodied cognition data sample, The signal value of the j-th sensor electrode at the i-th sampling point in the real embodied cognition data sample. This represents the total number of sampling points. This represents the total number of sensor electrodes.

[0098] Specifically, in this embodiment, an executable humanoid task trajectory planning result is generated by combining environmental information and cognitive cues as joint inputs. First, in the inference phase of the application model, this embodiment inputs the task environment information obtained in the aforementioned steps and the simulated embodied cognitive data into a pre-trained end-to-end humanoid robot task model. The end-to-end humanoid robot task model performs multi-dimensional queries on the task environment information to extract key information required for different decision dimensions from the same environmental representation. The multi-dimensional query results include at least: object query results representing the attributes and location of the target object; environmental layout query results representing the spatial structure and accessibility; and task query results representing the current task intent and step constraints. Subsequently, the end-to-end humanoid robot operation model incorporates simulated embodied cognitive data as conditional information for cognitively related physiological simulation signals, performing cross-attention calculations with different query results: First, cross-attention calculations are performed based on simulated embodied cognitive data, object query results, and task query results to obtain a first interaction result, highlighting key elements related to "task objective - object operation"; second, cross-attention calculations are performed based on simulated embodied cognitive data, object query results, and environmental layout query results to obtain a second interaction result; finally, the first and second interaction results are fused to output a humanoid operation trajectory planning result. Thus, the model can simultaneously consider the target object, task intent, and spatial layout in the same environment, and form a trajectory planning more consistent with human operation habits under the guidance of cognitive cues.

[0099] For example, object queries can obtain information such as the position (accuracy ≤ ±1mm), shape (including dimensional tolerance ≤ ±0.1mm, surface roughness Ra ≤ 1.6μm, and interface type in industrial scenarios; and material hardness (Shore hardness ≤ 50D) and fragility (whether it is a brittle material) of the target object through an RGB-D camera (depth accuracy ≤ ±2%) and a material sensor; environmental layout queries can obtain information such as the three-dimensional coordinates (accuracy ≤ ±0.5mm) of fixed structures (such as the edge of an industrial assembly table or the edge of a household desktop) and obstacles (dynamic obstacle movement speed ≤ 0.5m / s) within the work space through LiDAR (point cloud density ≥ 100 points / cm²) and visual SLAM; and task queries can be performed through voice commands (recognition accuracy ≤ ±1mm). Input the task requirements (≥95%) or text commands, such as "bearing assembly (coaxiality ≤0.02mm)" in industrial scenarios and "egg storage (gripping force ≤5N)" in household scenarios; then, perform cross-attention calculation based on simulated embodied cognition data, object query results, and task query results (the number of attention heads can be set to 8) to obtain the "robot-object-task" interaction result (i.e., the first interaction result). Perform cross-attention calculation based on simulated embodied cognition data, object query results, and environmental layout query results to obtain the "robot-object-environment" interaction result (i.e., the first interaction result). Based on the "robot-object-task" interaction result and the "robot-object-environment" interaction result, generate a cognitively enhanced humanoid task trajectory plan.

[0100] During the model training phase, to enable the end-to-end humanoid robot operation model to learn the mapping relationship from joint input to operation decision output, this embodiment first acquires operation environment information samples, real embodied cognitive data samples, and end-to-end operation decision samples. The real embodied cognitive data samples are cognitive-related physiological simulation signals; the end-to-end operation decision samples can correspond to the expected operation trajectory, action sequence, or other supervised targets. Subsequently, an embodied cognitive data generation model is used to generate simulated embodied cognitive data samples based on the operation environment information samples. These samples are then input into the end-to-end humanoid robot operation model to obtain the operation decision prediction value. Further, a second loss value is calculated using a preset end-to-end humanoid robot operation model loss function, combining the real embodied cognitive data samples, the end-to-end operation decision samples, the simulated embodied cognitive data samples, and the operation decision prediction value. Based on this second loss value, the backpropagation algorithm is used to update the parameters of the end-to-end humanoid robot operation model.

[0101] It can be recognized that this embodiment, by performing multi-dimensional queries on the work environment information and combining simulated embodied cognitive data for cross-attention calculation and fusion of interaction results, can simultaneously consider key factors such as objects, tasks, and spatial layout during the decision-making stage, and introduce cognitive cues to participate in the work decision, thereby improving the rationality of trajectory planning; by calculating the second loss value through a loss function that includes multiple losses such as object query, layout query, imitation learning, scene constraints, and cognitive consistency, joint training can be completed, which can improve the decision-making accuracy and stability of the end-to-end work model in complex scenarios.

[0102] S104. Convert the humanoid operation trajectory planning results into control commands, and execute the corresponding operation actions through the humanoid robot's actuator according to the control commands.

[0103] In some embodiments, the actuator includes at least a robotic arm, a mobile chassis, and an end effector, which converts the humanoid work trajectory planning results into control commands, and the humanoid robot's actuator executes the corresponding work actions according to the control commands, including:

[0104] S1041. Generate control commands for driving the motion of the actuator based on the humanoid work trajectory planning results;

[0105] S1042. Control commands are sent to the actuators via the EtherCAT protocol;

[0106] S1043. The actuator performs the corresponding work actions according to the control instructions.

[0107] Specifically, in this embodiment, the humanoid task trajectory planning results are translated into executable low-level control commands, which then drive the humanoid robot's actuators to complete actual tasks, thereby achieving a closed-loop implementation of "planning-control-execution". First, control commands for driving the actuators are generated based on the humanoid task trajectory planning results. The humanoid task trajectory planning results may include planning information such as target pose sequence, joint trajectories, end-effector motion trajectories, velocity and acceleration constraints, etc. Correspondingly, the control commands may include joint position, velocity, torque commands, end-effector pose control quantities, or other command forms that can be directly used to control the actuators. The actuators include at least a robotic arm, a mobile chassis, and an end-effector. The control commands can be used to drive the robotic arm to perform upper limb actions such as grasping and carrying, drive the mobile chassis to perform movement and alignment, and drive the end-effector to perform end-effector operations such as clamping, twisting, and pressing, to adapt to different task types.

[0108] Subsequently, in this embodiment, control commands are sent to the actuators via the EtherCAT protocol. EtherCAT, as an industrial Ethernet real-time communication protocol, can be used for periodic, low-latency data exchange between the controller and each actuator node, thereby ensuring the real-time issuance and synchronous execution of control commands. Finally, the actuators execute the corresponding operational actions according to the received control commands, realizing the planning results at the physical level.

[0109] It can be recognized that by converting the humanoid operation trajectory planning results into control commands that can directly drive the robotic arm, mobile chassis and end effector, and using EtherCAT for low-latency transmission, this embodiment can improve the real-time performance and consistency from planning to execution, reduce trajectory landing deviation, and thus improve the controllability and execution stability of the humanoid robot's operation.

[0110] The present invention will be further described below with reference to a specific embodiment.

[0111] In this embodiment, taking the industrial operation scenario of automobile bearing assembly as an example, the sample construction and training process of the embodied cognition data generation model is described as follows:

[0112] In the data preparation phase, the first step is to construct a sample of operational environment information. For example, 1000 sets of assembly scene videos of 6205 deep groove ball bearings (inner diameter 25mm, outer diameter 52mm) are collected, with a video sampling rate of... During data collection, variations in conditions such as part position offset (e.g., 0.5–2 mm) and tool wear (e.g., different surface roughness values) can be introduced. This ensures that the environmental samples include both standard assembly scenarios and scenarios with a certain degree of deviation, thereby improving the adaptability of the subsequent model to real-world working conditions. Simultaneously, realistic embodied cognitive data samples are constructed. For example, 20 skilled assemblers are recruited, and during their performance of the aforementioned bearing assembly operations, they wear a 16-electrode hand electromyography (EMG) wearable sensor with an EMG sampling rate of [missing information]. They also wore a 32-channel head-mounted electroencephalogram (EEG) device with an EEG sampling rate of [missing information]. Physiological signals during the assembly process are recorded synchronously. To reduce the impact of noise on supervised training, the collected physiological signals can be preprocessed, such as removing artifacts related to eye movements and heartbeats. Independent component analysis (ICA) can be used, for example, to separate and remove artifacts, thereby obtaining higher quality authentic embodied cognition data samples.

[0113] During the model training phase, because and Not divisible ( First, the temporal alignment between environmental and physiological samples needs to be addressed. The following formula calculates the number of EMG samples corresponding to each frame of video:

[0114]

[0115] in, The number of EMG samples corresponding to each frame of video. The sampling rate for the EMG signal is 1000Hz. This is the video sampling rate (30fps). This is to round down. The calculation result can then be obtained. Linear interpolation (step size = 1 / 30s) can be used for alignment, so that the video frame sequence and the EMG sequence correspond on the same time axis.

[0116] Regarding network structure and training strategies, for example, a "visual encoding—feature mapping—spatiotemporal feature extraction" approach can be used to learn the mapping relationship from the work environment to cognitively relevant physiological signals: For instance, ResNet50 is used to visually encode assembly video frames, resulting in a 2048-dimensional initial visual embedding; this vector is then mapped to a 512-dimensional work visual embedding through a projection layer; subsequently, a spatiotemporal convolutional network is used to model the sequence features, with a 3×3 convolutional kernel, and residual modules can be set to extract features step by step. For example, the number of output channels can be progressively increased from 128, 256, to 512 to obtain cognitive feature expressions corresponding to changes in assembly actions. During training, embodied cognitive data is used to generate the model loss term. Optimization is performed to make the EMG prediction signal output by the model as consistent as possible with the real EMG signal. In this example setting, the error between the EMG signal generated after training convergence and the real signal can be controlled at about 4.2%, thus obtaining an embodied cognitive data generation model that can be used to generate simulated embodied cognitive data in the subsequent inference stage.

[0117] To verify the application effect of this embodiment in industrial assembly scenarios, assembly task tests can be conducted under preset test conditions. For example, the test environment temperature is 20±2℃, humidity is 40%~60%, and the environment is dust-free; the test equipment is a 16-axis humanoid robot with a repeatability accuracy of ±0.03mm. Under these test conditions, a sample size of 100 assembly tasks is set, of which 50 tasks have part offset (e.g., 0.5~2mm), and the other 50 tasks have no offset. In one execution result example based on the above test settings, the assembly success rate can reach approximately 95%, the assembly time is reduced by approximately 15%, and the coaxiality deviation can be controlled within ≤0.02mm.

[0118] The present invention will be further described below with reference to another specific embodiment.

[0119] In this embodiment, taking a desktop organization task in a home setting as an example, the data preparation, model training settings, and execution results are explained as follows.

[0120] In the data preparation phase, the first step is to construct a sample of the work environment information. For example, 800 sets of video data on cluttered desktops can be collected. Desktop items may include typical everyday objects such as two glasses, three books, two pens, and three eggs. The video sampling rate is [missing information]. This study aims to cover tidying scenarios with different object materials, sizes, and placements. Simultaneously, it constructs a sample of realistic embodied cognitive data. For example, 15 household users were recruited and, during their desktop tidying process, wore an 8-electrode hand electromyography (EMG) sensor (sampling rate...). ), and wear an eye-tracking device (sampling rate) The system simultaneously records physiological signals during the sorting process to document the cognitive and physiological response characteristics of human workers in sorting behaviors such as prioritizing fragile items, allocating attention, and controlling gripping force.

[0121] Regarding model training and execution settings, due to and Since the data is not divisible, it is first necessary to align the data sequences with different sampling rates. The calculation process is as follows: Linear interpolation is used to achieve frame-by-frame alignment, ensuring that the video frame sequence corresponds to the EMG sequence on a unified timeline. Subsequently, weights can be assigned to each loss term in the loss function of the end-to-end humanoid robot task model to balance different training objectives. For example, the weight values ​​can be set as follows:

[0122]

[0123] Furthermore, priority is given to ensuring the weight settings of constraint-related loss terms, so that the model is more inclined to meet the scenario constraint requirement of "avoiding damage" when generating trajectory planning and operation actions.

[0124] To facilitate performance evaluation, this embodiment provides an example definition of desktop organization accuracy: Organization Accuracy = (Number of undamaged and correctly placed items / Total number of items) × 100%. "Correctly placed" means that items are placed in the preset area with a positional deviation of no more than 5 mm; "undamaged" means that items have no cracks, deformation, or other damage. In an example of the execution results under the above settings, the desktop organization accuracy can reach approximately 90%, the item damage rate is 0, and the organization time is reduced by approximately 20% compared to the baseline. From a behavioral perspective, the organization process reflects organization habits consistent with human experience, such as prioritizing fragile items and handling heavier items first, thus employing more stable organization strategies.

[0125] Please see Figure 2 This application also provides a humanoid robot operation decision-making and execution device based on embodied cognition enhancement, which can implement the above-mentioned method. The device includes:

[0126] The environmental perception module is used to acquire sensor data of the current working environment through the sensor group of the humanoid robot, and to perform data-level fusion and feature-level fusion on the sensor data to obtain the working environment information of the current working environment.

[0127] The embodied cognition data generation module is used to input work environment information into a pre-trained embodied cognition data generation model to obtain simulated embodied cognition data corresponding to the current work environment.

[0128] The task decision module is used to input task environment information and simulated embodied cognition data into a pre-trained end-to-end humanoid robot task model. The end-to-end humanoid robot task model performs multi-dimensional queries on the task environment information to obtain multi-dimensional query results. Based on the simulated embodied cognition data and multi-dimensional query results, it performs cross-attention calculation to generate humanoid task trajectory planning results.

[0129] The task execution module is used to convert the humanoid task trajectory planning results into control commands, and then execute the corresponding task actions through the humanoid robot's actuator according to the control commands.

[0130] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0131] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0132] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0134] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0135] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application.

[0136] The input / output interface 903 is used to implement information input and output;

[0137] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0138] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0139] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0140] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0141] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0142] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0143] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0145] This application provides a method and related equipment for humanoid robot operation decision-making and execution based on embodied cognition enhancement. The method effectively improves the accuracy and completeness of environmental perception by performing two-level fusion processing on sensor data of the working environment, providing a more reliable input basis for subsequent decision-making. It generates simulated embodied cognitive data corresponding to the current working environment based on the working environment information, and inputs both the working environment information and the simulated embodied cognitive data into an end-to-end humanoid robot operation model to complete the operation decision. This allows the robot to further incorporate embodied cognitive data that mimics human operator tendencies as auxiliary cues, based on its understanding of the working environment. This enables the operation model to better consider factors such as the object, environment, and task, improving the rationality, robustness, and human-likeness of trajectory planning. By converting the humanoid operation trajectory planning results into control commands and executing them through an actuator, the planning results can be implemented as actual actions in a controllable manner, improving the stability of operation execution.

[0146] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0147] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0150] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0151] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0153] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for decision-making and execution of humanoid robot tasks based on embodied cognition enhancement, characterized in that, Applied to humanoid robots, the humanoid robot task decision-making and execution method includes: The humanoid robot acquires sensor data of the current working environment through its sensor array, and performs data-level fusion and feature-level fusion on the sensor data to obtain the working environment information of the current working environment. The work environment information is input into a pre-trained embodied cognition data generation model to obtain simulated embodied cognition data corresponding to the current work environment; The work environment information and the simulated embodied cognition data are input into a pre-trained end-to-end humanoid robot work model. The end-to-end humanoid robot work model performs a multi-dimensional query on the work environment information to obtain multi-dimensional query results. Based on the simulated embodied cognition data and the multi-dimensional query results, cross-attention calculation is performed to generate humanoid work trajectory planning results. The humanoid work trajectory planning results are converted into control commands, and the humanoid robot's actuators execute the corresponding work actions according to the control commands.

2. The method for humanoid robot task decision-making and execution based on embodied cognition enhancement according to claim 1, characterized in that, The sensor group includes a depth camera, LiDAR, force sensor, and tactile sensor. The working environment information includes 3D point cloud information and environmental feature vectors. The humanoid robot acquires sensor data of the current working environment through its sensor group, and performs data-level fusion and feature-level fusion on the sensor data to obtain the working environment information of the current working environment, including: The depth camera acquires color images and depth information of the current working environment, and the lidar acquires the original three-dimensional point cloud information of the current working environment. The color image, depth information, and original 3D point cloud information are integrated into the environmental 3D point cloud information by Kalman filtering. Visual features of the current working environment are extracted based on the color image and the depth information; force sensor features of the current working environment are obtained through a force sensor; and tactile features of the current working environment are obtained through a tactile sensor. The visual features, force sensor features, and tactile features are fused into an environmental feature vector using a preset cross-attention algorithm.

3. The method for humanoid robot task decision-making and execution based on embodied cognition enhancement according to claim 1, characterized in that, The embodied cognition data generation model is trained through the following steps: Acquire samples of work environment information and real embodied cognition data; The work environment information sample is input into the embodied cognition data generation model to obtain the embodied cognition data prediction value; The first loss value of the embodied cognition data generation model is calculated based on the predicted value of the embodied cognition data and the real embodied cognition data sample using a preset loss function of the embodied cognition data generation model. The parameters of the embodied cognition data generation model are updated based on the first loss value using the backpropagation algorithm; The simulated embodied cognitive data and the real embodied cognitive data samples are both cognitive-related physiological simulation signals.

4. The method for humanoid robot task decision-making and execution based on embodied cognition enhancement according to claim 1, characterized in that, The multidimensional query results include object query results, environmental layout query results, and task query results. The process involves performing a multidimensional query on the work environment information using the end-to-end humanoid robot work model to obtain multidimensional query results. Based on the simulated embodied cognitive data and the multidimensional query results, cross-attention calculation is performed to generate humanoid work trajectory planning results, including: The end-to-end humanoid robot operation model is used to perform multi-dimensional queries on the operation environment information to obtain the object query results, the environment layout query results, and the operation task query results. The end-to-end humanoid robot operation model performs cross-attention calculation based on the simulated embodied cognition data, the object query results, and the operation task query results to obtain the first interaction result. The end-to-end humanoid robot operation model performs cross-attention calculations based on the simulated embodied cognition data, the object query results, and the environmental layout query results to obtain the second interaction result. The first interaction result and the second interaction result are fused by the end-to-end humanoid robot operation model to obtain the humanoid operation trajectory planning result.

5. The method for decision-making and execution of humanoid robot tasks based on embodied cognition enhancement according to claim 1, characterized in that, The end-to-end humanoid robot operation model is trained using the following steps: Acquire samples of work environment information, real-world embodied cognition data, and end-to-end work decision samples; The embodied cognition data generation model generates simulated embodied cognition data samples based on the work environment information samples; The work environment information sample and the simulated embodied cognition data sample are input into the end-to-end humanoid robot work model to obtain the work decision prediction value; The second loss value of the end-to-end humanoid robot operation model is calculated based on the real embodied cognitive data sample, the end-to-end operation decision sample, the simulated embodied cognitive data sample, and the operation decision prediction value using a preset end-to-end humanoid robot operation model loss function. The parameters of the end-to-end humanoid robot operation model are updated based on the second loss value using the backpropagation algorithm.

6. The method for humanoid robot task decision-making and execution based on embodied cognition enhancement according to claim 5, characterized in that, The expression for the preset end-to-end humanoid robot operation model loss function is as follows: in, This is the second loss value. The first loss value is generated for the model based on the embodied cognition data. For querying loss of objects, For environmental layout query loss, To imitate learning loss, Loss due to scenario constraints As the first weight, As the second weight, As the third weight, As the fourth weight, As the fifth weight, The signal value of the j-th sensor electrode at the i-th sampling point in the simulated embodied cognition data sample. The signal value of the j-th sensor electrode at the i-th sampling point in the real embodied cognition data sample. This represents the total number of sampling points. This represents the total number of sensor electrodes.

7. The method for humanoid robot task decision-making and execution based on embodied cognition enhancement according to claim 1, characterized in that, The actuator includes at least a robotic arm, a mobile chassis, and an end effector. The process of converting the humanoid work trajectory planning result into control commands, and then having the humanoid robot's actuator execute corresponding work actions according to the control commands, includes: Based on the humanoid work trajectory planning results, control commands are generated to drive the motion of the actuator. The control commands are sent to the actuator via the EtherCAT protocol; The actuator performs the corresponding work actions according to the control instructions.

8. A humanoid robot task decision-making and execution device based on embodied cognition enhancement, characterized in that, The device includes: The environmental perception module is used to acquire sensor data of the current working environment through the sensor group of the humanoid robot, and to perform data-level fusion and feature-level fusion on the sensor data to obtain the working environment information of the current working environment. An embodied cognition data generation module is used to input the work environment information into a pre-trained embodied cognition data generation model to obtain simulated embodied cognition data corresponding to the current work environment. The task decision module is used to input the task environment information and the simulated embodied cognition data into a pre-trained end-to-end humanoid robot task model, perform multi-dimensional queries on the task environment information through the end-to-end humanoid robot task model to obtain multi-dimensional query results, and perform cross-attention calculation based on the simulated embodied cognition data and the multi-dimensional query results to generate humanoid task trajectory planning results. The task execution module is used to convert the humanoid task trajectory planning results into control commands, and the humanoid robot's actuator executes the corresponding task actions according to the control commands.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images