A humanoid robot control method, system and related devices

Abstract

The embodiment of the application provides a humanoid robot control method, system and related equipment, and belongs to the technical field of robots. The method comprises the following steps: in response to a first electroencephalogram signal of a target user, collecting real-time visual data, real-time tactile data and real-time motion data of a humanoid robot; performing data fusion on the first electroencephalogram signal, the real-time visual data, the real-time tactile data and the real-time motion data to obtain real-time fusion data; inputting the real-time fusion data into a trained end-to-end model to output a motion instruction, so as to control the humanoid robot to execute a motion according to the motion instruction. The embodiment of the application can improve the accuracy and real-time performance of humanoid robot control.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a humanoid robot control method, system and related equipment. Background Technology

[0002] A robot is an intelligent machine that can work semi-autonomously or fully autonomously. It can perform tasks such as operations or movement through programming and automatic control, and is widely used in industries, medical fields, services, and scientific research.

[0003] In related technologies, robot control architecture suffers from modular fragmentation, requiring the manual design of multiple independent modules such as sensory feature extraction, intent understanding, task planning, and motion control. For example, brain signals are first parsed to identify intent, and then a classifier is used to identify the user's motion intent. Subsequently, this intent needs to be mapped into action instructions that can be executed by the robot's underlying controller through pre-set rules or lookup tables. This serial and multi-stage processing mechanism easily leads to system response delays, lengthy processes, and low efficiency, severely restricting the overall performance of humanoid robot control systems.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main objective of this application is to propose a humanoid robot control method, system, and related equipment that can improve the accuracy and real-time performance of humanoid robot control.

[0006] To achieve the above objectives, one aspect of this application proposes a humanoid robot control method, the method comprising:

[0007] In response to the target user's first EEG signal, the robot collects real-time visual data, real-time tactile data, and real-time motion data.

[0008] The first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data are fused to obtain real-time fused data;

[0009] The real-time fused data is input into the trained end-to-end model, which outputs action commands to control the humanoid robot to perform actions according to the action commands.

[0010] In some embodiments, the step of fusing the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data to obtain real-time fused data includes:

[0011] The first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data are subjected to nonlinear processing to obtain corresponding EEG features, visual features, tactile features, and motion features;

[0012] The real-time fusion data is obtained by multiplying the EEG features, visual features, tactile features, and motion features by their respective attention weights and then summing them.

[0013] In some embodiments, the attention weights are obtained through the following steps:

[0014] Obtain task-related vectors;

[0015] Based on the task-related vector, the similarity of the first EEG signal, the real-time visual data, the real-time tactile data and the real-time motion data are calculated respectively to obtain their respective similarity.

[0016] Weights are calculated based on the similarity to obtain the corresponding attention weights.

[0017] In some embodiments, the trained end-to-end model is obtained through the following steps:

[0018] Obtain a training dataset, wherein the training dataset includes a combination of several multimodal fusion data and target action sequences;

[0019] The multimodal fusion data is input into the end-to-end model to obtain the predicted action sequence;

[0020] Calculate the total loss value based on the target action sequence and the predicted action sequence;

[0021] With the goal of minimizing the total loss value, the end-to-end model parameters are adjusted to obtain a trained end-to-end model.

[0022] In some embodiments, the multimodal fusion data is obtained through the following steps:

[0023] Acquire preprocessed second EEG signals, visual training data, tactile training data, and motor training data;

[0024] The second EEG signal, the visual training data, the tactile training data, and the motion training data are aligned according to the timestamps.

[0025] The time-aligned second EEG signal, the visual training data, the tactile training data, and the motion training data are fused to obtain the multimodal fusion data.

[0026] In some embodiments, calculating the total loss value based on the target action sequence and the predicted action sequence includes:

[0027] Obtain the error prediction probability output by the end-to-end model;

[0028] The first loss value is obtained by calculating the mean square error based on the target action sequence and the predicted action sequence.

[0029] The second loss value is obtained by calculating the action continuity based on the predicted action sequence;

[0030] The third loss value is calculated based on the error prediction probability and the true error labels in the training dataset;

[0031] The first loss value, the second loss value, and the third loss value are weighted and summed to obtain the total loss value.

[0032] In some embodiments, the method further includes:

[0033] Based on the execution of the action commands by the humanoid robot, an error feedback signal is received;

[0034] The parameters of the end-to-end model are corrected in real time based on the error feedback signal.

[0035] To achieve the above objectives, another aspect of this application proposes a humanoid robot control system, the system comprising:

[0036] The acquisition module is used to acquire real-time visual data, real-time tactile data and real-time motion data of the humanoid robot in response to the first EEG signal of the target user.

[0037] The fusion module is used to fuse the first EEG signal, the real-time visual data, the real-time tactile data and the real-time motion data to obtain real-time fused data.

[0038] The execution module is used to input the real-time fused data into the trained end-to-end model and output action commands to control the humanoid robot to perform actions according to the action commands.

[0039] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0040] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0041] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0042] The embodiments of this application include at least the following beneficial effects: This application provides a humanoid robot control method, system, and related equipment. This solution collects real-time visual data, real-time tactile data, and real-time motion data of the humanoid robot in response to a first EEG signal from a target user; it fuses the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain real-time fused data; it inputs the real-time fused data into a trained end-to-end model and outputs motion commands to control the humanoid robot to perform actions according to the motion commands. This application achieves four-modal data fusion of the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data as the basis for accurate subsequent output of control commands. By directly mapping the real-time fused data to the humanoid robot's motion commands through an end-to-end model, it simplifies the intermediate links in the robot control chain, reduces the accumulated errors and delays in data processing caused by intermediate links, thereby improving the accuracy and real-time performance of humanoid robot control. Attached Figure Description

[0043] Figure 1 This is a flowchart of the humanoid robot control method provided in the embodiments of this application;

[0044] Figure 2 This is a flowchart illustrating the operation of the humanoid robot control system provided in an embodiment of this application.

[0045] Figure 3 This is a schematic diagram of the structure of the multimodal data acquisition module provided in the embodiments of this application;

[0046] Figure 4 This is a schematic diagram of the data preprocessing and fusion module provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the end-to-end model operation provided in the embodiments of this application;

[0048] Figure 6 This is a schematic diagram of the decoder operation provided in the embodiments of this application;

[0049] Figure 7 This is a schematic diagram of encoder operation provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the loss function training process provided in the embodiments of this application;

[0051] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0053] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0054] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0056] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0057] 1) EEG (Electroencephalography) records the electrical activity signals of neurons in the brain through electrodes. These signals are amplified and processed by a computer to form an analyzable waveform.

[0058] 2) FNIRS (Functional Near-Infrared Spectroscopy) is a method that uses near-infrared light to penetrate the skull to non-invasively monitor brain activity. It detects changes in blood oxygenation in the cerebral cortex through near-infrared light, reflecting brain activity.

[0059] 3) SSVEP (Steady-State Visual Evoked Potential) is a stable electrical signal generated by the brain in response to periodic visual stimuli, which can be used for intention recognition.

[0060] 4) ICA (Independent Component Analysis) is a signal processing method used to separate independent components in mixed signals, such as removing artifacts in EEG.

[0061] 5) MLP (Multilayer Perceptron) is an artificial neural network consisting of an input layer, hidden layers, and an output layer, used for feature mapping or classification.

[0062] 6) MSE (Mean Squared Error) is a loss function that calculates the average of the squared differences between the predicted and actual values, and is used to measure the magnitude of the error.

[0063] 7) ErrP (Error-Related Potential): A specific brain electrical signal generated by the brain when it becomes aware of an error or abnormality, which can be used for feedback and error correction.

[0064] A robot is an intelligent machine that can work semi-autonomously or fully autonomously. It can perform tasks such as operations or movement through programming and automatic control, and is widely used in industries, medical fields, services, and scientific research.

[0065] In related technologies, robot control architecture suffers from modular fragmentation, requiring the manual design of multiple independent modules such as sensory feature extraction, intent understanding, task planning, and motion control. For example, brain signals are first parsed to identify intent, and then a classifier is used to identify the user's motion intent. Subsequently, this intent needs to be mapped into action instructions that can be executed by the robot's underlying controller through pre-set rules or lookup tables. This serial and segmented processing mechanism not only causes system response delays, lengthy processes, and low efficiency, but also easily introduces and accumulates errors in multiple stages such as signal conversion, feature alignment, and instruction generation, which seriously restricts the overall performance of humanoid robot control systems.

[0066] In addition, the related technologies also have the following problems:

[0067] Monomodal: Most systems rely on a single sensor (such as vision or speech) and lack fusion perception of user intent (such as EEG signals) and environmental interaction (such as touch).

[0068] Insufficient adaptability: Fixed models struggle to handle complex dynamic scenarios (such as changes in object position in a home environment) and lack real-time optimization mechanisms based on user feedback.

[0069] In view of this, this application provides a humanoid robot control method, system, and related equipment. This scheme collects real-time visual data, real-time tactile data, and real-time motion data of the humanoid robot in response to the first EEG signal of the target user; it fuses the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data to obtain real-time fused data; it inputs the real-time fused data into a trained end-to-end model and outputs motion commands to control the humanoid robot to perform actions according to the motion commands. This application achieves four-modal data fusion of the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data as the basis for accurate subsequent output of control commands. Through an end-to-end model, the real-time fused data is directly mapped to the robot's motion commands, simplifying intermediate links in the robot control chain, reducing accumulated errors and delays in data processing caused by intermediate links, thereby improving the accuracy and real-time performance of humanoid robot control.

[0070] The humanoid robot control method provided in this application relates to the field of robotics. This humanoid robot control method can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the humanoid robot control method, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] Figure 1 This is an optional flowchart of the humanoid robot control method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S103.

[0073] Step S101: In response to the first EEG signal of the target user, collect real-time visual data, real-time tactile data and real-time motion data of the humanoid robot;

[0074] Step S102: The first EEG signal, real-time visual data, real-time tactile data and real-time motion data are fused to obtain real-time fused data;

[0075] Step S103: Input the real-time fused data into the trained end-to-end model and output motion commands to control the humanoid robot to perform actions according to the motion commands.

[0076] In step S101 of some embodiments, the first EEG signal can be acquired using an EEG + FNIRS (functional near-infrared spectroscopy) headgear. The first EEG signal includes motor imagery signals, SSVEP (stable-state visual evoked potential) signals, and EEG signals from the frontal lobe to the occipital lobe. Real-time tactile data includes pressure and temperature data from different parts of the humanoid robot. Real-time visual data includes the current RGB image and depth information of the humanoid robot; depth information refers to the physical distance of each pixel in the image from the camera. Real-time motion data refers to the humanoid robot's posture data across multiple degrees of freedom, such as the angles of each joint. When the user initiates a task using the first EEG signal, the robot synchronously acquires data from the preceding modalities to determine its current state, enabling the robot to perform a series of actions based on this state to complete the task.

[0077] In step S102 of some embodiments, before data fusion, data processing such as noise reduction, spatiotemporal alignment, and feature mapping is performed to facilitate subsequent data fusion processing. By fusing data from four modalities, the fusion perception of user intent (such as EEG signals) and environmental interaction (such as touch and vision) is improved, thereby increasing the accuracy of humanoid robot control and enabling the humanoid robot to perform tasks better.

[0078] In step S103 of some embodiments, the trained end-to-end model can directly map real-time fused data into action commands. An action command can be a sequence of actions used to control the robot to perform a series of actions, thereby completing the task expressed by the user through the first EEG signal. By directly mapping real-time fused data to the robot's action commands through the end-to-end model, the intermediate links in the robot control chain are simplified, reducing the accumulated errors and delays in data processing caused by intermediate links, thereby improving the accuracy and real-time performance of humanoid robot control.

[0079] In some embodiments, step S102 may include, but is not limited to, steps S201 to S202:

[0080] Step S201: Nonlinear processing is performed on the first EEG signal, real-time visual data, real-time tactile data and real-time motion data respectively to obtain the corresponding EEG features, visual features, tactile features and motion features;

[0081] Step S202: Multiply the EEG features, visual features, tactile features and motor features by their respective attention weights and sum them to obtain real-time fused data.

[0082] In steps S201 to S202 of some embodiments, the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data are respectively input into a multilayer perceptron (MLP). The MLP adopts a three-layer structure (input layer, hidden layer, and output layer), and nonlinear processing is performed in the hidden layer. For example, the data dimensions of each modality are processed in the MLP as follows: 64 dimensions in the input layer → 128 dimensions in the hidden layer → 32 dimensions in the output layer. The activation function of the MLP is ReLU, which is used to map the features of each modality to a unified dimension (32 dimensions) to facilitate attention-weighted fusion. The attention weights of the features of different modalities (EEG features, visual features, tactile features, and motion features) are learned through the self-attention mechanism of the Transformer model. The specific process of fusing features of different modalities to obtain real-time fused data is shown in Equation (1):

[0083] ,(1);

[0084] in, These are the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data, respectively. Modal attention weights ( MLP stands for Multilayer Perceptron; These are brainwave characteristics, visual characteristics, tactile characteristics, and motor characteristics.

[0085] In some embodiments, attention weights can be obtained through steps S210 to S220.

[0086] Step S210: Obtain task-related vectors;

[0087] Step S220: Calculate the similarity of the first EEG signal, real-time visual data, real-time tactile data and real-time motion data according to the task-related vectors to obtain their respective similarities;

[0088] Step S230: Calculate the weights based on the similarity to obtain the corresponding attention weights.

[0089] In steps S210 to S230 of one embodiment, the attention weights can be calculated using the modal attention layer of the Transformer model. The task-related vector can be obtained by processing the first EEG signal. The specific calculation process of the attention weights is shown in equation (2):

[0090] (2);

[0091] in, The attention weights for each data point. Represented as data in different modalities Similarity to the task-related vector Q. Attention weights can characterize the importance of different data to the current task. By assigning appropriate weights to data of different modalities, the accuracy of task execution can be further improved.

[0092] For example, if the task is "a robot grabbing a water cup," the task-related vector Q includes the core features of the "grasping action" (e.g., the hand needs to perform a grasping motion, vision needs to recognize the shape of the water cup, and touch needs to perceive the grip strength). Different modalities of data (EEG, vision, touch, and action) are matched with Q for similarity to determine the contribution of different modalities to the current task (e.g., the "wanting to grasp" signal in the EEG will be highly similar to Q and receive a high attention weight). The specific process of obtaining the task-related vector (Q) is as follows: the core source of the task-related vector is the user's EEG signal (because EEG directly reflects "the person's task intention"), which is then combined with "task prior knowledge" (e.g., typical features of common tasks) for optimization, so that Q accurately represents the current task.

[0093] Specifically, firstly, the raw EEG signals (such as motor imagery EEG, SSVEP, etc.) are preprocessed and the intent is coarsely extracted. These raw EEG signals are then filtered and denoised (e.g., by independent component analysis (ICA) to remove electrooculogram artifacts) to obtain "clean" EEG features. Use a simple classifier (such as Linear Discriminant Analysis (LDA)) to... Perform a "rough intent classification" to determine the type of task the user currently wants to perform (e.g., "grab," "place," "move"), and output a "task type label." ,like (Represents the task of fetching data).

[0094] Then, task prior vector library matching is performed, and a task prior vector library is pre-built ( The task prior vector library stores typical feature vectors for common tasks. For example, the prior vector for a grasping task. (Including fused features such as EEG templates of hand grasping movements, visual cup features, and joint angles of typical grasping actions); prior vectors for the placement task. (Including fused features such as EEG templates of hand release movements and visual features of the target location). Based on the "task type label" obtained in the previous examples. ,from Retrieve initial task-related vectors (e.g.) At that time, retrieve ).

[0095] Finally, by combining dynamic optimization of real-time EEG, the real-time EEG features are... and ( It can be or Dynamic fusion is performed using various methods (such as...) to obtain the final task-related vector Q. The fusion method can employ a weighted summation based on an attention mechanism (or calculation using a Transformer modal attention layer), as shown in the following equation:

[0096] ;

[0097] in, Represents the learnable weight matrix. It is the weight matrix for adaptive learning during model training, with dimensions of . , Represented as real-time EEG characteristics Vectors related to the initial task The concatenated feature dimensions (which need to match the sum of the feature lengths of both). This is represented as the target dimension after linear transformation (usually consistent with the dimension of subsequent attention calculations or feature representations). For the concatenated multimodal features... A linear transformation is performed to map it to a high-dimensional space for "attention weight calculation," enabling the model to more accurately capture the correlation between "real-time EEG intent" and "prior task features."

[0098] For the attention weight function, in the formula The calculation logic is as follows: first, the concatenated features are processed by the weight matrix W. Perform a linear transformation to obtain the intermediate vector. ;right Applying the Softmax function (i.e.) This transforms the vector elements into a probability distribution between 0 and 1 (the sum of all elements is 1).

[0099] The output is the attention weight corresponding to the real-time EEG features, which reflects the importance of the "current real-time EEG intention" in the final task-related vector Q (the higher the weight, the stronger the guidance of the real-time EEG on the current task).

[0100] Specifically As a complementary weight term, the attention weight must satisfy the normalization constraint that "real-time EEG weight + initial task vector weight = 1". This represents the attention weights corresponding to the initial task-related vectors. It achieves a dynamic balance between "real-time EEG intent" and "prior task features," when real-time EEG... When the weight of factors such as noise is low, The weight of [the signal] will increase (based on prior task knowledge, improving stability); when the real-time EEG intent is clear... The weight will be reduced (prioritizing real-time intent to improve flexibility).

[0101] As a direct carrier of human intent, EEG reflects user intentions more effectively than pure vision or touch. Pure real-time EEG may contain noise or ambiguous intents. Prior libraries provide basic task features, and dynamic optimization enables the task-related vector Q to stably and accurately represent the task.

[0102] The task-related vector Q obtained through the above process has the following beneficial effects:

[0103] Precise association of multimodal data: This enables the model to determine which modal data is more important for task execution (for example, in a grasping task, "cup position" in visual data and "grip feedback" in tactile data will receive high weights because they are similar to Q).

[0104] Adaptable to different task scenarios: Whether it is a "grabbing", "assembling" or "serving" task, Q can dynamically adjust through "EEG + prior library" to improve the targeting of multimodal fusion;

[0105] Improve model interpretability: Q serves as a "task intent anchor" to enhance the interpretability of attention weight calculation.

[0106] In some embodiments, the trained end-to-end model can be obtained through steps S301 to S304.

[0107] Step S301: Obtain the training dataset, wherein the training dataset includes a combination of several multimodal fusion data and target action sequences;

[0108] Step S302: Input the multimodal fusion data into the end-to-end model to obtain the predicted action sequence;

[0109] Step S303: Calculate the total loss value based on the target action sequence and the predicted action sequence;

[0110] Step S304: Adjust the end-to-end model parameters with the goal of minimizing the total loss value to obtain the trained end-to-end model.

[0111] In steps S301 to S304 of some embodiments, a timestamped training dataset is generated by synchronously collecting the user's EEG signals, tactile interaction data (pressure / temperature) between the robot and the environment, scene visual images, and robot joint angle data during task execution. ,in For multimodal fusion data, The target action sequence. It can be obtained through data fusion as shown in equation (1), that is for The end-to-end model adopts an improved Transformer architecture, including a modal attention layer, an encoder, and a decoder. The specific process of step S302 is shown in equation (3).

[0112] (3);

[0113] in, To predict action sequences, These are the model parameters. According to... and Calculate the total loss value. and The greater the difference, the greater the total loss value. This can be addressed by adjusting the model parameters. , making Towards Approximating the loss value improves the model's predictive performance.

[0114] In some embodiments, multimodal fusion data can be obtained through steps S310 to S320.

[0115] Step S310: Acquire the preprocessed second EEG signal, visual training data, tactile training data, and motion training data;

[0116] Step S320: Align the second EEG signal, visual training data, tactile training data, and motion training data according to the timestamp;

[0117] Step S330: The time-aligned second EEG signal, visual training data, tactile training data and motion training data are fused to obtain multimodal fusion data.

[0118] In steps S310 to S330 of some embodiments, the second EEG signal, visual training data, tactile training data, and motion training data are the EEG signal, scene visual image, tactile interaction data (pressure / temperature) between the robot and the environment, and robot joint angle data when the user performs the task. The preprocessing process is as follows: Independent component analysis (ICA) is performed on the EEG to remove electrooculogram artifacts, wavelet transform is used to extract temporal features from the tactile training data, image features are extracted from the visual training data through pre-trained ResNet50, and motion training data is normalized. Since the training data of each modality has a timestamp, the training data at the same time are aligned to facilitate subsequent data fusion. The specific data fusion is shown in Equation (1).

[0119] In some embodiments, step S303 may include, but is not limited to, steps S331 to S335:

[0120] Step S331: Obtain the error prediction probability output by the end-to-end model;

[0121] Step S332: Calculate the mean square error based on the target action sequence and the predicted action sequence to obtain the first loss value;

[0122] Step S333: Calculate the action continuity based on the predicted action sequence to obtain the second loss value;

[0123] Step S334: Calculate the third loss value based on the error prediction probability and the true error labels in the training dataset;

[0124] Step S335: The first loss value, the second loss value, and the third loss value are weighted and summed to obtain the total loss value.

[0125] In step S332 of some embodiments, the mean square error is calculated based on the target action sequence and the predicted action sequence, as shown in equation (4).

[0126] (4);

[0127] in, The action parameters predicted by the end-to-end model at different times are obtained from equation (3). Includes multiple , Represents the target action sequence in the training dataset D. The action parameters at different times, i.e. Includes multiple , The first loss value is the MSE (mean squared error) loss between the predicted action and the actual action, where T is the length of the action sequence. The first loss value characterizes the difference between the target action sequence and the predicted action sequence. Reducing the first loss value can make the model's predicted actions closer to the target action sequence, thus improving the prediction accuracy of the end-to-end model.

[0128] In step S333 of some embodiments, the calculation of action continuity based on the predicted action sequence is specifically shown in equation (5).

[0129] (5);

[0130] in, and This represents the action parameters predicted by the end-to-end model at adjacent time points. for The action parameters at the next moment, The second loss value can characterize the continuity of the predicted action sequence. By reducing the second loss value, the continuity of the action sequence predicted by the end-to-end model can be improved.

[0131] In steps S331 and S334 of some embodiments, the error prediction probability can refer to the possibility that the robot may make a mistake in performing the task based on the current training data, that is, the possibility that the user will provide ErrP (error-related potential) signal. The specific calculation of the third loss value is shown in Equation (6).

[0132] (6);

[0133] Where K represents the training dataset of The quantity, k corresponds to different The serial number, For ErrP (Error-Related Potential) signal detection loss (third loss value), the true error label Represented as 0 or 1, This represents the error prediction probability. Reducing the third loss value can improve the end-to-end model's ability to correct predicted action sequences, thus better accomplishing the task.

[0134] In step S335 of some embodiments, the total loss value is obtained through a dynamic loss function, and the specific calculation is shown in equation (7).

[0135] (7);

[0136] in, This is the total loss value. , and These represent the weights of each loss value.

[0137] In some embodiments, the humanoid robot control method further includes, but is not limited to, steps S104 to S105:

[0138] Step S104: Receive error feedback signals based on the humanoid robot's execution of action commands;

[0139] Step S105: Correct the parameters of the end-to-end model in real time based on the error feedback signal.

[0140] In steps S104 to S105 of some embodiments, based on the humanoid robot's execution of action commands, an error feedback signal is generated through visual recognition and ErrP (Error-Related Potential) detection. The error-related potential is automatically generated by the user's brain and fed back through EEG signals (a natural neurophysiological response). When the error exceeds a threshold, model fine-tuning is triggered, and incremental learning is used to update the parameters of the end-to-end model.

[0141] For example, in a scenario where a user controls a robot to grasp a water cup via EEG signals, when the user observes the robot failing to grasp the cup, the brain naturally generates an error-related potential (ErrP). This signal is collected by the EEG module and transmitted to the system. Simultaneously, the robot acquires images of the task execution via its vision module, analyzes the distance error between the finger joints and the water cup (exceeding a set threshold, such as 5mm), and combines this with the pressure error from the hand's tactile sensor (pressure value below 20% of the ideal grip strength threshold). These two types of errors are then encoded into a multimodal loss function.

[0142] ;

[0143] in, Represents the visual error loss term. It is the deviation value of "actual distance - ideal grasping distance" (if it exceeds the threshold by 5mm, the error is positive). The bias in visual perception is penalized using the mean squared error (MSE) or distance loss function.

[0144] This represents the tactile error loss term. It is the difference between "actual pressure and ideal grip strength" (a negative deviation if the pressure is insufficient). Similarly, the accuracy of constrained tactile feedback is calculated using MSE or force feedback loss function.

[0145] This represents the EEG error loss term, which encodes the deviation between the user's intention and the execution result by encoding the loss through the amplitude or duration of the EEG signal. For example, if the ErrP is stronger / longer, then... The larger.

[0146] α, β, and γ represent weighting coefficients. These are dynamically adjusted based on the strength of the ErrP signal (e.g., increasing the weighting coefficient when ErrP is strong). This leads to a loss of dominant EEG feedback; visual occlusion reduces [the impact of EEG feedback]. By weakening unreliable visual errors, adaptive weighting of multimodal errors can be achieved in different scenarios, thus enabling adaptive adjustment.

[0147] The model is optimized through incremental learning via backpropagation, as follows:

[0148] Adjust the cross-modal attention weights of the Transformer encoder (increase the proportion of visual feature attention to over 60% when the distance exceeds the threshold); correct the joint angle output weights of the MLP action prediction layer (compensate for the predicted value of finger gripping force based on pressure bias).

[0149] In dynamic scenarios such as multiple people moving around, if visual misjudgment is caused by pedestrian occlusion, the system can freeze the background region weight of visual spatial attention only when the ErrP duration and amplitude are short-duration weak ErrP (<300ms) (ignore occlusion and focus on the outline of the water cup) and forcibly trigger tactile-EEG bimodal calibration when the ErrP duration is long-duration strong (≥300ms) (reconstruct the grasping trajectory with pressure feedback and get rid of visual dependence).

[0150] By correcting model parameters in real time, the robot gains the ability to learn adaptively in real time. It can dynamically adjust model parameters through interactive feedback, which significantly improves the task execution accuracy and the naturalness of human-robot interaction in complex scenarios. This greatly enhances the robot's anti-interference ability and robustness in dynamic environments (such as when multiple people are moving around).

[0151] In some embodiments, the user initiates a task (such as "grabbing a water cup") through EEG signals. The robot system synchronously collects tactile (pressure when touching the water cup), visual (position of the water cup), and motion (arm movement trajectory) data. The end-to-end model directly outputs the finger joint angle. If an ErrP (error-related potential) signal is detected (the user realizes that the grasping has failed), the model parameters are corrected through the dynamic loss function L, which is shown in Equation (7).

[0152] In some embodiments, the end-to-end model training process is as follows: 10 subjects wear headgear and imagine "grabbing / placing" actions, while EEG signals are recorded simultaneously; the robot performs 500 grasping tasks in a home setting, collecting tactile (contact with tabletop / cup), visual (different positions of cup) and motion data.

[0153] Model Training: Dataset Split: 80% training set, 20% validation set; Hyperparameters: 6 Transformer layers, 8 attention heads, learning rate 1e-4, 100 training epochs; Dynamic Loss Weights: Initial... Later improvements To enhance error correction capabilities.

[0154] In some embodiments, the end-to-end model inference latency is <100ms. When the ErrP (Error-Related Potential) detection module detects a negative potential (NRE > 5μV) within 0.5s after instruction execution, it triggers model fine-tuning (the learning rate is reduced to 1e-5, and 10 batches are updated).

[0155] In some embodiments, please refer to Figure 2 A humanoid robot control system includes a multimodal data acquisition module, a data preprocessing and fusion module, an end-to-end training module, a real-time control module, and an execution feedback module.

[0156] The multimodal data acquisition module includes an EEG (electroencephalography) + FNIRS (functional near-infrared spectroscopy) synchronous acquisition headgear (containing 8-channel electrodes and 4 pairs of near-infrared probes), a 16-point tactile sensor array distributed on the robot's hand / torso (sampling frequency 1kHz), a binocular vision sensor (resolution 1920×1080, frame rate 30fps), and a 6-axis inertial measurement unit (sampling frequency 200Hz).

[0157] For the acquired EEG signals, the head cap used a timestamp synchronization mechanism for simultaneous acquisition of EEG and FNIRS (functional near-infrared spectroscopy). The EEG signals were notched and filtered at 50Hz to remove power frequency noise, and a bandpass filter of 0.1-30Hz was used to retain the effective signals of motor imagery and SSVEP (steady-state visual evoked potentials) (SNR ≥ 20dB after processing). The FNIRS signals, after conversion, must meet the detection accuracy of ΔHbO / ΔHbR ≤ 0.1μmol / L.

[0158] The data preprocessing and fusion module receives multimodal data via Ethernet, fiber optic cable, or wireless WIFI, performs noise reduction, spatiotemporal alignment, and feature mapping, and outputs a fusion feature vector with unified dimensions.

[0159] The end-to-end training module adopts an improved Transformer architecture, which includes a modal attention layer and an action prediction decoder, and optimizes model parameters through a dynamic loss function.

[0160] The real-time control module converts the motion commands output by the training model into robot joint drive signals, and the execution feedback module generates error signals through visual and tactile data for online fine-tuning of the model.

[0161] The execution feedback module generates error signals by calculating visual errors (such as object grasping position offset > 5mm) and tactile errors (such as contact pressure deviation > 1N) in real time. When the error signal exceeds the preset threshold, it is transmitted to the end-to-end training module via wired connection to trigger parameter fine-tuning.

[0162] In some embodiments, please refer to Figure 3 The multimodal data acquisition module includes an EEG unit, a tactile unit, a visual unit, and a motor unit. The EEG unit is an EEG (electroencephalography) + FNIRS (functional near-infrared spectroscopy) headgear used to acquire motor imagery (left hand / right hand / tongue) and SSVEP (steady-state visual evoked potential) signals, with a sampling rate of 250Hz. The 8-channel electrodes are located at Fz, Cz, Pz, O1, O2, T7, T8, and Oz positions of the international 10-20 system, respectively, to acquire EEG signals from the frontal lobe to the occipital lobe.

[0163] The tactile unit includes a flexible sensor array for the robot's hand / elbow, which collects pressure (0-10N) and temperature (20-40℃). The 16-point tactile sensor array includes 12 points for the hand (2 points for the thumb, 4 points for the index finger, 4 points for the middle finger, and 2 points for the palm) and 4 points for the elbow (2 points each for the medial and lateral epicondyles of the humerus). The sampling frequency is 1kHz, and the pressure detection accuracy is ±0.1N.

[0164] The vision unit includes a binocular camera and a depth sensor, used to acquire RGB images and depth information.

[0165] The motion unit (motion capture unit) includes joint encoders and IMUs, used to record the robot's joint angles (finger angles and arm angles, etc.) and 6-DOF pose data.

[0166] In some embodiments, please refer to Figure 4 The data preprocessing and fusion module uses timestamp-aligned multimodal data and achieves feature fusion through equation (1), where attention weights are dynamically adjusted (e.g., tactile weights are increased when performing fine operations). Timestamp synchronization uses GPS timing + local crystal oscillator calibration, with synchronization accuracy ≤1ms, so that the time deviation of EEG signals, tactile signals, visual signals, and action signals is within 0.5ms.

[0167] In some embodiments, please refer to Figure 5, Figure 6 and Figure 7 The end-to-end training module is based on a Transformer encoder-decoder architecture. The encoder processes fused features, and the decoder outputs joint angle sequences. The encoder first learns the complex relationships between multimodal environmental features through a multi-head self-attention mechanism, then performs feature enhancement and transformation via a feedforward network, ultimately outputting an environmental representation that incorporates global contextual information. The decoder first models the dependencies within the generated action sequence through masked self-attention, then uses an encoder-decoder attention layer to critically associate the current action with the environmental features provided by the encoder, finally outputting specific action commands via a feedforward network. Residual connections and layer normalization improve the stability and efficiency of training, enabling the model to generate reasonable and coherent action sequences based on the environmental state.

[0168] In some embodiments, the dynamic loss function is as shown in equation (7), through Balance motion accuracy, error correction, and motion smoothness. Figure 8 Demonstrated the training process The changing trend, among which, The initial value is 3.2. The initial value is 2.8. The initial value is 1.5, and the initial total loss is 5.0. After a certain number of training rounds (0-100 rounds), the total loss value is reduced.

[0169] This application has at least the following beneficial effects:

[0170] 1) Task execution accuracy has been significantly improved;

[0171] 2) The training cycle is significantly shortened;

[0172] 3) Enhanced anti-interference capability and significantly improved robustness in dynamic environments (such as when multiple people are moving around).

[0173] This application also provides a humanoid robot control system that can implement the above-described method. The system includes:

[0174] The acquisition module is used to acquire real-time visual data, real-time tactile data and real-time motion data of the humanoid robot in response to the first EEG signal of the target user.

[0175] The fusion module is used to fuse the first EEG signal, real-time visual data, real-time tactile data and real-time motion data to obtain real-time fused data.

[0176] The execution module is used to input real-time fused data into the trained end-to-end model and output motion commands to control the humanoid robot to perform actions according to the motion commands.

[0177] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0178] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0179] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0180] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0181] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0182] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application.

[0183] The input / output interface 903 is used to implement information input and output;

[0184] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0185] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0186] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0187] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0188] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0189] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0190] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0191] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0192] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0193] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0194] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0195] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0196] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0197] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0199] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0201] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for controlling a humanoid robot, characterized in that, The method includes the following steps: In response to the target user's first EEG signal, the robot collects real-time visual data, real-time tactile data, and real-time motion data. The first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data are fused to obtain real-time fused data; The real-time fused data is input into the trained end-to-end model, and action commands are output to control the humanoid robot to perform actions according to the action commands. The process of fusing the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data to obtain real-time fused data includes: The first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data are subjected to nonlinear processing to obtain corresponding EEG features, visual features, tactile features, and motion features; The EEG features, visual features, tactile features, and motion features are multiplied by their respective attention weights and then summed to obtain the real-time fused data; The attention weights are obtained through the following steps: Obtain task-related vectors; Based on the task-related vector, the similarity of the first EEG signal, the real-time visual data, the real-time tactile data and the real-time motion data are calculated respectively to obtain their respective similarity. Based on the similarity, weights are calculated to obtain the corresponding attention weights for each. The task-related vector is obtained through the following steps: The first EEG signal is combined with the preset task prior knowledge to obtain the task-related vector.

2. The method according to claim 1, characterized in that, The trained end-to-end model is obtained through the following steps: Obtain a training dataset, wherein the training dataset includes a combination of several multimodal fusion data and target action sequences; The multimodal fusion data is input into the end-to-end model to obtain the predicted action sequence; Calculate the total loss value based on the target action sequence and the predicted action sequence; With the goal of minimizing the total loss value, the end-to-end model parameters are adjusted to obtain a trained end-to-end model.

3. The method according to claim 2, characterized in that, The multimodal fusion data is obtained through the following steps: Acquire preprocessed second EEG signals, visual training data, tactile training data, and motor training data; The second EEG signal, the visual training data, the tactile training data, and the motion training data are aligned according to the timestamps. The time-aligned second EEG signal, the visual training data, the tactile training data, and the motion training data are fused to obtain the multimodal fusion data.

4. The method according to claim 2, characterized in that, The step of calculating the total loss value based on the target action sequence and the predicted action sequence includes: Obtain the error prediction probability output by the end-to-end model; The first loss value is obtained by calculating the mean square error based on the target action sequence and the predicted action sequence. The second loss value is obtained by calculating the action continuity based on the predicted action sequence; The third loss value is calculated based on the error prediction probability and the true error labels in the training dataset; The first loss value, the second loss value, and the third loss value are weighted and summed to obtain the total loss value.

5. The method according to claim 1, characterized in that, The method further includes: Based on the execution of the action commands by the humanoid robot, an error feedback signal is received; The parameters of the end-to-end model are corrected in real time based on the error feedback signal.

6. A humanoid robot control system, characterized in that, The system includes: The acquisition module is used to acquire real-time visual data, real-time tactile data and real-time motion data of the humanoid robot in response to the first EEG signal of the target user. The fusion module is used to fuse the first EEG signal, the real-time visual data, the real-time tactile data and the real-time motion data to obtain real-time fused data. The execution module is used to input the real-time fused data into the trained end-to-end model and output action commands to control the humanoid robot to perform actions according to the action commands. The fusion module specifically performs the following steps: The first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data are subjected to nonlinear processing to obtain corresponding EEG features, visual features, tactile features, and motion features; The EEG features, visual features, tactile features, and motion features are multiplied by their respective attention weights and then summed to obtain the real-time fused data; The attention weights are obtained through the following steps: Obtain task-related vectors; Based on the task-related vector, the similarity of the first EEG signal, the real-time visual data, the real-time tactile data and the real-time motion data are calculated respectively to obtain their respective similarity. Based on the similarity, weights are calculated to obtain the corresponding attention weights for each. The task-related vector is obtained through the following steps: The first EEG signal is combined with the preset task prior knowledge to obtain the task-related vector.

7. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images