Humanoid robot control method and system and related equipment

Abstract

The embodiment of the invention provides a humanoid robot control method and system and related equipment, and belongs to the technical field of robots. The method comprises the following steps: in response to a first electroencephalogram signal of a target user, collecting real-time visual data, real-time tactile data and real-time action data of the humanoid robot; performing data fusion on the first electroencephalogram signal, the real-time visual data, the real-time tactile data and the real-time action data to obtain real-time fusion data; and inputting the real-time fusion data into the trained end-to-end model, and outputting an action instruction to control the humanoid robot to execute actions according to the action instruction. According to the embodiment of the invention, the precision and real-time performance of humanoid robot control can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a humanoid robot control method, system, and related equipment. Background Art

[0002] A robot is an intelligent machine that can work semi-autonomously or fully autonomously. It can perform tasks such as operation or movement through programming and automatic control. It is widely used in industries such as industry, medical care, services, and scientific research.

[0003] In related technologies, the robot control architecture has the problem of modular fragmentation, and multiple independent modules such as perception feature extraction, intention understanding, task planning and motion control need to be manually designed separately. For example, the EEG signal is first analyzed to identify the intention, and then the classifier is used to identify the user's motion intention. Subsequently, the intention needs to be mapped into action instructions executable by the robot's underlying controller through pre-set rules or lookup tables. This serial and multi-stage processing mechanism can easily lead to system response delays, lengthy processes and low efficiency, which seriously restricts the overall performance of the humanoid robot control system.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to propose a humanoid robot control method, system and related equipment, which can improve the accuracy and real-time performance of humanoid robot control.

[0006] To achieve the above objectives, one aspect of an embodiment of the present application provides a method for controlling a humanoid robot, the method comprising: In response to the first EEG signal of the target user, collecting real-time visual data, real-time tactile data, and real-time motion data of the humanoid robot; fusing the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain real-time fused data; The real-time fusion data is input into a trained end-to-end model, and action instructions are output to control the humanoid robot to perform actions according to the action instructions.

[0007] In some embodiments, fusing the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data to obtain real-time fused data includes: performing nonlinear processing on the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data, respectively, to obtain corresponding EEG features, visual features, tactile features, and motion features; The EEG feature, the visual feature, the tactile feature, and the motion feature are respectively multiplied by corresponding attention weights and added to obtain the real-time fusion data.

[0008] In some embodiments, the attention weight is obtained by the following steps: Get task-related vectors; performing similarity calculations on the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data respectively according to the task-related vector to obtain respective corresponding similarities; A weight calculation is performed based on the similarity to obtain the corresponding attention weights.

[0009] In some embodiments, the trained end-to-end model is trained by the following steps: Acquire a training data set, wherein the training data set includes a combination of a plurality of multimodal fusion data and a target action sequence; Inputting the multimodal fusion data into the end-to-end model to obtain a predicted action sequence; Calculating a total loss value according to the target action sequence and the predicted action sequence; With the goal of minimizing the total loss value, the end-to-end model parameters are adjusted to obtain a trained end-to-end model.

[0010] In some embodiments, the multimodal fusion data is obtained by the following steps: Acquiring the preprocessed second EEG signal, visual training data, tactile training data, and motion training data; Aligning the second EEG signal, the visual training data, the tactile training data, and the motion training data according to the timestamp; The time-aligned second EEG signal, the visual training data, the tactile training data, and the motion training data are fused to obtain the multimodal fusion data.

[0011] In some embodiments, calculating the total loss value according to the target action sequence and the predicted action sequence includes: Obtaining an error prediction probability output by the end-to-end model; Calculating a mean square error based on the target action sequence and the predicted action sequence to obtain a first loss value; Performing action continuity calculation according to the predicted action sequence to obtain a second loss value; Calculating based on the error prediction probability and the true error label in the training data set to obtain a third loss value; A weighted sum is performed on the first loss value, the second loss value, and the third loss value to obtain the total loss value.

[0012] In some embodiments, the method further comprises: receiving an error feedback signal according to the humanoid robot executing the action instruction; The parameters of the end-to-end model are modified in real time according to the error feedback signal.

[0013] To achieve the above objectives, another aspect of the present application provides a humanoid robot control system, the system comprising: an acquisition module, configured to acquire real-time visual data, real-time tactile data, and real-time motion data of the humanoid robot in response to a first EEG signal of a target user; a fusion module, configured to fuse the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain real-time fused data; The execution module is used to input the real-time fusion data into the trained end-to-end model and output action instructions to control the humanoid robot to perform actions according to the action instructions.

[0014] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0015] To achieve the above objectives, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0016] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0017] The embodiments of the present application include at least the following beneficial effects: The present application provides a humanoid robot control method, system and related equipment, which collects real-time visual data, real-time tactile data and real-time motion data of the humanoid robot in response to a first EEG signal of a target user; fuses the first EEG signal, the real-time visual data, the real-time tactile data and the real-time motion data to obtain real-time fused data; inputs the real-time fused data into a trained end-to-end model, and outputs motion instructions to control the humanoid robot to perform actions according to the motion instructions. The present application implements the four-modal data fusion of the first EEG signal, real-time visual data, real-time tactile data and real-time motion data as the basis for the subsequent accurate output of control instructions. The real-time fused data is directly mapped to the motion instructions of the humanoid robot through the end-to-end model, which simplifies the intermediate links of the robot control link and reduces the accumulated errors and delays in data processing in the intermediate links, thereby improving the accuracy and real-time performance of the humanoid robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a humanoid robot control method provided by an embodiment of the present application; Figure 2 This is a flowchart of the operation of the humanoid robot control system provided by the embodiment of the present application; Figure 3 Schematic diagram of the structure of the multimodal data acquisition module provided in the embodiment of the present application; Figure 4 This is a schematic diagram of the structure of the data preprocessing and fusion module provided in an embodiment of the present application; Figure 5 This is a schematic diagram of the end-to-end model operation provided by the embodiment of the present application; Figure 6 This is a schematic diagram of the operation of the decoder provided in an embodiment of the present application; Figure 7 This is a schematic diagram of the operation of the encoder provided in an embodiment of the present application; Figure 8 This is a schematic diagram of the loss function training process provided in an embodiment of the present application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0020] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0021] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0023] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0024] 1) EEG (Electroencephalography) records the electrical activity signals of brain neurons through electrodes. These signals are amplified and processed by computers to form analyzable waveforms.

[0025] 2) FNIRS (Functional Near-Infrared Spectroscopy) is a method that uses near-infrared light to penetrate the skull to non-invasively monitor brain activity. It uses near-infrared light to detect changes in blood oxygen in the cerebral cortex, reflecting brain activity.

[0026] 3) SSVEP (Steady-State Visual Evoked Potential), a stable electrical signal generated by the brain in response to periodic visual stimulation, can be used for intention recognition.

[0027] 4) ICA (Independent Component Analysis), a signal processing method used to separate independent components in mixed signals, such as removing artifacts in EEG.

[0028] 5) MLP (Multilayer Perceptron), an artificial neural network consisting of an input layer, a hidden layer, and an output layer, used for feature mapping or classification.

[0029] 6) MSE (Mean Squared Error) is a loss function that calculates the average of the squared differences between the predicted value and the true value, and is used to measure the error size.

[0030] 7) ErrP (Error-Related Potential) is a specific EEG signal generated by the brain when it realizes an error or abnormality, which can be used for feedback and error correction.

[0031] A robot is an intelligent machine that can work semi-autonomously or fully autonomously. It can perform tasks such as operation or movement through programming and automatic control. It is widely used in industries such as industry, medical care, services, and scientific research.

[0032] In related technologies, the robot control architecture has the problem of modular fragmentation, and multiple independent modules such as perception feature extraction, intention understanding, task planning and motion control need to be manually designed separately. For example, the EEG signal is first analyzed to identify the intention, and then the classifier is used to identify the user's motion intention. Subsequently, the intention needs to be mapped into action instructions executable by the robot's underlying controller through pre-set rules or lookup tables. This serial and segmented processing mechanism not only causes system response delays, lengthy processes, and low efficiency, but also easily introduces and accumulates errors in multiple links such as signal conversion, feature alignment, and instruction generation, seriously restricting the overall performance of the humanoid robot control system.

[0033] In addition, the related technology still has the following problems: Single modality: Most systems rely on a single sensor (such as vision or voice) and lack the integrated perception of user intention (such as EEG signals) and environmental interaction (such as touch).

[0034] Insufficient adaptability: Fixed models are difficult to cope with complex dynamic scenarios (such as the changing position of objects in a home environment) and lack real-time optimization mechanisms based on user feedback.

[0035] In view of this, the present application provides a humanoid robot control method, system and related equipment. The scheme collects real-time visual data, real-time tactile data and real-time motion data of the humanoid robot in response to the first EEG signal of the target user; fuses the first EEG signal, real-time visual data, real-time tactile data and real-time motion data to obtain real-time fused data; inputs the real-time fused data into a trained end-to-end model, and outputs action instructions to control the humanoid robot to perform actions according to the action instructions. The present application realizes the four-modal data fusion of the first EEG signal, real-time visual data, real-time tactile data and real-time motion data as the basis for the subsequent accurate output of control instructions. The real-time fused data is directly mapped to the robot's motion instructions through the end-to-end model, which simplifies the intermediate links of the robot control link and reduces the accumulated errors and delays in data processing in the intermediate links, thereby improving the accuracy and real-time performance of the humanoid robot control.

[0036] The humanoid robot control method provided in the embodiment of the present application relates to the field of robotics technology. The humanoid robot control method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the humanoid robot control method, etc., but is not limited to the above forms.

[0037] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0038] Figure 1 This is an optional flowchart of the humanoid robot control method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S103.

[0039] Step S101, in response to a first EEG signal of a target user, collecting real-time visual data, real-time tactile data, and real-time motion data of a humanoid robot; Step S102, fusing the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain real-time fused data; Step S103: input the real-time fusion data into the trained end-to-end model, and output action instructions to control the humanoid robot to perform actions according to the action instructions.

[0040] In step S101 of some embodiments, the first EEG signal can be collected by an EEG (electroencephalogram) + FNIRS (functional near-infrared spectroscopy) headgear. The first EEG signal includes a motor imagery signal, an SSVEP (steady-state visual evoked potential) signal, and an EEG signal from the frontal lobe to the occipital lobe. Real-time tactile data includes pressure data and temperature data of different parts of the humanoid robot. Real-time visual data includes the current RGB image and depth information of the humanoid robot. The depth information refers to the physical distance of each pixel in the image from the camera. Real-time motion data refers to the posture data of multiple degrees of freedom of the humanoid robot, such as the angles of each joint. The user initiates a task through the first EEG signal, and the robot will synchronously collect data from the previous modes to determine the current state of the robot, so that the robot can perform a series of actions based on the current state to complete the task.

[0041] In some embodiments, before data fusion, step S102 involves performing data processing such as noise reduction, spatiotemporal alignment, and feature mapping to facilitate subsequent data fusion. By fusing data from four modalities, the fusion of user intent (e.g., EEG signals) and environmental interaction forces (e.g., touch and vision) is enhanced, thereby improving the precision of humanoid robot control and enabling the humanoid robot to better complete tasks.

[0042] In step S103 of some embodiments, the trained end-to-end model can directly map the real-time fused data into action commands. The action commands can be a sequence of actions used to control the robot to perform a series of actions, thereby completing the task expressed by the user through the first EEG signal. By directly mapping the real-time fused data into the robot's action commands through the end-to-end model, the robot's control chain is streamlined, reducing the accumulated errors and delays in data processing in these intermediate links, thereby improving the precision and real-time performance of humanoid robot control.

[0043] In some embodiments, step S102 may include but is not limited to steps S201 to S202: Step S201, performing nonlinear processing on the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain corresponding EEG features, visual features, tactile features, and motion features; In step S202 , the EEG features, visual features, tactile features, and motion features are multiplied by corresponding attention weights and added to obtain real-time fusion data.

[0044] In some embodiments, in steps S201 to S202, the first EEG signal, real-time visual data, real-time tactile data, and real-time motion data are respectively input into a multilayer perceptron (MLP). The MLP adopts a three-layer structure (input layer, hidden layer, and output layer), and nonlinear processing is performed in the hidden layer. Exemplarily, the data dimensions of each modality are processed in the MLP as follows: input layer 64 dimensions → hidden layer 128 dimensions → output layer 32 dimensions. The activation function of the MLP is ReLU, which is used to map the features of each modality to a unified dimension (32 dimensions) to facilitate attention weighted fusion. The attention weights of the features of different modalities (EEG features, visual features, tactile features, and motion features) are learned through the self-attention mechanism of the Transformer model. The specific process of fusing the features of different modalities to obtain real-time fusion data is shown in formula (1): , (1); in, They are the first EEG signal, real-time visual data, real-time tactile data and real-time motion data; is the modality attention weight ( ); MLP is a multi-layer perceptron; They are EEG features, visual features, tactile features and motion features.

[0045] In some embodiments, the attention weight can be obtained through steps S210 to S220.

[0046] Step S210, obtaining task-related vectors; Step S220, performing similarity calculations on the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data according to the task-related vector to obtain respective corresponding similarities; Step S230: perform weight calculation based on the similarity to obtain the corresponding attention weights.

[0047] In steps S210 to S230 of one embodiment, the attention weight can be calculated by the modal attention layer of the Transformer model. The task-related vector can be obtained by processing the first EEG signal. The specific calculation process of the attention weight is shown in formula (2): , (2); in, is the attention weight corresponding to each data, Represented as different modal data The similarity with the task-related vector Q. Attention weight can represent the importance of different data to the current task. By assigning corresponding weights to data of different modalities, the task execution accuracy can be further improved.

[0048] For example, if the task is "a robot grasping a water cup," the task-related vector Q includes the core features of the "grasping action" (for example, the hand needs to make a grasping movement, the vision needs to recognize the shape of the water cup, and the touch needs to perceive the grip force). Different modal data (EEG, vision, touch, action) are matched with Q for similarity to determine the contribution of different modal data to the current task (for example, the "want to grasp" signal in the EEG will be highly similar to Q and receive a high attention weight). The specific process of obtaining the task-related vector (Q) is as follows: The core source of the task-related vector is the user's EEG signal (because EEG directly reflects "the person's task intention"), which is then combined with "task prior knowledge" (such as the typical features of common tasks) to optimize so that Q accurately represents the current task.

[0049] Specifically, first, the EEG signal preprocessing and intention rough extraction are performed on the collected original EEG signals (such as motor imagery EEG, SSVEP, etc.), and then filtered and denoised (such as independent component analysis ICA to remove eye artifacts) to obtain "clean" EEG features. . Use a simple classifier (such as linear discriminant analysis LDA) to classify Perform "coarse classification of intent" to determine the type of task the user currently wants to perform (for example, "grab", "place", "move", and output a "task type label"). ,like represents a crawl task).

[0050] Then, perform task prior vector library matching and pre-build a task prior vector library ( ), the task prior vector library stores typical feature vectors of common tasks. For example: the prior vector of the grasping task (including fusion features such as EEG template of hand grasping movement, visual water cup features and joint angles of typical grasping movements); prior vector of placement task (Including fusion features such as the EEG template of the hand release movement and the visual features of the target position.) According to the "task type label" obtained in the previous example ,from Retrieve the initial task-related vector (e.g. When calling ).

[0051] Finally, combined with the dynamic optimization of real-time EEG, the real-time EEG features and ( Can be or etc.) to obtain the final task-related vector Q. The fusion method can adopt the weighted summation of the attention mechanism (or the Transformer modal attention layer calculation), as shown in the following formula: ; in, represents the learnable weight matrix, is the weight matrix of adaptive learning during model training, with dimension , Represented as real-time EEG features Vector related to the initial task The feature dimension after concatenation (needs to match the sum of the feature lengths of the two); Represented as the target dimension after linear transformation (usually consistent with the dimension of subsequent attention calculation or feature representation). Perform a linear transformation and map it to the high-dimensional space of "attention weight calculation", so that the model can more accurately capture the relationship between "real-time EEG intention" and "prior task characteristics".

[0052] is the attention weight function, in the formula The calculation logic is: first use the weight matrix W to combine the features Do a linear transformation to get the intermediate vector ;right Applying the Softmax function (i.e. , converts the vector elements into a probability distribution between 0 and 1 (all elements sum to 1).

[0053] The output is the attention weight corresponding to the real-time EEG feature, which reflects the importance of the "current real-time EEG intention" in the final task-related vector Q (the higher the weight, the stronger the guiding role of real-time EEG on the current task).

[0054] Specifically , as a complementary weight term, since the attention weight needs to satisfy the normalization constraint of "real-time EEG weight + initial task vector weight = 1", It is expressed as the attention weight corresponding to the initial task-related vector. Achieve a dynamic balance between "real-time EEG intention" and "prior task characteristics". When the weight is low due to factors such as noise, The weight of will increase (based on prior task knowledge to improve stability); when the real-time EEG intention is clear, The weight of will be reduced (prioritizing real-time intent and improving flexibility).

[0055] As a direct carrier of human intent, EEG is more capable of reflecting user intent than pure vision or touch. Pure real-time EEG may contain noise or ambiguous intent. The prior library provides basic task features, and dynamic optimization enables the task-related vector Q to stably and accurately represent the task.

[0056] The task-related vector Q obtained through the above process has the following beneficial effects: Accurately associating multimodal data: This allows the model to determine which modal data is more important for task execution (for example, in a grasping task, the "cup position" in visual data and the "grip force feedback" in tactile data will be given high weights due to their high similarity to Q). Adaptable to different task scenarios: Whether it is "grasping", "assembly" or "servicing" tasks, Q can dynamically adjust through "EEG + prior library" to improve the pertinence of multimodal fusion; Improve model interpretability: Q serves as a "task intent anchor" to improve the interpretability of attention weight calculation.

[0057] In some embodiments, the trained end-to-end model can be obtained through training from steps S301 to S304.

[0058] Step S301: obtaining a training data set, wherein the training data set includes a combination of a plurality of multimodal fusion data and a target action sequence; Step S302: input the multimodal fusion data into the end-to-end model to obtain a predicted action sequence; Step S303, calculating the total loss value according to the target action sequence and the predicted action sequence; Step S304: Adjust the end-to-end model parameters with the goal of minimizing the total loss value to obtain a trained end-to-end model.

[0059] In some embodiments, in steps S301 to S304, a training data set with a timestamp is generated by synchronously collecting the EEG signals of the user when performing the task, the tactile interaction data (pressure / temperature) between the robot and the environment, the scene visual image, and the robot joint angle data. ,in is multimodal fusion data, is the target action sequence. It can be obtained by data fusion as shown in formula (1), that is, for The end-to-end model adopts an improved Transformer architecture, including a modality attention layer, an encoder, and a decoder. The specific process of step S302 is shown in formula (3).

[0060] , (3); in, To predict action sequences, is the model parameter. and Calculate the total loss value, and The greater the difference, the greater the total loss value. By adjusting the model parameters , making Towards Close, that is, reducing the loss value and improving the prediction effect of the model.

[0061] In some embodiments, multimodal fusion data may be obtained through steps S310 to S320.

[0062] Step S310, obtaining the pre-processed second EEG signal, visual training data, tactile training data, and motion training data; Step S320, aligning the second EEG signal, the visual training data, the tactile training data, and the motion training data according to the timestamp; Step S330 , fusing the time-aligned second EEG signal, the visual training data, the tactile training data, and the motion training data to obtain multimodal fusion data.

[0063] In steps S310 to S330 of some embodiments, the second EEG signal, visual training data, tactile training data, and motion training data are the EEG signal of the user performing the task, the visual image of the scene, the tactile interaction data (pressure / temperature) between the robot and the environment, and the robot's joint angle data. The preprocessing process is as follows: Independent Component Analysis (ICA) is performed on the EEG to remove eyeball artifacts, wavelet transform is used to extract time domain features of the tactile training data, image features of the visual training data are extracted using a pre-trained ResNet50, and the motion training data is normalized. Because the training data of each modality is timestamped, the training data at the same time are aligned to facilitate subsequent data fusion. The specific data fusion process is shown in Equation (1).

[0064] In some embodiments, step S303 may include but is not limited to steps S331 to S335: Step S331, obtaining the error prediction probability output by the end-to-end model; Step S332, calculating a mean square error based on the target action sequence and the predicted action sequence to obtain a first loss value; Step S333, performing motion continuity calculation based on the predicted motion sequence to obtain a second loss value; Step S334, calculating based on the error prediction probability and the true error label in the training data set to obtain a third loss value; Step S335 , performing weighted summation on the first loss value, the second loss value, and the third loss value to obtain a total loss value.

[0065] In step S332 of some embodiments, the mean square error is calculated based on the target action sequence and the predicted action sequence as shown in formula (4).

[0066] , (4); in, represents the action parameters at different moments predicted by the end-to-end model, obtained by formula (3) Including multiple , Represents the target action sequence in the training dataset D The action parameters at different times, that is Including multiple , is the mean squared error (MSE) loss (first loss) between the predicted and true actions, where T is the length of the action sequence. The first loss represents the difference between the target and predicted action sequences. Reducing the first loss can make the model's predicted actions closer to the target action sequence, improving the prediction accuracy of the end-to-end model.

[0067] In step S333 of some embodiments, the action continuity calculation is performed according to the predicted action sequence as shown in formula (5).

[0068] , (5); in, and It is represented by the action parameters of adjacent moments predicted by the end-to-end model, for The action parameters of the next moment, The second loss value can characterize the continuity of the predicted action sequence. By reducing the second loss value, the continuity of the action sequence predicted by the end-to-end model can be improved.

[0069] In steps S331 and S334 of some embodiments, the error prediction probability may refer to the possibility of an error in controlling the robot to perform a task based on the current training data, that is, the possibility that the user will feedback an ErrP (error-related potential) signal. The specific calculation of the third loss value is shown in formula (6).

[0070] , (6); Among them, K represents the training data set of The number of k corresponds to different Serial number, is the ErrP (error-related potential) signal detection loss (third loss value), the true error label Represented as 0 or 1, is the error prediction probability. Reducing the third loss value can improve the ability of the end-to-end model to correct the predicted action sequence and better complete the task.

[0071] In step S335 of some embodiments, the total loss value is obtained by a dynamic loss function, and the specific calculation is shown in formula (7).

[0072] , (7); in, is the total loss value, 、 and Represent the weights of each loss value.

[0073] In some embodiments, the humanoid robot control method further includes but is not limited to steps S104 to S105: Step S104, receiving an error feedback signal according to the humanoid robot's execution of the action command; Step S105 , performing real-time correction on the parameters of the end-to-end model according to the error feedback signal.

[0074] In steps S104 to S105 of some embodiments, according to the situation of the humanoid robot executing the action instruction, an error feedback signal is generated through visual recognition and ErrP (error-related potential) detection, the error-related potential is automatically generated by the user's brain and fed back through the electroencephalogram signal (neurophysiological natural reaction), and when the error exceeds the threshold, the model fine-tuning is triggered, and the parameters of the end-to-end model are updated by incremental learning.

[0075] Illustratively, in the scene of the user controlling the robot to grab a water cup through the electroencephalogram signal, when the user observes that the robot fails to grab the water cup, the brain will naturally generate an error-related potential (ErrP), which is transmitted to the system after being collected by the EEG module. The robot synchronously acquires the task execution image through the visual module, analyzes the distance error between the finger joint and the water cup (when exceeding the set threshold value such as 5 mm), and combines the pressure error of the hand tactile sensor (when the pressure value is lower than the ideal grip threshold value by 20%), and encodes the two types of errors into a multi-modal loss function: ; Wherein, represents the visual error loss term, is the deviation value of the "actual distance-ideal grabbing distance" (such as positive error when exceeding the threshold value of 5 mm), calculated by mean square error (MSE) or distance loss function, to punish the deviation of visual perception.

[0076] represents the tactile error loss term, is the difference value of the "actual pressure-ideal grip" (such as negative deviation when the pressure is insufficient), also calculated by MSE or force feedback loss function, to constrain the accuracy of tactile feedback.

[0077] represents the electroencephalogram error loss term, which encodes the loss by the amplitude or duration of the electroencephalogram signal to reflect the deviation between the user's intention and the execution result, for example, if the ErrP is stronger / longer, the larger.

[0078] α, β, γ represent weight coefficients. According to the intensity of the ErrP signal, the weight coefficients are dynamically adjusted (such as increasing so that the electroencephalogram feedback dominates the loss; when the vision is blocked, the weight coefficient is reduced to weaken the unreliable visual error), so as to realize adaptive weighting of multi-modal errors in different scenes and realize adaptive adjustment.

[0079] Through back propagation incremental learning, the model is optimized specifically as follows: Adjust the cross-modal attention weight of the Transformer encoder (when the distance exceeds the threshold, increase the attention weight of the visual features to more than 60%); correct the joint angle output weight of the MLP action prediction layer (based on pressure deviation, compensate for the predicted value of finger grip strength).

[0080] In dynamic scenes such as multiple people walking, if visual misjudgment is caused by pedestrian occlusion, the system can freeze the background region weight of visual spatial attention (ignore the occlusion and focus on the cup outline) based on the duration and amplitude of ErrP (short duration and weak ErrP (<300ms)); when the duration is long and the amplitude is strong (≥300ms), the haptic-EEG dual-mode calibration is triggered (use pressure feedback to reconstruct the grasping trajectory and get rid of visual dependence).

[0081] By this way of real-time correction of model parameters, the robot has real-time adaptive learning ability, which can dynamically adjust model parameters through interactive feedback, significantly improve the task execution accuracy and human-robot interaction naturalness of humanoid robots in complex scenes, and greatly improve the anti-interference ability and robustness of robots in dynamic environments (such as multiple people walking).

[0082] In some embodiments, the user initiates a task (such as "grasp the cup") through brain electrical signals, and the robot system synchronously collects haptic (pressure when contacting the cup), visual (cup position), and motion (arm movement trajectory) data, and the end-to-end model directly outputs the finger joint angle. If an ErrP (error-related potential) signal is detected (the user realizes that the grasping fails), the model parameters are corrected through a dynamic loss function L, which is shown in equation (7).

[0083] In some embodiments, the end-to-end model training process is as follows: 10 subjects wear headgear and imagine "grasp / place" actions, and brain electrical signals are recorded synchronously; the robot performs 500 grasping tasks in a home scene, and collects haptic (contacting the table / cup), visual (cup at different positions), and motion data.

[0084] Model training: data set division: 80% training set, 20% validation set; hyperparameters: 6 layers of Transformer, 8 attention heads, learning rate 1e-4, 100 training rounds; dynamic loss weight: initial , later to enhance error correction ability.

[0085] In some embodiments, the end-to-end model inference delay is <100ms, and when the ErrP (error-related potential) detection module detects a negative potential (NRE>5μV) within 0.5s after the instruction is executed, the model fine-tuning is triggered (learning rate is reduced to 1e-5, and 10 batches are updated).

[0086] In some embodiments, please refer to Figure 2 , a humanoid robot control system, including a multimodal data acquisition module, a data preprocessing and fusion module, an end-to-end training module, a real-time control module and an execution feedback module; The multimodal data acquisition module includes an EEG (electroencephalogram) + FNIRS (functional near-infrared spectroscopy) synchronous acquisition head cap (containing 8-channel electrodes and 4 pairs of near-infrared probes), a 16-point tactile sensor array distributed on the robot's hand / torso (sampling frequency 1kHz), a binocular vision sensor (resolution 1920×1080, frame rate 30fps), and a 6-axis inertial measurement unit (sampling frequency 200Hz); For the collected EEG signals, the EEG (electroencephalogram) + FNIRS (functional near-infrared spectroscopy) synchronous acquisition head cap adopts a timestamp synchronization mechanism. The EEG (electroencephalogram) signal is notch filtered at 50Hz to remove power frequency noise, and a 0.1-30Hz bandpass filter is used to retain the effective signals of motor imagery and SSVEP (steady-state visual evoked potential) (SNR ≥ 20dB after processing). After conversion, the FNIRS signal must meet the detection accuracy of ΔHbO / ΔHbR ≤ 0.1μmol / L.

[0087] The data preprocessing and fusion module receives multimodal data via Ethernet, optical fiber or wireless WIFI, performs noise reduction, spatiotemporal alignment and feature mapping, and outputs a fusion feature vector with unified dimension; The end-to-end training module uses an improved Transformer architecture, including a modal attention layer and an action prediction decoder, and optimizes model parameters through a dynamic loss function. The real-time control module converts the motion instructions output by the training model into robot joint drive signals, and the execution feedback module generates error signals through visual and tactile data for online model fine-tuning. The execution feedback module generates an error signal by calculating visual errors (such as object grasping position offset > 5mm) and tactile errors (such as contact pressure deviation > 1N) in real time. When the error signal exceeds the preset threshold, it is transmitted via wire to the end-to-end training module to trigger parameter fine-tuning.

[0088] In some embodiments, please refer to Figure 3 The multimodal data acquisition module includes an EEG unit, a tactile unit, a visual unit and a movement unit. The EEG unit is an EEG (electroencephalogram) + FNIRS (functional near-infrared spectroscopy) head cap, which is used to collect motor imagery (left hand / right hand / tongue) and SSVEP (steady-state visual evoked potential) signals with a sampling rate of 250Hz; the 8-channel electrodes are located at the Fz, Cz, Pz, O1, O2, T7, T8, and Oz positions of the international 10-20 system, which are used to collect EEG signals from the frontal lobe to the occipital lobe.

[0089] The tactile unit includes a flexible sensor array on the robot's hand / elbow, which collects pressure (0-10N) and temperature (20-40℃); the 16-point tactile sensor array includes 12 points on the hand (2 points on the thumb, 4 points on the index finger, 4 points on the middle finger, and 2 points on the palm), and 4 points on the elbow (2 points each on the internal and external condyles of the humerus). The sampling frequency is 1kHz, and the pressure detection accuracy is ±0.1N.

[0090] The vision unit includes a binocular camera and a depth sensor to obtain RGB images and depth information.

[0091] The action unit (motion capture unit) includes joint encoders and IMUs, which are used to record the robot's joint angles (finger angles, arm angles, etc.) and 6-DOF posture data.

[0092] In some embodiments, please refer to Figure 4 The data preprocessing and fusion module uses timestamps to align multimodal data and implements feature fusion through Equation (1). The attention weight is dynamically adjusted (e.g., increasing the tactile weight when performing fine manipulation). Timestamp synchronization uses GPS timing + local crystal oscillator calibration with a synchronization accuracy of ≤1ms, ensuring that the time deviation of EEG signals, tactile signals, visual signals, and motion signals is within 0.5ms.

[0093] In some embodiments, please refer to Figure 5 、 Figure 6 and Figure 7 The end-to-end training module is based on the Transformer encoder-decoder. The encoder processes the fused features and the decoder outputs the joint angle sequence. The encoder first learns the complex associations between multimodal environmental features through a multi-head self-attention mechanism, then performs feature enhancement and transformation through a feedforward network, and finally outputs an environmental representation that incorporates global context information. The decoder first models the dependencies within the generated action sequence through masked self-attention, then uses the encoder-decoder attention layer to make key associations between the current action and the environmental features provided by the encoder, and finally outputs specific action instructions through a feedforward network. Residual connections and layer normalization improve the stability and efficiency of training, enabling the model to generate reasonable and coherent action sequences based on the environmental state.

[0094] In some embodiments, the dynamic loss function is as shown in Formula (7), which is obtained by Balancing motion accuracy, error correction, and motion smoothness. Figure 8 Shows the training process The changing trend of The initial value is 3.2, The initial value is 2.8, The initial value is 1.5, and the initial value of total loss is 5.0. After a certain number of training rounds (0-100 rounds), the total loss value is reduced.

[0095] This application has at least the following beneficial effects: 1) The task execution accuracy is significantly improved; 2) The training period is significantly shortened; 3) The anti-interference capability is enhanced, and the robustness in dynamic environments (such as multiple people walking) is greatly improved.

[0096] The present application also provides a humanoid robot control system that can implement the above method. The system includes: an acquisition module, configured to acquire real-time visual data, real-time tactile data, and real-time motion data of the humanoid robot in response to a first EEG signal of a target user; a fusion module, configured to fuse the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain real-time fused data; The execution module is used to input real-time fusion data into the trained end-to-end model and output action instructions to control the humanoid robot to perform actions according to the action instructions.

[0097] It can be understood that the contents of the above method embodiments are applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0098] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0099] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0100] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes: The processor 901 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application. The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the above-mentioned methods of the embodiments of this application. Input / output interface 903, used to implement information input and output; Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 ); The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0101] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above method is implemented.

[0102] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0103] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0104] It is understandable that the contents of the above method embodiments are all applicable to the present program product embodiments, the functions specifically implemented by the present program product embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0105] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0106] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0107] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0109] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0110] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0111] It should be understood that in this application, "at least one (item)" means one or more, and "more" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0116] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for controlling a humanoid robot, characterized in that: The method comprises the following steps: In response to the first EEG signal of the target user, collecting real-time visual data, real-time tactile data, and real-time motion data of the humanoid robot; fusing the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain real-time fused data; The real-time fusion data is input into a trained end-to-end model, and action instructions are output to control the humanoid robot to perform actions according to the action instructions.

2. The method according to claim 1, characterized in that The step of fusing the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain real-time fused data includes: performing nonlinear processing on the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data, respectively, to obtain corresponding EEG features, visual features, tactile features, and motion features; The EEG feature, the visual feature, the tactile feature, and the motion feature are respectively multiplied by corresponding attention weights and added to obtain the real-time fusion data.

3. The method according to claim 2, characterized in that The attention weights are obtained by the following steps: Get task-related vectors; performing similarity calculations on the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data respectively according to the task-related vector to obtain respective corresponding similarities; A weight calculation is performed based on the similarity to obtain the corresponding attention weights.

4. The method according to claim 1, wherein The trained end-to-end model is obtained by training through the following steps: Acquire a training data set, wherein the training data set includes a combination of a plurality of multimodal fusion data and a target action sequence; Inputting the multimodal fusion data into the end-to-end model to obtain a predicted action sequence; Calculating a total loss value according to the target action sequence and the predicted action sequence; With the goal of minimizing the total loss value, the end-to-end model parameters are adjusted to obtain a trained end-to-end model.

5. The method according to claim 4, characterized in that The multimodal fusion data is obtained by the following steps: Acquiring the preprocessed second EEG signal, visual training data, tactile training data, and motion training data; Aligning the second EEG signal, the visual training data, the tactile training data, and the motion training data according to the timestamp; The time-aligned second EEG signal, the visual training data, the tactile training data, and the motion training data are fused to obtain the multimodal fusion data.

6. The method according to claim 4, characterized in that The calculating the total loss value according to the target action sequence and the predicted action sequence includes: Obtaining an error prediction probability output by the end-to-end model; Calculating a mean square error based on the target action sequence and the predicted action sequence to obtain a first loss value; Performing action continuity calculation according to the predicted action sequence to obtain a second loss value; Calculating based on the error prediction probability and the true error label in the training data set to obtain a third loss value; A weighted sum is performed on the first loss value, the second loss value, and the third loss value to obtain the total loss value.

7. The method according to claim 1, characterized in that The method further comprises: receiving an error feedback signal according to the humanoid robot executing the action instruction; The parameters of the end-to-end model are modified in real time according to the error feedback signal.

8. A humanoid robot control system, characterized in that: The system comprises: an acquisition module, configured to acquire real-time visual data, real-time tactile data, and real-time motion data of the humanoid robot in response to a first EEG signal of a target user; a fusion module, configured to fuse the first EEG signal, the real-time visual data, the real-time tactile data, and the real-time motion data to obtain real-time fused data; The execution module is used to input the real-time fusion data into the trained end-to-end model and output action instructions to control the humanoid robot to perform actions according to the action instructions.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images