Adaptive Control Method, System, Electronic Device and Storage Medium for Humanoid Robot

Abstract

An embodiment of the present application provides a humanoid robot adaptive control method, system, electronic device and storage medium, belonging to the technical field of robot intelligent control. The method collects the electroencephalogram (EEG) signal and electromyogram (EMG) signal of a target object; determines the first time-frequency domain feature of the EEG signal according to the EEG signal, and determines the second time-frequency domain feature of the muscle activation state according to the EMG signal; fuses the first time-frequency domain feature and the second time-frequency domain feature by using a dynamic weighting algorithm to obtain a multi-modal feature; inputs the multi-modal feature into an action intention recognition model to obtain a predicted action intention; and determines a robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot. The present application can improve the accuracy of adaptively controlling a humanoid robot.

Description

Technical Field

[0001] This application relates to the field of robot intelligent control technology, and particularly to a humanoid robot adaptive control method, system, electronic device, and storage medium. Background Art

[0002] Existing robot control technologies have many limitations. The traditional manual control method relies on devices such as joysticks and buttons, requiring operators to have proficient operation skills, and it is difficult to achieve autonomous control in this way. Although the preset program control can execute some established tasks, it lacks adaptability to environmental changes and real-time human motion intentions and cannot flexibly adjust the robot's actions.

[0003] In related technologies, the robot can be adaptively controlled based on the brain-computer interface. However, due to the brain electrical signals being easily affected by external interference, weak signals, and difficult feature extraction, the control accuracy is low and it is difficult to meet the actual application requirements. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a humanoid robot adaptive control method, system, electronic device, and storage medium, aiming to improve the accuracy of adaptively controlling humanoid robots.

[0005] To achieve the above object, on the one hand, an embodiment of this application proposes a humanoid robot adaptive control method, including the following steps:

[0006] Collect the electroencephalogram (EEG) signals and electromyogram (EMG) signals of the target object;

[0007] Determine the first time-frequency domain feature of the EEG signal according to the EEG signal, and determine the second time-frequency domain feature of the muscle activation state according to the EMG signal;

[0008] Adopt a dynamic weighting algorithm to fuse the first time-frequency domain feature and the second time-frequency domain feature to obtain a multi-modal feature;

[0009] Input the multi-modal feature into an action intention recognition model to obtain a predicted action intention;

[0010] Determine a robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot.

[0011] In some embodiments, the step of determining the first time-frequency domain feature of the EEG signal according to the EEG signal includes the following steps:

[0012] Perform multi-level adaptive filtering and blind source separation processing on the EEG signal to obtain a first target signal;

[0013] Perform continuous wavelet transform on the first target signal based on wavelet basis functions and scale parameters to obtain wavelet coefficients and center frequencies at different scales;

[0014] Extract the energy entropy of the target frequency based on the wavelet coefficients and center frequencies at different scales to obtain the first time-frequency domain feature.

[0015] In some embodiments, the second time-frequency domain feature for determining the muscle activation state based on the electromyogram signal includes the following steps:

[0016] Determine the muscle activation state at the corresponding moment based on the electromyogram signal;

[0017] Perform noise reduction processing on the electromyogram signal at the corresponding moment according to the noise threshold under the muscle activation state to obtain a second target signal;

[0018] Extract the information entropy of the second target signal at different frequency components to obtain the second time-frequency domain feature of the muscle activation state.

[0019] In some embodiments, the step of fusing the first time-frequency domain feature and the second time-frequency domain feature by using a dynamic weighting algorithm to obtain a multi-modal feature includes the following steps:

[0020] Use the dynamic weighting algorithm to calculate the first weight of the first time-frequency domain feature and the second weight of the second time-frequency domain feature respectively;

[0021] Multiply the first weight by the first time-frequency domain feature to obtain an electroencephalogram feature, and multiply the second weight by the second time-frequency domain feature to obtain an electromyogram feature;

[0022] Perform time alignment on the electroencephalogram feature and the electromyogram feature and then splice them to obtain a multi-modal feature.

[0023] In some embodiments, the step of using the dynamic weighting algorithm to calculate the first weight of the first time-frequency domain feature and the second weight of the second time-frequency domain feature respectively includes the following steps:

[0024] Determine the electroencephalogram signal quality based on the signal power of the electroencephalogram signal in the target frequency band and the electromyogram resting noise power, and determine the electromyogram signal quality based on the signal power of the electromyogram signal and the electromyogram resting noise power, where the electromyogram resting noise power represents the baseline noise of the electromyogram signal in the resting state;

[0025] Determine the first weight of the first time-frequency domain feature according to the ratio of the electroencephalogram signal quality to the total signal quality, and determine the second weight of the second time-frequency domain feature according to the ratio of the electromyogram signal quality to the total signal quality.

[0026] In some embodiments, inputting the multi-modal features into an action intention recognition model to obtain a predicted action intention includes the following steps:

[0027] Input the multi-modal features into a dilated convolutional layer and a bidirectional long short-term memory network respectively to obtain multi-scale rhythm features and muscle activation timing pattern features correspondingly;

[0028] Adopt a cross-modal attention mechanism to dynamically adjust the contribution degrees of the multi-scale rhythm features and the muscle activation timing pattern features;

[0029] Determine the predicted action intention according to the multi-scale rhythm features and their contribution degrees and the muscle activation timing pattern features and their contribution degrees.

[0030] In some embodiments, determining a robot control instruction according to the predicted action intention and the current state of the robot includes the following steps:

[0031] Determine a target action template of a corresponding motion primitive according to the predicted action intention, where the target action template is used to describe the kinematic parameters and dynamic parameters required for the target action;

[0032] According to the current state of the robot, calculate the parameter control amount described by the target action template through an inverse kinematics algorithm to obtain a robot control instruction.

[0033] To achieve the above object, another aspect of the embodiments of the present application proposes a humanoid robot adaptive control system, including:

[0034] A first module, configured to collect electroencephalogram signals and electromyogram signals of a target object;

[0035] A second module, configured to determine first time-frequency domain features of the electroencephalogram signals according to the electroencephalogram signals, and determine second time-frequency domain features of the muscle activation state according to the electromyogram signals;

[0036] A third module, configured to fuse the first time-frequency domain features and the second time-frequency domain features by using a dynamic weighting algorithm to obtain multi-modal features;

[0037] A fourth module, configured to input the multi-modal features into an action intention recognition model to obtain a predicted action intention;

[0038] A fifth module, configured to determine a robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot.

[0039] To achieve the above object, on the other hand, an embodiment of the present application provides an electronic device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing the connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the above embodiment is realized.

[0040] To achieve the above object, on the other hand, an embodiment of the present application provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the method described in the above embodiment.

[0041] The humanoid robot adaptive control method, system, electronic device and storage medium provided by the present application collect the electroencephalogram (EEG) signal and electromyogram (EMG) signal of a target object, determine the first time-frequency domain feature of the EEG signal according to the EEG signal, and determine the second time-frequency domain feature of the muscle activation state according to the EMG signal. The dynamic weighted algorithm is used to fuse the first time-frequency domain feature and the second time-frequency domain feature to obtain the multi-modal feature. The multi-modal feature is input into the action intention recognition model to obtain the predicted action intention, and the robot control instruction is determined according to the predicted action intention and the current state of the robot to control the humanoid robot. The present application combines the EMG signal on the basis of the EEG signal to predict the action intention, and uses the dynamic weighted algorithm to consider the contribution degree of different signals when input into the prediction action model, improving the accuracy of the recognition of the control action intention, and thus improving the accuracy of the adaptive control of the humanoid robot. Description of the Drawings

[0042] Figure 1 is a flowchart of the humanoid robot adaptive control method provided by an embodiment of the present application;

[0043] Figure 2 is a flowchart of the multi-modal feature acquisition method provided by an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of the training process of the action intention recognition model provided by an embodiment of the present application;

[0045] Figure 4 is a schematic diagram of the robot instruction generation method provided by an embodiment of the present application;

[0046] Figure 5 is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present application. Detailed Embodiments

[0047] In order to make the objectives, technical solutions, and advantages of the present application more clearly understood, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] It should be noted that although functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order from the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims, and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] The embodiments of the present application provide a humanoid robot adaptive control method, system, electronic device, and storage medium, aiming to improve the accuracy of adaptively controlling a humanoid robot.

[0051] The humanoid robot adaptive control method, system, electronic device, and storage medium provided by the embodiments of the present application will be specifically described through the following embodiments. First, the humanoid robot adaptive control method in the embodiments of the present application will be described.

[0052] The humanoid robot adaptive control method provided by the embodiments of the present application relates to the technical field of robot intelligent control. The humanoid robot adaptive control method provided by the embodiments of the present application can be applied to a terminal, a server, or software running on a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the humanoid robot adaptive control method, etc., but is not limited to the above forms.

[0053] This application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0054] Figure 1 is an optional flowchart of the humanoid robot adaptive control method provided by an embodiment of this application. Figure 1 The method in may include but is not limited to steps S101 to S105.

[0055] Step S101, collect the electroencephalogram (EEG) signal and electromyogram (EMG) signal of the target object;

[0056] Step S102, determine the first time-frequency domain feature of the EEG signal according to the EEG signal, and determine the second time-frequency domain feature of the muscle activation state according to the EMG signal;

[0057] Step S103, fuse the first time-frequency domain feature and the second time-frequency domain feature using a dynamic weighting algorithm to obtain a multi-modal feature;

[0058] Step S104, input the multi-modal feature into an action intention recognition model to obtain a predicted action intention;

[0059] Step S105, determine a robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot.

[0060] In step S101 of some embodiments, the electroencephalogram (EEG) is an electrical signal generated by the activities of brain neurons. It is time-series data recorded by electrodes, reflecting the potential changes caused by the synchronous activities of a large number of neurons in the brain. The electromyogram, namely the surface electromyogram (sEMG), is a muscle electrical activity signal recorded by electrodes placed on the skin surface. It reflects the electrical activities of motor units (MUs) in the muscle and is an important tool for studying muscle function, movement control, and rehabilitation medicine. A muscle consists of multiple motor units, and each motor unit includes a motor neuron and the muscle fibers it innervates. When the motor neuron is activated, the muscle fibers generate action potentials, resulting in muscle contraction. Therefore, by analyzing the action potentials of muscle fibers, the actions that the target object wants to perform can be confirmed. The acquisition process of EEG and sEMG signals can be to arrange a multi-channel EEG sensing array on the human scalp using a standardized electrode layout method and deploy distributed sEMG sensing units on the surface of the target muscle groups. The time series of EEG and sEMG signals are made consistent through signal alignment. Specifically, the signal alignment method can be the hardware clock synchronization alignment method or a combination of hardware clock synchronization and software resampling, and the alignment accuracy is at the millisecond level.

[0061] Exemplarily, a multi-channel EEG sensing array is arranged on the scalp of the target object using a standardized electrode layout method. When arranging, it can follow the international 10-20 system standard. Before pasting the electrodes, the scalp is cleaned to remove grease and dirt to ensure good contact between the electrodes and the scalp and improve the signal acquisition quality. Distributed sEMG sensing units are deployed on the surface of the target muscle groups, specifically including the biceps brachii, triceps brachii in the upper limb, and the quadriceps femoris, triceps surae in the lower limb and other muscle groups related to movement. After cleaning the skin, the sEMG sensing units are deployed according to the distributed layout, and the sensing units are firmly pasted to avoid electrode detachment or signal interference caused by movement. During the signal acquisition process, the time series of EEG and sEMG signals are made consistent through a hardware synchronization mechanism, and the target object can be kept in a relaxed state to avoid large movements and electromagnetic interference. The sampling frequency can be 1000 Hz.

[0062] In step S102 of some embodiments, the first time-frequency domain feature refers to the time-domain and frequency-domain composite feature reflecting the signal dynamic characteristics extracted from the collected EEG signals. Specifically, the time series of the EEG signals can be directly analyzed, such as time-domain features like mean, variance, peak value, etc., the frequency-domain features of the frequency band energy can be analyzed using Fourier transform (FFT), and the time-frequency characteristics can also be analyzed using short-time Fourier transform (STFT) or continuous wavelet transform (CWT). The second time-frequency domain feature refers to the time-frequency domain feature related to the muscle activation state extracted from the EMG signals. Specifically, the time series of the EMG signals can be directly analyzed, such as time-domain features like mean, variance, peak value, etc., the frequency-domain features of the frequency band energy can be analyzed using Fourier transform (FFT), and the time-frequency characteristics can also be analyzed using short-time Fourier transform (STFT) or continuous wavelet transform (CWT).

[0063] According to some embodiments of the present application, in step S102, the step of determining the first time-frequency domain feature of the EEG signals based on the EEG signals may include, but is not limited to, the following steps:

[0064] Step S201, perform multi-level adaptive filtering and blind source separation processing on the EEG signals to obtain a first target signal;

[0065] Step S202, perform continuous wavelet transform on the first target signal based on the wavelet basis function and scale parameter to obtain wavelet coefficients and central frequencies at different scales;

[0066] Step S203, extract the energy entropy of the target frequency according to the wavelet coefficients and central frequencies at different scales to obtain the first time-frequency domain feature.

[0067] In this embodiment, the collected EEG signals may be interfered by physiological noise and environmental noise. Physiological noise such as electrooculogram (EOG), electromyogram (EMG), electrocardiogram (ECG), etc., and environmental noise interference such as power line interference, etc. Therefore, this embodiment can adopt multi-level adaptive filtering technology to dynamically suppress the low-frequency drift and high-frequency noise interference of the EEG signals, then construct an artifact component feature library, and eliminate physiological (electrooculogram, electrocardiogram) artifacts through blind source separation technology, so as to obtain a relatively pure EEG signal, that is, the first target signal.

[0068] The multi - stage adaptive filtering technology is a signal processing technology. By cascading or paralleling multiple adaptive filters, the filtering effect is gradually optimized to better process complex signal and noise environments. Among them, the adaptive filter can dynamically adjust its cut - off frequency according to the characteristics of the input signal to minimize the error signal (usually the difference between the desired signal and the filter output). Through the multi - stage adaptive filtering technology, efficient denoising and signal separation of electroencephalogram (EEG) signals can be achieved. For example, the dynamic range of the cut - off frequency of the multi - stage adaptive filter can be designed as 0.5 - 100 Hz. In the low - frequency band (0.5 - 4 Hz), it focuses on suppressing low - frequency drift, and in the high - frequency band (30 - 100 Hz), it removes high - frequency noise interference.

[0069] The blind source separation technology is a signal processing technology aimed at recovering the original source signals from the mixed signals without prior knowledge of the mixing process or prior information about the source signals. In this embodiment, an artifact component feature library is constructed to collect common artifact features such as electrooculogram (EOG) and electrocardiogram (ECG). Then, using the blind source separation technology, for example, the independent component analysis (ICA) algorithm, the EEG signal is decomposed into multiple independent components. By comparing with the artifact component feature library, physiological artifacts are identified and removed.

[0070] After pre - processing the EEG signal to obtain the first target signal, feature extraction is performed on the first target signal. The frequency range of the EEG signal is mainly distributed between 0.5 Hz and 100 Hz, and is divided into multiple frequency bands: the δ - wave (Delta) frequency is 0.5 - 4 Hz, which is related to deep sleep; the θ - wave (Theta) frequency is 4 - 8 Hz, which is related to light sleep and meditation; the α - wave (Alpha) frequency is 8 - 13 Hz, which is related to relaxation and closed - eye state; the β - wave (Beta) frequency is 13 - 30 Hz, which is related to wakefulness and concentration; the γ - wave (Gamma) frequency is 30 - 100 Hz, which is related to higher cognitive functions. In this embodiment, continuous wavelet transform (CWT) can be used to analyze the pre - processed EEG signal (i.e., the first target signal), extract the energy entropy of the α and β frequency bands, so as to obtain the first time - frequency domain feature. When performing continuous wavelet transform on the time - window segment of the first target signal, an appropriate wavelet function (Morlet wavelet) and its scale parameter can be selected for processing to obtain wavelet coefficients of different frequencies at different time scales, and then feature extraction is performed according to the wavelet coefficients to accurately capture the energy changes of the EEG signal at different frequencies and time scales.

[0071] In one example, the energy entropy of the α (8 - 13 Hz) and β (13 - 30 Hz) frequency bands is extracted by continuous wavelet transform (CWT) The calculation formula is as follows:

[0072] ;

[0073] ;

[0074] Among them, represents the wavelet coefficient of the continuous wavelet transform at the i th scale (i.e., corresponding to a specific frequency), characterizes the energy at the i th scale, characterizes the proportion of the energy of the i th frequency band in the total energy, that is, the normalized energy distribution probability, N represents the number of frequency scales after wavelet transform.

[0075] According to some embodiments of the present application, in step S102, the step of determining the second time-frequency domain feature of the muscle activation state according to the electromyogram signal may include, but is not limited to, the following steps:

[0076] Step S301, determining the muscle activation state at the corresponding moment according to the electromyogram signal;

[0077] Step S302, performing noise reduction processing on the electromyogram signal at the corresponding moment according to the noise threshold in the muscle activation state to obtain a second target signal;

[0078] Step S303, extracting the information entropy of the second target signal at different frequency components to obtain the second time-frequency domain feature of the muscle activation state.

[0079] In this embodiment, performing time-varying threshold noise reduction processing on the electromyogram signal, dynamically adjusting the noise threshold according to the muscle activation state, and then using the adjusted noise threshold to perform noise reduction on the electromyogram signal can not only remove noise, but also better retain the muscle activation characteristics in the electromyogram signal. Specifically, determining the muscle activation state at the corresponding moment according to the electromyogram signal, when the muscle is at rest, reducing the threshold (frequency threshold or amplitude threshold), and then filtering out the part smaller than the threshold from the electromyogram signal to remove minute noise; when the muscle contracts, appropriately increasing the threshold, and then filtering out the part smaller than the threshold from the electromyogram signal to retain the effective electromyogram characteristics, thereby obtaining the second target signal. This implementation can judge the muscle activation state by analyzing the amplitude and frequency changes of the electromyogram signal. For example, if the amplitude value of the electromyogram signal at a certain moment is greater than the preset value, it is considered that the muscle activation state at the current moment is the activation state, otherwise it is the non-activation state (i.e., the resting state).

[0080] After performing the above preprocessing on the collected electromyogram signal to obtain the second target signal, further calculating the integrated electromyogram value (iEMG) and other second time-frequency domain features related to the muscle activation state according to the second target signal. Exemplarily, extracting the wavelet-Shannon entropy of the second target signal and analyzing the information entropy of the electromyogram signal at different frequency components to obtain richer muscle activation characteristics.

[0081] In step S103 of some embodiments, after determining the first time-frequency domain feature of the electroencephalogram signal and the second time-frequency domain feature of the electromyogram signal, the weights of the first time-frequency domain feature and the second time-frequency domain feature can be dynamically allocated to form multimodal features. In the feature fusion process, a dynamic weighting algorithm can be used to determine the weights of each feature, and then the feature fusion can be performed in combination with the weight values, so that the model can pay more attention to important features in the follow-up, thereby improving the accuracy of action intention recognition. The dynamic weighting algorithm mainly allocates the weights of the corresponding time-frequency domain features according to the signal characteristics of the electroencephalogram signal and the electromyogram signal itself. The signal characteristics can include but are not limited to signal intensity, signal quality, information richness, etc. For example, if the electromyogram signal has not changed for a long time or has a small change amplitude, it is considered that the information richness contained in the signal is low, and the weight of the signal is appropriately reduced. It can be understood that in the process of allocating feature weights, the weights can be calculated from one aspect of signal intensity, signal quality, and information richness, or the weights can be calculated comprehensively by combining multiple aspects of signal intensity, signal quality, and information richness.

[0082] According to some embodiments of the present application, in step S103, the steps of using a dynamic weighting algorithm to fuse the first time-frequency domain feature and the second time-frequency domain feature to obtain multimodal features may include but are not limited to the following steps:

[0083] Step S401, using a dynamic weighting algorithm to calculate the first weight of the first time-frequency domain feature and calculate the second weight of the second time-frequency domain feature respectively;

[0084] Step S402, multiplying the first weight by the first time-frequency domain feature to obtain an electroencephalogram feature, and multiplying the second weight by the second time-frequency domain feature to obtain an electromyogram feature;

[0085] Step S403, performing time alignment on the electroencephalogram feature and the electromyogram feature and then splicing them to obtain multimodal features.

[0086] In this embodiment, after using a dynamic weighting algorithm to calculate the first weight of the first time-frequency domain feature and calculate the second weight of the second time-frequency domain feature respectively, multiplying the first weight by the first time-frequency domain feature to obtain an electroencephalogram feature, and multiplying the second weight by the second time-frequency domain feature to obtain an electromyogram feature, and then performing time alignment on the electroencephalogram feature and the electromyogram feature and splicing them to obtain multimodal features, that is, splicing the electroencephalogram feature and the electromyogram feature corresponding to the same time window segment to obtain the multimodal feature of this time window segment, and then inputting the multimodal features of each time window segment into the action intention recognition model in sequence according to the time order for prediction. The electroencephalogram (EEG) and electromyogram (sEMG) multimodal features after dynamic weight fusion have a dimension of D = Deeg + DsEMG.

[0087] According to some embodiments of the present application, in step S401, when calculating the first weight of the first time-frequency domain feature and the second weight of the second time-frequency domain feature by using a dynamic weighting algorithm, it may include but is not limited to the following steps:

[0088] Step S501, determining the quality of the EEG signal according to the signal power of the EEG signal in the target frequency band and the power of the sEMG resting noise, and determining the quality of the EMG signal according to the signal power of the EMG signal and the power of the sEMG resting noise, where the power of the sEMG resting noise represents the baseline noise of the EMG signal in the resting state;

[0089] Step S502, determining the first weight of the first time-frequency domain feature according to the ratio of the EEG signal quality to the total signal quality, and determining the second weight of the second time-frequency domain feature according to the ratio of the EMG signal quality to the total signal quality.

[0090] In this embodiment, the weights of the corresponding time-frequency domain features can be adjusted according to the signal qualities of the EEG signal and the EMG signal.

[0091] The quality of the EEG signal can be determined by converting the ratio of the signal power of the EEG signal in the target frequency band to the power of the sEMG resting noise into decibel values. Specifically, the quality of the EEG signal can be calculated by the following formula:

[0092] ;

[0093] The quality of the EMG signal can be determined by converting the ratio of the signal power of the EMG signal to the power of the sEMG resting noise into decibel values. Specifically, the quality of the EMG signal can be calculated by the following formula:

[0094] ;

[0095] In the above formula, is the signal power of the EEG signal in the δ frequency band (0.5 - 4 Hz), is the signal power of the EMG signal in the frequency band related to movement; is the power of the sEMG resting noise, which is taken from the baseline of the EMG signal in the resting state.

[0096] In this embodiment, the sEMG resting noise is used as the environmental noise benchmark to analyze the signal-to-noise ratios of the EEG signal and the EMG signal, realizing the dynamic evaluation of the qualities of the EEG signal and the EMG signal, thereby optimizing the weight allocation of multi-modal feature fusion. In this embodiment, the sEMG resting noise (sEMG noise) is used as the noise benchmark for electroencephalogram (EEG), rather than the traditional self-noise calculation. The reason why this dynamic weight allocation method can effectively improve the anti-interference ability and robustness of the system in practical applications is as follows:

[0097] The electroencephalogram (EEG) self-noise is difficult to measure directly. The noise of EEG signals mainly comes from physiological artifacts (such as electrooculogram, electrocardiogram) and environmental interference. However, these noises are mixed with the EEG signals themselves and are difficult to separate. Traditional methods need to separate the noise through complex techniques such as independent component analysis (ICA), which has a high computational cost and unstable results. The electromyogram (EMG) resting noise is stable, and the baseline noise of EMG signals in the resting state (such as at the μV level) is very stable, which can be used as a reliable environmental noise benchmark. Therefore, the embodiments of this application adopt a cross-modal noise compensation mechanism. That is, when environmental noise (such as electromagnetic interference) affects both EEG and EMG signals simultaneously, the EMG resting noise can reflect the current environmental noise level, thereby more accurately evaluating the quality of EEG signals.

[0098] The first weight of the first time-frequency domain feature of the EEG signal and the second weight of the second time-frequency domain feature of the EMG signal are calculated respectively by the following formulas:

[0099] ;

[0100] ;

[0101] where is used to prevent division-by-zero errors, .

[0102] According to some embodiments of this application, please refer to Figure 2 , the multi-modal features for inputting into the action intention recognition model are obtained through the following process: Refer to the electrode layout schematic diagram to arrange electrodes on the scalp and muscle groups, then synchronously collect EEG signals and EMG signals, perform multi-level adaptive filtering and artifact elimination processing on the EEG signals in sequence, perform time-varying threshold noise reduction and muscle activation detection processing on the EMG signals, and then consider the signal quality and use a dynamic weight adjustment mechanism to calculate the weight values of the two processed feature data, and fuse the two processed feature data in combination with the weight values to obtain the feature fusion result.

[0103] In step S104 of some embodiments, the action intention recognition model can be obtained through deep learning based on a neural network. The action intention recognition model is a classification model, which mainly maps and reduces the input multi-modal features layer by layer through neurons to obtain the probability values of the multi-modal features belonging to each action intention label, and then uses a classifier for classification and discrimination to determine the predicted action intention.

[0104] Specifically, in addition to the multi-modal features of electroencephalogram (EEG) and surface electromyogram (sEMG) (with a dimension of D = Deeg + DsEMG) fused by dynamic weights, the input of the action intention recognition model can further input the above information, such as the data of environmental sensors and task state encoding. Exemplarily, the model input data is shown in Table 1:

[0105] Table 1 Model Input Data

[0106]

[0107] Considering information that is likely to affect signals, such as the environment (e.g., external temperature) and the subject's posture, in the input of the model can enable the model to predict action intentions based on this information, improving the accuracy of action intention recognition.

[0108] The output of the action intention recognition model is the subject's action intention and its Softmax probability distribution. Based on the definition of sample labels, the action intention can be grasping, walking, pushing a door, triggering a voice command, etc. For scenarios of continuous actions that require fine control (such as pouring water and writing), the model can also be further trained to output regression values of joint angles or speeds, etc.

[0109] According to some embodiments of the present application, in step S104, the step of obtaining the predicted action intention by inputting the multi-modal features into the action intention recognition model may include, but is not limited to, the following steps:

[0110] Step S601: Input the multi-modal features into the dilated convolutional layer and the bidirectional long short-term memory network respectively, and correspondingly obtain multi-scale rhythm features and muscle activation time series pattern features;

[0111] Step S602: Adopt a cross-modal attention mechanism to dynamically adjust the contribution degrees of the multi-scale rhythm features and the muscle activation time series pattern features;

[0112] Step S603: Determine the predicted action intention according to the multi-scale rhythm features and their contribution degrees and the muscle activation time series pattern features and their contribution degrees.

[0113] In this embodiment, the action intention recognition model can adopt a deep learning architecture with temporal modeling capabilities, including an extensible feature embedding layer and a dynamic feature selection module. Specifically, the action intention recognition model includes a dilated convolutional layer, a bidirectional LSTM layer, a dynamic feature selection module, and an output layer. The dilated convolutional layer is used to extract the multi-scale rhythm features of the electroencephalogram in the multi-modal features. The bidirectional LSTM layer is used to capture the muscle activation temporal patterns of the electromyogram in the multi-modal features. Different muscle activation temporal patterns can correspond to and represent an action. A cross-modal attention mechanism is introduced in the dynamic feature selection module to dynamically adjust the contribution degrees of the output features of the dilated convolutional layer and the bidirectional LSTM layer, and calculate the comprehensive features based on the multi-scale rhythm features and their contribution degrees and the muscle activation temporal pattern features and their contribution degrees. The output layer is used to map the predicted action intention based on the comprehensive features.

[0114] In this embodiment, please refer to Figure 3 , and use the multi-modal fusion feature vector samples with movement intention labels to train the model. During the training process, an adaptive learning rate adjustment strategy is adopted, the model performance is evaluated through cross-validation, and the model hyperparameters are optimized according to the validation results to improve the adaptive ability and generalization ability of the model. The specific training process is as follows:

[0115] Design a deep learning architecture with temporal modeling capabilities, including an input layer, an extensible feature embedding layer (i.e., a temporal modeling layer), a dynamic feature selection module, and an output layer. The dynamic feature selection module automatically selects the features that contribute more to the model output based on the attention mechanism, improving the model efficiency.

[0116] Use the multi-modal feature samples with movement intention labels to train the model. The samples can be collected from the electroencephalogram and electromyogram signal features when different subjects perform different actions (such as grasping, walking, etc.), and the multi-modal feature samples are obtained through the above signal processing and fusion process. Use these samples to train the model. During the training process, an adaptive learning rate adjustment strategy is adopted. For example, the Adam optimizer can be used for training. The initial learning rate is set to 0.001, and the learning rate is dynamically adjusted according to the convergence of the model during the training process to accelerate the convergence speed and avoid overfitting.

[0117] Evaluate the model performance through cross-validation and optimize the hyperparameters such as the number of network layers, the number of nodes, and the learning rate of the model.

[0118] In some embodiments, a temporal neural network architecture with incremental learning ability is further designed to continuously optimize model parameters through an online learning mechanism to adapt to individual differences and environmental changes. During the model learning process, it is divided into an offline learning stage and an online learning stage. In the offline learning stage, based on a large-scale pre-trained dataset, sample data from a large number of different subjects are used to establish a basic action intention recognition model. After the action intention recognition model is put into use, the Elastic Weight Consolidation (EWC) algorithm is adopted for online learning to perform personalized adaptation while retaining existing knowledge, thereby improving the accuracy of action intention recognition for individual brain-muscle signals.

[0119] In step S105 of some embodiments, a robot control instruction is determined according to the predicted action intention and the current state of the robot to control the humanoid robot. Specifically, please refer to Figure 4 , obtain the current state of the robot, which includes but is not limited to state information such as the current joint angles, speeds, positions, etc. of the humanoid robot, and perform fusion processing on the motion intention information output by the model and the current state information of the robot. The fusion process can be implemented through a weighted algorithm or through neural network processing. Determine the target pose and joint space trajectory of the robot end effector according to the fusion result, and then calculate the target motion parameters (such as the angles and speeds that each joint needs to rotate) of each joint of the robot during the process of achieving the target pose and joint space trajectory of the end effector through the inverse kinematics algorithm to generate a motion instruction. Further, perform safety verification (motion range limit and collision prediction detection) and smoothing processing on the generated motion instruction to ensure the feasibility of the instruction and the stability of the robot motion.

[0120] In some embodiments, the current state information of the humanoid robot can be obtained in real time through sensor information such as the joint angle sensor, acceleration sensor, and gyroscope of the robot. When fusing the predicted motion intention with the current state of the robot, the weights of the current position and pose information of the robot can be appropriately increased when the robot is approaching the target object to ensure precise operation; if the neural network processing method is used, the motion intention information and the robot state information are used as the input of the neural network, and after internal calculation and processing of the network, the fused information is output.

[0121] According to some embodiments of the present application, in step S105, the step of determining a robot control instruction according to the predicted action intention and the current state of the robot may include but is not limited to the following steps:

[0122] Step S701, determine the target action template of the corresponding motion primitive according to the predicted action intention, where the target action template is used to describe the kinematic parameters and dynamic parameters required for the target action;

[0123] Step S702: According to the current state of the robot, calculate the parameter control quantity described by the target action template through the inverse kinematics algorithm to obtain the robot control instruction.

[0124] In this embodiment, since the action executions of different action intents vary greatly. For example, the walking action is mainly the lower limb action, while the grasping action is mainly the fine hand action. Therefore, after identifying the action intent, the target action template can be further matched to limit the joints and parameter constraints for execution, so as to achieve efficient control, as follows:

[0125] Map the predicted action intent output by the model to motion primitives, which include grasping, walking, speech, etc., and call the corresponding target action template. The target action template is predefined in the action template library, and the template library is a set of predefined motion primitive collections. Each primitive corresponds to a standardized action template, which is used to describe the following content:

[0126] Kinematics parameters: target joint angles, speeds, acceleration curves;

[0127] Dynamics parameters: impedance control gains (stiffness, damping);

[0128] Environmental constraints: collision detection rules, safety thresholds.

[0129] Exemplarily, for the action template of the grasping motion primitive, its parameter description can be the end effector trajectory (Cartesian space), finger joint closing speed, grasping force threshold (such as 10 - 100N); for the action template of the speech motion primitive, its parameter description can be the keyword trigger list ("stop", "accelerate"), voiceprint recognition confidence threshold (>0.8); for the action template of the walking motion primitive, its parameter description can be the gait cycle, foot end trajectory (ZMP planning), trunk balance strategy.

[0130] After obtaining the target action template, the parameters in the target action template can also be adjusted according to the current environmental state. For example, adjust the gait cycle of the walking motion according to the object position. According to the current state of the robot, calculate the parameter control quantity described by the target action template through the inverse kinematics algorithm to obtain the robot control instruction. Inverse kinematics solution is to establish a robot model by the D - H parameter method and solve the target joint angles. When performing control at the bottom layer, combine impedance control to adjust the joint damping parameters to achieve compliant operation. In addition, safety constraints and collision prediction can also be performed on the control instruction during bottom - layer execution. The safety constraint conditions include a double - check mechanism, and the kinematic constraint check includes joint angle and speed limits. Collision prediction is based on neural network prediction using the robot geometric model and depth sensor data.

[0131] According to some embodiments of the present application, after the step of inputting multimodal features into an action intention recognition model to obtain a predicted action intention, the humanoid robot adaptive control method of the embodiments of the present application may further include the following steps:

[0132] Obtain the action task that the robot is currently executing;

[0133] Determine whether the first priority of the predicted action intention is greater than the second priority of the currently executing action task;

[0134] When the first priority is greater than the second priority, interrupt the currently executing action task and execute the action task of the predicted action intention;

[0135] When the first priority is less than or equal to the second priority, put the action task of the predicted action intention into the task queue for task queue scheduling.

[0136] In this embodiment, the priorities of the predicted action intention and the currently executing action task can be determined by querying. Considering that the user may generate a new action intention during the robot's action execution process, and to improve the rationality during the robot's action execution, different action intentions can be set with priorities, so that the robot can execute the corresponding actions in sequence according to the priorities of the action intentions. Exemplarily, in a dangerous situation, the user generally generates a stop intention. Therefore, the stop intention can be set as the highest priority. When the predicted action intention is the stop intention, the currently executing action tasks such as grasping and walking will pause. In another example, the priority of the grasping action can be set to be greater than the priority of the behavior action. When the robot is executing the grasping action, if the user generates a walking action intention, the robot will not immediately execute the walking action, but will execute the walking action after the grasping action is completed, so as to avoid the item from falling due to the robot's walking during the process of grasping the item.

[0137] According to some embodiments of the present application, the embodiments of the present application have the following beneficial effects:

[0138] The embodiments of the present application fuse electroencephalogram signals and electromyogram signals, make full use of the complementary information of multimodal signals, capture human motion intentions more comprehensively and accurately, and improve the accuracy of robot control.

[0139] The multimodal brain-muscle signal fusion model constructed in the embodiments of the present application has an adaptive learning ability, can self-adjust according to the brain-muscle signal characteristics and motion intentions of different individuals, and enhances the generalization ability of the system.

[0140] Embodiments of the present application generate adaptive motion instructions in combination with the current state of a humanoid robot, enabling the robot to better adapt to complex and changing environments and achieve flexible and stable motion control. In addition, embodiments of the present application set reasonable priorities for executing action intentions to meet the actual requirements of operating in dangerous scenarios or other special scenarios.

[0141] Embodiments of the present application also propose a humanoid robot adaptive control system, including:

[0142] A first module for collecting electroencephalogram (EEG) signals and electromyogram (EMG) signals of a target object;

[0143] A second module for determining first time-frequency domain features of the EEG signals based on the EEG signals and second time-frequency domain features of the muscle activation state based on the EMG signals;

[0144] A third module for fusing the first time-frequency domain features and the second time-frequency domain features using a dynamic weighting algorithm to obtain multi-modal features;

[0145] A fourth module for inputting the multi-modal features into an action intention recognition model to obtain predicted action intentions;

[0146] A fifth module for determining a robot control instruction based on the predicted action intentions and the current state of the robot to control the humanoid robot.

[0147] It can be understood that the content in the above embodiments of the humanoid robot adaptive control method is applicable to the embodiments of this system. The functions specifically implemented by the embodiments of this system are the same as those of the above embodiments of the humanoid robot adaptive control method, and the beneficial effects achieved are also the same as those of the above embodiments of the humanoid robot adaptive control method.

[0148] Embodiments of the present application also provide an electronic device, which includes: a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, it implements the above humanoid robot adaptive control method. This electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0149] Please refer to Figure 5 , Figure 5 which shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0150] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0151] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the humanoid robot adaptive control method in the embodiments of the present application;

[0152] The input / output interface 903 is used to implement information input and output;

[0153] The communication interface 904 is used to implement communication interaction between this device and other devices, and can implement communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0154] The bus 905 transmits information between the various components of the device (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0155] Among them, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other inside the device through the bus 905.

[0156] The embodiments of the present application also provide a storage medium. The storage medium is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned humanoid robot adaptive control method.

[0157] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include memories that are remotely located relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0159] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0160] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0162] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above figures are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0163] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (item) of the following" or its similar expressions refer to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0164] In several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection with each other can be through some interfaces. The indirect coupling or communication connection of systems or units can be in electrical, mechanical or other forms.

[0165] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0167] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0168] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, which does not limit the scope of rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall be within the scope of rights of the embodiments of this application.

Claims

1. A humanoid robot adaptive control method, characterized in that: The following steps are involved: Collecting EEG signals and EMG signals of the target object; Determine a first time-frequency domain feature of the EEG signal according to the EEG signal, and determine a second time-frequency domain feature of a muscle activation state according to the EMG signal; Using a dynamic weighting algorithm to fuse the first time-frequency domain features and the second time-frequency domain features to obtain a multimodal feature; According to inputting the multimodal features into the action intention recognition model, a predicted action intention is obtained; Determining a robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot; The step of fusing the first time-frequency domain feature and the second time-frequency domain feature using a dynamic weighting algorithm to obtain a multimodal feature includes the following steps: Determining the quality of the EEG signal according to the signal power of the EEG signal in the target frequency band and the EMG resting noise power, and determining the quality of the EMG signal according to the signal power of the EMG signal and the EMG resting noise power, wherein the EMG resting noise power represents the baseline noise of the EMG signal in a resting state; Determining a first weight of the first time-frequency domain feature according to a ratio of the EEG signal quality to the total signal quality, and determining a second weight of the second time-frequency domain feature according to a ratio of the EMG signal quality to the total signal quality; The first time-frequency domain feature and the second time-frequency domain feature are fused according to the first weight and the second weight to obtain a multimodal feature.

2. The humanoid robot adaptive control method according to claim 1, characterized in that: Determining the first time-frequency domain feature of the EEG signal according to the EEG signal comprises the following steps: Performing multi-stage adaptive filtering and blind source separation processing on the EEG signal to obtain a first target signal; Performing continuous wavelet transform on the first target signal based on wavelet basis functions and scale parameters to obtain wavelet coefficients and center frequencies of different scales; The energy entropy of the target frequency is extracted according to the wavelet coefficients and center frequencies of different scales to obtain the first time-frequency domain feature.

3. The humanoid robot adaptive control method according to claim 1, characterized in that: Determining the second time-frequency domain feature of the muscle activation state according to the electromyographic signal comprises the following steps: Determine the muscle activation state at the corresponding moment according to the electromyographic signal; Performing noise reduction processing on the electromyographic signal at the corresponding moment according to the noise threshold in the muscle activation state to obtain a second target signal; The information entropy of the second target signal at different frequency components is extracted to obtain the second time-frequency domain features of the muscle activation state.

4. The humanoid robot adaptive control method according to claim 1, characterized in that: The fusing the first time-frequency domain feature and the second time-frequency domain feature according to the first weight and the second weight to obtain a multimodal feature comprises the following steps: Multiplying the first weight by the first time-frequency domain feature to obtain an electroencephalogram feature, and multiplying the second weight by the second time-frequency domain feature to obtain an electromyography feature; The EEG features and the EMG features are time aligned and then spliced ​​to obtain multimodal features.

5. The humanoid robot adaptive control method according to any one of claims 1 to 4, characterized in that: The step of inputting the multimodal features into an action intention recognition model to obtain a predicted action intention comprises the following steps: The multimodal features are input into the dilated convolutional layer and the bidirectional long short-term memory network respectively, and multi-scale rhythm features and muscle activation timing pattern features are obtained correspondingly; A cross-modal attention mechanism is used to dynamically adjust the contribution of the multi-scale rhythm feature and the contribution of the muscle activation timing pattern feature; The predicted action intention is determined based on the multi-scale rhythm features and their contribution and the muscle activation timing pattern features and their contribution.

6. The method for adaptive control of a humanoid robot according to any one of claims 1 to 4, characterized in that: Determining the robot control instruction according to the predicted action intention and the current state of the robot comprises the following steps: Determining a corresponding target action template according to the predicted action intention, wherein the target action template is used to describe kinematic parameters and dynamic parameters that need to be controlled for the target action; According to the current state of the robot, the parameter control quantity described by the target action template is calculated through the kinematic inverse solution algorithm to obtain the robot control instruction.

7. A humanoid robot adaptive control system, characterized in that: include: The first module is used to collect EEG signals and EMG signals of the target object; A second module is used to determine a first time-frequency domain feature of the EEG signal according to the EEG signal, and to determine a second time-frequency domain feature of a muscle activation state according to the EMG signal; A third module is used to fuse the first time-frequency domain features and the second time-frequency domain features using a dynamic weighting algorithm to obtain a multimodal feature; A fourth module is used to obtain a predicted action intention by inputting the multimodal features into an action intention recognition model; A fifth module is used to determine a robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot; The third module is specifically used to perform the following steps: Determining the quality of the EEG signal according to the signal power of the EEG signal in the target frequency band and the EMG resting noise power, and determining the quality of the EMG signal according to the signal power of the EMG signal and the EMG resting noise power, wherein the EMG resting noise power represents the baseline noise of the EMG signal in a resting state; Determining a first weight of the first time-frequency domain feature according to a ratio of the EEG signal quality to the total signal quality, and determining a second weight of the second time-frequency domain feature according to a ratio of the EMG signal quality to the total signal quality; The first time-frequency domain feature and the second time-frequency domain feature are fused according to the first weight and the second weight to obtain a multimodal feature.

8. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 6 are realized.

9. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 6.

Images