Humanoid robot adaptive control method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a humanoid robot self-adaptive control method and system, electronic equipment and a storage medium, and belongs to the technical field of robot intelligent control. The method comprises the steps that electroencephalogram signals and electromyographic signals of a target object are collected; determining a first time-frequency domain feature of the electroencephalogram signal according to the electroencephalogram signal, and determining a second time-frequency domain feature of a muscle activation state according to the electromyographic signal; fusing the first time-frequency domain feature and the second time-frequency domain feature by adopting a dynamic weighting algorithm to obtain a multi-modal feature; inputting the multi-modal features into an action intention recognition model to obtain a predicted action intention; and determining a robot control instruction according to the predicted action intention and the current state of the robot so as to control the humanoid robot. The precision of self-adaptive control of the humanoid robot can be improved.

Description

Technical Field

[0001] The present application relates to the field of robot intelligent control technology, and in particular to a humanoid robot adaptive control method, system, electronic device and storage medium. Background Art

[0002] Existing robot control technology has many limitations. Traditional manual control methods rely on devices such as joysticks and buttons, which require operators to have proficient operating skills. This method is difficult to achieve autonomous control. Although preset program control can perform some predetermined tasks, it lacks adaptability to environmental changes and real-time human movement intentions, and cannot flexibly adjust robot movements.

[0003] In related technologies, robots can be adaptively controlled based on brain-computer interfaces. However, since EEG signals are susceptible to external interference, the signals are weak, and feature extraction is difficult, the control accuracy is low and it is difficult to meet actual application needs. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a humanoid robot adaptive control method, system, electronic device and storage medium, aiming to improve the accuracy of adaptive control of the humanoid robot.

[0005] To achieve the above object, an embodiment of the present application provides a method for adaptively controlling a humanoid robot, comprising the following steps: Collecting EEG signals and EMG signals of the target object; Determine a first time-frequency domain feature of the EEG signal according to the EEG signal, and determine a second time-frequency domain feature of a muscle activation state according to the EMG signal; Using a dynamic weighting algorithm to fuse the first time-frequency domain features and the second time-frequency domain features to obtain a multimodal feature; According to inputting the multimodal features into the action intention recognition model, a predicted action intention is obtained; A robot control instruction is determined according to the predicted action intention and the current state of the robot to control the humanoid robot.

[0006] In some embodiments, determining the first time-frequency domain feature of the EEG signal according to the EEG signal comprises the following steps: Performing multi-stage adaptive filtering and blind source separation processing on the EEG signal to obtain a first target signal; Performing continuous wavelet transform on the first target signal based on wavelet basis functions and scale parameters to obtain wavelet coefficients and center frequencies of different scales; The energy entropy of the target frequency is extracted according to the wavelet coefficients and center frequencies of different scales to obtain the first time-frequency domain feature.

[0007] In some embodiments, determining the second time-frequency domain feature of the muscle activation state according to the electromyographic signal comprises the following steps: Determine the muscle activation state at the corresponding moment according to the electromyographic signal; Performing noise reduction processing on the electromyographic signal at the corresponding moment according to the noise threshold in the muscle activation state to obtain a second target signal; The information entropy of the second target signal at different frequency components is extracted to obtain the second time-frequency domain features of the muscle activation state.

[0008] In some embodiments, the adopting a dynamic weighting algorithm to fuse the first time-frequency domain feature and the second time-frequency domain feature to obtain a multimodal feature includes the following steps: Using a dynamic weighting algorithm to calculate a first weight of the first time-frequency domain feature and a second weight of the second time-frequency domain feature; Multiplying the first weight by the first time-frequency domain feature to obtain an electroencephalogram feature, and multiplying the second weight by the second time-frequency domain feature to obtain an electromyography feature; The EEG features and the EMG features are time aligned and then spliced ​​to obtain multimodal features.

[0009] In some embodiments, the step of using a dynamic weighting algorithm to respectively calculate the first weight of the first time-frequency domain feature and the step of calculating the second weight of the second time-frequency domain feature comprises the following steps: Determining the quality of the EEG signal according to the signal power of the EEG signal in the target frequency band and the EMG resting noise power, and determining the quality of the EMG signal according to the signal power of the EMG signal and the EMG resting noise power, wherein the EMG resting noise power represents the baseline noise of the EMG signal in a resting state; A first weight of the first time-frequency domain feature is determined according to the ratio of the EEG signal quality to the total signal quality, and a second weight of the second time-frequency domain feature is determined according to the ratio of the EMG signal quality to the total signal quality.

[0010] In some embodiments, the step of inputting the multimodal features into an action intention recognition model to obtain a predicted action intention comprises the following steps: The multimodal features are input into the dilated convolutional layer and the bidirectional long short-term memory network respectively, and multi-scale rhythm features and muscle activation timing pattern features are obtained correspondingly; A cross-modal attention mechanism is used to dynamically adjust the contribution of the multi-scale rhythm feature and the contribution of the muscle activation timing pattern feature; The predicted action intention is determined based on the multi-scale rhythm features and their contribution and the muscle activation timing pattern features and their contribution.

[0011] In some embodiments, determining the robot control instruction according to the predicted action intention and the current state of the robot comprises the following steps: Determine a target action template corresponding to the motion primitive according to the predicted action intention, wherein the target action template is used to describe the kinematic parameters and dynamic parameters required to be controlled for the target action; According to the current state of the robot, the parameter control quantity described by the target action template is calculated through the kinematic inverse solution algorithm to obtain the robot control instruction.

[0012] To achieve the above purpose, another aspect of the embodiment of the present application provides a humanoid robot adaptive control system, comprising: The first module is used to collect EEG signals and EMG signals of the target object; A second module is used to determine a first time-frequency domain feature of the EEG signal according to the EEG signal, and to determine a second time-frequency domain feature of a muscle activation state according to the EMG signal; A third module is used to fuse the first time-frequency domain features and the second time-frequency domain features using a dynamic weighting algorithm to obtain a multimodal feature; A fourth module is used to obtain a predicted action intention by inputting the multimodal features into an action intention recognition model; The fifth module is used to determine the robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot.

[0013] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program implements the method described in the above embodiment when executed by the processor.

[0014] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the above embodiment.

[0015] The humanoid robot adaptive control method, system, electronic device and storage medium proposed in the present application collect the EEG signal and EMG signal of the target object, determine the first time-frequency domain feature of the EEG signal based on the EEG signal, and determine the second time-frequency domain feature of the muscle activation state based on the EMG signal, use a dynamic weighting algorithm to fuse the first time-frequency domain feature and the second time-frequency domain feature to obtain a multimodal feature, input the multimodal feature into the action intention recognition model to obtain a predicted action intention, and determine the robot control instruction based on the predicted action intention and the current state of the robot to control the humanoid robot. The present application combines EEG signals with EMG signals to predict action intentions, and uses a dynamic weighting algorithm to consider the contribution of different signals as inputs to the predicted action model, thereby improving the accuracy of control action intention recognition and thus improving the accuracy of adaptive control of humanoid robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a humanoid robot adaptive control method provided by an embodiment of the present application; Figure 2 is a flow chart of a multimodal feature acquisition method provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the training process of the action intention recognition model provided in the embodiment of the present application; Figure 4 It is a schematic diagram of a robot instruction generation method provided in an embodiment of the present application; Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] It should be noted that, although the functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] The embodiments of the present application provide a humanoid robot adaptive control method, system, electronic device and storage medium, aiming to improve the accuracy of adaptive control of a humanoid robot.

[0021] The humanoid robot adaptive control method, system, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the humanoid robot adaptive control method in the embodiments of the present application is described.

[0022] The humanoid robot adaptive control method provided in the embodiment of the present application relates to the field of robot intelligent control technology. The humanoid robot adaptive control method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the humanoid robot adaptive control method, etc., but is not limited to the above forms.

[0023] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0024] Figure 1 is an optional flow chart of the humanoid robot adaptive control method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S105.

[0025] Step S101, collecting EEG signals and EMG signals of a target object; Step S102, determining a first time-frequency domain feature of the EEG signal according to the EEG signal, and determining a second time-frequency domain feature of the muscle activation state according to the EMG signal; Step S103, using a dynamic weighting algorithm to fuse the first time-frequency domain feature and the second time-frequency domain feature to obtain a multimodal feature; Step S104, obtaining predicted action intention by inputting the multimodal features into the action intention recognition model; Step S105, determining the robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot.

[0026] In step S101 of some embodiments, the electroencephalogram (EEG) is an electrical signal generated by the activity of brain neurons. The time series data recorded by the electrodes reflects the potential changes caused by the synchronous activity of a large number of neurons in the brain. The electromyographic signal, namely the surface electromyography (sEMG), is a muscle electrical activity signal recorded by electrodes placed on the surface of the skin. It reflects the electrical activity of the motor units (MUs) in the muscle and is an important tool for studying muscle function, motor control and rehabilitation medicine. Muscles are composed of multiple motor units, each of which includes a motor neuron and its innervated muscle fibers. When the motor neuron is activated, the muscle fiber will generate an action potential, causing muscle contraction. Therefore, by analyzing the action potential of the muscle fiber, the action that the target object wants to complete can be confirmed. The acquisition process of the electroencephalogram and electromyography signals can be to arrange a multi-channel electroencephalogram sensor array on the human scalp using a standardized electrode layout method, deploy distributed electromyography sensor units on the surface of the target muscle group, and align the signals to keep the timing of the electroencephalogram and electromyography signals consistent. Specifically, the signal alignment method can adopt a hardware clock synchronization alignment method, or a combination of hardware clock synchronization and software resampling, with an alignment accuracy of milliseconds.

[0027] Exemplarily, a multi-channel EEG sensor array is arranged on the scalp of the target object using a standardized electrode layout method. The layout can be carried out according to the international 10-20 system standard. Before pasting the electrodes, the scalp is cleaned to remove grease and dirt to ensure good contact between the electrodes and the scalp and improve the signal acquisition quality. Distributed electromyographic sensor units are deployed on the surface of the target muscle groups, including the biceps and triceps of the upper limbs, the quadriceps and calf triceps of the lower limbs, and other muscle groups related to exercise. After cleaning the skin, the electromyographic sensor units are deployed according to the distributed layout. The sensor units are firmly pasted to avoid electrode shedding or signal interference caused by exercise. During the signal acquisition process, the timing of the EEG signal and the electromyographic signal is consistent through the hardware synchronization mechanism, and the target object can be kept in a relaxed state to avoid large-scale movements and electromagnetic interference. The sampling frequency can be 1000Hz.

[0028] In step S102 of some embodiments, the first time-frequency domain feature refers to the composite time-domain and frequency-domain features extracted from the collected EEG signal that reflect the dynamic characteristics of the signal. Specifically, the time series of the EEG signal can be directly analyzed, such as the time-domain features such as the mean, variance, and peak value, and the frequency-domain features of the band energy can be analyzed using Fourier transform (FFT). The time-frequency characteristics can also be analyzed using short-time Fourier transform (STFT) or wavelet transform (CWT). The second time-frequency domain feature refers to the time-frequency domain feature related to the muscle activation state extracted from the electromyographic signal. Specifically, the time series of the electromyographic signal can be directly analyzed, such as the time-domain features such as the mean, variance, and peak value, and the frequency-domain features of the band energy can be analyzed using Fourier transform (FFT). The time-frequency characteristics can also be analyzed using short-time Fourier transform (STFT) or wavelet transform (CWT).

[0029] According to some embodiments of the present application, in step S102, the step of determining the first time-frequency domain feature of the EEG signal according to the EEG signal may include but is not limited to the following steps: Step S201, performing multi-stage adaptive filtering and blind source separation processing on the EEG signal to obtain a first target signal; Step S202, performing continuous wavelet transform on the first target signal based on the wavelet basis function and the scale parameter to obtain wavelet coefficients and center frequencies of different scales; Step S203: extract energy entropy of the target frequency according to wavelet coefficients and center frequencies of different scales to obtain a first time-frequency domain feature.

[0030] In this embodiment, the collected EEG signals may be interfered by physiological noise and environmental noise, such as eye movement (EOG), electromyography (EMG), electrocardiography (ECG), etc., and environmental noise interference such as power line interference, etc. Therefore, this embodiment can use multi-stage adaptive filtering technology to dynamically suppress the low-frequency drift and high-frequency noise interference of the EEG signal, and then construct an artifact component feature library, and eliminate physiological (electrooculogram, electrocardiogram) artifacts through blind source separation technology, so as to obtain a relatively pure EEG signal, that is, the first target signal.

[0031] Multi-stage adaptive filtering technology is a signal processing technology that gradually optimizes the filtering effect by cascading or paralleling multiple adaptive filters to better handle complex signal and noise environments. The adaptive filter can dynamically adjust its cutoff frequency according to the characteristics of the input signal to minimize the error signal (usually the difference between the expected signal and the filter output). Multi-stage adaptive filtering technology can achieve efficient denoising and signal separation of EEG signals. For example, the dynamic range of the cutoff frequency of the multi-stage adaptive filter can be designed to be 0.5-100Hz, focusing on suppressing low-frequency drift in the low-frequency band (0.5-4Hz) and removing high-frequency noise interference in the high-frequency band (30-100Hz).

[0032] Blind source separation technology is a signal processing technology that aims to recover the original source signal from a mixed signal without knowing the prior information of the mixing process or the source signal. In this embodiment, an artifact component feature library is constructed to collect common artifact features such as electrooculogram and electrocardiogram, and then blind source separation technology is used, such as independent component analysis (ICA) algorithm, to decompose the EEG signal into multiple independent components, and physiological artifacts are identified and removed by comparing with the artifact component feature library.

[0033] After the EEG signal is preprocessed to obtain the first target signal, the first target signal is feature extracted. The frequency range of the EEG signal is mainly distributed between 0.5Hz and 100Hz, and is divided into multiple frequency bands: Delta wave frequency 0.5-4Hz, related to deep sleep; Theta wave (Theta) frequency 4-8Hz, related to light sleep and meditation; Alpha wave (Alpha) frequency 8-13Hz, related to relaxation and closed eyes; Beta wave (Beta) frequency 13-30Hz, related to wakefulness and concentration; Gamma wave (Gamma) frequency 30-100Hz, related to advanced cognitive functions. In this embodiment, continuous wavelet transform (CWT) can be used to analyze the preprocessed EEG signal (i.e., the first target signal), extract the energy entropy of the α and β frequency bands, and thus obtain the first time-frequency domain feature. When performing continuous wavelet transform on the time window segment of the first target signal, a suitable wavelet function (Morlet wavelet) and its scale parameters can be selected for processing to obtain wavelet coefficients of different frequencies at different time scales, and then feature extraction is performed based on the wavelet coefficients to accurately capture the energy changes of the EEG signal at different frequencies and time scales.

[0034] In one example, the energy entropy of the α (8-13 Hz) and β (13-30 Hz) frequency bands is extracted by continuous wavelet transform (CWT). The calculation formula is as follows: ; ; in, Indicates that the continuous wavelet transform i The wavelet coefficients at a scale (i.e. corresponding to a specific frequency), Characterization i Energy at this scale, Characterization i The ratio of the energy of each frequency band to the total energy, that is, the normalized energy distribution probability, N Represents the number of frequency scales after wavelet transformation.

[0035] According to some embodiments of the present application, in step S102, the step of determining the second time-frequency domain feature of the muscle activation state according to the electromyographic signal may include but is not limited to the following steps: Step S301, determining the muscle activation state at a corresponding moment according to the electromyographic signal; Step S302, performing noise reduction processing on the electromyographic signal at the corresponding moment according to the noise threshold in the muscle activation state to obtain a second target signal; Step S303: extract the information entropy of the second target signal at different frequency components to obtain the second time-frequency domain features of the muscle activation state.

[0036] In this embodiment, the electromyographic signal is subjected to time-varying threshold noise reduction processing, the noise threshold is dynamically adjusted according to the muscle activation state, and then the electromyographic signal is subjected to noise reduction using the adjusted noise threshold, which can not only remove noise, but also better retain the muscle activation features in the electromyographic signal. Specifically, the muscle activation state at the corresponding moment is determined according to the electromyographic signal. When the muscle is at rest, the threshold (frequency threshold or amplitude threshold) is lowered, and then the part less than the threshold is filtered out from the electromyographic signal to remove tiny noise; when the muscle contracts, the threshold is appropriately increased, and then the part less than the threshold is filtered out from the electromyographic signal to retain the effective electromyographic features, thereby obtaining the second target signal. This implementation can determine the muscle activation state by analyzing the amplitude and frequency changes of the electromyographic signal. For example, if the amplitude value of the electromyographic signal at a certain moment is greater than the preset value, the muscle activation state at the current moment is considered to be an activated state, otherwise it is an inactivated state (i.e., a resting state).

[0037] After the collected EMG signal is preprocessed as above to obtain the second target signal, the second time-frequency domain features related to the muscle activation state, such as the integrated EMG value (iEMG), are further calculated based on the second target signal. Exemplarily, the wavelet-Shannon entropy of the second target signal is extracted, and the information entropy of the EMG signal at different frequency components is analyzed, so as to obtain richer muscle activation features.

[0038] In step S103 of some embodiments, after determining the first time-frequency domain feature of the EEG signal and the second time-frequency domain feature of the electromyographic signal, the weights of the first time-frequency domain feature and the second time-frequency domain feature can be dynamically allocated to form a multimodal feature. In the feature fusion process, a dynamic weighting algorithm can be used to determine the weight of each feature, and then the feature fusion is performed in combination with the weight value, so that the model can focus more on important features in the future, thereby improving the accuracy of action intention recognition. The dynamic weighting algorithm mainly allocates the weights of the corresponding time-frequency domain features based on the signal characteristics of the EEG signal and the electromyographic signal themselves. The signal characteristics may include but are not limited to signal strength, signal quality, information richness, etc. For example, if the electromyographic signal does not change for a long time or the change amplitude is small, it is considered that the information contained in the signal is low in richness, and the weight of the signal is appropriately reduced. It is understandable that in the process of allocating feature weights, the weight can be calculated from one of the aspects of signal strength, signal quality and information richness, or the weight can be calculated comprehensively based on multiple aspects of signal strength, signal quality and information richness.

[0039] According to some embodiments of the present application, in step S103, the step of fusing the first time-frequency domain feature and the second time-frequency domain feature using a dynamic weighting algorithm to obtain a multimodal feature may include but is not limited to the following steps: Step S401, using a dynamic weighting algorithm to calculate a first weight of a first time-frequency domain feature, and a second weight of a second time-frequency domain feature; Step S402, multiplying the first weight by the first time-frequency domain feature to obtain an electroencephalogram feature, and multiplying the second weight by the second time-frequency domain feature to obtain an electromyography feature; Step S403, time-aligning and splicing the EEG features and the EMG features to obtain multi-modal features.

[0040] In this embodiment, a dynamic weighting algorithm is used to calculate the first weight of the first time-frequency domain feature and the second weight of the second time-frequency domain feature, and then the first weight is multiplied by the first time-frequency domain feature to obtain the EEG feature, and the second weight is multiplied by the second time-frequency domain feature to obtain the EMG feature, and then the EEG feature and the EMG feature are spliced ​​after time alignment to obtain the multimodal feature, that is, the EEG feature and the EMG feature corresponding to the same time window segment are spliced ​​to obtain the multimodal feature of the time window segment, and then the multimodal features of each time window segment are sequentially input into the action intention recognition model for prediction in chronological order. The dimension of the EEG (EEG) and EMG (sEMG) multimodal features after dynamic weight fusion is D=Deeg+DsEMG.

[0041] According to some embodiments of the present application, in step S401, using a dynamic weighting algorithm to respectively calculate the first weight of the first time-frequency domain feature and the second weight of the second time-frequency domain feature may include but is not limited to the following steps: Step S501, determining the quality of the EEG signal according to the signal power of the EEG signal in the target frequency band and the EMG resting noise power, and determining the quality of the EMG signal according to the signal power of the EMG signal and the EMG resting noise power, wherein the EMG resting noise power represents the baseline noise of the EMG signal in a resting state; Step S502, determining a first weight of a first time-frequency domain feature according to a ratio of an EEG signal quality to a total signal quality, and determining a second weight of a second time-frequency domain feature according to a ratio of an EMG signal quality to a total signal quality.

[0042] In this embodiment, the weights of the corresponding time-frequency domain features may be adjusted according to the signal qualities of the EEG signal and the EMG signal.

[0043] The quality of the EEG signal can be determined by calculating the ratio of the signal power of the EEG signal in the target frequency band to the EMG resting noise power and then converting it into a decibel value. Specifically, the quality of the EEG signal can be calculated by the following formula: ; The quality of the electromyographic signal can be determined by calculating the ratio of the signal power of the electromyographic signal to the electromyographic resting noise power and then converting it into a decibel value. Specifically, the quality of the electromyographic signal can be calculated by the following formula: ; In the above formula, is the signal power of the EEG signal in the delta frequency band (0.5-4Hz), It is the signal power of the EMG signal in the frequency band related to the action; is the EMG resting noise power, which is taken from the EMG signal resting state baseline.

[0044] In this embodiment, this embodiment uses electromyographic resting noise as the environmental noise reference to analyze the signal-to-noise ratio of EEG signals and electromyographic signals, realizes dynamic evaluation of the quality of EEG signals and electromyographic signals, and thus optimizes the weight allocation of multimodal feature fusion. This embodiment uses electromyographic resting noise (sEMG noise) as the noise reference of EEG (EEG), rather than traditional self-noise calculation. This dynamic weight allocation method can effectively improve the anti-interference ability and robustness of the system in practical applications for the following reasons: The EEG noise itself is difficult to measure directly. The noise of EEG signals mainly comes from physiological artifacts (such as electrooculography and electrocardiography) and environmental interference, but these noises are mixed with the EEG signals themselves and are difficult to separate. Traditional methods require complex techniques such as independent component analysis (ICA) to separate noise, which has high computational costs and unstable results. However, the EMG resting noise is stable, and the baseline noise of the EMG signal in the resting state (such as μV level) is very stable and can be used as a reliable environmental noise benchmark. Therefore, the embodiment of the present application adopts a cross-modal noise compensation mechanism, that is, when environmental noise (such as electromagnetic interference) affects both EEG and EMG signals at the same time, the EMG resting noise can reflect the noise level of the current environment, thereby more accurately evaluating the quality of the EEG signal.

[0045] The first weight of the first time-frequency domain feature of the EEG signal and the second weight of the second time-frequency domain feature of the electromyographic signal Calculated by the following formulas: ; ; in, To prevent division by zero errors, .

[0046] According to some embodiments of this application, please refer to Figure 2,The multimodal features used to input the action intention recognition model are obtained through the following process: the scalp and muscle groups are arranged with reference to the electrode arrangement diagram, and then the EEG signals and EMG signals are collected synchronously, the EEG signals are sequentially subjected to multi-level adaptive filtering and artifact elimination processing, the EMG signals are subjected to time-varying threshold noise reduction and muscle activation detection processing, and then the weight values ​​of the two feature data after processing are calculated by using the dynamic weight adjustment mechanism considering the signal quality, and the two feature data after processing are fused in combination with the weight values ​​to obtain the feature fusion result.

[0047] In step S104 of some embodiments, the action intention recognition model can be obtained through deep learning based on a neural network. The action intention recognition model is a classification model, which mainly uses neurons to map and reduce the dimension of the input multimodal features layer by layer to obtain the probability value of the multimodal features belonging to each action intention label, and then uses a classifier to perform classification and judgment, so as to determine the predicted action intention.

[0048] Specifically, in addition to the multimodal features of EEG and sEMG (dimension D=Deeg+DsEMG) fused by dynamic weights, the input of the action intention recognition model can also further input the above information, such as environmental sensor data and task state encoding. For example, the model input data is shown in Table 1: Table 1 Model input data

[0049] Considering information that can easily affect the signal, such as the environment (such as external temperature) and the subject's posture, in the model input can enable the model to predict action intentions based on this information, thereby improving the accuracy of action intention recognition.

[0050] The output of the action intention recognition model is the subject's action intention and its Softmax probability distribution. Based on the sample label definition, the action intention can be grasping, walking, pushing the door, voice command triggering, etc. For scenes with continuous actions that require fine control (such as pouring water, writing), the model can be further trained to output the regression value of joint angle or speed, etc.

[0051] According to some embodiments of the present application, in step S104, the step of obtaining the predicted action intention by inputting the multimodal features into the action intention recognition model may include but is not limited to the following steps: Step S601, inputting the multimodal features into the dilated convolutional layer and the bidirectional long short-term memory network respectively, and obtaining multi-scale rhythm features and muscle activation timing pattern features accordingly; Step S602, using a cross-modal attention mechanism to dynamically adjust the contribution of multi-scale rhythm features and the contribution of muscle activation timing pattern features; Step S603, determining the predicted action intention based on the multi-scale rhythm features and their contribution and the muscle activation timing pattern features and their contribution.

[0052] In this embodiment, the action intention recognition model can adopt a deep learning architecture with time series modeling capabilities, including an expandable feature embedding layer and a dynamic feature selection module. Specifically, the action intention recognition model includes an expanded convolution layer, a bidirectional LSTM layer, a dynamic feature selection module and an output layer. The expanded convolution layer is used to extract the multi-scale rhythmic features of brain currents in multimodal features, and the bidirectional LSTM layer is used to capture the muscle activation timing pattern of muscle currents in multimodal features. Different muscle activation timing patterns can correspond to a kind of action. The cross-modal attention mechanism is introduced in the dynamic feature selection module to dynamically adjust the contribution of the output features of the expanded convolution layer and the bidirectional LSTM layer, and calculate the comprehensive features based on the multi-scale rhythmic features and their contributions and the muscle activation timing pattern features and their contributions. The output layer is used to map out the predicted action intention based on the comprehensive features.

[0053] In this embodiment, please refer to Figure 3 , the model is trained using multimodal fusion feature vector samples with motion intention labels. An adaptive learning rate adjustment strategy is used during the training process. The model performance is evaluated through cross-validation. The model hyperparameters are optimized based on the validation results to improve the model's adaptability and generalization capabilities. The specific training process is as follows: A deep learning architecture with time series modeling capabilities is designed, which includes an input layer, an extensible feature embedding layer (i.e., a time series modeling layer), a dynamic feature selection module, and an output layer. The dynamic feature selection module is based on the attention mechanism and automatically selects features that contribute more to the model output, thereby improving model efficiency.

[0054] The model is trained using multimodal feature samples with motion intention labels. The samples can be collected by letting different subjects perform different actions (grasping, walking, etc.) brain muscle signal feature samples, and multimodal feature samples are obtained through the above signal processing and fusion process. Use these samples to train the model, and use an adaptive learning rate adjustment strategy during the training process. For example, the Adam optimizer can be used for training. The initial learning rate is set to 0.001, and the learning rate is dynamically adjusted according to the convergence of the model during the training process to speed up the convergence speed and avoid overfitting.

[0055] The model performance is evaluated through cross-validation, and the model's hyperparameters such as the number of network layers, number of nodes, and learning rate are optimized.

[0056] In some embodiments, a temporal neural network architecture with incremental learning capabilities is further designed, and model parameters are continuously optimized through an online learning mechanism to adapt to individual differences and environmental changes. The model learning process is divided into an offline learning stage and an online learning stage. The offline learning stage is based on a large-scale pre-training data set, sample data from a large number of different subjects, and a basic action intention recognition model is established. After the action intention recognition model is launched, the elastic weight consolidation (EWC) algorithm is used for online learning, and personalized adaptation is performed while retaining existing knowledge to improve the accuracy of action intention recognition for individual brain-muscle signals.

[0057] In step S105 of some embodiments, a robot control instruction is determined according to the predicted action intention and the current state of the robot to control the humanoid robot. Figure 4 , obtain the current state of the robot, which includes but is not limited to the current joint angle, speed, position and other state information of the humanoid robot, and fuse the motion intention information output by the model with the current state information of the robot. The fusion process can be achieved through a weighted algorithm or a neural network. Determine the target posture and joint space trajectory of the robot end effector based on the fusion result, and then calculate the target motion parameters of each joint of the robot in the process of achieving the target posture and joint space trajectory of the end effector through the kinematic inverse solution algorithm (such as the angle and speed of each joint to rotate, etc.), and generate motion instructions. Further perform safety verification (motion range limitation and collision prediction detection) and smoothing on the generated motion instructions to ensure the feasibility of the instructions and the stability of the robot's motion.

[0058] In some embodiments, the current state information of the humanoid robot can be obtained in real time through the robot's joint angle sensor, acceleration sensor, gyroscope and other sensor information. When the predicted motion intention is fused with the robot's current state, the weight of the robot's current position and posture information can be appropriately increased when the robot approaches the target object to ensure accurate operation; if a neural network processing method is used, the motion intention information and robot state information are used as inputs of the neural network, and the fused information is output after calculation and processing within the network.

[0059] According to some embodiments of the present application, in step S105, the step of determining the robot control instruction according to the predicted action intention and the current state of the robot may include but is not limited to the following steps: Step S701, determining a target action template corresponding to a motion primitive according to the predicted action intention, wherein the target action template is used to describe kinematic parameters and dynamic parameters required to be controlled for the target action; Step S702, according to the current state of the robot, the parameter control amount described by the target action template is calculated by the kinematic inverse solution algorithm to obtain the robot control instruction.

[0060] In this embodiment, since the execution of actions of different action intentions is quite different, for example, walking actions are mainly lower limb actions, while grasping actions are mainly fine hand movements, therefore, after identifying the action intention, the target action template can be further matched to limit the joints and parameter constraints to be executed, thereby achieving efficient control, as follows: The predicted action intention output by the model is mapped to motion primitives, including grabbing, walking, speech, etc., and the corresponding target action template is called. The target action template is predefined in the action template library. The template library is a set of predefined motion primitives. Each primitive corresponds to a standardized action template, which is used to describe the following: Kinematic parameters: target joint angle, velocity, acceleration curve; Dynamic parameters: impedance control gain (stiffness, damping); Environmental constraints: collision detection rules, safety thresholds.

[0061] Exemplarily, for the action template of the grasping motion primitive, its parameter description can be the end effector trajectory (Cartesian space), finger joint closure speed, and grasping force threshold (such as 10-100N); for the action template of the speech motion primitive, its parameter description can be the keyword trigger list ("stop", "accelerate"), and the voiceprint recognition confidence threshold (>0.8); for the action template of the walking motion primitive, its parameter description can be the gait cycle, foot end trajectory (ZMP planning), and trunk balance strategy.

[0062] After obtaining the target action template, the parameters in the target action template can also be adjusted according to the current environmental state, for example, the gait cycle of the walking motion can be adjusted according to the position of the object. According to the current state of the robot, the parameter control quantity described by the target action template is calculated through the kinematic inverse solution algorithm to obtain the robot control instruction. The inverse kinematics solution is to establish the robot model by the DH parameter method and solve the target joint angle. When executing the control at the bottom layer, the joint damping parameters are adjusted in combination with impedance control to achieve smooth operation. In addition, the control instructions can also be subjected to safety constraints and collision prediction during the bottom layer execution. The safety constraint conditions include a double verification mechanism, and the kinematic constraint checks include joint angles and speed limits. Collision prediction is based on the robot geometry model and the depth sensor data for neural network prediction.

[0063] According to some embodiments of the present application, after the step of obtaining the predicted action intention by inputting the multimodal features into the action intention recognition model, the humanoid robot adaptive control method of the embodiment of the present application may further include the following steps: Get the action task that the robot is currently executing; Determine whether the first priority of the predicted action intention is greater than the second priority of the action task currently being executed; When the first priority is greater than the second priority, the currently executed action task is interrupted and the action task of the predicted action intention is executed; When the first priority is less than or equal to the second priority, the action task of predicting the action intention is put into the task queue to be scheduled by the task queue.

[0064] In this embodiment, the priority of the predicted action intention and the action task currently being executed can be determined by querying. In this embodiment, taking into account that the user may generate a new action intention during the robot's execution of the action, in order to improve the rationality of the robot's execution of the action, priorities can be set for different action intentions, so that the robot can perform corresponding actions in sequence according to the priority of the action intention. For example, in dangerous situations, the user generally has the intention to stop, so the intention to stop can be set to the highest priority. When the predicted action intention is the intention to stop, the action tasks currently being executed, such as grabbing and walking, will be suspended. In another example, the priority of the grabbing action can be set to be greater than the priority of the behavioral action. When the robot is performing a grabbing action, if the user has the intention to walk, the robot will not immediately perform the walking action, but will wait until the grabbing action is completed before performing the walking action, thereby avoiding the falling of items due to the robot walking during the grabbing process.

[0065] According to some embodiments of the present application, the embodiments of the present application have the following beneficial effects: The embodiment of the present application fuses EEG signals and EMG signals, fully utilizes the complementary information of multimodal signals, captures human body movement intentions more comprehensively and accurately, and improves the accuracy of robot control.

[0066] The multimodal brain-muscle signal fusion model constructed in the embodiment of the present application has adaptive learning capabilities and can self-adjust according to the brain-muscle signal characteristics and movement intentions of different individuals, thereby enhancing the generalization ability of the system.

[0067] The embodiment of the present application generates adaptive motion instructions based on the current state of the humanoid robot, so that the robot can better adapt to complex and changing environments and achieve flexible and stable motion control. In addition, the embodiment of the present application sets a reasonable action intention execution priority to meet the actual needs of dangerous scenes or other special scenes. The present application also provides a humanoid robot adaptive control system, including: The first module is used to collect EEG signals and EMG signals of the target object; The second module is used to determine the first time-frequency domain feature of the EEG signal according to the EEG signal, and to determine the second time-frequency domain feature of the muscle activation state according to the EMG signal; The third module is used to fuse the first time-frequency domain feature and the second time-frequency domain feature using a dynamic weighting algorithm to obtain a multimodal feature; The fourth module is used to obtain the predicted action intention according to inputting the multimodal features into the action intention recognition model; The fifth module is used to determine the robot control instructions according to the predicted action intention and the current state of the robot to control the humanoid robot.

[0068] It can be understood that the contents of the above-mentioned humanoid robot adaptive control method embodiment are all applicable to the present system embodiment, the functions specifically implemented by the present system embodiment are the same as those in the above-mentioned humanoid robot adaptive control method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned humanoid robot adaptive control method embodiment.

[0069] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned humanoid robot adaptive control method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0070] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the humanoid robot adaptive control method of the embodiment of the present application; Input / output interface 903, used to implement information input and output; Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904); The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0071] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned humanoid robot adaptive control method.

[0072] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0073] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0074] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0075] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0076] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0077] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0078] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0079] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0080] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0081] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0083] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A humanoid robot adaptive control method, characterized in that: The following steps are involved: Collecting EEG signals and EMG signals of the target object; Determine a first time-frequency domain feature of the EEG signal according to the EEG signal, and determine a second time-frequency domain feature of a muscle activation state according to the EMG signal; Using a dynamic weighting algorithm to fuse the first time-frequency domain features and the second time-frequency domain features to obtain a multimodal feature; According to inputting the multimodal features into the action intention recognition model, a predicted action intention is obtained; A robot control instruction is determined according to the predicted action intention and the current state of the robot to control the humanoid robot.

2. The humanoid robot adaptive control method according to claim 1, characterized in that: Determining the first time-frequency domain feature of the EEG signal according to the EEG signal comprises the following steps: Performing multi-stage adaptive filtering and blind source separation processing on the EEG signal to obtain a first target signal; Performing continuous wavelet transform on the first target signal based on wavelet basis functions and scale parameters to obtain wavelet coefficients and center frequencies of different scales; The energy entropy of the target frequency is extracted according to the wavelet coefficients and center frequencies of different scales to obtain the first time-frequency domain feature.

3. The humanoid robot adaptive control method according to claim 1, characterized in that: Determining the second time-frequency domain feature of the muscle activation state according to the electromyographic signal comprises the following steps: Determine the muscle activation state at the corresponding moment according to the electromyographic signal; Performing noise reduction processing on the electromyographic signal at the corresponding moment according to the noise threshold in the muscle activation state to obtain a second target signal; The information entropy of the second target signal at different frequency components is extracted to obtain the second time-frequency domain features of the muscle activation state.

4. The humanoid robot adaptive control method according to claim 1, characterized in that: The method of fusing the first time-frequency domain feature and the second time-frequency domain feature using a dynamic weighting algorithm to obtain a multimodal feature includes the following steps: Using a dynamic weighting algorithm to calculate a first weight of the first time-frequency domain feature and a second weight of the second time-frequency domain feature; Multiplying the first weight by the first time-frequency domain feature to obtain an electroencephalogram feature, and multiplying the second weight by the second time-frequency domain feature to obtain an electromyography feature; The EEG features and the EMG features are time aligned and then spliced ​​to obtain multimodal features.

5. The method for adaptive control of a humanoid robot according to claim 4, characterized in that: The step of using a dynamic weighting algorithm to respectively calculate the first weight of the first time-frequency domain feature and the second weight of the second time-frequency domain feature comprises the following steps: Determining the quality of the EEG signal according to the signal power of the EEG signal in the target frequency band and the EMG resting noise power, and determining the quality of the EMG signal according to the signal power of the EMG signal and the EMG resting noise power, wherein the EMG resting noise power represents the baseline noise of the EMG signal in a resting state; A first weight of the first time-frequency domain feature is determined according to the ratio of the EEG signal quality to the total signal quality, and a second weight of the second time-frequency domain feature is determined according to the ratio of the EMG signal quality to the total signal quality.

6. The method for adaptive control of a humanoid robot according to any one of claims 1 to 5, characterized in that: The step of inputting the multimodal features into an action intention recognition model to obtain a predicted action intention comprises the following steps: The multimodal features are input into the dilated convolutional layer and the bidirectional long short-term memory network respectively, and multi-scale rhythm features and muscle activation timing pattern features are obtained correspondingly; A cross-modal attention mechanism is used to dynamically adjust the contribution of the multi-scale rhythm feature and the contribution of the muscle activation timing pattern feature; The predicted action intention is determined based on the multi-scale rhythm features and their contribution and the muscle activation timing pattern features and their contribution.

7. The method for adaptive control of a humanoid robot according to any one of claims 1 to 5, characterized in that: Determining the robot control instruction according to the predicted action intention and the current state of the robot comprises the following steps: Determine a target action template corresponding to the motion primitive according to the predicted action intention, wherein the target action template is used to describe the kinematic parameters and dynamic parameters required to be controlled for the target action; According to the current state of the robot, the parameter control quantity described by the target action template is calculated through the kinematic inverse solution algorithm to obtain the robot control instruction.

8. A humanoid robot adaptive control system, characterized in that: include: The first module is used to collect EEG signals and EMG signals of the target object; A second module is used to determine a first time-frequency domain feature of the EEG signal according to the EEG signal, and to determine a second time-frequency domain feature of a muscle activation state according to the EMG signal; A third module is used to fuse the first time-frequency domain features and the second time-frequency domain features using a dynamic weighting algorithm to obtain a multimodal feature; A fourth module is used to obtain a predicted action intention by inputting the multimodal features into an action intention recognition model; The fifth module is used to determine the robot control instruction according to the predicted action intention and the current state of the robot to control the humanoid robot.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 7.

Images