Robot control method and system based on electroencephalogram signals

Abstract

The invention discloses a robot control method and system based on electroencephalogram signals, a robot is an interactive robot, and the method comprises the following steps that feature extraction is conducted based on the electroencephalogram signals, and corresponding frequency domain features, connectivity features and time domain mode features are obtained; performing feature fusion and dimension reduction based on the frequency domain feature, the time domain mode feature and the connectivity feature to obtain a corresponding fusion feature; performing feature analysis on the fusion feature according to a preset control strategy to obtain a corresponding analysis result; based on the control strategy, a control instruction corresponding to the analysis result is generated, and a target robot is controlled based on the control instruction; the connectivity characteristic comprises a phase locking value between each pair of channel signals; the time domain mode feature comprises the mode complexity of the electroencephalogram signal. According to the method, dynamic interaction with the target robot is achieved through the electroencephalogram signals by designing the frequency domain features, the connectivity features and the time domain mode features.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interfaces, and more particularly to a robot control method and system based on electroencephalogram (EEG) signals; Background Technology

[0002] Interactive robots such as robot dogs currently rely mainly on manual remote control, voice or mobile app control, lacking the ability to directly utilize brain signals for intelligent response.

[0003] Brain-computer interface (BCI) technology enables human-computer interaction by capturing brain signals, showing potential in fields such as medicine, gaming, and industrial control. However, existing brain-controlled systems are mostly limited to simple on / off commands, making it difficult to achieve multi-level, dynamic interaction and unsuitable for controlling interactive robots such as robotic dogs. Summary of the Invention

[0004] This invention addresses the lack of control schemes for interactive robots based on electroencephalogram (EEG) signals in the prior art by providing a robot control method and system that combines frequency domain features, connectivity features, and time domain pattern features based on EEG signals.

[0005] This invention proposes a robot control method based on electroencephalogram (EEG) signals, which controls the actions of a target robot based on the EEG signals of a target user. The target robot is an interactive robot.

[0006] Includes the following steps:

[0007] Feature extraction is performed based on the EEG signals to obtain corresponding frequency domain features, connectivity features, and time domain pattern features;

[0008] Based on the frequency domain features, the time domain pattern features, and the connectivity features, feature fusion and dimensionality reduction are performed to obtain the corresponding fused features.

[0009] The fused features are analyzed according to a preset control strategy to obtain corresponding analysis results;

[0010] Based on the control strategy, control commands corresponding to the analysis results are generated, and the target robot is controlled based on the control commands.

[0011] The EEG signal contains multiple channel signals. By combining the channel signals in pairs, at least one pair of channel signals can be obtained.

[0012] The connectivity features include a phase lock value between each pair of channel signals;

[0013] The temporal pattern features include the pattern complexity of the EEG signal.

[0014] As one possible implementation method, the preset control strategy includes an active control strategy, which includes brain control scoring rules and brain control feedback rules. The brain control feedback rules include several attention levels, the brain control score range corresponding to each attention level, and the execution actions of the target robot.

[0015] Based on the brain control scoring rules, corresponding brain control scores are generated based on fusion features;

[0016] Based on the brain control feedback rules, the focus level corresponding to the brain control score is determined, and corresponding control instructions are generated based on the execution action corresponding to the focus level.

[0017] As one possible implementation, a focus judgment is made on a preset length of EEG signal segment based on a preset feedback cycle, and a corresponding feedback signal is generated based on the focus judgment result and sent to the corresponding brain-computer device, which then provides feedback to the target user based on the feedback signal.

[0018] Among them, the EEG signal segment includes several sub-segments with a unit feedback length;

[0019] The specific steps for judging focus and generating corresponding feedback signals based on target EEG signal segments are as follows:

[0020] The pattern complexity corresponding to each sub-segment in the target EEG signal segment is obtained respectively;

[0021] The phase lock value corresponding to each sub-segment in the target EEG signal segment is obtained respectively, and the average phase lock value corresponding to each sub-segment is obtained;

[0022] Determine whether the pattern complexity and average phase lock value corresponding to each sub-segment meet the preset attention level standards. If they do, mark the corresponding sub-segment as qualified; otherwise, mark the corresponding sub-segment as unqualified.

[0023] The focus level of each sub-segment is encoded to generate corresponding feedback action sequences and obtain feedback signals.

[0024] As one possible implementation, a mode configuration step is also included, specifically:

[0025] The target operating mode is determined based on the user configuration information, and the corresponding control strategy is obtained.

[0026] As one possible implementation method, the control strategy includes an adaptive control strategy, which includes state classification rules and state feedback strategies that correspond one-to-one with brain states, including fatigue, focus, sleep, meditation, and alcohol effects.

[0027] Based on the state classification rules, a preset classification model performs brain state classification based on the corresponding fusion features to obtain the target brain state corresponding to the fusion features.

[0028] Acquire and execute a state feedback strategy corresponding to the target brain state.

[0029] As one possible implementation, feature extraction is performed on the corresponding EEG signals based on an adaptive control strategy to obtain the corresponding frequency domain features, connectivity features, temporal pattern features, and channel mutual information.

[0030] Based on the frequency domain features, the time domain mode features, the connectivity features, and the channel mutual information, feature fusion and dimensionality reduction are performed to obtain the corresponding fused features.

[0031] As one possible implementation, the input of the classification model is the corresponding fusion features, and the output is the classification probability corresponding to each brain state. The target brain state is determined based on the classification probability.

[0032] The state feedback strategy includes several stages corresponding to the brain state, as well as stage judgment rules and state feedback rules corresponding to each stage state.

[0033] The stage determination rule is used to indicate the required state maintenance time and / or classification probability for the corresponding stage state.

[0034] As one possible implementation, the EEG signal is divided into several unit signal segments, and each unit signal segment is further divided into sub-frequency bands to obtain several sub-frequency bands. The frequency band normalized power corresponding to each sub-frequency band is calculated to obtain the corresponding frequency domain features. The sub-frequency bands are δ band, θ band, α band, and β band.

[0035] The steps to determine each sub-frequency band in the current unit signal segment are as follows:

[0036] Identify the dominant frequency band in the current unit signal segment;

[0037] The power spectral density corresponding to the dominant frequency band is calculated by fast Fourier transform, and the energy distribution data is obtained by analysis.

[0038] Based on the power spectral density and the energy distribution data, a clustering algorithm is used to adjust the boundary of the corresponding reference sub-frequency band to obtain the corresponding sub-frequency band, wherein the reference sub-frequency band is the sub-frequency band divided by the previous unit signal segment or a preset initial sub-frequency band.

[0039] As one possible implementation, when the channel signal is greater than 2, the connectivity feature further includes spectral features, and the method for obtaining the spectral features includes:

[0040] Construct an adjacency matrix based on the phase-locked value of each pair of channel signals;

[0041] The adjacency matrix is ​​mapped to a weighted undirected graph, where the weight of an edge is the phase lock value between the two channel signals connected.

[0042] Extract corresponding graph features from the weighted undirected graph, the graph features including node degree and clustering coefficient.

[0043] The present invention also proposes a robot control system based on electroencephalogram (EEG) signals, which controls the action of a target robot based on the EEG signals of a target user, wherein the target robot is an interactive robot;

[0044] include:

[0045] Signal processing module:

[0046] This is used to extract features based on the EEG signal to obtain corresponding frequency domain features, connectivity features, and temporal domain pattern features; the EEG signal contains multiple channel signals, and the channel signals are combined in pairs to obtain several pairs of channel signals; the connectivity features include the phase lock value between each pair of channel signals; the temporal domain pattern features include the pattern complexity of the EEG signal;

[0047] This is used to perform feature fusion and dimensionality reduction based on the frequency domain features, the time domain mode features, and the connectivity features to obtain corresponding fused features;

[0048] Used to perform feature analysis on the fused features according to a preset control strategy, and obtain corresponding analysis results;

[0049] The control module is used to generate control commands corresponding to the analysis results based on the control strategy, and to control the target robot based on the control commands.

[0050] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0051] This invention, by adopting the above technical solutions, has significant technical effects:

[0052] This invention, through the design of frequency domain features, connectivity features, and time domain pattern features, can significantly improve feature resolution, making it suitable for dynamic interaction of interactive robots. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating a robot control method based on electroencephalogram (EEG) signals according to Embodiment 1 of the present invention.

[0055] Figure 2 This is a flowchart illustrating a robot control method based on electroencephalogram (EEG) signals according to Embodiment 2 of the present invention.

[0056] Figure 3 This is a flowchart illustrating a robot control method based on electroencephalogram (EEG) signals in Embodiment 4 of the present invention. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0058] Example 1: A robot control method based on electroencephalogram (EEG) signals, used in scenarios such as entertainment, rehabilitation training, and concentration training where the target user actively controls the target robot through their EEG signals;

[0059] Reference Figure 1 This includes the following steps:

[0060] S100: Acquire the raw EEG signals collected by the brain-computer interface device, and preprocess the raw EEG signals to obtain corresponding EEG signals;

[0061] Preprocessing methods include:

[0062] An adaptive bandpass filter (0.5–50Hz) is used, and the cutoff frequency is dynamically adjusted according to the signal power spectrum.

[0063] To filter out eye movement, electromyography (EMG), and power frequency interference, this embodiment uses independent component analysis (ICA) to separate eye movement (0.5–4 Hz) and EMG (20–100 Hz) artifacts while retaining the main components of the EEG signal.

[0064] This preprocessing procedure reduces computational complexity, making it suitable for real-time processing while ensuring the quality of EEG signals.

[0065] Note that the above preprocessing methods are all existing and publicly available technologies, so they will not be described in detail.

[0066] S200. Based on the EEG signal, feature extraction is performed to obtain corresponding frequency domain features, connectivity features, and time domain pattern features;

[0067] S210. Extract frequency domain features;

[0068] The frequency domain feature is the normalized power of each sub-frequency band, and the sub-frequency bands are δ band, θ band, α band and β band;

[0069] The formula for calculating the normalized power of the frequency band is:

[0070]

[0071] P k This represents the normalized power of the frequency band corresponding to the k-th sub-band;

[0072] B k Let f ∈ B represent the k-th sub-band. k This indicates that frequency point f belongs to B. k ;

[0073] S(f) represents the power spectral amplitude at a frequency point;

[0074] This represents the sum of squares of the power spectral magnitudes at all frequency points in the k-th sub-band, that is, the power corresponding to the k-th sub-band;

[0075] ∑|S(f)| 2 This represents the sum of squares of the power spectral amplitudes at all frequency points in the dominant frequency band (e.g., 0.5-40Hz), i.e., the total power.

[0076] As one possible implementation method, the accuracy of frequency domain features can be improved by adaptive frequency band division. That is, the EEG signal is divided into several unit signal segments, each unit signal segment is divided into sub-frequency bands to obtain several sub-frequency bands, and the frequency band normalized power corresponding to each sub-frequency band is calculated.

[0077] In this embodiment, the EEG signal is segmented using a 1-second sliding window to obtain the corresponding unit signal segment.

[0078] The specific steps are as follows:

[0079] S211. Determine the dominant frequency band in the current unit signal segment;

[0080] The dominant frequency band can be a preset value (0.5–45Hz) or a dynamically dominant frequency band obtained through identification.

[0081] As one possible implementation method, the power spectrum of the current unit signal segment is analyzed to identify the dominant frequency band;

[0082] The dominant frequency band refers to the main frequency range in the power spectrum where energy (power) is concentrated. In this embodiment, the dominant frequency band is the frequency band involved in the δ band, θ band, α band and β band. Due to the differences in individual EEG signals, the range of the dominant frequency band may also vary.

[0083] S212. Calculate the power spectral density corresponding to the dominant frequency band through fast Fourier transform, and analyze to obtain energy distribution data;

[0084] In this embodiment, the energy distribution data includes the frequency peak and the energy range that reaches the preset energy threshold (80%).

[0085] The methods for obtaining the above-mentioned frequency peaks and main energy ranges are all existing technologies, and therefore will not be described in detail in this specification.

[0086] S213. Based on the power spectral density and the energy distribution data, a clustering algorithm is used to adjust the boundary of the corresponding reference sub-frequency band to obtain the corresponding sub-frequency band;

[0087] The reference sub-band is the sub-band divided by the previous unit signal segment or a preset initial sub-band;

[0088] The initial sub-bands are the ranges corresponding to the δ band, θ band, α band, and β band in the prior art.

[0089] In this embodiment:

[0090] Using four frequency peaks as initial cluster centers, clustering is performed using existing publicly available clustering algorithms to ensure that each frequency band covers the main energy range (80%) and does not overlap with each other;

[0091] The boundaries of the reference sub-bands are adjusted based on the clustering results;

[0092] As one possible implementation, to ensure stability, a constraint condition is set that the variance is less than a preset variance threshold (5%), thereby enabling fine-tuning of the corresponding boundaries based on the clustering results.

[0093] Note that, given the initial cluster centers and the required constraints (each frequency band does not overlap, and the variance corresponding to the corresponding reference sub-frequency band is less than the preset variance threshold), those skilled in the art can choose existing publicly available clustering algorithms to implement clustering to determine the boundaries of the corresponding sub-frequency bands according to actual needs.

[0094] S214. Calculate the normalized power of each sub-band.

[0095] Compared with existing technologies that extract features based on standard ranges of the δ, θ, α, and β bands, this embodiment uses dynamic frequency band division to adapt to individual signal differences, resulting in higher accuracy.

[0096] S220, Extract connectivity features;

[0097] The EEG signal contains multiple channel signals. By combining the channel signals in pairs, one or more pairs of channel signals can be obtained. The connectivity feature includes the phase lock value between each pair of channel signals.

[0098] As one possible implementation, the EEG signal contains two channel signals, and when corresponding to a pair of channel signals, the connectivity feature includes the phase lock value between the pair of channel signals;

[0099] As another possible implementation, when the EEG signal contains more than two channel signals, the channel signals are combined in pairs to obtain multiple pairs of channel signals. In this case, the connectivity features may include not only the phase lock values ​​between each pair of channel signals, but also atlas features.

[0100] S221, Phase Lock Value;

[0101] In this embodiment, a 1-second sliding window is used to segment the EEG signal to obtain the corresponding unit signal segment, and the dominant frequency band (0.5-45Hz) in the unit signal segment is extracted.

[0102] The method for calculating the phase lock value between the first channel signal and the second channel signal in the dominant frequency band is as follows:

[0103] Based on the Hilbert transform, the instantaneous phases corresponding to the first and second channel signals are obtained. The formula for calculating the instantaneous phase is as follows:

[0104]

[0105] Based on the obtained instantaneous phase, the phase difference between the first channel signal and the second channel signal is calculated. The formula for calculating the phase difference is:

[0106] ΔΦ ij (t)=Φ i (t)-Φ j (t)

[0107] in:

[0108] Φ i (t) represents the instantaneous phase corresponding to the i-th channel signal, that is, the instantaneous phase corresponding to the first channel signal;

[0109] Φ j (t) represents the instantaneous phase corresponding to the j-th channel signal, which is the instantaneous phase corresponding to the second channel signal;

[0110] ΔΦ ij (t) represents the phase difference between the i-th channel signal and the j-th channel signal, that is, the phase difference between the first channel signal and the second channel signal.

[0111] Based on the phase difference, the phase lock value between the first channel signal and the second channel signal is calculated using the following formula:

[0112]

[0113] in:

[0114] PLV ij PLV represents the phase-locked value between the i-th channel signal and the j-th channel signal, i.e., the phase-locked value between the first channel signal and the second channel signal. ij The value is between 0 and 1, with a higher value indicating a higher degree of connectivity;

[0115] N represents the signal length, which is 1 second in this embodiment;

[0116] t represents the time index;

[0117] ΔΦ ij (t) represents the phase difference between the i-th channel signal and the j-th channel signal, that is, the phase difference between the first channel signal and the second channel signal.

[0118] The phase difference is converted into a unit vector on the complex plane based on Euler's formula.

[0119] S221, Spectral characteristics;

[0120] In this embodiment, the graph features include node degree and clustering coefficient, which reflect the network synchronization of brain states.

[0121] Methods for obtaining spectral features include:

[0122] Construct an adjacency matrix A based on the phase-locked value of each pair of channel signals;

[0123] For M channels, an M×M adjacency matrix A is generated based on the PLV values ​​of all channel pairs. ij This represents the PLV value between channel i and channel j. Since PLV is symmetrical (PLV... ij =PLV ji Therefore, the adjacency matrix A is a symmetric matrix, with diagonal elements A ij Set it to 0.

[0124] Map the adjacency matrix A to a weighted undirected graph G(V,E), where V is the set of nodes (i.e., channels), E is the set of edges (i.e., PLV values), and the weight of each edge (i,j) is PLV. ij The weight of an edge is between 0 and 1, meaning that the weight of an edge is the phase lock value between the two connected channel signals.

[0125] Extract the corresponding node degree and clustering coefficient from the weighted undirected graph G(V,E).

[0126] Nodal degree reflects the overall synchronization strength of a channel with other channels. During meditation, the alpha wave synchronization is enhanced, which may lead to a significant increase in the degree of certain nodes (such as the frontal lobe channel Fz).

[0127] In this embodiment, the node degree of each node in the weighted undirected graph G(V,E) is calculated, where the node degree of node i is the sum of the weights of all edges connected to that node, and the calculation formula is as follows:

[0128]

[0129] Clustering coefficients reflect the synchronicity of local brain regions. For example, in a meditative state, the clustering coefficient of the posterior parietal lobe region may increase due to alpha wave synchronization.

[0130] In this embodiment, the clustering coefficient corresponding to each node in the weighted undirected graph G(V,E) is calculated to measure the degree of connection between its neighboring nodes. The formula for calculating the clustering coefficient of node i is as follows:

[0131]

[0132] This embodiment utilizes dynamic interaction between PLV and the map feature capture channel.

[0133] S220, Extract temporal pattern features;

[0134] The temporal pattern features include the pattern complexity of the EEG signal, and the pattern frequency can also be calculated as a temporal pattern feature according to actual needs.

[0135] The specific implementation steps are as follows:

[0136] The time-domain pattern in each channel signal is detected. In this embodiment, a repeating waveform pattern with a length of 0.5 seconds is taken as the time-domain pattern.

[0137] The fast pattern matching algorithm is used to calculate the pattern frequency and pattern complexity.

[0138] Wherein, the pattern frequency of pattern m is the number of times pattern m occurs / the total number of times the pattern occurs;

[0139] This embodiment calculates the pattern complexity corresponding to the EEG signal based on Shannon entropy. The calculation formula is as follows:

[0140]

[0141] Where M is the set of patterns, and p(m) is the probability of pattern m occurring.

[0142] This embodiment captures the patterns of nonlinear signals through time-domain pattern detection, supplementing frequency-domain features.

[0143] S300. Based on the frequency domain features, the time domain pattern features, and the connectivity features, feature fusion and dimensionality reduction are performed to obtain the corresponding fused features.

[0144] All extracted features are integrated to form a high-dimensional feature vector. Those skilled in the art can then concatenate the extracted features to achieve feature fusion.

[0145] In this embodiment, Sparse Principal Component Analysis (Sparse PCA) is applied to reduce the high-dimensional eigenvectors obtained by feature fusion to low dimensions while retaining 90% of the variance, thus obtaining the corresponding fused features. The formula is as follows:

[0146] X reduced =X*W

[0147] Among them, X reduced These are the resulting fusion features;

[0148] X is the original feature matrix, that is, the high-dimensional feature vector;

[0149] W is a sparse projection matrix.

[0150] S400. Perform feature analysis on the fused features according to the preset control strategy to obtain the corresponding analysis results;

[0151] In this embodiment, the control strategy adopts a preset active control strategy, which is used to calculate the brain control score of the EEG signal and control the target robot to give corresponding feedback according to the brain control score. The higher the brain control score, the higher the concentration, the more active the target robot is, and the more complex the movements are, so as to encourage the target user to concentrate.

[0152] In this embodiment, the active control strategy includes brain control scoring rules and brain control feedback rules. The brain control feedback rules include several attention levels, the brain control score range corresponding to each attention level, and the execution actions of the target robot.

[0153] In practical applications, a corresponding brain control score will be generated based on brain control scoring rules and fusion features; and the focus level corresponding to the brain control score will be determined according to the brain control feedback rules, and the execution action corresponding to the focus level will be determined.

[0154] The method for calculating the mind control score is as follows:

[0155] Each sub-feature in the fused feature set is normalized to [0,1]. Here, the sub-features are the principal components obtained from sparse principal component analysis. After weighted summation of each sub-feature, a score transformation is performed, and the resulting sum is mapped to 0-100 to obtain the corresponding mind control score.

[0156] Those skilled in the art can set the weighting weights corresponding to each sub-feature according to actual needs, so that the sum of all weighting weights is 1. This specification does not impose detailed limitations on this.

[0157] In this embodiment, the brain-controlled feedback rules are as follows:

[0158] Low score (0-30): Walk slowly.

[0159] Medium score (31-60): Running.

[0160] High score (61-85): Jump.

[0161] Highest score (86-100): Roll.

[0162] Note that users can configure the brain control score range and robot dog actions corresponding to each focus level according to their actual needs.

[0163] S500. Based on the control strategy, generate control instructions corresponding to the analysis results, and control the target robot based on the control instructions;

[0164] As can be seen from the above, the analysis results in this embodiment are the level and the executed action corresponding to the brain control score;

[0165] In this embodiment, corresponding control instructions will be generated based on the corresponding execution action to control the target robot to perform the execution action.

[0166] As one possible implementation method, the control strategy also includes action triggering rules and interference detection rules;

[0167] Action triggering rules are used to indicate the conditions for generating and issuing action triggering control commands based on the execution of actions. In this embodiment, the action triggering rules include the number of scores to be analyzed (e.g., two) and the score deviation threshold (e.g., variance < 5). That is, the score must be stable within 2 seconds (variance < 5) before the action is triggered.

[0168] Interference detection rules are used to indicate the judgment conditions for abnormal situations and the corresponding handling methods. For example, when the average PLV exceeds the normal range (<0.2 or >0.9) or the pattern complexity changes abruptly (>2 times the standard deviation), control is suspended, and an abnormal feedback signal is sent to the brain-computer device, so that the brain-computer device can provide sound or vibration prompts to the target user based on the abnormal feedback signal, such as a 1-second vibration prompt or an abnormal prompt sound.

[0169] This embodiment adaptively divides the frequency bands to make the extracted frequency domain features fit individual differences; this embodiment enhances stability in complex environments by extracting connectivity features and temporal pattern features, thereby improving the accuracy of brain control scores and enhancing the effect of focus training in scenarios where the target robot is actively controlled.

[0170] As one possible implementation, a control feedback step is also included;

[0171] Focus is assessed on a preset length of EEG signal segment based on a preset feedback cycle. Based on the focus assessment result, a corresponding feedback signal is generated and sent to the corresponding brain-computer interface device, which then provides feedback to the target user based on the feedback signal.

[0172] Among them, the EEG signal segment includes several sub-segments with a unit feedback length;

[0173] The specific steps for judging focus and generating corresponding feedback signals based on target EEG signal segments are as follows:

[0174] The pattern complexity corresponding to each sub-segment in the target EEG signal segment is obtained respectively;

[0175] The phase lock value corresponding to each sub-segment in the target EEG signal segment is obtained respectively, and the average phase lock value corresponding to each sub-segment is obtained;

[0176] Determine whether the pattern complexity and average phase lock value corresponding to each sub-segment meet the preset attention level standards. If they do, mark the corresponding sub-segment as qualified; otherwise, mark the corresponding sub-segment as unqualified.

[0177] The focus level of each sub-segment is encoded to generate corresponding feedback action sequences and obtain feedback signals.

[0178] Those skilled in the art can set the feedback cycle and the conditions for achieving the required level of focus according to the actual situation, and this specification does not impose detailed limitations on them.

[0179] In this embodiment, the length of the EEG signal segment is 8 seconds, and the unit feedback length is 2 seconds, which is divided into 4 sub-windows of 2 seconds each.

[0180] When the average phase lock value of the corresponding sub-segment is greater than the preset first threshold (0.7) and the pattern complexity (1.5) is greater than the preset second threshold, the focus level is determined to meet the standard and is encoded as "1"; otherwise, it is encoded as "0".

[0181] An 8-second signal generates a 4-bit code (e.g., "1001"), corresponding to a feedback action sequence. This sequence then enables the brain-computer interface to provide tactile or auditory feedback. For example, when providing tactile feedback:

[0182] "0": No vibration.

[0183] “1”: Short vibration (0.5 seconds, 50 Hz, intensity 0.8).

[0184] Users can adjust the status in a timely manner based on tactile feedback.

[0185] In practical applications, the robot will perform corresponding actions based on the user's brain control score. However, since the action corresponds to a range, the user cannot clearly know whether their concentration has increased or decreased before the concentration level corresponding to the user's brain control score changes. This hinders the improvement of the user's concentration, especially in scenarios of concentration training or rehabilitation training, where the training effect cannot be optimal.

[0186] To address the aforementioned issues, this embodiment proposes a multimodal feedback method. In scenarios where users actively interact with a target robot by controlling their focus, the method encodes the feedback action sequence based on the phase-locking value and pattern complexity to determine whether the level of focus meets the standard. Even if the target robot's actions do not change, users can still understand the changes in their focus based on the feedback from the brain-computer interface according to the feedback action sequence and adjust accordingly. For example, if a user feels an increased vibration in a vibration feedback, it indicates that their focus is increasing; conversely, if they feel a decrease in the number of vibrations, it indicates that their focus has decreased.

[0187] Case 1:

[0188] User A interacts with a robot dog. Initially, when the user's brain control score is 40, the robot dog runs. The code "1001" triggers a vibration (short-still-still-short). The user focuses their attention, and the brain control score rises to 55. The robot dog continues running, but this time the code "1101" triggers a vibration (short-short-still-short). The user learns through the vibration feedback that their focus is improving, encouraging them to maintain focus and increasing user activity and satisfaction. This is suitable for scenarios such as family entertainment, children's focus training, and rehabilitation training for brain injury patients.

[0189] As one possible implementation method, brain state classification is also based on fusion features;

[0190] Pre-train a lightweight convolutional neural network (CNN) or a lightweight LSTM network as a classification model;

[0191] In one embodiment, brain states include focus and fatigue;

[0192] The fused features are input into the trained classification model, which outputs the classification probability corresponding to each type of brain state. The brain state is then determined based on the classification probability.

[0193] It also pre-programs corresponding actions for fatigue states to remind or relax the robot dog, such as having the robot dog bark softly as a reminder or having the robot dog circle around to relax.

[0194] Note that the above networks all use existing publicly available network architectures. Those skilled in the art, knowing the input (fusion features) and output (classification probabilities of various brain states), can train the corresponding classification model based on existing model training methods.

[0195] Case 2:

[0196] User B (8 years old) interacted with a robotic dog to train their concentration. When their brain control score dropped to 25, the robotic dog slowed down. At this stage, when the robot detected brain fatigue, it would bark softly while walking slowly to remind and encourage the child to refocus. Parents can adjust the score ranges corresponding to each concentration level based on their child's training progress to improve the training effect.

[0197] User C (a brain injury patient) interacts with the robot dog for rehabilitation training. When the user focuses on improving their brain control score to 65, the robot dog changes from running to jumping to encourage the user to maintain focus. To detect fatigue, the robot dog performs circling movements (0.2 m / s) according to the user's settings, thus relaxing the user.

[0198] Example 2: A robot control method based on EEG signals, used in scenarios such as daily companionship where the target robot is adaptively controlled based on the user's EEG signals;

[0199] In Example 1, steps S400 to S500, the control strategy is changed from an active control strategy to an adaptive control strategy.

[0200] The adaptive control strategy includes state classification rules and state feedback strategies that correspond one-to-one with brain states. In this embodiment, the brain states include fatigue, focus, sleep, meditation, and alcohol effects.

[0201] Reference Figure 2 In this embodiment, the fused features are analyzed according to an adaptive control strategy to obtain corresponding analysis results, and control commands corresponding to the analysis results are generated to control the target robot to perform corresponding actions. The specific steps are as follows:

[0202] S600. Based on the state classification rules, the brain state is classified by a preset classification model based on the corresponding fusion features to obtain the target brain state corresponding to the fusion features.

[0203] Pre-train a lightweight convolutional neural network (CNN) or a lightweight LSTM network as a classification model;

[0204] In this embodiment, the input to the classification model is the corresponding fusion feature, and the output is the classification probability of various brain states.

[0205] Those skilled in the art can collect EEG signals corresponding to various brain states, generate corresponding fusion sample features and label their respective brain states to obtain corresponding training data, and train existing and publicly available lightweight convolutional neural networks (CNNs) or lightweight LSTM networks based on existing and publicly available training methods to obtain classification models. This embodiment does not limit the specific training scheme.

[0206] As one possible implementation method, channel mutual information is also extracted based on EEG signals to generate corresponding fusion features;

[0207] Specifically:

[0208] Feature extraction is performed on the corresponding EEG signals to obtain the corresponding frequency domain features, connectivity features, temporal pattern features, and channel mutual information;

[0209] Based on the frequency domain features, the time domain mode features, the connectivity features, and the channel mutual information, feature fusion and dimensionality reduction are performed to obtain the corresponding fused features.

[0210] This embodiment enhances the distinguishability of complex states (such as meditation) by designing connectivity features and capturing the dynamic interaction between channels through phase-locked values ​​and spectral features.

[0211] This embodiment effectively improves the resolution of fatigue and focus states through the design of time-domain pattern features.

[0212] This embodiment, through the design of channel mutual information, can reflect the nonlinear dependence between channels, thereby further improving the accuracy of brain state classification.

[0213] The formula for calculating channel mutual information is:

[0214]

[0215] in:

[0216] MI ij This represents the mutual information between the i-th channel and the j-th channel;

[0217] x i Corresponding to the i-th channel signal;

[0218] x j Corresponding to the j-th channel signal;

[0219] p(x i ,x j ) represents the joint probability distribution, i.e., x i and x j The probability of both occurring simultaneously.

[0220] p(x i ) represents xi The marginal probability distribution;

[0221] p(x j ) represents x i The marginal probability distribution.

[0222] S600: Obtain and execute a state feedback strategy corresponding to the target brain state.

[0223] The input to the classification model is the corresponding fusion features, and the output is the classification probability corresponding to each brain state. The target brain state is determined based on the classification probability.

[0224] The state feedback strategy includes several stages corresponding to the brain state, as well as stage judgment rules and state feedback rules corresponding to each stage state.

[0225] The stage determination rule is used to indicate the required state maintenance time and / or classification probability for the corresponding stage state.

[0226] For example:

[0227] Fatigue is classified into the first fatigue state and the second fatigue state based on the duration of the fatigue state.

[0228] In the first fatigue stage, when the brain state is determined to be fatigued based on the classification probability, the action is to sway left and right (frequency 0.5Hz, lasting 20 seconds) to help the user relax.

[0229] In the second fatigue stage, when the brain is fatigued for 30 minutes, the action is to sway from side to side (frequency 0.5Hz, lasting 20 seconds) + short barks (3 times, volume 60dB) to help the user relax and remind the user to rest.

[0230] Focus can be divided into the first state of focus and the second state of focus based on the duration of the focused state.

[0231] In the first focus phase, when the brain state is determined to be focused based on the classification probability, the action is to remain still and turn off the lights and sounds.

[0232] In the second focus phase, when the brain is focused for 15 minutes, the action is to control the robot dog to enter ultra-low power mode.

[0233] Sleep is categorized into sleep aid, guardian, and welcoming based on the duration of the sleep state and the changes in classification probability.

[0234] To aid sleep, when the brain state is determined to be sleep based on the aforementioned classification probability, white noise is played (volume gradually decreasing to 20dB, 5 minutes), and the lights are gradually dimmed (3000K, 10 lux).

[0235] When the user is asleep for 30 minutes and the corresponding classification probability is higher than the preset probability threshold (indicating that the user has fallen asleep), the action is to monitor for abnormal sounds. When an abnormal sound (>70dB) is detected, a soft prompt (40dB, 1 second) is triggered.

[0236] Welcoming: If the corresponding classification probability drops below the preset probability threshold or the determined brain state changes, walk slowly to 1 meter away from the user and beep softly (50dB, 2 seconds).

[0237] Meditation is divided into the first meditation state and the second meditation state based on the duration of the meditation state;

[0238] In the first meditative state, when the brain state is determined to be meditative based on the classification probability, the action is stillness + playing breathing sounds (30dB, rhythm 4 seconds / time).

[0239] The second meditation state, the mental state, is when the meditation lasts for ten minutes, and the slight adjustment of posture (raising the head 5°) prompts the end.

[0240] The effects of alcohol are classified into mild, moderate, and severe based on the corresponding probability of alcohol effects.

[0241] Mild: When the brain state is determined to be affected by alcohol based on the classification probability, and the classification probability is less than a preset first alcohol probability threshold, the action is recorded for time and the system is monitored while stationary.

[0242] In the case of moderate alcohol poisoning, if the brain state is determined to be affected by alcohol based on the classification probability, and the classification probability reaches a preset first alcohol probability threshold but is less than a preset second alcohol probability threshold, the action is a prompt sound (3 times, 60dB) + slowly walking to within sight (0.2m / s).

[0243] Serious: If the brain state is determined to be influenced by alcohol based on the classification probability, and the classification probability is greater than the preset second alcohol probability threshold, the action is to notify the emergency contact (SMS, which can be canceled after a 30-second delay).

[0244] In practical applications, users can adjust the judgment conditions and execution actions of each brain state according to their actual needs.

[0245] As one possible implementation, reinforcement learning optimization is performed based on Q-learning updates:

[0246] Objective: Maximize user satisfaction (activity score or manual feedback).

[0247] As one possible implementation method:

[0248] The average value of the classification probability of each brain state is calculated based on the time window within a preset duration (3s), and the brain state with the largest average value is taken as the target brain state.

[0249] When the target brain state changes, after the target brain state duration reaches the preset observation time (5s), the state feedback strategy corresponding to the target brain state is obtained and the state switch is executed.

[0250] Example 3: A robot control method based on electroencephalogram (EEG) signals, which can switch between active control strategy and adaptive control strategy based on the user-configured working model;

[0251] That is, the target working mode is determined based on the user configuration information, and the corresponding control strategy is obtained.

[0252] Reference Figure 3 :

[0253] When the target control strategy is an active control strategy, the robot is controlled according to the method described in Example 1;

[0254] When the target control strategy is an adaptive control strategy, the robot is controlled according to the method described in Example 2.

[0255] Example 4: A robot control system based on electroencephalogram (EEG) signals, comprising:

[0256] Signal processing module:

[0257] This is used to extract features based on the EEG signal to obtain corresponding frequency domain features, connectivity features, and temporal domain pattern features; the EEG signal contains multiple channel signals, and the channel signals are combined in pairs to obtain several pairs of channel signals; the connectivity features include the phase lock value between each pair of channel signals; the temporal domain pattern features include the pattern complexity of the EEG signal;

[0258] This is used to perform feature fusion and dimensionality reduction based on the frequency domain features, the time domain mode features, and the connectivity features to obtain corresponding fused features;

[0259] Used to perform feature analysis on the fused features according to a preset control strategy, and obtain corresponding analysis results;

[0260] The control module is used to generate control commands corresponding to the analysis results based on the control strategy, and to control the target robot based on the control commands.

[0261] The signal processing module includes:

[0262] The preprocessing unit is used to preprocess the raw EEG signals received from the target user to obtain EEG signals;

[0263] The first feature extraction unit is used to extract corresponding frequency domain features based on the electroencephalogram (EEG) signal;

[0264] The second feature extraction unit is used to extract corresponding connectivity features based on the electroencephalogram (EEG) signal.

[0265] The third feature extraction unit is used to extract corresponding temporal pattern features based on the electroencephalogram (EEG) signal;

[0266] The fourth feature extraction unit is used to extract corresponding channel mutual information features based on the electroencephalogram (EEG) signal;

[0267] The feature fusion unit is used to perform feature fusion and dimensionality reduction based on the extracted frequency domain features, connectivity features, and time domain mode features to obtain the corresponding first fused features. It is also used to perform feature fusion and dimensionality reduction based on the extracted frequency domain features, connectivity features, time domain mode features, and channel mutual features to obtain the corresponding second fused features.

[0268] The brain control score calculation unit is used to perform feature analysis on the first fusion feature according to a preset active control strategy, obtain the corresponding analysis results, that is, generate the corresponding brain control score based on the first fusion feature, and determine the corresponding execution action based on the brain control score.

[0269] The brain state classification unit is used to perform feature analysis on the first fusion feature or the second fusion feature according to a preset adaptive control strategy, obtain the corresponding analysis results, input the first fusion feature or the second fusion feature into a preset classification model, determine the target brain state based on the classification probability of each brain state output by the classification model, and obtain the corresponding state feedback rules based on the target brain state to determine the corresponding execution action.

[0270] The feedback analysis unit is used to perform attention judgment on EEG signal segments of a preset length based on a preset feedback cycle and generate corresponding attention judgment results.

[0271] The control module includes:

[0272] The first control unit generates corresponding execution instructions based on the execution actions determined by the brain control score calculation unit and the brain state classification unit, and controls the target robot's actions based on the execution instructions.

[0273] The second control unit generates a feedback signal based on the attention judgment result generated by the feedback analysis unit, and sends the feedback signal to the corresponding brain-computer interface device.

[0274] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0275] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0276] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0277] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0278] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0279] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0280] It should be noted that:

[0281] The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0282] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0283] Furthermore, it should be noted that the shapes and names of the parts and components described in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A robot control method based on electroencephalogram (EEG) signals, wherein the target robot is an interactive robot, and the target robot controls the action of the target robot based on the EEG signals of the target user; Its features are, Includes the following steps: Feature extraction is performed based on the EEG signals to obtain corresponding frequency domain features, connectivity features, and time domain pattern features; Based on the frequency domain features, the time domain pattern features, and the connectivity features, feature fusion and dimensionality reduction are performed to obtain the corresponding fused features; The fused features are analyzed according to a preset control strategy to obtain corresponding analysis results; Based on the control strategy, control commands corresponding to the analysis results are generated, and the target robot is controlled based on the control commands. The EEG signal contains multiple channel signals. By combining the channel signals in pairs, one or more pairs of channel signals can be obtained. The connectivity features include a phase lock value between each pair of channel signals; The temporal pattern features include the pattern complexity of the EEG signal.

2. The robot control method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The preset control strategy includes an active control strategy, which includes brain control scoring rules and brain control feedback rules. The brain control feedback rules include several focus levels, the brain control score range corresponding to each focus level, and the execution actions of the target robot. Based on the brain control scoring rules, corresponding brain control scores are generated based on fusion features; Based on the brain control feedback rules, the focus level corresponding to the brain control score is determined, and corresponding control instructions are generated based on the execution action corresponding to the focus level.

3. A robot control method based on electroencephalogram (EEG) signals according to claim 1 or 2, characterized in that, Focus is assessed on a preset length of EEG signal segments based on a preset feedback cycle. Based on the focus assessment result, a corresponding feedback signal is generated and sent to the corresponding brain-computer interface device, which then provides feedback to the target user based on the feedback signal. Among them, the EEG signal segment includes several sub-segments with a unit feedback length; The specific steps for judging focus and generating corresponding feedback signals based on target EEG signal segments are as follows: The pattern complexity corresponding to each sub-segment in the target EEG signal segment is obtained respectively; The phase lock value corresponding to each sub-segment in the target EEG signal segment is obtained respectively, and the average phase lock value corresponding to each sub-segment is obtained; Determine whether the pattern complexity and average phase lock value corresponding to each sub-segment meet the preset attention level standards. If they do, mark the corresponding sub-segment as qualified; otherwise, mark the corresponding sub-segment as unqualified. The attention levels of each sub-segment are encoded to generate corresponding feedback action sequences and obtain feedback signals.

4. The robot control method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, It also includes mode configuration steps, specifically: The target operating mode is determined based on the user configuration information, and the corresponding control strategy is obtained.

5. A robot control method based on electroencephalogram (EEG) signals according to claim 1 or 4, characterized in that, The control strategy includes an adaptive control strategy, which includes state classification rules and a state feedback strategy that corresponds one-to-one with brain states, including fatigue, focus, sleep, meditation, and alcohol effects. Based on the state classification rules, a preset classification model performs brain state classification based on the corresponding fusion features to obtain the target brain state corresponding to the fusion features. Acquire and execute a state feedback strategy corresponding to the target brain state.

6. The robot control method based on electroencephalogram (EEG) signals according to claim 5, characterized in that: Based on an adaptive control strategy, feature extraction is performed on the corresponding EEG signals to obtain the corresponding frequency domain features, connectivity features, temporal pattern features, and channel mutual information. Based on the frequency domain features, the time domain mode features, the connectivity features, and the channel mutual information, feature fusion and dimensionality reduction are performed to obtain the corresponding fused features.

7. The robot control method based on electroencephalogram (EEG) signals according to claim 5, characterized in that: The input to the classification model is the corresponding fusion features, and the output is the classification probability corresponding to each brain state. The target brain state is determined based on the classification probability. The state feedback strategy includes several stages corresponding to the brain state, as well as stage judgment rules and state feedback rules corresponding to each stage state. The stage determination rule is used to indicate the required state maintenance time and / or classification probability for the corresponding stage state.

8. The robot control method based on electroencephalogram (EEG) signals according to claim 1, characterized in that, The EEG signal is divided into several unit signal segments, and each unit signal segment is further divided into sub-frequency bands to obtain several sub-frequency bands. The frequency band normalized power corresponding to each sub-frequency band is calculated to obtain the corresponding frequency domain features. The sub-frequency bands are δ band, θ band, α band, and β band. The steps to determine each sub-frequency band in the current unit signal segment are as follows: Identify the dominant frequency band in the current unit signal segment; The power spectral density corresponding to the dominant frequency band is calculated by fast Fourier transform, and the energy distribution data is obtained by analysis. Based on the power spectral density and the energy distribution data, a clustering algorithm is used to adjust the boundary of the corresponding reference sub-frequency band to obtain the corresponding sub-frequency band, wherein the reference sub-frequency band is the sub-frequency band divided by the previous unit signal segment or a preset initial sub-frequency band.

9. A robot control method based on electroencephalogram (EEG) signals according to claim 2, characterized in that, When the channel signal is greater than 2, the connectivity feature also includes spectral features, and the method for obtaining the spectral features includes: Construct an adjacency matrix based on the phase-locked value of each pair of channel signals; The adjacency matrix is ​​mapped to a weighted undirected graph, where the weight of an edge is the phase lock value between the two channel signals connected. Extract corresponding graph features from the weighted undirected graph, the graph features including node degree and clustering coefficient.

10. A robot control system based on electroencephalogram (EEG) signals, wherein the target robot is an interactive robot, and the target robot controls the action of the target robot based on the EEG signals of the target user; Its features are, include: Signal processing module: Used for feature extraction based on the EEG signals to obtain corresponding frequency domain features, connectivity features, and time domain pattern features; The EEG signal contains multiple channel signals, which are combined in pairs to obtain one or more pairs of channel signals; the connectivity feature includes the phase lock value between each pair of channel signals; The temporal pattern features include the pattern complexity of the EEG signal; This is used to perform feature fusion and dimensionality reduction based on the frequency domain features, the time domain mode features, and the connectivity features to obtain corresponding fused features; Used to perform feature analysis on the fused features according to a preset control strategy, and obtain corresponding analysis results; The control module is used to generate control commands corresponding to the analysis results based on the control strategy, and to control the target robot based on the control commands.

Images