Motion imagination coupling system and method based on emotion prediction

Abstract

The invention discloses a motor imagery coupling system and method based on emotion prediction. The method comprises the steps that dynamic emotion electroencephalogram data of an emotion-related brain area and motor imagery electroencephalogram data of a motion perception-related brain area are collected; extracting emotional features and motor imagery features to form an emotional feature vector and a motor feature vector; based on the historical emotion feature vector and the current emotion feature vector, obtaining a predicted emotion feature vector at a subsequent moment; fusing the current emotional feature vector and the predicted emotional feature vector to obtain an emotional context vector, thereby determining an emotional index vector, and determining an emotional index weight vector based on the emotional index vector; and performing weighted fusion on the emotion index weight vector and the motion feature vector to obtain a weighted enhanced motion imagination feature vector, decoding the motion imagination feature vector, and outputting a control signal to drive terminal equipment according to a decoding result. Decoding deviation caused by emotion interference is reduced, and decoding accuracy, adaptability and reliability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electroencephalogram data processing, in particular to a motor imagery coupling system and method based on emotion prediction. BACKGROUND

[0002] With the development of the times, the brain-computer interface technology (BCI) has been gradually applied to life, especially in the treatment of some neurological diseases. Among them, the upper and lower limb hemiplegia caused by stroke also has a very mature application scenario, for example, using motor imagery paradigm to assign tasks to the affected limb, so as to achieve nerve repair work according to imagination training. In addition, event-related desynchronization (ERD) and event-related synchronization (ERP) are also widely used in the field of motor imagery, such as predicting the motor function recovery of stroke patients, evaluating the motor function of Parkinson's disease patients, etc. It is more obvious to the motor area, especially the left and right sides of the upper limbs.

[0003] However, the actual performance of motor imagery BCI depends on the physiological and psychological state of the user. As a core psychological state indicator, emotion has been proven to significantly affect brain activity, thereby interfering with the characteristics and stability of motor imagery electroencephalogram signals. For example, the user's anxiety and tension will increase the noise in the electroencephalogram signal, resulting in a decrease in motor imagery classification accuracy; while a relaxed and focused state is conducive to generating stable ERD / ERP patterns.

[0004] Currently, emotions are mostly used as an auxiliary solution in existing technologies, such as monitoring and prompting, intervening as prompts during training, and not deeply integrated with the control process of motor imagery, with limited system efficiency improvement. In addition, some use data preprocessing to filter and screen data, which is essentially a passive "filtering" mechanism that cannot fully utilize emotional information. Finally, some use it as a feedback mechanism to intervene in task status, with the adjustment logic and motor imagery signal decoding model being separate.

[0005] In summary, existing motor imagery BCI technology based on emotion generally considers emotion recognition and motor imagery decoding as two independent parallel processes. Emotion is only used as an auxiliary, external reference information for system start-stop, data screening, or interface-level interactive adjustment, and is not deeply integrated with motor imagery signals at the model level. SUMMARY

[0006] The technical purpose of the present application is to address the technical problem that existing motor imagery BCI technology generally considers emotion recognition and motor imagery decoding as two independent parallel processes, thereby affecting the effect of motor imagery, and provides a motor imagery coupling system based on emotion prediction.

[0007] To achieve the above technical purposes, the embodiments of the present application adopt the following technical solutions.

[0008] In one aspect, the embodiments of the present application provide a motor imagery coupling system based on emotion prediction, comprising:

[0009] A data acquisition module is configured to acquire dynamic emotional electroencephalogram data of an emotional brain region and motor imagery electroencephalogram data of a motor perception brain region;

[0010] A feature extraction module is configured to extract emotional features based on the dynamic emotional electroencephalogram data and extract motor imagery features based on the motor imagery electroencephalogram data, thereby forming an emotional feature vector and a motor feature vector;

[0011] An emotion prediction module is configured to obtain a predicted emotional feature vector at a subsequent time based on a historical emotional feature vector and a current emotional feature vector;

[0012] A context vector determination module is configured to obtain an emotional context vector by fusing the current emotional feature vector and the predicted emotional feature vector;

[0013] A feature modulation adaptive module is configured to determine an emotional index vector based on the emotional context vector and determine an emotional index weight vector based on the emotional index vector;

[0014] A feature weighting fusion module is configured to perform weighted fusion of the emotional index weight vector and the motor feature vector, thereby obtaining a weighted and enhanced motor imagery feature vector;

[0015] A motor imagery decoding module is configured to complete decoding of the weighted and enhanced motor imagery feature vector and output a control signal to drive a terminal device according to a decoding result.

[0016] In another aspect, the embodiments of the present application provide a motor imagery coupling method based on emotion prediction, comprising: acquiring dynamic emotional electroencephalogram data of an emotional brain region and motor imagery electroencephalogram data of a motor perception brain region;

[0017] Extracting emotional features based on the dynamic emotional electroencephalogram data and extracting motor imagery features based on the motor imagery electroencephalogram data, thereby forming an emotional feature vector and a motor feature vector;

[0018] Obtaining a predicted emotional feature vector at a subsequent time based on a historical emotional feature vector and a current emotional feature vector;

[0019] Fusing the current emotion feature vector and the predicted emotion feature vector to obtain an emotional context vector, determining an emotion index vector based on the emotional context vector, and determining an emotion index weight vector based on the emotion index vector; and performing weighted fusion of the emotion index weight vector and the motion feature vector to obtain a weighted and enhanced motor imagery feature vector.

[0020] Performing decoding on the weighted and enhanced motor imagery feature vector, and outputting a control signal according to a decoding result to drive a terminal device.

[0021] Compared with the prior art, the motor imagery coupling system and method based on emotion prediction provided in the embodiments of the present application have the following beneficial technical effects: by dynamically collecting emotional and motor imagery dual-modal electroencephalogram data, the emotion recognition is converted from an independent and parallel process into a dynamic regulation factor of motor imagery decoding, the real-time state and change trend of the emotion are captured by means of emotion prediction and context vector fusion, and then the motor imagery features are adaptively modulated by the emotion index weight, thereby realizing deep coupling between the two. This design not only conforms to the dynamic influence mechanism of emotion on the motor cortex activity, but also enhances the discriminability of the motion features by weighted fusion, reduces the decoding deviation caused by emotional interference, and adapts to the non-stationarity and individual differences of the electroencephalogram signal, thereby significantly improving the decoding accuracy, dynamic adaptability and actual operation reliability of the motor imagery BCI system.

[0022] It should be understood that the summary section is not intended to identify key or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. In addition, the shapes and proportions of the components in the drawings are only illustrative and are used to help understand the present application, and are not specific limitations on the shapes and proportions of the components of the present application. Those skilled in the art can select various possible shapes and proportions to implement the present application according to specific circumstances under the teaching of the present application. In the drawings:

[0024] Figure 1 A motor imagery coupling method based on emotion prediction is provided for the embodiments;

[0025] Figure 2 A schematic diagram of the process of collecting and preprocessing electroencephalogram signals and feature extraction in the embodiments is provided;

[0026] Figure 3 A schematic diagram of the process of dual-path updating of emotion index and motor imagery coupling in the embodiments is provided;

[0027] Figure 4A flowchart of a slight bias strategy for implementing the initialization of adaptive weight parameters m and n in the embodiment. DETAILED DESCRIPTION

[0028] In order for those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0029] It should be fully understood that the user information (including but not limited to user physiological sensor information, user personal information, etc.) involved in the present application is information and data authorized by the user or authorized by all parties, and the use of user information should comply with the privacy policies and practices generally considered to meet or exceed the industry for maintaining user privacy, the collection, use and processing of relevant data should comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0030] In the embodiment, the emotion prediction-based motor imagery coupling system comprises a data acquisition module, a feature extraction module, an emotion prediction module, a context vector determination module, a feature modulation adaptive module, a feature weighted fusion module and a motor imagery decoding module.

[0031] The data acquisition module is used to acquire dynamic emotional electroencephalogram data of emotional related brain regions and motor imagery electroencephalogram data of motor perception related brain regions;

[0032] The feature extraction module is used to extract emotional features based on the dynamic emotional electroencephalogram data, and extract motor imagery features based on the motor imagery electroencephalogram data, to form an emotional feature vector and a motor feature vector;

[0033] The emotion prediction module is used to obtain a predicted emotional feature vector at a subsequent time based on a historical emotional feature vector and a current emotional feature vector;

[0034] The context vector determination module is used to obtain an emotional context vector by fusing the current emotional feature vector and the predicted emotional feature vector;

[0035] The feature modulation adaptive module is used to determine an emotional index vector based on the emotional context vector, and determine an emotional index weight vector based on the emotional index vector;

[0036] The feature weighted fusion module is used to perform weighted fusion of the emotional index weight vector and the motor feature vector, to obtain a weighted enhanced motor imagery feature vector;

[0037] The motor imagery decoding module is configured to decode the weighted enhanced motor imagery feature vector, and output a control signal to drive a terminal device according to a decoding result.

[0038] Based on the same inventive concept as the motor imagery coupling system based on emotion prediction provided in the above embodiments, the embodiments of the present application also provide a motor imagery coupling method based on emotion prediction, as shown in the accompanying drawings. Figure 1 As shown, the method mainly includes collecting (and also preprocessing) electroencephalogram data of different regions of the brain for emotion and motor imagery tasks, extracting emotion features and motor imagery features according to the processed data, establishing an emotion prediction model, constructing a modulation network by the predicted emotion features and current emotion features to fine-tune the motor imagery data weight, and finally decoding the weighted motor features to obtain the motor imagery result to drive the hardware terminal device.

[0039] The motor imagery coupling method based on emotion prediction provided by the embodiments includes:

[0040] Step 1: Collect dynamic emotional electroencephalogram data of emotion-related brain regions and motor imagery electroencephalogram data of motor perception-related brain regions.

[0041] In the embodiments, a multi-channel lead electroencephalogram cap according to the international 10-20 standard can be selected, and a wet electrode combined with conductive paste can be selected as a conductive medium to reduce interference caused by the external environment.

[0042] In the embodiments, in order to reduce the computational complexity, only the emotion-related brain regions and the motor perception-related brain regions are selected for electroencephalogram signal collection. For example, the emotion-related brain regions include the frontal lobe regions (Fp1, Fp2, F3, F4), and the motor perception-related brain regions include C3, C4 and Cz. The dynamic emotional electroencephalogram data and the motor imagery electroencephalogram data are used for ERD (event-related desynchronization) and ERP (event-related synchronization).

[0043] In the embodiments, a sampling rate of at least 250 Hz can be selected to ensure the details of the electroencephalogram features.

[0044] Optionally, the data collection module is also configured to perform data preprocessing on the collected dynamic emotional electroencephalogram data and motor imagery electroencephalogram data. The data preprocessing includes: removing power frequency interference by using 50 Hz notch filtering, retaining effective electroencephalogram features by electroencephalogram emotion and motor imagery related full-band band-pass filtering, and retaining electroencephalogram related feature details by 0.5-45 Hz band-pass filtering.

[0045] In some embodiments, the collected dynamic emotional EEG data and motor imagery EEG data are de-noised, specifically including: using a dynamic threshold (such as ±100 μV) to remove large amplitude interference caused by motion artifacts, and if there is an electrooculogram channel, the eye movement artifacts can be identified and removed by ICA (independent component analysis).

[0046] In embodiments, the continuous EEG data stream can be segmented into overlapping time windows for subsequent feature extraction. A fixed time window of 2-4 seconds can be used to store emotional EEG data of the emotional related region for a period of time.

[0047] In embodiments, smooth dynamic emotional data stream can be obtained by sliding 0.5 seconds according to dynamic emotional data.

[0048] Step 2: Perform emotional feature extraction based on dynamic emotional EEG data, and perform motor imagery feature extraction based on motor imagery EEG data to form emotional feature vectors and motor feature vectors.

[0049] In embodiments, theta (4-8 Hz), alpha (8-13 Hz), beta (13-30 Hz), and gamma (> 30 Hz) bands related to emotions are selected, wherein the alpha band is related to relaxation and resting state, the beta band is related to concentration and tension, the theta band is related to drowsiness, and the gamma band is related to high cognitive load and excitement.

[0050] As shown in Figure 2 , in embodiments, fast Fourier transform is performed on the dynamic emotional EEG data to obtain the frequency domain feature values of each channel, and the expression is: .

[0051] wherein X n represents the frequency domain feature values of the dynamic emotional EEG data, X k represents the discrete time domain signal of the dynamic emotional EEG data, N represents the total number of sampling points, i represents the imaginary unit, k is the index of the time domain sampling point, and n is the index of the frequency domain component.

[0052] Based on the frequency domain feature values of each channel, the PSD (power spectral density) of the theta band, alpha band, beta band, and gamma band of the dynamic emotional EEG data is obtained using the Welch method, and the differential entropy D band =log(PSD) is used as the emotional frequency domain feature.

[0053] Since there is frontal lobe EEG asymmetry, the differential entropy of the theta, alpha, beta, and gamma bands of the left (Fp1, F3) dynamic emotional EEG data of the frontal lobe region is also calculated.

[0054] Taking the calculation of the differential entropy of the α band as an example (the calculation methods for the differential entropy and symmetry index of the θ, β, and γ bands are the same), the differential entropy D of the α band of the dynamic emotional EEG data of the left side (Fp1, F3) of the prefrontal region is calculated respectively. α_Fp1 and D α_F3 The differential entropy D of the alpha band of dynamic emotional EEG data from the right prefrontal cortex region (Fp2, F4) α_Fp2 and D α_F4 .

[0055] The symmetry index A of the α band is determined based on the differential entropy of the α band. α1 and A α2 (Only the symmetry index of the α band needs to be determined; the symmetry index of other bands is not considered.) The expression is:

[0056] A α1 =(D α_Fp1 -D α_Fp2 ) / (D α_Fp1 +D α_Fp2 A α2 =(D α_F3 -D α_F4 ) / (D α_F3 +D α_F4 ).

[0057] Based on the standard deviation of the time-domain signal and the Hjorth mobility parameter as the time-domain features of emotion, the standard deviations (S0, S1, S2, S3) of the time-domain signals of the four channels (left (Fp1, F3) and right (Fp2, F4)) were calculated respectively. Fp1 S Fp2 S F3 S F4 ) and the corresponding Hjorth mobility HM (HM is a mobility index, a judgment index belonging to the time domain characteristics), including H Fp1 H Fp2 H F3 H F4 HM uses the ratio of the standard deviation of the time-domain signal to the standard deviation of the first derivative of the signal for quantization.

[0058] Finally, the feature vectors are concatenated based on the temporal, frequency, and spatial features of emotion to obtain the emotion feature vector F. emo The emotion frequency domain features include the differential entropy relationships of different frequency bands of Fp1, Fp2, F3, and F4; the emotion spatial domain features are symmetry indices; and the emotion temporal domain features include the Hjorth mobility estimation index for the number of relevant channels. The emotion feature vector F emo The expression is as follows (for ease of representation, F will be used below). emo The expression omits the corresponding data for the Fp2, F3, and F4 regions.

[0059] F emo =[D θ_Fp1 ,D α_Fp1, D β_Fp1, D γ_Fp1 ..,A α1 ,A α2, S Fp1 ,H Fp1 ,...];

[0060] wherein D θ_Fp1 is the differential entropy of the frontal lobe region Fp1 dynamic emotional electroencephalogram data in the theta band, D α_Fp1 is the differential entropy of the frontal lobe region Fp1 dynamic emotional electroencephalogram data in the alpha band, D β_Fp1 is the differential entropy of the frontal lobe region Fp1 dynamic emotional electroencephalogram data in the beta band, D γ_Fp1 is the differential entropy of the frontal lobe region Fp1 dynamic emotional electroencephalogram data in the gamma band, S Fp1 is the standard deviation of the frontal lobe region Fp1 dynamic emotional electroencephalogram data in the time domain signal, H Fp1 is the Hjorth mobility HM of the frontal lobe region Fp1 dynamic emotional electroencephalogram data.

[0061] In order to eliminate individual differences, the Z-Score is used to standardize the emotional feature vector to obtain the standardized emotional feature vector F emo .

[0062] In the embodiment, the real-time motor imagery feature extraction includes: the correlation data of the motor perception region (C3, C4, Cz) and the emotional data in the same time window are processed to obtain the ERD (Event-Related Desynchronization, event-related desynchronization) and ERP (Event-Related Potential, event-related potential) phenomenon, that is, when unilateral imagination, the energy of the contralateral motor perception region decreases (ERD), and the energy of the ipsilateral motor perception region increases (ERP).

[0063] In specific embodiments, the differential entropy of the mu rhythm (8-13 Hz) and the beta rhythm (13-30 Hz) of C3 and C4 is obtained, wherein the ERD / ERP is preferably used to eliminate individual differences, and the resting state calculation is required in the paradigm to obtain the baseline data, and the ERD / ERP ratio is calculated by and used as the motor imagery frequency domain feature together with the differential entropy, wherein P base is the baseline frequency domain information, P current is the actual frequency domain information, and Per c is the motor imagery frequency domain feature.

[0064] In this embodiment, the specific process for the frequency domain features of motor imagery includes: first, acquiring user baseline data using the resting-state paradigm, and then calculating frequency domain information for this user baseline data using FFT; next, calculating the power Pbase_μ and power Pbase_β of the user baseline data in the μ band and β band respectively; the same steps are followed in the subsequent real-time motor imagery EEG data feature processing to obtain the power Pcurrent_μ and power Pcurrent_β of the real-time motor imagery EEG data in the μ band and β band respectively; using the formula:

[0065] ; Get Per c _μ,Per c _β. Finally, these features are calculated for both C3 and C4 channels to obtain four specific features [Per c _c3_μ,Per c _C3_β,Per c _c4_μ,Per c _C4_β] is used as its motion imagination frequency domain feature, where Per c _c3_μ represents the ERD / ERP ratio of the μ-rhythm in lead C3, Per c _C3_β is the ERD / ERP ratio of the β rhythm in lead C3, Per c _c4_μ represents the ERD / ERP ratio of the μ-rhythm in lead C4, Per c _C4_β is the ERD / ERP ratio of the β rhythm in lead C4.

[0066] In this embodiment, CSP (Common Spatial Pattern) is used to eliminate common noise in the EEG space and enhance the signal-to-noise ratio related to motor imagery to obtain spatial features of motor imagery, specifically including:

[0067] Step 201: Take the preprocessed EEG data of the motor perception region as input. The data dimension is the number of channels multiplied by the number of signal points, and the motor imagery task category has been labeled (such as left hand imagery, right hand imagery).

[0068] Step 202: Based on the labeled training dataset, the optimal spatial projection matrix W is learned through the co-space pattern algorithm. The input EEG signal X is linearly transformed with the projection matrix W (Z=W×X) to obtain the transformed signal Z. Each row of the transformed signal Z corresponds to a spatially filtered component, and the components are sorted from strong to weak according to their ability to distinguish the motor imagery task.

[0069] Step 203, take the most distinctive feature components. In the embodiment, the first m components and the last m components of Z after transformation can be selected (m is a self-defined positive integer, generally set to 2 or 3), which can best reflect the specificity difference of different motor imagery tasks. The variance of each component after screening is calculated as a feature quantization index, and the logarithmic standardization processing is performed on the variance value, so that the feature distribution is closer to the Gaussian distribution, and finally the motor imagery spatial feature components filtered by common spatial pattern are obtained, which are used for splicing of the subsequent motor imagery feature vectors.

[0070] In the embodiment, the Hjorth mobility parameter is used as the motor imagery time domain feature, and the calculation method is the same as that of obtaining the emotion time domain feature.

[0071] The motor imagery frequency domain feature, the motor imagery spatial feature and the motor imagery time domain feature are spliced into a feature vector to obtain a spliced and standardized motor feature vector F mi ; wherein the motor imagery frequency domain feature includes the differential entropy of different rhythms of C3 and C4 channels and the ERD / ERP ratio Per c _c3_μ, Per c _C3_β, Per c _c4_μ and Per c _C4_β information, the motor imagery spatial feature includes the filtered components of CSP; the motor imagery spatial feature uses Hjorth mobility estimation.

[0072] Step 3: based on the historical emotion feature vector and the current emotion feature vector, a predicted emotion feature vector at a subsequent time is obtained.

[0073] In the embodiment, the emotion prediction module includes a feature vector cache queue S emo and a causal time series learning model: the feature vector cache queue S emo stores historical emotion feature vectors, adopts a first-in-first-out mechanism to manage queue data, and the cache threshold is set to n, n is the maximum capacity of the queue (for example, n = 3, 5, 10, which is set according to actual time series dependence requirements), which means that the queue can store at most n emotion feature vectors at the same time, and when a new emotion feature vector (such as the n+1th) is to be added to the queue, the queue is full of n features, reaching the capacity limit, and the oldest feature is deleted from the queue to make room for the new feature.

[0074] The causal time series learning model is based on the cached historical emotion feature vector and the current emotion feature vector F emo (t), and outputs a predicted emotion feature vector at a subsequent time t+k.

[0075] In the embodiments, the causal time series learning model is a gated recurrent unit model or a causal convolutional network model, which is trained by an offline data set to minimize the mean squared error of the predicted emotion feature vector F emo_pred and the true emotion feature F emo_true . The design using the GRU (gated recurrent unit) architecture achieves rapidity and a small number of parameters.

[0076] In actual use, the causal time series learning model is first trained by an offline data set, and the training target is to minimize the mean squared error of the predicted value F emo_pred (t+1) and the true value F emo (t+1). In actual use, the next moment predicted emotion feature vector F emo (t+1) is finally obtained by inputting S emo_pred (t).

[0077] Step 4: fuse the current emotion feature vector and the predicted emotion feature vector to obtain an emotion context vector; determine an emotion index vector based on the emotion context vector, determine an emotion index weight vector based on the emotion index vector, and perform weighted fusion on the emotion index weight vector and the motion feature vector to obtain a weighted and enhanced motor imagery feature vector.

[0078] In the embodiments, the current emotion feature vector F emo (t) and the predicted emotion feature vector F emo_pred (t+1) are fused to construct an emotion context vector C emo , and the expression is as follows:

[0079] C emo= m×F emo (t) + n×F emo_pred (t+1);

[0080] wherein m and n are adaptive weight parameters preset by the prior relationship between motor imagery and predicted emotion, which reflect the effectiveness of the predicted emotion and the influence on the motor imagery result.

[0081] In some embodiments, when setting the initial values of m and n, a simple and intuitive way is to set m and n to 0.5, but this strategy has a problem: even if the attention weight mechanism is introduced for dynamic adjustment later, the weight initialization of 0.5 and 0.5 is equivalent to injecting a prior hypothesis that the current emotion feature F emo (t) and the predicted emotion feature F emo_pred (t+1) are equally important in any case.

[0082] In some other embodiments, a better initialization method is selected, such as using a small random initialization or zero initialization method (Xavier Uniform initialization), injecting a neutral prior, and fully trusting and relying on the attention mechanism to dynamically learn the most suitable weight distribution for the current context. This is a very intuitive algorithm design.

[0083] In order to provide a precise and efficient motor imagery coupling system, although the neutral prior can explore an effective dynamic weighting strategy, the model may be affected in terms of convergence speed. Therefore, in some other embodiments, an improved method for initializing adaptive weight parameters m and n is provided, i.e., a slightly biased initialization weight based on adaptive adjustment of motor imagery domain knowledge.

[0084] Before further describing the improved strategy, first briefly describe the conventional domain knowledge-based slightly biased initialization weight method: set the weight of the current emotional feature vector F emo (t) to 0.6, and set the weight of the predicted emotional feature vector F emo_pred (t+1) to 0.4, i.e., a weak prior that the current emotion is relatively more important is injected into the model. The starting point of this prior method is that the predicted emotional feature vector F emo_pred (t+1) may not be completely accurate and consistent with the true emotion, and the predicted emotional feature vector F emo_pred (t+1) is only a prediction factor. Through this prior method, the convergence can be accelerated to some extent and the initial stability can be improved.

[0085] If a fixed weight is set, the flexibility is poor. Therefore, referring to the complete flowchart from calculating the attention score to the final feature fusion shown in FIG. 8, in some embodiments, a more flexible slightly biased initialization weight strategy is used, including: Figure 4

[0086] The current emotional feature vector F emo (t) and the predicted emotional feature vector F emo_pred (t+1) are respectively projected to the key (Key) and value (Value) spaces through a linear layer, a query vector (Q) is calculated, and after the attention score (e=Q×K^T) is calculated, a positive bijection b is artificially added to the score corresponding to F emo (t).

[0087] The attention score corresponding to the current emotional feature vector is calculated according to the query vector and the key vector of the current emotional feature vector, wherein a positive bias is added, and the expression is: e_t=Q×K_t^T+b, wherein b is an initialized small positive number, for example, b=0.2.

[0088] ​The attention score corresponding to the predicted emotion feature vector is calculated according to the pre-query vector and the key vector of the predicted emotion feature vector, without adding a bias, and the expression is: e_f=Q×K_f^T, wherein the future emotion score is not biased; wherein e_t is the attention score of the current emotion feature vector, Q is the query vector, K_t is the key vector of the current emotion feature vector, T is the transpose, b is the bias, e_f is the attention score of the predicted emotion feature vector, and K_f is the key vector of the predicted emotion feature vector.

[0089] The above-mentioned light bias initialization weight mode is equivalent to slightly lifting the current emotion feature F emo (t) before the attention mechanism starts to work. emo (t) and the predicted emotion feature F emo_pred (t+1) are similar in relevance to the query vector, and by default, more attention is paid to F emo (t). If F emo (t) does show extremely strong relevance, the attention mechanism can completely learn a large score to cover this initial bias, thereby balancing the flexibility of adaptive weights and better playing the role of accelerating convergence and improving initial stability.

[0090] Supplementary note: Query vector calculation method: Q=W_q×X_combined+b_q;

[0091] Wherein W_q is the weight matrix of the linear layer, X_combined is the spliced vector composed of the motion feature vector and the emotion context feature vector, and b_q is the bias vector.

[0092] In the embodiment, as shown in Figure 3 , the feature modulation adaptive module includes an emotion index decoding module and a feature gate control module.

[0093] The emotion index decoding module is configured to determine an emotion index vector I emo according to the emotion context vector. Through the above-mentioned emotion context needs to be mapped to a set of emotion index weights (such as attention, fatigue, valence, anxiety, and brain vitality indexes), for this purpose, an attention modulation module is constructed through a network model to automatically derive an emotion index vector I emo by inputting the emotion context C emo (which contains emotion-related time domain, frequency domain, and spatial domain features).

[0094] The feature gate control module is configured to obtain an emotion index weight vector A emo with the same dimension as the motion feature vector F mi by using an activation function (such as a Tanh activation function) based on the emotion index vector Iemo , to ensure that the output is in the range [-1, 1], where negative values represent low levels and positive values represent high levels. The emotion index weight vector A emo All feature output values are in the interval (0, 1).

[0095] In an embodiment, the feature weighting fusion module performs dot multiplication on the emotion index weight vector A emo and the motion feature vector F mi to obtain the final weighted and enhanced motor imagery feature vector F mi_wei .

[0096] In some embodiments, the system further includes a safety arbitration module that determines the user's state based on the emotion index weight vector. If it is determined to be a negative state, the decoding process is interrupted and a prompt signal is issued. If it is determined to be a positive state, the feature weighting fusion module normally outputs the weighted and enhanced motor imagery feature vector to the motor imagery decoding module.

[0097] In an embodiment, because the emotion index weight vector A emo is output in the range [-1, 1], this vector contains emotion indicators such as attention, fatigue, anxiety, and relaxation, each corresponding to a weight, which can be positive or negative. The final value can be obtained by weighted averaging of the elements of the emotion index weight vector A emo . If it is positive, it is a positive emotion, and if it is negative, it is a negative emotion.

[0098] In an embodiment, some network models can be used to learn the relationship between indicators, and the emotion index weight vector A emo can be used to automatically give weighted values for weighting and judgment to determine whether it is positive or negative.

[0099] In some embodiments, specific indicators can also be judged individually. For example, several negative indicators such as anxiety or fatigue are directly judged as negative if they exceed the set weight threshold. If the result is positive, the next step is processed. If the result is negative, it indicates that the user may not be suitable for training or hardware control, and the system is directly interrupted and prompted, and the motion decoding step is not performed. This also reduces resource consumption and improves system performance.

[0100] Step 5: Complete decoding of the weighted and enhanced motor imagery feature vector, and output a control signal to drive the terminal device according to the decoding result.

[0101] Through the motor imagery decoding module, motor imagery decoding is achieved. A lightweight multilayer perceptro (MLP) model is selected as the decoding model. The weighted and enhanced motor imagery feature vector F mi_weiAs input, the model outputs a probability vector, and the one with the highest probability is selected as the final motor imagery result for control. This can be either rehabilitation training hardware or brain-controlled hardware.

[0102] The weighted and enhanced motion imagery feature vector F mi_wei After inputting into the motion imagery decoding module, motion imagery task feedback can be obtained. Based on this feedback, F can be determined. mi_wei The effectiveness. However, due to F mi_wei In reality, it is obtained from the emotional context vector containing information on historical emotions, current emotions, and future predicted emotions, as well as the features of motor imagery. At this point, if we only adjust m and n after initialization based on the feedback from the motor imagery task, it is actually a relatively compromised adjustment method. That is, the entire algorithm design idea is "emotion prediction-execution-feedback", which is similar to the design idea of ​​reinforcement learning. It cannot correct the model itself. Once a negative motor imagery task feedback is obtained, the model cannot distinguish whether the negative feedback is caused by incorrect m and n or by an error in the future predicted emotion.

[0103] Therefore, in some embodiments, reference may be made to the appendix. Figure 3 m and n are updated using a dual-path approach: one path uses rapid adaptive updates based on feedback from a motion visualization task, and the other path updates based on the predicted F. emo_pred The corresponding true emotion feature vector F emo_true Slow adaptive updates are performed to fully ensure the adaptability, flexibility, and accuracy of the motion-imagination coupling system in this scheme.

[0104] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, or a tablet computer, or any combination of these devices.

[0105] The above provides a detailed description of the motion-image coupling system and method based on emotion prediction provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the concept of this application and should not be construed as a limitation on the scope of protection of this application.

Claims

1. A motion imagination coupling system based on emotion prediction, characterized in that, The system comprises: a data acquisition module configured to acquire dynamic emotional electroencephalogram (EEG) data of an emotional brain region and motor imagery EEG data of a motor perception brain region; a feature extraction module configured to extract emotional features based on the dynamic emotional EEG data and extract motor imagery features based on the motor imagery EEG data, to form an emotional feature vector and a motor feature vector; an emotion prediction module configured to obtain a predicted emotional feature vector at a subsequent time based on a historical emotional feature vector and a current emotional feature vector; a context vector determination module configured to fuse the current emotional feature vector and the predicted emotional feature vector to obtain an emotional context vector; a feature modulation adaptive module configured to determine an emotional index vector based on the emotional context vector and determine an emotional index weight vector based on the emotional index vector; a feature weighted fusion module configured to perform weighted fusion of the emotional index weight vector and the motor feature vector to obtain a weighted and enhanced motor imagery feature vector; a motor imagery decoding module configured to complete decoding of the weighted and enhanced motor imagery feature vector and output a control signal according to a decoding result to drive a terminal device.

2. The motor imagery coupling system based on emotion prediction according to claim 1, wherein the feature modulation adaptive module comprises an emotional index decoding module and a feature gate control module; the emotional index decoding module is configured to determine an emotional index vector according to the emotional context vector; The feature door control module is configured to obtain a motion feature vector F by using an activation function based on the emotion index vector mi The emotion index weight vector A has the same dimension emo ; The feature weighting fusion module is configured to perform point multiplication on the emotion index weight vector A emo and the motion feature vector F mi to obtain a final weighted and enhanced motion imagination feature vector F mi_wei .

3. The emotion prediction based motor imagery coupling system of claim 2, wherein, the system further comprises a safety arbitration module configured to determine a user state based on an emotional index corresponding to the emotional context vector, and if the user state is determined to be a negative state, the decoding process is interrupted and a prompt signal is sent; if the user state is determined to be a positive state, the feature weighted fusion module normally outputs the weighted and enhanced motor imagery feature vector to the motor imagery decoding module.

4. The emotion prediction based motor imagery coupling system of claim 1, wherein, The data acquisition module is further configured to perform data preprocessing on the acquired dynamic emotional EEG data and motor imagery EEG data, and the data preprocessing comprises: removing power frequency interference through 50 Hz notch filtering; removing motion artifacts and electrooculogram artifacts through a dynamic threshold method and / or independent component analysis; and dividing continuous dynamic emotional EEG data and motor imagery EEG data streams into overlapping time windows for subsequent feature extraction.

5. The emotion prediction based motor imagery coupling system of claim 1, wherein, The emotional feature extraction based on the dynamic emotional EEG data comprises extracting emotional time domain features, emotional frequency domain features, and emotional spatial domain features; the emotional time domain features comprise standard deviation of time domain signals and Hjorth mobility parameters; the emotional frequency domain features comprise differential entropy of power spectral density of θ band, α band, β band, and γ band; the emotional spatial domain features comprise α band asymmetry index of left and right sides of a frontal lobe region; and the emotional time domain features, the emotional frequency domain features, and the emotional spatial domain features are subjected to Z-Score standardization processing to form an emotional feature vector.

6. The emotion prediction based motor imagery coupling system of claim 1, wherein, The motor imagery feature extraction based on motor imagery EEG data includes extracting motor imagery time domain features, motor imagery frequency domain features and motor imagery space domain features, the motor imagery time domain features include Hjorth mobility parameters, the motor imagery frequency domain features include differential entropy of mu rhythm and beta rhythm and ERD / ERP ratio, the ERD / ERP ratio is calculated by current EEG power and resting state baseline power; the motor imagery space domain features include feature components filtered by common spatial pattern.

7. The emotion prediction based motor imagery coupling system of claim 1, wherein, The emotion prediction module includes a feature vector cache queue and a causal time series learning model: the feature vector cache queue is used to store historical emotion feature vectors, and adopts a first-in first-out mechanism to manage queue data; the causal time series learning model outputs a predicted emotion feature vector at a subsequent time based on the cached historical emotion feature vector and the current emotion feature vector; the causal time series learning model is a gated recurrent unit model or a causal convolutional network model, which is trained through an offline data set to minimize the mean square error between the predicted feature and the real feature.

8. The emotion prediction based motor imagery coupling system of claim 1, wherein, The feature modulation adaptive module fuses the current emotion feature vector and the predicted emotion feature vector to obtain an emotion context vector by using the following formula: C emo = m x F emo (t) + n x F emo_pred (t + 1) generates an emotional context vector, where F emo (t) is a current emotional feature vector, F emo_pred (t + 1) is a predicted emotional feature vector, and m and n are adaptive weight parameters. The motion feature vector is weighted and fused based on the emotion context vector to obtain a weighted and enhanced motor imagery feature vector, including: generating an emotion index vector based on the emotion context vector, normalizing the emotion index vector to the [-1, 1] interval through a Tanh activation function, and then generating an emotion index weight vector consistent with the motor imagery feature dimension through a Sigmoid activation function; The emotion index weight vector and the motor imagery feature vector are point multiplied to obtain a weighted and enhanced motor imagery feature.

9. The emotion prediction based motor imagery coupling system of claim 8, wherein, The initialization of adaptive weight parameters m and n adopts a mild bias strategy, including projecting the current emotion feature vector and the predicted emotion feature vector to key space and value space respectively through a linear layer to obtain corresponding key vectors and value vectors; calculating a unique query vector Q based on a combined feature vector formed by splicing the motor imagery feature vector and the task context feature vector; The attention score corresponding to the current emotion feature vector is calculated according to the key vector of the query vector and the current emotion feature vector, wherein a positive bias is added; The attention score corresponding to the predicted emotion feature vector is calculated according to the key vector of the pre-query vector and the predicted emotion feature vector, wherein no bias is added; the two groups of attention scores are normalized through a Softmax function to obtain initial adaptive weight parameters m and n, which are used to weight and fuse the value vectors of the current emotion feature vector and the predicted emotion feature vector.

10. The emotion prediction based motor imagery coupling system of claim 3, wherein, The safety arbitration module uses at least one of the following methods to determine the user state: The weighted average of each index weight in the emotion index weight vector is calculated, and if the result is negative, it is determined as a negative state; Thresholds are set for each index weight in the emotion index weight vector, and if a specific negative emotion index exceeds the threshold, it is determined as a negative state; The association between emotion indexes is learned through a network model, and the state determination result is automatically output according to the emotion index weight vector.

11. A method for emotion prediction-based motor imagery coupling, comprising: including: Collecting dynamic emotional brain electrical data of emotion-related brain regions and motor imagination brain electrical data of motor perception-related brain regions; Performing emotion feature extraction based on the dynamic emotional brain electrical data and motor imagination feature extraction based on the motor imagination brain electrical data to form an emotion feature vector and a motor feature vector; Obtaining a predicted emotion feature vector of a subsequent moment based on a historical emotion feature vector and a current emotion feature vector; Fusing the current emotion feature vector and the predicted emotion feature vector to obtain an emotional context vector, determining an emotion index vector based on the emotional context vector, determining an emotion index weight vector based on the emotion index vector, and performing weighted fusion of the emotion index weight vector and the motor feature vector to obtain a weighted and enhanced motor imagination feature vector; Performing decoding on the weighted and enhanced motor imagination feature vector, and outputting a control signal to drive a terminal device according to a decoding result.

Images