A motion-image coupling system and method based on emotion prediction

Abstract

This application discloses a motion-imagery coupling system and method based on emotion prediction. The method includes collecting dynamic emotional EEG data from emotion-related brain regions and motion-imagery EEG data from motion-perception-related brain regions; extracting emotional and motion-imagery features to form emotional and motion feature vectors; obtaining predicted emotional feature vectors for subsequent time moments based on historical and current emotional feature vectors; fusing the current and predicted emotional feature vectors to obtain an emotional context vector, thereby determining an emotional index vector; determining an emotional index weight vector based on the emotional index vector; weighting and fusing the emotional index weight vector with the motion feature vector to obtain a weighted enhanced motion-imagery feature vector; decoding the vector; and outputting control signals to drive a terminal device based on the decoding result. This application reduces decoding bias caused by emotional interference and improves decoding accuracy, adaptability, and reliability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electroencephalogram data processing, in particular to a motor imagery coupling system and method based on emotion prediction. BACKGROUND

[0002] With the development of the times, the brain-computer interface technology (BCI) has been gradually applied to life, especially in the treatment of some neurological diseases. Among them, the upper and lower limb hemiplegia caused by stroke also has a very mature application scenario, for example, using motor imagery paradigm to assign tasks to the affected limb, so as to achieve nerve repair work according to imagination training. In addition, event-related desynchronization (ERD) and event-related synchronization (ERP) are also widely used in the field of motor imagery, such as predicting the motor function recovery of stroke patients, evaluating the motor function of Parkinson's disease patients, etc. It is more obvious to the motor area, especially the left and right sides of the upper limbs.

[0003] However, the actual performance of motor imagery BCI depends on the physiological and psychological state of the user. As a core psychological state indicator, emotion has been proven to significantly affect brain activity, thereby interfering with the characteristics and stability of motor imagery electroencephalogram signals. For example, the user's anxiety and tension will increase the noise in the electroencephalogram signal, resulting in a decrease in motor imagery classification accuracy; while a relaxed and focused state is conducive to generating stable ERD / ERP patterns.

[0004] Currently, emotions are mostly used as an auxiliary solution in existing technologies, such as monitoring and prompting, intervening as prompts during training, and not deeply integrated with the control process of motor imagery, with limited system efficiency improvement. In addition, some use data preprocessing to filter and screen data, which is essentially a passive "filtering" mechanism that cannot fully utilize emotional information. Finally, some use it as a feedback mechanism to intervene in task status, with the adjustment logic and motor imagery signal decoding model being separate.

[0005] In summary, existing motor imagery BCI technology based on emotion generally considers emotion recognition and motor imagery decoding as two independent parallel processes. Emotion is only used as an auxiliary, external reference information for system start-stop, data screening, or interface-level interactive adjustment, and is not deeply integrated with motor imagery signals at the model level. SUMMARY

[0006] The technical purpose of the present application is to address the technical problem that existing motor imagery BCI technology generally considers emotion recognition and motor imagery decoding as two independent parallel processes, thereby affecting the effect of motor imagery, and provides a motor imagery coupling system based on emotion prediction.

[0007] To achieve the above technical purposes, the embodiments of the present application adopt the following technical solutions.

[0008] In one aspect, the embodiments of the present application provide a motor imagery coupling system based on emotion prediction, comprising:

[0009] A data acquisition module is configured to acquire dynamic emotional electroencephalogram data of an emotional brain region and motor imagery electroencephalogram data of a motor perception brain region;

[0010] A feature extraction module is configured to extract emotional features based on the dynamic emotional electroencephalogram data and extract motor imagery features based on the motor imagery electroencephalogram data, thereby forming an emotional feature vector and a motor feature vector;

[0011] An emotion prediction module is configured to obtain a predicted emotional feature vector at a subsequent time based on a historical emotional feature vector and a current emotional feature vector;

[0012] A context vector determination module is configured to obtain an emotional context vector by fusing the current emotional feature vector and the predicted emotional feature vector;

[0013] A feature modulation adaptive module is configured to determine an emotional index vector based on the emotional context vector and determine an emotional index weight vector based on the emotional index vector;

[0014] A feature weighting fusion module is configured to perform weighted fusion of the emotional index weight vector and the motor feature vector, thereby obtaining a weighted and enhanced motor imagery feature vector;

[0015] A motor imagery decoding module is configured to complete decoding of the weighted and enhanced motor imagery feature vector and output a control signal to drive a terminal device according to a decoding result.

[0016] In another aspect, the embodiments of the present application provide a motor imagery coupling method based on emotion prediction, comprising: acquiring dynamic emotional electroencephalogram data of an emotional brain region and motor imagery electroencephalogram data of a motor perception brain region;

[0017] Extracting emotional features based on the dynamic emotional electroencephalogram data and extracting motor imagery features based on the motor imagery electroencephalogram data, thereby forming an emotional feature vector and a motor feature vector;

[0018] Obtaining a predicted emotional feature vector at a subsequent time based on a historical emotional feature vector and a current emotional feature vector;

[0019] Fusing the current emotion feature vector and the predicted emotion feature vector to obtain an emotional context vector, determining an emotion index vector based on the emotional context vector, and determining an emotion index weight vector based on the emotion index vector; and performing weighted fusion of the emotion index weight vector and the motion feature vector to obtain a weighted and enhanced motor imagery feature vector.

[0020] Performing decoding on the weighted and enhanced motor imagery feature vector, and outputting a control signal according to a decoding result to drive a terminal device.

[0021] Compared with the prior art, the motor imagery coupling system and method based on emotion prediction provided in the embodiments of the present application have the following beneficial technical effects: by dynamically collecting emotional and motor imagery dual-modal electroencephalogram data, the emotion recognition is converted from an independent and parallel process into a dynamic regulation factor of motor imagery decoding, the real-time state and change trend of the emotion are captured by means of emotion prediction and context vector fusion, and then the motor imagery features are adaptively modulated by the emotion index weight, so that deep coupling of the two is realized. This design not only conforms to the dynamic influence mechanism of emotion on the motor cortex activity, but also enhances the discriminability of the motion features by weighted fusion, reduces the decoding deviation caused by emotional interference, and at the same time adapts to the non-stationarity and individual differences of the electroencephalogram signal, so as to finally significantly improve the decoding accuracy, dynamic adaptability and actual operation reliability of the motor imagery BCI system.

[0022] It should be understood that the summary section is not intended to identify key or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. In addition, the shapes and proportions of the components in the drawings are only illustrative and are used to help understand the present application, and are not specific limitations on the shapes and proportions of the components of the present application. Those skilled in the art can select various possible shapes and proportions to implement the present application according to specific circumstances under the teaching of the present application. In the drawings:

[0024] Figure 1 A motor imagery coupling method based on emotion prediction is provided for the embodiments;

[0025] Figure 2 A flowchart of the process of collecting and preprocessing electroencephalogram signals and feature extraction is provided for the embodiments;

[0026] Figure 3 A flowchart of the process of dual-path updating of emotion index and motor imagery coupling is provided for the embodiments;

[0027] Figure 4A flowchart of a slight bias strategy for implementing the initialization of adaptive weight parameters m and n in the embodiment. DETAILED DESCRIPTION

[0028] In order for those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0029] It should be fully understood that the user information (including but not limited to user physiological sensor information, user personal information, etc.) involved in the present application is information and data authorized by the user or authorized by all parties, and the use of user information should comply with the privacy policies and practices generally considered to meet or exceed the industry for maintaining user privacy, the collection, use and processing of relevant data should comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0030] In the embodiment, the emotion prediction-based motor imagery coupling system comprises a data acquisition module, a feature extraction module, an emotion prediction module, a context vector determination module, a feature modulation adaptive module, a feature weighted fusion module and a motor imagery decoding module.

[0031] The data acquisition module is used to acquire dynamic emotional electroencephalogram data of emotional related brain regions and motor imagery electroencephalogram data of motor perception related brain regions;

[0032] The feature extraction module is used to extract emotional features based on the dynamic emotional electroencephalogram data, and extract motor imagery features based on the motor imagery electroencephalogram data, to form an emotional feature vector and a motor feature vector;

[0033] The emotion prediction module is used to obtain a predicted emotional feature vector at a subsequent time based on a historical emotional feature vector and a current emotional feature vector;

[0034] The context vector determination module is used to obtain an emotional context vector by fusing the current emotional feature vector and the predicted emotional feature vector;

[0035] The feature modulation adaptive module is used to determine an emotional index vector based on the emotional context vector, and determine an emotional index weight vector based on the emotional index vector;

[0036] The feature weighted fusion module is used to perform weighted fusion of the emotional index weight vector and the motor feature vector, to obtain a weighted enhanced motor imagery feature vector;

[0037] The motor imagery decoding module is configured to decode the weighted enhanced motor imagery feature vector, and output a control signal to drive a terminal device according to a decoding result.

[0038] Based on the same inventive concept as the motor imagery coupling system based on emotion prediction provided in the above embodiments, the embodiments of the present application also provide a motor imagery coupling method based on emotion prediction, as shown in the accompanying drawings. Figure 1 As shown, the method mainly includes collecting (and also preprocessing) electroencephalogram data of different regions of the brain for emotion and motor imagery tasks, extracting emotion features and motor imagery features according to the processed data, establishing an emotion prediction model, constructing a modulation network by the predicted emotion features and current emotion features to fine-tune the motor imagery data weight, and finally decoding the weighted motor features to obtain the motor imagery result to drive the hardware terminal device.

[0039] The motor imagery coupling method based on emotion prediction provided by the embodiments includes:

[0040] Step 1: Collect dynamic emotional electroencephalogram data of emotion-related brain regions and motor imagery electroencephalogram data of motor perception-related brain regions.

[0041] In the embodiments, a multi-channel lead electroencephalogram cap according to the international 10-20 standard can be selected, and a wet electrode combined with conductive paste can be selected as a conductive medium to reduce interference caused by the external environment.

[0042] In the embodiments, in order to reduce the computational complexity, only the emotion-related brain regions and the motor perception-related brain regions are selected for electroencephalogram signal collection. For example, the emotion-related brain regions include the frontal lobe regions (Fp1, Fp2, F3, F4), and the motor perception-related brain regions include C3, C4 and Cz. The dynamic emotional electroencephalogram data and the motor imagery electroencephalogram data are used for ERD (event-related desynchronization) and ERP (event-related synchronization).

[0043] In the embodiments, a sampling rate of at least 250 Hz can be selected to ensure the details of the electroencephalogram features.

[0044] Optionally, the data collection module is also configured to perform data preprocessing on the collected dynamic emotional electroencephalogram data and motor imagery electroencephalogram data. The data preprocessing includes: removing power frequency interference by using 50 Hz notch filtering, retaining effective electroencephalogram features by electroencephalogram emotion and motor imagery related full-band band-pass filtering, and retaining electroencephalogram related feature details by 0.5-45 Hz band-pass filtering.

[0045] In some embodiments, the collected dynamic emotional EEG data and motor imagery EEG data are de-noised, specifically including: using a dynamic threshold (such as ±100 μV) to remove large amplitude interference caused by motion artifacts, and if there is an electrooculogram channel, the eye movement artifacts can be identified and removed by ICA (independent component analysis).

[0046] In embodiments, the continuous EEG data stream can be segmented into overlapping time windows for subsequent feature extraction. A fixed time window of 2-4 seconds can be used to store emotional EEG data of the emotional related region for a period of time.

[0047] In embodiments, smooth dynamic emotional data stream can be obtained by sliding 0.5 seconds according to dynamic emotional data.

[0048] Step 2: Perform emotional feature extraction based on dynamic emotional EEG data, and perform motor imagery feature extraction based on motor imagery EEG data to form emotional feature vectors and motor feature vectors.

[0049] In embodiments, theta (4-8 Hz), alpha (8-13 Hz), beta (13-30 Hz), and gamma (> 30 Hz) bands related to emotions are selected, wherein the alpha band is related to relaxation and resting state, the beta band is related to concentration and tension, the theta band is related to drowsiness, and the gamma band is related to high cognitive load and excitement.

[0050] As shown in Figure 2 , in embodiments, fast Fourier transform is performed on the dynamic emotional EEG data to obtain the frequency domain feature values of each channel, and the expression is: .

[0051] wherein X n represents the frequency domain feature values of the dynamic emotional EEG data, X k represents the discrete time domain signal of the dynamic emotional EEG data, N represents the total number of sampling points, i represents the imaginary unit, k is the index of the time domain sampling point, and n is the index of the frequency domain component.

[0052] Based on the frequency domain feature values of each channel, the PSD (power spectral density) of the theta band, alpha band, beta band, and gamma band of the dynamic emotional EEG data is obtained using the Welch method, and the differential entropy D band =log(PSD) is used as the emotional frequency domain feature.

[0053] Since there is frontal lobe EEG asymmetry, the differential entropy of the left (Fp1, F3) dynamic emotional EEG data of the theta, alpha, beta, and gamma bands of the frontal lobe region is also calculated.

[0054] Taking the calculation of the differential entropy of the α band as an example (the calculation methods for the differential entropy and symmetry index of the θ, β, and γ bands are the same), the differential entropy D of the α band of the dynamic emotional EEG data of the left side (Fp1, F3) of the prefrontal region is calculated respectively. α_Fp1 and D α_F3 The differential entropy D of the alpha band of dynamic emotional EEG data from the right prefrontal cortex region (Fp2, F4) α_Fp2 and D α_F4 .

[0055] The symmetry index A of the α band is determined based on the differential entropy of the α band. α1 and A α2 (Only the symmetry index of the α band needs to be determined; the symmetry index of other bands is not considered.) The expression is:

[0056] A α1 =(D α_Fp1 -D α_Fp2 ) / (D α_Fp1 +D α_Fp2 A α2 =(D α_F3 -D α_F4 ) / (D α_F3 +D α_F4 ).

[0057] Based on the standard deviation of the time-domain signal and the Hjorth mobility parameter as the time-domain features of emotion, the standard deviations (S0, S1, S2, S3) of the time-domain signals of the four channels (left (Fp1, F3) and right (Fp2, F4)) were calculated respectively. Fp1 S Fp2 S F3 S F4 ) and the corresponding Hjorth mobility HM (HM is a mobility index, a judgment index belonging to the time domain characteristics), including H Fp1 H Fp2 H F3 H F4 HM uses the ratio of the standard deviation of the time-domain signal to the standard deviation of the first derivative of the signal for quantization.

[0058] Finally, the feature vectors are concatenated based on the temporal, frequency, and spatial features of emotion to obtain the emotion feature vector F. emo The emotion frequency domain features include the differential entropy relationships of different frequency bands of Fp1, Fp2, F3, and F4; the emotion spatial domain features are symmetry indices; and the emotion temporal domain features include the Hjorth mobility estimation index for the number of relevant channels. The emotion feature vector F emo The expression is as follows (for ease of representation, F will be used below). emo The expression omits the corresponding data for the Fp2, F3, and F4 regions.

[0059] F emo =[D θ_Fp1 D α_Fp1, D β_Fp1, D γ_Fp1 A α1 A α2, S Fp1 H Fp1 ,...];

[0060] Where D θ_Fp1 D is the differential entropy of the theta band in the Fp1 dynamic emotional EEG data of the prefrontal region. α_Fp1 D is the differential entropy of the alpha band of Fp1 dynamic emotional EEG data in the prefrontal region. β_Fp1 D is the differential entropy of the β band of Fp1 dynamic emotional EEG data in the prefrontal region. γ_Fp1 S represents the differential entropy of the gamma band in the Fp1 dynamic emotional EEG data of the prefrontal region. Fp1 H is the standard deviation of the time-domain signal of the Fp1 dynamic emotion EEG data in the prefrontal cortex region. Fp1 It is the dynamic emotional EEG data of the prefrontal cortex Fp1 region and the Hjorth mobility HM.

[0061] To eliminate individual differences, the Z-Score is used to standardize the emotion feature vector, resulting in the standardized emotion feature vector F. emo .

[0062] In this embodiment, the real-time motion imagery feature extraction includes: performing feature processing on the relevant data of the motion perception area (C3, C4, Cz) and the same time window as the emotion data, and capturing the ERD (Event-Related Desynchronization) and ERP (Event-Related Potential) phenomena, that is, when imagining one side, the energy of the contralateral motion perception area decreases (ERD), and the energy of the ipsilateral area increases (ERP).

[0063] In a specific embodiment, C3 and C4 obtain the differential entropy of their μ rhythm (8~13Hz) and β rhythm (13~30Hz). Preferably, ERD / ERP is used to eliminate individual differences. Therefore, resting-state calculations are required in the paradigm to obtain baseline data. The calculated ERD / ERP ratio, along with the aforementioned differential entropy, serves as a frequency domain feature of motion imagery, where P base For baseline frequency domain information, P current For actual frequency domain information, Per c This represents the frequency domain characteristics of motion imagination.

[0064] In this embodiment, the specific process for the frequency domain features of motor imagery includes: first, acquiring user baseline data using the resting-state paradigm, and then calculating frequency domain information for this user baseline data using FFT; next, calculating the power Pbase_μ and power Pbase_β of the user baseline data in the μ band and β band respectively; the same steps are followed in the subsequent real-time motor imagery EEG data feature processing to obtain the power Pcurrent_μ and power Pcurrent_β of the real-time motor imagery EEG data in the μ band and β band respectively; using the formula:

[0065] ; Get Per c _μ,Per c _β. Finally, these features are calculated for both C3 and C4 channels to obtain four specific features [Per c _c3_μ,Per c _C3_β,Per c _c4_μ,Per c _C4_β] is used as its motion imagination frequency domain feature, where Per c _c3_μ represents the ERD / ERP ratio of the μ-rhythm in lead C3, Per c _C3_β is the ERD / ERP ratio of the β rhythm in lead C3, Per c _c4_μ represents the ERD / ERP ratio of the μ-rhythm in lead C4, Per c _C4_β is the ERD / ERP ratio of the β rhythm in lead C4.

[0066] In this embodiment, CSP (Common Spatial Pattern) is used to eliminate common noise in the EEG space and enhance the signal-to-noise ratio related to motor imagery to obtain spatial features of motor imagery, specifically including:

[0067] Step 201: Take the preprocessed EEG data of the motor perception region as input. The data dimension is the number of channels multiplied by the number of signal points, and the motor imagery task category has been labeled (such as left hand imagery, right hand imagery).

[0068] Step 202: Based on the labeled training dataset, the optimal spatial projection matrix W is learned through the co-space pattern algorithm. The input EEG signal X is linearly transformed with the projection matrix W (Z=W×X) to obtain the transformed signal Z. Each row of the transformed signal Z corresponds to a spatially filtered component, and the components are sorted from strong to weak according to their ability to distinguish the motor imagery task.

[0069] Step 203: Select the feature components with the strongest discriminative power. In this embodiment, the first m and last m components of the transformed Z can be selected (m is a user-defined positive integer, usually set to 2 or 3). These components can best reflect the specific differences of different motion imagery tasks. Calculate the variance of each component after filtering, and use it as a feature quantification index. Then, perform logarithmic standardization on the variance value to make the feature distribution closer to a Gaussian distribution. Finally, obtain the motion imagery spatial domain feature components after co-space pattern filtering, which are used for subsequent concatenation of motion imagery feature vectors.

[0070] In this embodiment, the Hjorth mobility parameter is used as a temporal feature of motor imagery, and the calculation method is the same as that used to obtain the temporal features of emotion.

[0071] The motion visualization frequency domain features, motion visualization spatial domain features, and motion visualization temporal domain features are concatenated to obtain the concatenated and standardized motion feature vector F. mi The motion imagery frequency domain features include the differential entropy of different rhythms in channels C3 and C4, as well as the ERD / ERP ratio Per. c _c3_μ、Per c _C3_β、Per c _c4_μ and Per c The _C4_β information, the motion imagination spatial features include the filtered components of CSP; the motion imagination spatial features use Hjorth mobility estimation.

[0072] Step 3: Based on the historical emotion feature vector and the current emotion feature vector, obtain the predicted emotion feature vector for subsequent time periods.

[0073] In this embodiment, the emotion prediction module includes a feature vector cache queue S. emo And causal temporal learning model: feature vector cache queue S emo This is used to store historical emotion feature vectors. A first-in-first-out (FIFO) mechanism is used to manage the queue data. The cache threshold is set to n, where n is the maximum capacity of the queue (e.g., n=3, 5, 10, set according to actual time-series dependency requirements). This means that the queue can store a maximum of n emotion feature vectors at the same time. After exceeding n, when a new emotion feature vector (e.g., the (n+1)th) is to be added to the queue, if the queue is already full of n features and the capacity limit is reached, the oldest feature will be deleted from the queue to make room for the new feature.

[0074] The causal temporal learning model is based on cached historical sentiment feature vectors and current sentiment feature vector F. emo (t), outputting the predicted sentiment feature vector at time t+k.

[0075] In this embodiment, the causal temporal learning model is a gated recurrent unit model or a causal convolutional network model, trained using an offline dataset, to minimize the predicted sentiment feature vector F. emo_pred Compared with true emotional characteristics F emo_true The mean squared error is minimized. A GRU (Gated Recurrent Unit) architecture is used in the design to achieve speed and a small number of parameters.

[0076] In practical applications, a causal time-series learning model is first trained using an offline dataset, with the training objective being to minimize the predicted value F. emo_pred (t+1) and the true value F emo The mean square error of (t+1). In practical use, this is achieved by inputting S. emo (t) Finally, the predicted emotion feature vector F for the next moment is obtained. emo_pred (t+1).

[0077] Step 4: Merge the current emotion feature vector with the predicted emotion feature vector to obtain the emotion context vector; determine the emotion index vector based on the emotion context vector; determine the emotion index weight vector based on the emotion index vector; and perform weighted fusion of the emotion index weight vector and the motion feature vector to obtain the weighted enhanced motion imagery feature vector.

[0078] In the embodiment, based on the current emotion feature vector F emo (t) and the predicted sentiment feature vector F emo_pred (t+1) is used to fuse the data and construct an emotion context vector C. emo The expression is:

[0079] C emo= m×F emo (t)+n×F emo_pred (t+1);

[0080] Where m and n are adaptive weight parameters preset through the prior relationship between motor imagery and predicted emotions, which reflect the effectiveness of predicted emotions and their impact on the results of motor imagery.

[0081] In some embodiments, a simple and intuitive way to set the initial values ​​of m and n is to set both m and n to 0.5. However, this strategy has a problem: even if an attention weight mechanism is subsequently introduced for dynamic adjustment, initializing the weights to 0.5 is equivalent to injecting a priori assumption: the current emotional feature F... emo (t) and predictive sentiment features F emo_pred (t+1) is equally important in all cases.

[0082] In other embodiments, a better initialization method is chosen, such as micro-random initialization or zero initialization (Xavier Uniform initialization), injecting neutral priors, and fully trusting and relying on the attention mechanism to dynamically learn the weight allocation that is best suited to the current context. This is a very intuitive algorithm design.

[0083] To provide a precise and efficient motion imagination coupling system, although neutral priors can explore effective dynamic weighting strategies, the model may be affected in terms of convergence speed. Therefore, in some embodiments, an improved method for initializing adaptive weight parameters m and n is provided, namely, a slightly biased initialization weight based on adaptive adjustment of knowledge in the motion imagination domain.

[0084] Before further explaining the improvement strategy, let's briefly describe the conventional approach to initializing weights with a slight bias based on domain knowledge: The current sentiment feature vector F... emo The weight of (t) is set to 0.6, and the predicted sentiment feature vector F is... emo_pred The weight of (t+1) is set to 0.4, which means that a weak prior of "current emotion is relatively more important" is injected into the model. The starting point of this prior approach is to predict the emotion feature vector F. emo_pred (t+1) may not be entirely accurate or perfectly consistent with the actual emotion; the predicted emotion feature vector F emo_pred (t+1) is merely a predictor; this prior method can accelerate convergence and improve initial stability to some extent.

[0085] Setting a fixed weight lacks flexibility. Therefore, refer to... Figure 4 The flowchart shown illustrates the complete process from calculating the attention score to the final feature fusion. In some embodiments, a more flexible, lightly biased weight initialization strategy is employed, including:

[0086] The current emotion feature vector F emo (t) and the predicted sentiment feature vector F emo_pred (t+1) Project the vectors onto the key and value spaces through a linear layer, respectively, and compute a query vector (Q). After calculating the attention score (e=Q×K^T), F is manually assigned a value. emo The score corresponding to (t) is increased by a positive bijective b.

[0087] The attention score corresponding to the current emotion feature vector is calculated based on the key vector of the query vector and the current emotion feature vector, with a positive bias added. The expression is: e_t=Q×K_t^T+b, where b is a small positive number initialized, for example, b=0.2.

[0088] The attention score corresponding to the predicted emotion feature vector is calculated based on the key vector of the pre-query vector and the predicted emotion feature vector, without adding bias. The expression is: e_f = Q × K_f^T, where the future emotion score is not biased; where e_t is the attention score of the current emotion feature vector, Q is the query vector, K_t is the key vector of the current emotion feature vector, T is the transpose, b is the bias, e_f is the attention score of the predicted emotion feature vector, and K_f is the key vector of the predicted emotion feature vector.

[0089] The aforementioned method of slightly biasing the initial weights is equivalent to slightly boosting the current emotional feature F before the attention mechanism begins to work. emo The baseline importance of (t) means the model can still dynamically learn weights m and n, but if the current sentiment feature F... emo (t) and predictive sentiment features F emo_pred The (t+1) features have similar relevance to the query vector, and by default, more attention is paid to F. emo (t). If F emo (t) indeed shows a very strong correlation. The attention mechanism can learn a large score to cover the initial bias, thus taking into account the flexibility of adaptive weights and better playing the role of accelerating convergence and improving initial stability.

[0090] Additional notes: Query vector calculation method: Q = W_q × X_combined + b_q;

[0091] Where W_q is the weight matrix of the linear layer, X_combined is the concatenated vector composed of motion feature vector and emotion context feature vector, and b_q is the bias vector.

[0092] In the embodiments, such as Figure 3 As shown, the feature modulation adaptive module includes a sentiment index decoding module and a feature gate control module.

[0093] The sentiment index decoding module is used to determine the sentiment index vector I based on the sentiment context vector. emo The aforementioned emotional context needs to be mapped to weights of a set of emotional indicators (such as brain activity indicators like attention, fatigue, valence, and anxiety). To achieve this, an attention modulation module is constructed using a network model, taking the emotional context C as input. emo (Including emotion-related time-domain, frequency-domain, and spatial-domain features) Automatically derives the emotion index vector I emo .

[0094] The feature gate control module is used to control the emotion index vector I. emo The motion feature vector F is obtained through an activation function (such as the Tanh activation function). mi Sentiment indicator weight vector A with identical dimensionsemo The output is guaranteed to be within the range [-1, 1], where negative values ​​represent low levels and positive values ​​represent high levels. Sentiment index weight vector A emo All feature output values ​​are within the range (0,1).

[0095] In this embodiment, the feature weighted fusion module will convert the emotion index weight vector A emo With motion feature vector F mi Performing a dot product yields the final weighted and enhanced motion imagery feature vector F. mi_wei .

[0096] In some embodiments, the system also includes a security arbitration module that determines the user's state based on the emotion index weight vector. If the state is determined to be negative, the decoding process is interrupted and a prompt signal is issued. If the state is determined to be positive, the feature weighted fusion module normally outputs the weighted enhanced motion imagery feature vector to the motion imagery decoding module.

[0097] In the embodiment, because the sentiment index weight vector A emo The output is in the range [-1, 1]. This vector contains emotional indicators such as attention, fatigue, anxiety, and relaxation, with each indicator corresponding to a weight, which can be positive or negative. The emotional indicator weight vector A can be easily defined. emo The final value is calculated by weighting all elements. If it is positive, it indicates positive emotions; if it is negative, it indicates negative emotions.

[0098] In the embodiments, some network models can also be used to learn the correlation between indicators, based on the sentiment indicator weight vector A. emo It automatically provides weighted values ​​for weighted summation and judgment to determine whether it is positive or negative.

[0099] In some embodiments, specific indicators can be judged individually. For example, negative emotions such as anxiety or fatigue are directly judged as negative as long as they exceed a set weight threshold. If the result is positive, the next step is carried out. If the result is negative, it means that the user may not be suitable for training or hardware control. In this case, the system is interrupted and prompted directly, and the motion decoding step is not performed. This also reduces resource consumption and improves system performance.

[0100] Step 5: Decode the weighted and enhanced motion imagery feature vector, and output control signals to drive the terminal device based on the decoding results.

[0101] Motion imagery decoding is achieved through a motion imagery decoding module. A lightweight multilayer perceptron (MLP) model is selected as the decoding model, and the weighted and enhanced motion imagery feature vector F obtained in step 4 is used. mi_weiAs input, the model outputs a probability vector, and the one with the highest probability is selected as the final motor imagery result for control. This can be either rehabilitation training hardware or brain-controlled hardware.

[0102] The weighted and enhanced motion imagery feature vector F mi_wei After inputting into the motion imagery decoding module, motion imagery task feedback can be obtained. Based on this feedback, F can be determined. mi_wei The effectiveness. However, due to F mi_wei In reality, it is obtained from the emotional context vector containing information on historical emotions, current emotions, and future predicted emotions, as well as the features of motor imagery. At this point, if we only adjust m and n after initialization based on the feedback from the motor imagery task, it is actually a relatively compromised adjustment method. That is, the entire algorithm design idea is "emotion prediction-execution-feedback", which is similar to the design idea of ​​reinforcement learning. It cannot correct the model itself. Once a negative motor imagery task feedback is obtained, the model cannot distinguish whether the negative feedback is caused by incorrect m and n or by an error in the future predicted emotion.

[0103] Therefore, in some embodiments, reference may be made to the appendix. Figure 3 m and n are updated using a dual-path approach: one path uses rapid adaptive updates based on feedback from a motion visualization task, and the other path updates based on the predicted F. emo_pred The corresponding true emotion feature vector F emo_true Slow adaptive updates are performed to fully ensure the adaptability, flexibility, and accuracy of the motion-imagination coupling system in this scheme.

[0104] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, or a tablet computer, or any combination of these devices.

[0105] The above provides a detailed description of the motion-image coupling system and method based on emotion prediction provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the concept of this application and should not be construed as a limitation on the scope of protection of this application.

Claims

1. A motion imagination coupling system based on emotion prediction, characterized in that, The system comprises: a data acquisition module configured to acquire dynamic emotional electroencephalogram (EEG) data of an emotional brain region and motor imagery EEG data of a motor perception brain region; a feature extraction module configured to extract emotional features based on the dynamic emotional EEG data and extract motor imagery features based on the motor imagery EEG data, to form an emotional feature vector and a motor feature vector; an emotion prediction module configured to obtain a predicted emotional feature vector at a subsequent time based on a historical emotional feature vector and a current emotional feature vector; a context vector determination module configured to fuse the current emotional feature vector and the predicted emotional feature vector to obtain an emotional context vector; a feature modulation adaptive module configured to determine an emotional index vector based on the emotional context vector and determine an emotional index weight vector based on the emotional index vector; a feature weighted fusion module configured to perform weighted fusion of the emotional index weight vector and the motor feature vector to obtain a weighted and enhanced motor imagery feature vector; a motor imagery decoding module configured to complete decoding of the weighted and enhanced motor imagery feature vector and output a control signal according to a decoding result to drive a terminal device.

2. The motor imagery coupling system based on emotion prediction according to claim 1, wherein the feature modulation adaptive module comprises an emotional index decoding module and a feature gate control module; the emotional index decoding module is configured to determine an emotional index vector according to the emotional context vector; The feature door control module is configured to obtain a motion feature vector F by using an activation function based on the emotion index vector mi The emotion index weight vector A has the same dimension emo ; The feature weighting fusion module is configured to perform point multiplication on the emotion index weight vector A emo and the motion feature vector F mi to obtain a final weighted and enhanced motion imagination feature vector F mi_wei .

3. The emotion prediction based motor imagery coupling system of claim 2, wherein, the system further comprises a safety arbitration module configured to determine a user state based on an emotional index corresponding to the emotional context vector, and if the user state is determined to be a negative state, the decoding process is interrupted and a prompt signal is sent; if the user state is determined to be a positive state, the feature weighted fusion module normally outputs the weighted and enhanced motor imagery feature vector to the motor imagery decoding module.

4. The emotion prediction based motor imagery coupling system of claim 1, wherein, The data acquisition module is further configured to perform data preprocessing on the acquired dynamic emotional EEG data and motor imagery EEG data, and the data preprocessing comprises: removing power frequency interference through 50 Hz notch filtering; removing motion artifacts and electrooculogram artifacts through a dynamic threshold method and / or independent component analysis; and dividing continuous dynamic emotional EEG data and motor imagery EEG data streams into overlapping time windows for subsequent feature extraction.

5. The emotion prediction based motor imagery coupling system of claim 1, wherein, The emotional feature extraction based on the dynamic emotional EEG data comprises extracting emotional time domain features, emotional frequency domain features, and emotional spatial domain features; the emotional time domain features comprise standard deviation of time domain signals and Hjorth mobility parameters; the emotional frequency domain features comprise differential entropy of power spectral density of θ band, α band, β band, and γ band; the emotional spatial domain features comprise α band asymmetry index of left and right sides of a frontal lobe region; and the emotional time domain features, the emotional frequency domain features, and the emotional spatial domain features are subjected to Z-Score standardization processing to form an emotional feature vector.

6. The emotion prediction based motor imagery coupling system of claim 1, wherein, The motor imagery feature extraction based on motor imagery EEG data includes extracting motor imagery time domain features, motor imagery frequency domain features and motor imagery space domain features, the motor imagery time domain features include Hjorth mobility parameters, the motor imagery frequency domain features include differential entropy of mu rhythm and beta rhythm and ERD / ERP ratio, the ERD / ERP ratio is calculated by current EEG power and resting state baseline power; the motor imagery space domain features include feature components filtered by common spatial pattern.

7. The emotion prediction based motor imagery coupling system of claim 1, wherein, The emotion prediction module includes a feature vector cache queue and a causal time series learning model: the feature vector cache queue is used to store historical emotion feature vectors, and adopts a first-in first-out mechanism to manage queue data; the causal time series learning model outputs a predicted emotion feature vector at a subsequent time based on the cached historical emotion feature vector and the current emotion feature vector; the causal time series learning model is a gated recurrent unit model or a causal convolutional network model, which is trained through an offline data set to minimize the mean square error between the predicted feature and the real feature.

8. The emotion prediction based motor imagery coupling system of claim 1, wherein, The feature modulation adaptive module fuses the current emotion feature vector and the predicted emotion feature vector to obtain an emotion context vector by using the following formula: C emo = m x F emo (t) + n x F emo_pred (t + 1) generates an emotional context vector, where F emo (t) is a current emotional feature vector, F emo_pred (t + 1) is a predicted emotional feature vector, and m and n are adaptive weight parameters. The motion feature vector is weighted and fused based on the emotion context vector to obtain a weighted and enhanced motor imagery feature vector, including: generating an emotion index vector based on the emotion context vector, normalizing the emotion index vector to the [-1, 1] interval through a Tanh activation function, and then generating an emotion index weight vector consistent with the motor imagery feature dimension through a Sigmoid activation function; The emotion index weight vector and the motor imagery feature vector are point multiplied to obtain a weighted and enhanced motor imagery feature.

9. The emotion prediction based motor imagery coupling system of claim 8, wherein, The initialization of adaptive weight parameters m and n adopts a mild bias strategy, including projecting the current emotion feature vector and the predicted emotion feature vector to key space and value space respectively through a linear layer to obtain corresponding key vectors and value vectors; calculating a unique query vector Q based on a combined feature vector formed by splicing the motor imagery feature vector and the task context feature vector; The attention score corresponding to the current emotion feature vector is calculated according to the key vector of the query vector and the current emotion feature vector, wherein a positive bias is added; The attention score corresponding to the predicted emotion feature vector is calculated according to the key vector of the pre-query vector and the predicted emotion feature vector, wherein no bias is added; the two groups of attention scores are normalized through a Softmax function to obtain initial adaptive weight parameters m and n, which are used to weight and fuse the value vectors of the current emotion feature vector and the predicted emotion feature vector.

10. The emotion prediction based motor imagery coupling system of claim 3, wherein, The safety arbitration module uses at least one of the following methods to determine the user state: The weighted average of each index weight in the emotion index weight vector is calculated, and if the result is negative, it is determined as a negative state; Thresholds are set for each index weight in the emotion index weight vector, and if the anxiety index or the fatigue index exceeds the threshold, it is determined as a negative state; The association between emotion indexes is learned through a network model, and the state determination result is automatically output according to the emotion index weight vector.

11. A method for emotion prediction-based motor imagery coupling, comprising: including: Collecting dynamic emotional brain electrical data of emotion-related brain regions and motor imagination brain electrical data of motor perception-related brain regions; Performing emotion feature extraction based on the dynamic emotional brain electrical data and motor imagination feature extraction based on the motor imagination brain electrical data to form an emotion feature vector and a motor feature vector; Obtaining a predicted emotion feature vector of a subsequent moment based on a historical emotion feature vector and a current emotion feature vector; Fusing the current emotion feature vector and the predicted emotion feature vector to obtain an emotional context vector, determining an emotion index vector based on the emotional context vector, determining an emotion index weight vector based on the emotion index vector, and performing weighted fusion of the emotion index weight vector and the motor feature vector to obtain a weighted and enhanced motor imagination feature vector; Performing decoding on the weighted and enhanced motor imagination feature vector, and outputting a control signal to drive a terminal device according to a decoding result.

Images