Emotion perception interaction method and system of brain-computer interface earphone

Abstract

The invention relates to the technical field of brain-computer interfaces, in particular to an emotion perception interaction method and system for a brain-computer interface earphone, and the method comprises the following steps: obtaining an electroencephalogram signal and a voice signal of a user; preprocessing the electroencephalogram signal and the voice signal; electroencephalogram features are extracted from the preprocessed electroencephalogram signals, and voice features are extracted from the preprocessed voice signals; performing multi-modal feature fusion based on the electroencephalogram features and the voice features to obtain fusion features; identifying an emotional state of the user based on the fused feature; adjusting earphone output parameters according to the emotional state; obtaining user feedback; and updating the model based on user feedback; the deep fusion of the electroencephalogram and the voice signal is realized through the innovative DAMCLN algorithm, and the accuracy of emotion recognition is remarkably improved. Test results show that the emotion recognition accuracy of the method reaches 92.5% and is far higher than that of the prior art; intelligent parameter adjustment based on the emotional state of the user is realized, and the score of the user reaches 8.7.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to an emotion perception interaction method and system for a brain-computer interface headset. Background Art

[0002] With the rapid development of artificial intelligence and human-computer interaction technology, affective computing has become a research field that has attracted much attention. In the field of education, the application of intelligent learning assistance systems is becoming more and more widespread, aiming to improve learning efficiency and provide personalized learning experience. However, existing technologies still face many challenges in achieving high-precision emotion recognition and personalized interaction.

[0003] At present, emotion recognition technology is mainly divided into the following categories:

[0004] 1. Methods based on a single physiological signal: such as using only electroencephalogram (EEG) for emotion recognition. Although such methods can directly reflect brain activity, they are often affected by signal noise and individual differences, resulting in limited recognition accuracy. For example, the EEG-based emotion recognition method only achieved an accuracy of 78% in the five-category emotion classification task.

[0005] 2. Methods based on speech signals: such as using acoustic features such as Mel-Frequency Cepstral Coefficients (MFCC) for emotion recognition. Such methods are easily affected by environmental noise and individual differences between speakers, and lack robustness in practical applications. Some scholars have proposed a speech emotion recognition method based on LSTM, which can achieve an accuracy of 83% under ideal conditions, but the performance drops significantly in noisy environments.

[0006] 3. Simple multimodal fusion methods: For example, simply concatenating features of different modalities for classification. Although this type of method utilizes multimodal information, it ignores the complex interaction between modalities and fails to fully utilize the advantages of multimodal fusion. The feature cascade method proposed by some scholars has achieved an accuracy rate of 86% in emotion recognition tasks, but there is still much room for improvement.

[0007] 4. Fixed-rule parameter adjustment method: Some existing intelligent learning assistance systems use preset fixed rules to adjust system parameters (such as volume, equalizer settings, etc.), which lack the ability to adapt to individual differences of users and changes in situations. This method is difficult to meet the diverse and dynamically changing needs of users, resulting in poor user experience.

[0008] These prior arts have the following main problems

[0009] 1. Insufficient emotion recognition accuracy: Single modality or simple multimodal fusion methods are difficult to fully capture the complex emotional state of humans, resulting in limited recognition accuracy.

[0010] 2. Lack of personalization and adaptability: The parameter adjustment method with fixed rules cannot adapt to the individual differences and situational changes of different users, and it is difficult to provide a truly personalized experience.

[0011] 3. Balance between real-time and system performance: Although some complex emotion recognition algorithms can improve accuracy, they often lead to increased system response time and energy consumption, and are not suitable for long-term use in portable devices.

[0012] 4. Effective use of user feedback: Existing systems often ignore the importance of user feedback and lack effective mechanisms to continuously optimize and improve system performance.

[0013] In view of the above problems, there is an urgent need for an emotion perception interaction method that can achieve high-precision emotion recognition, personalized parameter adjustment, real-time response and continuous optimization, so as to improve the performance and user experience of intelligent learning assistance systems. Summary of the invention

[0014] In response to these problems, the present invention proposes an emotion perception interaction method and system for brain-computer interface headphones.

[0015] The present invention provides an emotion perception interaction method of a brain-computer interface headset, comprising the following steps:

[0016] Obtaining the user's EEG signal and speech signal; preprocessing the EEG signal and the speech signal; extracting EEG features from the preprocessed EEG signal, and extracting speech features from the preprocessed speech signal; performing multimodal feature fusion based on the EEG features and the speech features to obtain fusion features; identifying the user's emotional state based on the fusion features; adjusting the headphone output parameters according to the emotional state; obtaining user feedback; and updating the model based on the user feedback.

[0017] Specifically, the step of extracting EEG features from the preprocessed EEG signal includes:

[0018] The EEG features are extracted using a multi-scale time-frequency dynamic feature extraction algorithm. The algorithm includes: performing a 5-layer wavelet packet decomposition on the preprocessed EEG signal S(t) to obtain 32 sub-band signals {S i (t),i=1,2,…,32};

[0019] Perform recursive quantization analysis on each subband signal: RQA i =RQA(S i (t)),i=1,2,…,32;

[0020] Construct a multi-scale time-frequency dynamic feature vector: MTFD = [E1,…,E 32 ,RQA1,…,RQA 32]; RQA() is the recursive quantitative analysis function, E i is the energy of the ith subband.

[0021] Specifically, the step of extracting speech features from the preprocessed speech signal includes:

[0022] The speech features are extracted using an adaptive acoustic feature fusion algorithm, which includes:

[0023] Extracting basic acoustic features: F MFCC 、F pitch ,F formants and F energy ;

[0024] Calculate feature importance scores:

[0025] Score i =MI(F i ,Y),i∈{MFCC,pitch,formants,energy};

[0026] Adaptive feature fusion:

[0027] F A AFF=∑(w i *F i ),i∈{MFCC,pitch,formants,energy}; where MI() is

[0028] Mutual information function, Y is the sentiment label, w i =softmax(Score i ) is the adaptive weight.

[0029] Specifically, the step of performing multimodal feature fusion based on EEG features and speech features includes:

[0030] The dynamic adaptive multimodal contrastive learning network algorithm is used for feature fusion, which includes:

[0031] Dynamic weight adaptation:

[0032] w EEG =σ(W1*MTFD+b1),w speech =σ(W2*F AAFF +b2);

[0033] Feature Mapping and Fusion:

[0034] h E EG=tanh(W3*MTFD+b3),

[0035] h speech=tanh(W4*F AAFF +b4),

[0036] h fused =w EEG *h EEG +w speech *h speech ;

[0037] Cross-modal contrastive learning:

[0038] L contrast =-log.exp(sim(h EEG ,h speech ) / τ) / ∑exp(sim(h EEG ,h k ) / τ) / ;

[0039] Temporal Attention Enhancement:

[0040] g t =GRU(h fused ,g t-1 ),

[0041] α t =softmax(v T *tanh(W5*[g t ;c])),

[0042] c t =∑(α t *g t );

[0043] Sentiment Classification:

[0044] y pred =softmax(W6*c t +b6);

[0045] Among them, σ() is the Sigmoid function, sim() is the cosine similarity function, τ is the temperature parameter, GRU is the gated recurrent unit, W1 to W6 and b1 to b6 are learnable parameters.

[0046] Specifically, the step of adjusting the headphone output parameters according to the emotional state includes:

[0047] The headphone output parameters are adjusted using a multi-objective reinforcement learning emotion adaptation control algorithm, which includes:

[0048] Define state space S, action space A and reward function R;

[0049] Value function approximation: Q(s,a)=W T *φ(s,a);

[0050] Policy update: π(a|s) = softmax(Q(s,a) / τ);

[0051] Parameter update: Among them, φ(s,a) is the feature vector of the state-action pair, W is the weight vector, α is the learning rate, γ is the discount factor, and τ is the exploration temperature parameter.

[0052] Specifically, the multi-objective reinforcement learning emotion adaptation control algorithm also includes:

[0053] Use the Pareto optimization method to find the optimal balance between user satisfaction, emotional improvement and energy consumption:

[0054]

[0055] Among them, J i represents the i-th objective function.

[0056] Specifically, the step of obtaining user feedback includes:

[0057] Implicit feedback is extracted using a multimodal user satisfaction estimation algorithm, which includes:

[0058] Feature extraction: F usage 、F physio and F behavior ;

[0059] Satisfaction estimation model: S = σ(W u *F usage +W p *F physio +W b *F behavior + b);

[0060] Model training: Minimize the mean square error L M USE=(SS explicit ) 2 ; where σ is the Sigmoid function, W u , W p , W b and b are learnable parameters, S explicit For explicit feedback.

[0061] Specifically, the step of updating the model based on user feedback includes:

[0062] Incremental updates to a dynamically adaptive multimodal contrastive learning network:

[0063] Use the sliding window strategy to retain the most recent N samples to build a small batch B;

[0064] Compute the gradient:

[0065] Update parameters: θ = θ - η * g;

[0066] Among them, θ is the model parameter and η is the learning rate.

[0067] Specifically, it also includes the model performance monitoring steps:

[0068] Monitor model performance using an adaptive performance evaluation mechanism that includes:

[0069] Defining multi-dimensional performance indicators: emotion recognition accuracy Acc emotion , user satisfaction Satisfaction, system response time T response Battery Life life ;

[0070] Calculate the overall performance score: Score APE =w1*Acc emotion +w2*Satisfaction-w3*T response +w4*Battery life ;

[0071] Dynamic weight adjustment: Among them, w1, w2, w3 and w4 are weight parameters, and λ is the learning rate.

[0072] An emotion perception interaction system of a brain-computer interface headset for implementing the method, characterized by comprising:

[0073] A signal acquisition module is used to obtain the user's EEG signal and voice signal;

[0074] A signal preprocessing module, used for preprocessing the EEG signal and the speech signal;

[0075] A feature extraction module, used to extract EEG features from the preprocessed EEG signal and to extract speech features from the preprocessed speech signal;

[0076] A feature fusion module, used for performing multimodal feature fusion based on the EEG feature and the speech feature to obtain a fusion feature;

[0077] An emotion recognition module, used for recognizing the user's emotional state based on the fused features;

[0078] A parameter adjustment module, used to adjust the headphone output parameters according to the emotional state;

[0079] Feedback acquisition module, used to obtain user feedback;

[0080] A model updating module, configured to update the model based on the user feedback; and

[0081] Performance monitoring module, used to monitor system performance and trigger model retraining or manual intervention.

[0082] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0083] 1. High-precision emotion recognition: The innovative DAMCLN algorithm realizes the deep fusion of EEG and speech signals, which significantly improves the accuracy of emotion recognition. Test results show that the emotion recognition accuracy of the present invention reaches 92.5%, far exceeding the existing technology.

[0084] 2. Personalized parameter adjustment: The MORLEAC algorithm is used to achieve intelligent parameter adjustment based on the user's emotional state, which greatly improves user satisfaction. The user rating reached 8.7 points (out of 10 points), which is 1.2 points higher than the fixed rule method.

[0085] 3. Balance between real-time performance and energy efficiency: Despite the introduction of complex algorithms, the present invention still maintains a fast response time of 45ms and a battery life of 9.5 hours, meeting the usage requirements of portable devices.

[0086] 4. Continuous optimization capability: Through the MUSE algorithm, user feedback is effectively collected and utilized to achieve continuous optimization of the system and ensure long-term use effects.

[0087] 5. Improve learning experience: Through accurate emotion recognition and personalized parameter adjustment, the present invention can create a more comfortable and efficient learning environment for users, effectively improving learning efficiency and quality.

[0088] In summary, the present invention overcomes the limitations of the prior art and provides a high-precision, personalized, real-time responsive and continuously optimized emotion perception interaction method, which opens up a new direction for the development of intelligent learning assistance systems and has important theoretical significance and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 This is a flow chart of an emotion perception interaction method of a brain-computer interface headset of the present invention.

[0090] Figure 2 This is a schematic diagram of the framework of the emotion perception interaction system of the brain-computer interface headset of the present invention. DETAILED DESCRIPTION

[0091] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0092] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0093] The present invention proposes an emotion perception interaction method and system for a brain-computer interface headset, aiming to provide users with an intelligent and personalized audio experience. Imagine a college student named Xiao Ming who is using this smart headset to prepare for his final exam. This scenario will run through our entire description to better understand the working principle and practical application of the present invention.

[0094] like Figure 1-2 As shown, the brain-computer interface headset of the present invention includes a signal acquisition module 1, a signal preprocessing module 2, a feature extraction module 3, a feature fusion module 4, an emotion recognition module 5, a parameter adjustment module 6, a feedback acquisition module 7, a model update module 8 and a performance monitoring module 9. These modules work together to realize a complete emotion perception interactive closed-loop system.

[0095] Below, we will describe the workflow of the present invention in detail with reference to specific embodiments:

[0096] 1. Signal collection step, in this step, the signal collection module 1 is responsible for obtaining the user's EEG signal and voice signal. Take Xiao Ming as an example, when he puts on the headphones to start learning, the headphones will continue to collect his EEG and voice data.

[0097] Specifically, the EEG signal is collected using the dry electrode array built into the headset, the sampling rate is set to 1000Hz, and the acquisition frequency band is 0.5-100Hz. The present invention uses the NeuroSky TGAM module (model: TGAM1) as the core acquisition chip, which has the characteristics of low power consumption and high precision, and is very suitable for long-term wear scenarios.

[0098] Voice signal acquisition is completed through the built-in MEMS microphone (model: Knowles SPH0645LM4H-B) of the headset, with a sampling rate of 44.1kHz and a bit depth of 24 bits. This configuration can capture subtle changes in voice and provide rich information for subsequent emotional analysis.

[0099] 2. Signal preprocessing step: The signal preprocessing module 2 processes the collected original signal to improve the accuracy of subsequent analysis. This step is crucial for removing various interferences and noises.

[0100] 2.1 EEG signal preprocessing:

[0101] a) First, use a notch filter to remove 50Hz power supply interference. This is because power supply frequency interference is very common in EEG signals and will affect the accuracy of subsequent analysis.

[0102] b) Then, wavelet transform is applied to remove electrooculogram and electromyography artifacts. Daubechies wavelet (db4) is used here, the decomposition level is 8, and the soft threshold method is used for denoising.

[0103] c) Finally, a bandpass filter (0.5-45Hz) is used to extract the effective EEG frequency band. This frequency band contains most of the EEG activities related to cognition and emotion.

[0104] 2.2 Speech signal preprocessing:

[0105] a) First, perform endpoint detection to remove silent segments. We use a method that combines short-time energy and zero-crossing rate, with the energy threshold set to 0.01 and the zero-crossing rate threshold set to 0.02.

[0106] b) Then perform pre-emphasis filtering to enhance the high frequency components. The pre-emphasis coefficient α is set to 0.97, and the transfer function is H(z) = 1-αz -1 .

[0107] c) Finally, frame division and windowing are performed. The frame length is set to 25ms, the frame shift is 10ms, and the Hamming window function is used.

[0108] Through these preprocessing steps, we obtained clear and reliable signals, laying the foundation for subsequent feature extraction.

[0109] 3. Feature extraction step: In this step, the feature extraction module 3 extracts features from the preprocessed EEG signal and speech signal respectively. These features will be used for subsequent emotion recognition.

[0110] 3.1 EEG feature extraction

[0111] This paper proposes a novel "Multi-scale Time-Frequency Dynamic Feature Extraction" (MTFDFE) algorithm for extracting EEG features. The algorithm combines wavelet packet transform and recursive quantization analysis, and can effectively capture the multi-scale time-frequency characteristics and nonlinear dynamic characteristics of EEG signals.

[0112] The algorithm steps are as follows:

[0113] a) Perform 5-layer wavelet packet decomposition on the preprocessed EEG signal S(t) to obtain 32 sub-band signals {S i (t), i = 1, 2, ..., 32}. Daubechies wavelet (db4) is used as the basis function.

[0114] b) Perform recursive quantization analysis on each subband signal:

[0115] RQA i =RQA(S i (t)),i=1,2,…,32

[0116] Among them, the RQA() function calculates the following recursive quantitative analysis features:

[0117] Regression rate (RR): indicates the percentage of recurrence points to all points;

[0118] Determinism (DET): indicates the proportion of diagonal structure;

[0119] Average diagonal length (L): reflects the predictability of the system;

[0120] Entropy (ENTR): represents the complexity of the diagonal length distribution;

[0121] Trend (TREND): reflects the non-stationary nature of the system

[0122] c) Construct multi-scale time-frequency dynamic feature vector:

[0123] MTFD=[E1,…,E 32 ,RQA1,…,RQA 32 ]

[0124] Among them, E i is the energy of the ith subband, calculated as: E i =∑|S i (t)| 2

[0125] This feature extraction method can fully capture the time-frequency characteristics and dynamic characteristics of EEG signals, providing rich information for subsequent emotion recognition. For example, when Xiao Ming feels increased pressure during learning, the energy of the beta wave (13-30Hz) in his EEG signal may increase, which will be reflected in the energy E of the corresponding sub-band. i At the same time, the complexity of EEG signals may decrease, which will be reflected in RQA features such as the increase of certainty (DET) and the decrease of entropy (ENTR).

[0126] 3.2 Speech feature extraction. The present invention proposes an "Adaptive Acoustic Feature Fusion" (AAFF) algorithm for extracting speech features. The algorithm can dynamically fuse multiple acoustic features to enhance the feature discrimination capability.

[0127] The algorithm steps are as follows:

[0128] a) Extract basic acoustic features:

[0129] F MFCC = MFCC(S(t))∥Mel-frequency cepstral coefficients

[0130] F pitch = Pitch(S(t))∥fundamental frequency

[0131] F formants =Formants(S(t))∥Formants

[0132] F energy =Energy(S(t))∥short-term energy

[0133] Specifically, MFCC feature extraction uses 13th-order coefficients, Pitch uses the autocorrelation method to extract, Formants uses the linear prediction method to extract the first three resonance peaks, and Energy calculates the logarithmic energy of each frame.

[0134] b) Calculate feature importance scores:

[0135] Score i =MI(F i ,Y),i∈{MFCC,pitch,formants,energy}

[0136] Where MI() is the mutual information function and Y is the emotion label. The mutual information is calculated using the nearest neighbor estimation method, and the k value is set to 5. This step can evaluate the contribution of each feature to emotion recognition.

[0137] c) Adaptive feature fusion:

[0138] F AAFF =∑(w i *F i ),i∈{MFCC,pitch,formants,energy}

[0139] where w i =softmax(Score i) is the adaptive weight. The temperature of the Softmax function This adaptive feature fusion method can dynamically adjust the feature weights according to the importance of various acoustic features under different emotional states. For example, when Xiao Ming feels frustrated during learning, his voice may become low and slow, and the weights of pitch and energy features may increase, while when he feels excited, the weight of formants features may increase.

[0140] 4. Feature fusion step: In this step, the feature fusion module 4 uses the "Dynamic Adaptive Multi-modal Contrastive Learning Network" (DAMCLN) algorithm to fuse EEG features and speech features. This algorithm can adaptively fuse multimodal features and learn cross-modal representations that are consistent with emotions.

[0141] The algorithm steps are as follows:

[0142] 4.1 Dynamic Weight Adaptation:

[0143] w EEG =σ(W1*MTFD+b1),

[0144] w speech =σ(W2*F AAFF +b2);

[0145] 4.2 Feature Mapping and Fusion:

[0146] h EEG =tanh(W3*MTFD+b3)

[0147] h speech =tanh(W4*F AAFF +b4)

[0148] h fused =w EEG *h EEG +w speech *h speec

[0149] Among them, σ() is the Sigmoid function, W1, W2, b1, b2 are learnable parameters. During initialization, W1 and W2 are initialized using Xavier, and b1 and b2 are initialized to 0. Among them, W3, W4, b3, b4 are also initialized using Xavier. The tanh function is used to map the features to the range of [-1, 1], which is conducive to subsequent contrastive learning.

[0150] 4.3 Cross-modal contrastive learning module:

[0151]

[0152] Where sim() is the cosine similarity function and τ is the temperature parameter, which is set to 0.07. k is a negative sample obtained by random sampling in a mini-batch. The purpose of this module is to make the EEG and speech features from the same emotional state closer in the feature space.

[0153] 4.4 Temporal Attention Enhancement Module:

[0154] g t =GRU(h fused ,g t-1 )

[0155] α t =softmax(v T *tanh(W5*[g t ;c]))

[0156] c t =∑(α t *g t )

[0157] GRU is a gated recurrent unit, and the hidden state dimension is set to 128. v and W5 are learnable parameters, and c is the context vector. This module can capture the temporal dependencies of emotional states.

[0158] 4.5 Sentiment Classification:

[0159] y pred =softmax(W6*c t +b6)

[0160] Where W6 and b6 are the parameters of the fully connected layer. pred is the predicted emotion category probability distribution.

[0161] 4.6 Joint optimization objectives:

[0162] L total =L CE +λ*L comtrast

[0163] Where L CE is the cross entropy loss, and λ is the balance parameter, which is set to 0.1.

[0164] In this way, the DAMCLN algorithm can effectively fuse EEG and speech features and learn emotionally consistent cross-modal representations. For example, when Xiao Ming encounters difficult questions in the learning process, his EEG signals may show characteristics of concentration (such as enhanced theta waves), while his speech may become hesitant or pause more. The DAMCLN algorithm can capture these cross-modal consistent expressions and thus more accurately identify his emotional state.

[0165] 5. Emotion recognition step, the emotion recognition module 5 recognizes the user’s emotional state based on the fusion features. Specifically, we use the output y of the DAMCLN algorithm pred To determine the sentiment category and intensity:

[0166] 5.1 Emotion category C = argmax(y pred )

[0167] 5.2 Emotional Intensity K is the number of sentiment categories

[0168] Here, we divide emotions into 5 categories: calm, happy, depressed, anxious, and tired. The range of emotion intensity is [0, 0.8], where 0 represents neutral and 0.8 represents extremely strong emotion. For example, if the system recognizes that Xiao Ming's emotional state is mildly anxious (C = "anxious", I = 0.3), this may mean that he is facing a challenging learning task.

[0169] 6. Parameter adjustment steps, the algorithm steps are as follows:

[0170] Parameter adjustment module 6 uses the "Multi-Objective Reinforcement Learning Emotion-Adaptive Control" (MORLEAC) algorithm to dynamically adjust the headphone output parameters according to the recognized emotional state.

[0171] 6.1 Define the state space S: current emotional state (C, I), headphone parameters, environmental noise, etc.

[0172] 6.2 Define action space A: audio equalizer parameters, volume, voice assistant tone, etc.

[0173] 6.3 Define the reward function R:

[0174] R=w1*user_satisfaction+w2*emotion_improvement-w3

[0175] *energy_consumption

[0176] Among them, w1, w2, w3 are weight parameters, and their initial values ​​are set to 0.4, 0.4, and 0.2 respectively.

[0177] 6.4 Value function approximation:

[0178] Q(s,a)=W T *φ(s,a)

[0179] Where φ(s,a) is the feature vector of the state-action pair, and W is the weight vector. The design of φ(s,a) includes one-hot encoding of the emotional state, normalized values ​​of the action parameters, etc.

[0180] 6.5 Strategy Update:

[0181] π(a|s)=softmax(Q(s,a) / τ)

[0182] Where τ is the exploration temperature parameter, which is initially set to 1 and gradually decreases to 0.1 as training progresses to balance exploration and utilization.

[0183] 6.6 Parameter update:

[0184]

[0185] Where α is the learning rate, which is set to 0.001; γ is the discount factor, which is set to 0.99.

[0186] 6.7 Multi-objective optimization: Use Pareto optimization method to find the optimal balance between user satisfaction, emotion improvement and energy consumption:

[0187]

[0188] Among them J i Denotes the i-th objective function. We use the NSGA-Il algorithm to solve this multi-objective optimization problem, with the population size set to 50 and the evolutionary generation number set to 100.

[0189] 6.8 Adaptive parameter adjustment: According to the output of the MORLEAC algorithm, the following parameters are dynamically adjusted:

[0190] a) Audio equalizer: EQ(f) = ∑(G i *H i (f)),i=1,2,…,N

[0191] Among them G i is the gain of the ith filter, ranging from [-12dB, +12dB]; H i (f) is the filter response function, using a parameterized second-order IIR filter.

[0192] b) Volume: V = Vbase *(1+ΔV),ΔV∈[-0.3,0.3]

[0193] c) Voice assistant tone parameters:

[0194] Speech speed: S = S base *(1+ΔS),ΔS∈[-0.2,0.2]

[0195] Tone: P = P base +ΔP,ΔP∈[-20Hz,20Hz]

[0196] Emotion intensity: E = f(I), where f is a mapping function that maps the emotion intensity l to the emotion parameters of speech synthesis. For example, when the system detects that Xiao Ming is becoming anxious (such as when preparing for a difficult test), the MORLEAC algorithm may take the following actions: adjust the equalizer to slightly enhance the low frequencies (which helps to relax); slightly lower the volume to reduce auditory stimulation; adjust the voice assistant’s tone to make it more peaceful and encouraging. These adjustments are intended to help Xiao Ming stay calm and focused, thereby improving his learning efficiency.

[0197] 7. Feedback acquisition step,The feedback acquisition module 7 is responsible for collecting users’ explicit and implicit feedback, which is crucial for the continuous optimization of the system.

[0198] 7.1 Explicit Feedback Collection:

[0199] Design a simple voice interaction interface and periodically (for example, every 30 minutes) ask users to rate the system performance (1-5 points). Example question: "Xiao Ming, how is your learning experience during this period? Please give a rating of 1 to 5 points."

[0200] 7.2 Implicit Feedback Extraction: Use the Multi-modal User Satisfaction Estimation (MUSE) algorithm to extract implicit feedback.

[0201] The algorithm steps are as follows:

[0202] a) Feature extraction:

[0203] F usage =[Usage duration, interaction frequency, function usage distribution]

[0204] F physio =[heart rate variability, galvanic skin response, body temperature changes]

[0205] F behavior =[head movement mode, voice interaction mode]

[0206] Among them, heart rate variability is obtained by analyzing the ECG artifacts in the EEG signal, and the galvanic skin response and body temperature changes are collected by additional sensors.

[0207] b) Satisfaction estimation model:

[0208] S=σ(W u *F usage +W p *F physio +W b *F behavior +b)

[0209] Where σ is the Sigmoid function, W u ,W p ,W b , b is a learnable parameter. He initialization is used for initialization.

[0210] c) Model training: Use explicit feedback as a supervision signal to minimize the mean square error:

[0211] L MUSE =(SS explicit ) 2

[0212] The Adam optimizer is used for training, the learning rate is set to 0.001, and the batch size is 32.

[0213] Through the MUSE algorithm, the system can estimate Xiao Ming’s satisfaction in real time without frequently interrupting his learning process. For example, if it detects that Xiao Ming’s usage time increases and his heart rate variability increases (indicating reduced stress), the system will infer that his satisfaction may be increasing.

[0214] 8. Model update step: Model update module 8 is responsible for continuously optimizing system performance based on user feedback, mainly including incremental updates to the DAMCLN and MORLEAC algorithms.

[0215] 8.1DAMCLN incremental update:

[0216] Using the sliding window strategy, the most recent N samples (N=1000) are retained to construct a small batch.

[0217] 8.1DAMCLN incremental update:

[0218] Using the sliding window strategy, the most recent N samples (N=1000) are retained to construct a small batch.

[0219] For each mini-batch B:

[0220] 1. Calculate the gradient:

[0221] 2. Update parameters: θ = θ - η * g

[0222] Where θ is the model parameter and η is the learning rate, which is set to 0.0001. The Adam optimizer is used for parameter update.

[0223] 8.2MORLEAC incremental update: adopting experience replay and priority sampling strategy.

[0224] For each interaction (s,a,r,s′):

[0225] 1. Store the experience into the playback buffer D, the buffer size is 10000;

[0226] 2. Calculate TD error: δ = r + γ * max a′ Q(s′,a′)-Q(s,a);

[0227] 3. Update priority: p = |δ| + ∈, where ∈ = 0.01;

[0228] 4. Sample a small batch B (size 64) from D, with a sampling probability proportional to p;

[0229] 5. Update Q network parameters.

[0230] This incremental learning mechanism enables the system to continuously adapt to Xiao Ming's personal characteristics and learning habits, providing more and more personalized services.

[0231] 9. Performance monitoring step: Performance monitoring module 9 uses the "Adaptive Performance Evaluation" (APE) mechanism to monitor system performance.

[0232] 9.1 Define multi-dimensional performance indicators:

[0233] Emotion recognition accuracy: Acc emotion

[0234] User satisfaction: Satisfaction

[0235] System response time: T response

[0236] Battery life: Battery1ife

[0237] 9.2 Calculate the overall performance score:

[0238] Score APE =w1*Acc emotion +w2*Satisfaction-w3*T response +w4*Battery life

[0239] Initial weights w1 = 0.3, w2 = 0.3, w3 = 0.2, w4 = 0.2

[0240] 9.3 Dynamic Weight Adjustment:

[0241] Optimize the weights using gradient descent:

[0242]

[0243] Where λ is the learning rate, which is set to 0.01. This dynamic weight adjustment mechanism can adaptively adjust the importance of each indicator according to the needs of different usage scenarios.

[0244] 9.4 Performance warning mechanism:

[0245] If Score APE Below the preset threshold T APE (Initially set to 0.7), triggering model retraining or manual intervention. Specifically:

[0246] If 0.6≤Score APE <0.7, the system will increase the model update frequency.

[0247] If 0.5≤Score APE <0.6, the system will trigger a comprehensive model retraining.

[0248] If Score APE <0.5, the system will send an alert to the development team and request manual intervention.

[0249] Through this mechanism, the system can promptly detect and resolve performance issues, ensuring continuous high-quality service for Xiao Ming. For example, if the system detects that the accuracy of emotion recognition has dropped, it may be because Xiao Ming's emotional expression pattern has changed (such as increased pressure during exam week). At this time, the system will automatically increase the frequency of model updates to adapt to this change.

[0250] The brain-computer interface headset emotion perception interaction system of the present invention includes the following modules:

[0251] Signal acquisition module 1: This module includes an EEG signal acquisition unit and a speech signal acquisition unit. The EEG signal acquisition unit uses the NeuroSkyTGAM module (model: TGAM1), with a sampling rate of 1000Hz and an acquisition frequency band of 0.5-100Hz. The speech signal acquisition unit uses a MEMS microphone (model: Knowles SPH0645LM4H-B), with a sampling rate of 44.1kHz and a bit depth of 24bit.

[0252] Signal preprocessing module 2: This module performs noise reduction, filtering and other processing on the collected original signals. For EEG signals, notch filters, wavelet transforms and bandpass filters are used for processing. For speech signals, endpoint detection, pre-emphasis filtering, framing and windowing are performed.

[0253] Feature extraction module 3: This module includes an EEG feature extraction unit and a speech feature extraction unit. The EEG feature extraction unit implements the MTFDFE algorithm, and the speech feature extraction unit implements the AAFF algorithm. The detailed steps of these two algorithms have been described in the method description.

[0254] Feature fusion module 4: This module implements the DAMCLN algorithm to fuse EEG features and speech features. The algorithm includes steps such as dynamic weight adaptation, feature mapping and fusion, cross-modal contrastive learning, and temporal attention enhancement.

[0255] Emotion Recognition Module 5: This module recognizes the user’s emotional state based on the fusion features, including the emotion category and intensity. The softmax classifier is used for emotion classification and the emotion intensity is calculated.

[0256] Parameter adjustment module 6: This module implements the MORLEAC algorithm and dynamically adjusts the headphone output parameters according to the recognized emotional state, including the adjustment of audio equalizer parameters, volume, voice assistant tone, etc.

[0257] Feedback acquisition module 7: This module includes an explicit feedback collection unit and an implicit feedback extraction unit. Explicit feedback is collected through the voice interaction interface, and implicit feedback is extracted using the MUSE algorithm.

[0258] Model update module 8: This module is responsible for incremental updates of the DAMCLN and MORLEAC algorithms. Online learning is achieved using a sliding window strategy and experience replay mechanism.

[0259] Performance monitoring module 9: This module implements the APE mechanism, including multi-dimensional performance indicator calculation, comprehensive performance score calculation, dynamic weight adjustment and performance early warning mechanism.

[0260] All these modules are integrated into a compact headset device. The core processing unit uses a low-power, high-performance ARM Cortex-M7 processor (model: STM32H743VI) with a main frequency of up to 480MHz, built-in 1MB Flash and 1MB SRAM, which can meet the real-time processing requirements of complex algorithms. To further improve processing capabilities, the system also integrates a dedicated neural network accelerator chip (such as Google's EdgeTPU) to accelerate the inference process of deep learning models such as DAMCLN.

[0261] In terms of power management, the system uses a high-energy-density lithium polymer battery (capacity: 500mAh), combined with an efficient power management chip (such as Texas Instruments' BQ25120), which can support the device to work continuously for 8-10 hours. In addition, the system also supports wireless charging (using the Qi standard), providing users with a convenient charging experience.

[0262] In terms of software architecture, the system is built on the real-time operating system FreeRTOS to ensure the real-time and reliability of each module. The data storage uses the lightweight database SQLite to store user configurations, historical data, etc. The system also integrates a Bluetooth 5.0 module to support data synchronization and remote control with devices such as smartphones.

[0263] Through this close integration of hardware and software, the system can provide users with a highly personalized, real-time responsive emotional perception interactive experience. For example, when Xiao Ming encounters a difficult question during the review process and becomes frustrated, the system can detect this emotional change in real time and automatically adjust the music rhythm and volume. At the same time, the voice assistant may provide some study suggestions or relaxation techniques in an encouraging tone to help Xiao Ming overcome learning obstacles and maintain a positive learning state.

[0264] In summary, the brain-computer interface headset emotion perception interaction system of the present invention realizes accurate perception and intelligent response to the user's emotional state through advanced technologies such as multimodal signal processing, deep learning and reinforcement learning, providing users with a new and intelligent learning auxiliary tool, which is expected to significantly improve learning efficiency and experience.

[0265] To verify the superiority of the present invention, we conducted a group of comparative experiments, including 1 embodiment and 3 comparative examples.

[0266] Example 1 (the present invention): The complete DAMCLN algorithm is used for feature fusion, the MORLEAC algorithm is used for parameter adjustment, and the MUSE algorithm is applied for user feedback collection.

[0267] Comparative Example 1: Only EEG signals are used for emotion recognition, and a traditional support vector machine (SVM) classifier is adopted.

[0268] Comparative Example 2: Using only speech signals for emotion recognition, a long short-term memory network (LSTM) is adopted.

[0269] Comparative Example 3: Use a simple feature cascade method to fuse EEG and speech features, use random forest for sentiment classification, and use fixed rules for parameter adjustment.

[0270] Test indicators and methods:

[0271] 1. Emotion recognition accuracy: 5-fold cross validation was used and tested on a dataset containing 1,000 samples.

[0272] 2. User satisfaction: 50 volunteers were recruited to study each method for 2 hours and scored through a questionnaire (1-10 points).

[0273] 3. System response time: Measure the average time from receiving the input signal to outputting the adjusted parameters, repeat 100 times and take the average value.

[0274] 4. Battery life: Measures the operating time of a device under continuous use.

[0275] 5. Parameter adjustment effectiveness: The user’s subjective rating of the system parameter adjustment is collected through a questionnaire survey (1-10 points).

[0276] The test results are shown in the following table:

[0277] index Example Comparative Example 1 Comparative Example 2 Comparative Example 3 Emotion recognition accuracy (%) 92.5 78.3 83.7 86.2 User satisfaction (1-10 points) 8.7 6.2 7.1 7.5 System response time (ms) 45 30 35 60 Battery life (hours) 9.5 12 11 18.5 Parameter adjustment effectiveness (1-10 points) 8.9 N / A N / A 6.8

[0278] The results are analyzed and interpreted as follows:

[0279] 1. Emotion recognition accuracy: The accuracy of the present invention is significantly higher than that of other comparative examples. This is mainly due to the multimodal fusion capability of the DAMCLN algorithm, which can make full use of the complementary information of EEG and speech signals. Comparative examples 1 and 2 only use a single modality, and the information is not comprehensive enough. Although comparative example 3 uses two modalities, the simple feature cascade cannot capture the complex relationship between the modalities.

[0280] 2. User satisfaction: The present invention achieved the highest user satisfaction score. This reflects the superiority of the MORLEAC algorithm in parameter adjustment, which can provide a more personalized experience based on the user's emotional state. Comparative Examples 1 and 2 have relatively poor user experience due to the lack of parameter adjustment mechanism.

[0281] 3. System response time: The response time of the present invention is slightly longer than that of Comparative Examples 1 and 2, but still within an acceptable range (less than 50 ms). This is because the present invention needs to process more data and a more complex algorithm. However, compared with Comparative Example 3, the response time of the present invention is shorter, which illustrates the high efficiency of the DAMCLN algorithm.

[0282] 4. Battery life: The battery life of the present invention is slightly shorter than that of Comparative Examples 1 and 2, which is due to the need to process more data and run more complex algorithms. However, compared with Comparative Example 3, the battery life of the present invention is longer, reflecting the optimization effect of the algorithm. The battery life of 9.5 hours is sufficient to meet daily use needs.

[0283] 5. Parameter adjustment effectiveness: The present invention is significantly better than Example 3 in this indicator (Examples 1 and 2 do not have parameter adjustment function). This proves the excellent performance of the MORLEAC algorithm in providing personalized experience and can more accurately adjust system parameters according to the user's emotional state.

[0284] These test results fully demonstrate the superiority of the present invention. The DAMCLN algorithm achieves high-precision emotion recognition, the MORLEAC algorithm provides effective parameter adjustment, and the MUSE algorithm ensures that the system can be continuously optimized. Although there is a slight sacrifice in response time and battery life, the exchange is a significant improvement in recognition accuracy and user satisfaction, which is a more critical indicator for emotion perception interaction systems.

[0285] The advantages of the present invention are mainly reflected in:

[0286] 1. High accuracy brought by multimodal fusion;

[0287] 2. Personalized parameter adjustment improves user experience;

[0288] 3. Balance between real-time performance and energy efficiency.

[0289] These advantages enable the present invention to provide users with a smarter and more personalized learning assistance experience, effectively improving learning efficiency and quality. Future research directions can focus on further optimizing algorithm efficiency to reduce energy consumption while maintaining high performance, thereby further improving battery life.

[0290] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An emotion perception interaction method for a brain-computer interface headset, characterized in that: The following steps are involved: Obtaining the user's EEG signal and voice signal; Preprocessing the EEG signal and the speech signal; Extracting EEG features from the preprocessed EEG signal, and extracting speech features from the preprocessed speech signal; Performing multimodal feature fusion based on the EEG feature and the speech feature to obtain a fusion feature; The method further comprises: identifying the user's emotional state based on the fused features; adjusting the headphone output parameters according to the emotional state; obtaining user feedback; and updating the model based on the user feedback.

2. The method according to claim 1, characterized in that The step of extracting EEG features from the preprocessed EEG signal comprises: The EEG features are extracted using a multi-scale time-frequency dynamic feature extraction algorithm. The algorithm includes: performing a 5-layer wavelet packet decomposition on the preprocessed EEG signal S(t) to obtain 32 sub-band signals {S i (t), i=1, 2, ..., 32}; Perform recursive quantization analysis on each subband signal: RQA i =RQA(S i (t)), i = 1, 2, ..., 32; Construct a multi-scale time-frequency dynamic feature vector: MTFD = [E1, ..., E 32 , RQA1, ..., RQA 32 ]; RQA() is the recursive quantitative analysis function, E i is the energy of the ith subband.

3. The method according to claim 1, characterized in that The step of extracting speech features from the preprocessed speech signal comprises: The speech features are extracted using an adaptive acoustic feature fusion algorithm, which includes: Extracting basic acoustic features: F MFCC 、F pitch , F formuants and F energy ; Calculate feature importance scores: Score i =MI(F i ,Y),i∈{MFCC,pitch,formants,energy}; Adaptive feature fusion: F A AFF=∑(w i *F i ), i∈{MFCC, pitch, formants, energy}; where MI() is the mutual information function, Y is the emotion label, and w i =softmax(Score i ) is the adaptive weight.

4. The method according to claim 1, characterized in that: The step of performing multimodal feature fusion based on EEG features and speech features includes: The dynamic adaptive multimodal contrastive learning network algorithm is used for feature fusion, which includes: Dynamic weight adaptation: W EEG =σ(W1*MTFD+b1)=w speech =σ(W2*F AAFF +b2); Feature Mapping and Fusion: <h2 style=";text-align:left;direction:ltr">h<h2 style=";text-align:left;direction:ltr"> E <h2 style=";text-align:left;direction:ltr"> EG = tanh(W3*MTFD+b3), h speech =tanh(W4*F AAFF +b4), h fused =w EEG *h EEG +w speech *h speech ; Cross-modal contrastive learning: L contrast =-log(exp(sim(h EEG ,h speech ) / τ) / ∑exp(sim(h EEG ,h k ) / τ)); Temporal Attention Enhancement: g t =GRU(h fused ,g t-1 ), α t =softmax(v T *tanh(W5*[g t ;c])), c t =∑(α t *g t ); Sentiment Classification: <h2 style=";text-align:left;direction:ltr">y<h2 style=";text-align:left;direction:ltr"> pred <h2 style=";text-align:left;direction:ltr"> =softmax(W6*c<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> +b6); Among them, σ() is the Sigmoid function, sim() is the cosine similarity function, τ is the temperature parameter, GRU is the gated recurrent unit, W1 to W6 and b1 to b6 are learnable parameters.

5. The method according to claim 1, characterized in that The step of adjusting the headphone output parameters according to the emotional state comprises: The headphone output parameters are adjusted using a multi-objective reinforcement learning emotion adaptation control algorithm, which includes: Define state space S, action space A and reward function R; Value function approximation: Q(s, a) = W T *φ(s,a); Policy update: π(a|s)=softmax(Q(s,a) / τ); Parameter update: Among them, φ(s, a) is the feature vector of the state-action pair, W is the weight vector, α is the learning rate, γ is the discount factor, and τ is the exploration temperature parameter.

6. The method according to claim 5, characterized in that The multi-objective reinforcement learning emotion adaptation control algorithm also includes: Use the Pareto optimization method to find the optimal balance between user satisfaction, emotional improvement and energy consumption: Among them, J i represents the i-th objective function.

7. The method according to claim 1, characterized in that The step of obtaining user feedback comprises: Implicit feedback is extracted using a multimodal user satisfaction estimation algorithm, which includes: Feature extraction: F usage 、F physio and F behavior ; Satisfaction estimation model: S = σ(W u *F usage +W p *F physio +W b *F behavior + b); Model training: Minimize the mean square error L M USE=(SS explicit ) 2 ; where σ is the Sigmoid function, W u , W p , W b and b are learnable parameters, S explicit For explicit feedback.

8. The method according to claim 1, characterized in that The step of updating the model based on user feedback comprises: Incremental updates to a dynamically adaptive multimodal contrastive learning network: Use the sliding window strategy to retain the most recent N samples to build a small batch B; Compute the gradient: Update parameters: θ = θ - η * g; Among them, θ is the model parameter and η is the learning rate.

9. The method according to claim 1, characterized in that: It also includes steps for monitoring model performance: Monitor model performance using an adaptive performance evaluation mechanism that includes: Defining multi-dimensional performance indicators: emotion recognition accuracy Acc emotion , user satisfaction Satisfaction, system response time T response Battery Life life ; Calculate the overall performance score: Score APE =w1*Acc emotion +w2*Satisfaction-w3*T response +w4*Battery life ; Dynamic weight adjustment: Among them, w1, w2, w3 and w4 are weight parameters, and λ is the learning rate.

10. An emotion perception interaction system of a brain-computer interface headset implementing the method described in any one of claims 1 to 9, characterized in that: include: A signal acquisition module is used to obtain the user's EEG signal and voice signal; A signal preprocessing module, used for preprocessing the EEG signal and the speech signal; A feature extraction module, used to extract EEG features from the preprocessed EEG signal and to extract speech features from the preprocessed speech signal; A feature fusion module, used for performing multimodal feature fusion based on the EEG feature and the speech feature to obtain a fusion feature; An emotion recognition module, used for recognizing the user's emotional state based on the fused features; A parameter adjustment module, used to adjust the headphone output parameters according to the emotional state; Feedback acquisition module, used to obtain user feedback; A model updating module, used for updating the model based on the user feedback; as well as Performance monitoring module, used to monitor system performance and trigger model retraining or manual intervention.

Images