Real-time emotion recognition and regulation method and device combined with brain-computer interface

Abstract

The invention discloses a real-time emotion recognition, regulation and control method and device combined with a brain-computer interface. The method comprises the following steps that S1, electroencephalogram signals of a user are collected in real time through an electroencephalogram sensor; s2, preprocessing the original electroencephalogram data; s3, processing the electroencephalogram signals in the S2 by adopting a time-frequency analysis technology, and extracting time-frequency characteristics related to the emotional state; s4, inputting the extracted time-frequency features into a trained long-short-term memory network model, performing time sequence analysis, and identifying the emotional state of the user; s5, according to the emotion state recognized in the step S4, combining historical emotion data and individual features of the user, and dynamically adjusting an emotion regulation strategy; and S6, comprehensively evaluating the emotional state of the user through a multi-modal data fusion technology. By combining the advanced time-frequency analysis technology, the long-short term memory network model and the multi-modal data fusion technology, accurate recognition and dynamic regulation and control of the emotional state of the user are achieved.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface technology, and in particular to a real-time emotion recognition and regulation method and device combined with a brain-computer interface. Background Art

[0002] With the rapid development of brain-computer interface (BCI) technology, research on emotion recognition and regulation using electroencephalogram (EEG) signals has received widespread attention. Traditional emotion recognition methods usually rely on single biological signals (such as EEG signals) for processing and analysis. This method is often not very accurate when dealing with emotion recognition in complex situations, and it is difficult to respond to user emotional changes in real time. In addition, existing emotion regulation methods are usually based on preset fixed strategies and cannot be dynamically adjusted according to the emotional history and real-time status of individual users, resulting in poor regulation effects and poor user experience.

[0003] At present, emotion recognition mainly relies on simple time domain or frequency domain analysis methods, which have the problems of slow response speed and low recognition accuracy when facing changeable emotional states and complex environmental backgrounds. Especially in the field of multimodal bio-signal fusion, existing technologies are usually difficult to effectively integrate bio-signals from different sources (such as heart rate, skin electrical response, etc.) to provide more accurate emotion assessment. In addition, traditional emotion regulation devices often lack personalized adjustment functions and cannot perform adaptive regulation based on the user's emotional history data and real-time emotional state, resulting in low emotion regulation effects and user satisfaction.

[0004] The main problems faced by existing technologies include: (1) The real-time and accuracy of emotion recognition are insufficient, especially in complex situations, it is difficult to effectively identify the user's emotional state; (2) There is a lack of personalized emotion regulation strategies, and it is impossible to dynamically adjust according to the user's historical emotion data and real-time emotional state; (3) Multimodal biosignal fusion technology is not perfect, making it difficult to comprehensively evaluate the user's emotional state. Summary of the invention

[0005] The real-time emotion recognition and regulation method and device based on brain-computer interface proposed in the present invention realizes accurate recognition and dynamic regulation of the user's emotional state by combining advanced time-frequency analysis technology, long short-term memory network (LSTM) model and multimodal data fusion technology.

[0006] According to an embodiment of the present invention, a method and device for real-time emotion recognition and regulation combined with a brain-computer interface includes the following steps:

[0007] S1, collect the user's EEG signals in real time through the EEG sensor to obtain the original EEG data;

[0008] S2, preprocessing the original EEG data to remove artifacts and noise to obtain processed EEG signals;

[0009] S3, using time-frequency analysis technology to process S2 EEG signals and extract time-frequency features related to emotional state;

[0010] S4, inputting the extracted time-frequency features into the trained long short-term memory network model to perform time series analysis and identify the user's emotional state;

[0011] S5, dynamically adjust the emotion regulation strategy based on the emotion state identified by S4 and the user's historical emotion data and individual characteristics;

[0012] S6. Through multimodal data fusion technology, EEG signals are fused with other biological signals including heart rate and skin electrical response to comprehensively evaluate the user's emotional state.

[0013] Optionally, the S1 specifically includes:

[0014] S11, collecting multi-channel EEG signals in real time through a multi-channel EEG sensor array installed on the user's head, where the multi-channel EEG signals are represented by X(t), where t is a time variable and X(t) is the EEG potential at each moment;

[0015] S12, bandpass filter the collected multi-channel EEG signals, set the filter range to 05Hz to 45Hz, remove power frequency interference and low-frequency drift, and the filtered signal is represented by X f (t);

[0016] S13, the EEG signal X after filtering f (t) Gain adjustment is performed to unify the signal amplitude. The adjusted signal is expressed as X a (t);

[0017] S14, the gain-adjusted signal X a (t) Perform artifact removal processing, including electrooculographic artifacts and electromyographic artifacts. The signal after artifact removal is represented by X c (t), that is, the final preprocessed EEG signal is obtained.

[0018] Optionally, the S2 specifically includes:

[0019] S21, the preprocessed EEG signal X c (t) is divided into multiple time windows of fixed length, and the length of each time window is T w , capture the short-term variation characteristics of EEG signals;

[0020] S22, for each EEG signal X in the time window c(t) Perform fast Fourier transform to obtain the spectrum information in the time window. The spectrum is expressed as P(ω), where ω is the frequency component;

[0021] S23, perform feature selection on the spectrum P(ω) to extract the energy features in the characteristic frequency bands related to the emotional state. The energy features are represented by E b , where b is the number of the characteristic frequency band;

[0022] S24, extract the energy feature E b Convert to feature vector F b , and perform standardization to reduce the differences between different signal sources. The standardized feature vector is expressed as F s ;

[0023] S25, the standardized feature vector F s As input data, it is input into the long short-term memory network model to perform time series analysis and identify the user's emotional state.

[0024] Optionally, the S3 specifically includes:

[0025] S31, extract the feature vector F through time-frequency analysis s Input into the pre-trained long short-term memory network model, use the long short-term memory network model to process the time series data, and generate the emotion prediction sequence Y(t), where Y(t) represents the predicted value of the emotion state at time t;

[0026] S32, in the input layer of the long short-term memory network model, gradually process the input feature vector F s , generate the hidden state vector h t , where h t Represents the intermediate emotional state at time t:

[0027] h t =σ(W h ·F s +U h ·h t-1 +b h );

[0028] Among them, h t represents the intermediate emotional state at the current time t, h t-1 represents the intermediate emotional state at the previous moment, W h and U h The input feature vector F s and the previous moment's intermediate emotional state h t-1 Convert to the current intermediate emotional state h t The weight parameter, b hTo adjust the bias of the model output, σ is the activation function to better fit the complex emotional patterns;

[0029] S33, through the output layer of the long short-term memory network model, the implicit state vector h t Converted to the final sentiment prediction value Y(t):

[0030] Y(t)=σ(W y ·h t +b y );

[0031] Where Y(t) represents the predicted value of the emotional state at time t, W y The implicit state vector h t The weight parameter mapped to the sentiment prediction value, b y is the bias for adjusting the output of emotion prediction, and σ is the activation function of the output layer, so that the output emotion prediction value changes within a reasonable range;

[0032] S34, post-processing the output of the long short-term memory network model, including smoothing and normalization, to obtain the final emotional state prediction value Y s (t), Y s (t) represents the predicted value of the emotional state after standardization at time t.

[0033] Optionally, the S4 specifically includes:

[0034] S41, based on the emotional state prediction value Y(t) generated in step S3, obtaining the current emotional state of the user at time t;

[0035] S42, retrieving the user's historical emotional data stored in the system, expressed as H(t), where H(t) is the user's emotional state at the previous time t′<t;

[0036] S43, compare the current emotional state prediction value Y(t) with the historical emotional data H(t) to determine the trend of the current emotional change, expressed as T(t):

[0037] T(t)=Y(t)-H(t);

[0038] Among them, T(t) represents the change in emotional state at time t, reflecting the changing trend of the user's current emotion relative to historical emotions;

[0039] S44, based on the emotion change trend T(t) and the user's individual characteristic data P u Including age, gender, and emotional sensitivity, dynamically adjust the emotion regulation strategy C(t), where C(t) represents the regulation strategy at time t:

[0040] C(t)=f(T(t),P u );

[0041] Among them, function f represents the process of optimizing the control strategy according to the trend of emotion change and user characteristics;

[0042] S45. Emotion regulation strategy C(t) includes adjusting the following parameters: neurofeedback parameter N f Indicates the intensity and frequency of neurofeedback, brain stimulation parameter S b represents the current intensity and stimulation pattern of brain stimulation, sensory feedback parameter S f Indicates the type and intensity of sensory feedback;

[0043] S46. Apply the adjusted emotion regulation strategy C(t) to the user, with the goal of guiding the user's emotional state to the target emotional state G(t), where G(t) represents the expected emotional state of the user at time t:

[0044] G(t)=Y(t)+ΔC(t);

[0045] Among them, ΔC(t) is the change in emotional state after applying the regulation strategy, which adjusts the user's current emotional state Y(t) to the target emotional state G(t).

[0046] Optionally, the S5 specifically includes:

[0047] S51, obtain the current emotional state Y(t) and historical emotional data H(t) identified in step S4, and combine the user's individual characteristic parameter P u , where P u Represents the user's personalized characteristics, including age, gender, emotional sensitivity, and emotional response patterns;

[0048] S52. Calculate the emotion change rate R(t) based on the current emotion state Y(t) and the historical emotion data H(t):

[0049]

[0050] in, represents the instantaneous rate of change of emotional state over time, μH is the mean of historical emotional data, α, β, γ are constant parameters for adjusting the rate of change of emotion, and |H(t)| is the absolute value of historical emotional data;

[0051] S53, according to the emotion change rate R(t) and individual characteristic parameter P u , the emotion regulation strategy C(t) is dynamically adjusted by nonlinear combination, and the adjustment formula is:

[0052]

[0053] Among them, W u is the weight factor set according to the individual characteristics of the user, indicating the degree of influence of different characteristics on the regulation strategy, λ and θ are the slope and threshold of the regulation strategy respectively, η is the fine-tuning parameter, ∈ is a small value to prevent the denominator from being zero, and the overall expression reflects the complex relationship between emotional changes and individual characteristics of users;

[0054] S54. In the emotion regulation strategy C(t), based on the adjusted parameters, the neural feedback N f , Brain Stimulation b , sensory feedback f The control measures are updated nonlinearly:

[0055]

[0056] Among them, σ is the activation function, tanh is the hyperbolic tangent function, ω is the frequency parameter, κ, ζ, δ, v are the parameters for controlling nonlinear changes, and N f , S b , S f are the updated parameters of neurofeedback, brain stimulation, and sensory feedback, respectively;

[0057] S55. Apply the dynamically adjusted emotion regulation strategy C(t) to the user, regulate the user's emotional state in real time, and continuously monitor changes in the emotional state to transform the user's emotional state to the target emotional state G(t).

[0058] Optionally, the S6 specifically includes:

[0059] S61, obtaining a user's brain electrical signal E(t) through an electroencephalogram (EEG) sensor, where E(t) represents brain electrical activity at time t;

[0060] S62, obtaining a heart rate signal HR(t) of the user through a heart rate sensor, where HR(t) represents a change in heart rate over time t, and the heart rate signal is pre-processed and filtered to remove noise interference;

[0061] S63, obtaining a skin conductance signal GSR(t) of the user through a galvanic skin response (GSR) sensor, wherein GSR(t) represents a change in skin conductance at time t, and the skin conductance signal is preprocessed to remove baseline drift;

[0062] S64, perform time synchronization processing on the EEG signal E(t), the heart rate signal HR(t) and the skin conductance signal GSR(t), and the time synchronized signals are represented as E(t), HR(t), and GSR(t). s (t), HR s (t) and GSR s (t);

[0063] S65, the synchronized EEG signal E s (t), heart rate signal HR s (t) and skin conductance signal GSR s (t) Perform feature extraction, and the extracted features are represented as F E (t), F HR (t) and F GSR (t):

[0064]

[0065] Among them, W Ei , W HRi 、w GSRk are the characteristic weight coefficients of EEG signal, heart rate signal and skin electrical signal respectively, and n, m and p are the dimensions of the characteristic vector of each signal respectively;

[0066] S66, extract the multimodal features F E (t), F HR (t) and F GSR (t) Perform weighted fusion through the fusion algorithm to generate a comprehensive emotion assessment feature vector F fusion (t), the fusion formula is:

[0067] F fusion (t) = α·F E (t)+β·F HR (t)+γ·F GSR (t);

[0068] Among them, α, β, and γ are weighted coefficients of multimodal feature fusion, reflecting the relative contribution of each modal signal to emotion assessment;

[0069] S67, the comprehensive emotion evaluation feature vector F fusion (t) Input the emotion recognition model to generate the user’s comprehensive emotional state evaluation result Y fusion (t), where Y fusion (t) represents the emotional state after comprehensive evaluation at time t.

[0070] Optional, real-time emotion recognition and regulation device based on brain-computer interface, including the following modules:

[0071] The EEG signal acquisition module collects the user's EEG signal E(t) in real time through the EEG sensor;

[0072] The bio-signal acquisition module includes a heart rate sensor and a skin galvanic response sensor, which respectively collect the user's heart rate signal HR(t) and skin conductance signal GSR(t);

[0073] The signal preprocessing module preprocesses the EEG signal, heart rate signal and skin conductance signal to remove noise and artifacts and generate the preprocessed signal E s (t), HR s (t) and GSR s (t);

[0074] Feature extraction module, extracts emotion-related feature vectors F from the preprocessed signal E (t), (F HR (t) and F GSR (t), the extraction process refers to step S65 in claim 7;

[0075] The multimodal data fusion module extracts the multimodal features F E (t), F HR (t) and F GSR (t) Perform weighted fusion to generate a comprehensive emotion assessment feature vector F fusion (t), the fusion process refers to step S66 in claim 7;

[0076] The emotion recognition module receives the comprehensive emotion evaluation feature vector F fusion (t) and generate a comprehensive emotional state assessment result Y fusion (t);

[0077] The historical emotion database stores the user's historical emotion data H(t), where H(t) is the user's emotional state before time t;

[0078] The emotion regulation strategy generation module is based on the current emotion state prediction value Y(t), historical emotion data H(t) and individual characteristic parameters P u Dynamically adjust the emotion regulation strategy C(t), the adjustment process refers to step S53 in claim 6, and generates the regulation strategy C(t);

[0079] Emotion regulation execution module, which adjusts the neural feedback N according to the emotion regulation strategy C(t) f , Brain Stimulation b and sensory feedback f , execute emotion regulation measures, and guide the user's emotional state to transform to the target state G(t).

[0080] The beneficial effects of the present invention are:

[0081] The real-time emotion recognition and regulation method and device based on brain-computer interface proposed in this invention realizes accurate recognition and dynamic regulation of the user's emotional state by combining advanced time-frequency analysis technology, long short-term memory network (LSTM) model and multimodal data fusion technology. This method makes full use of the advantages of multiple biological signals such as EEG signals, heart rate, and skin electrical response, and can accurately evaluate the user's emotional state in real time in complex environments, significantly improving the sensitivity and robustness of emotion recognition.

[0082] In addition, the present invention provides a personalized emotion management solution by dynamically adjusting the emotion regulation strategy and combining the user's historical emotion data and individual characteristics. The device can automatically adjust the parameters of neural feedback, brain stimulation and sensory feedback according to real-time emotion changes, ensuring that the emotion regulation process is more in line with the user's individual needs, thereby improving the effect of emotion regulation and user experience.

[0083] The beneficial effect of the present invention is that it not only improves the real-time and accuracy of emotion recognition, but also realizes the personalization and intelligence of emotion regulation. Through the comprehensive evaluation of multimodal biological signals and the application of dynamic regulation strategies, the present invention can provide users with more accurate and efficient emotion management services, overcome the problems of inaccurate recognition and poor regulation effect in traditional emotion recognition and regulation methods, and greatly improve the practicality and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0085] Figure 1 This is a schematic diagram of the structure of the real-time emotion recognition and regulation device based on brain-computer interface proposed by the present invention;

[0086] Figure 2 It is a schematic diagram of multimodal data fusion and emotion recognition in the present invention. DETAILED DESCRIPTION

[0087] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0088] refer to Figure 1 , a real-time emotion recognition and regulation method and device combined with brain-computer interface

[0089] , including the following steps:

[0090] In this implementation, S1 specifically includes:

[0091] S1, collect the user's EEG signals in real time through the EEG sensor to obtain the original EEG data;

[0092] S11, collecting multi-channel EEG signals in real time through a multi-channel EEG sensor array installed on the user's head, where the multi-channel EEG signals are represented by X(t), where t is a time variable and X(t) is the EEG potential at each moment;

[0093] S12, bandpass filter the collected multi-channel EEG signals, set the filter range to 05Hz to 45Hz, remove power frequency interference and low-frequency drift, and the filtered signal is represented by X f (t);

[0094] S13, the EEG signal X after filtering f (t) Gain adjustment is performed to unify the signal amplitude. The adjusted signal is expressed as X a (t);

[0095] S14, the gain-adjusted signal X a (t) Perform artifact removal processing, including electrooculographic artifacts and electromyographic artifacts. The signal after artifact removal is represented by X c (t), that is, the final preprocessed EEG signal is obtained.

[0096] In this implementation, S2 specifically includes:

[0097] S2, preprocessing the original EEG data to remove artifacts and noise to obtain processed EEG signals;

[0098] S21, the preprocessed EEG signal X c (t) is divided into multiple time windows of fixed length, and the length of each time window is T w , capture the short-term variation characteristics of EEG signals;

[0099] S22, for each EEG signal X in the time window c (t) Perform fast Fourier transform to obtain the spectrum information in the time window. The spectrum is expressed as P(ω), where ω is the frequency component;

[0100] S23, perform feature selection on the spectrum P(ω) to extract the energy features in the characteristic frequency bands related to the emotional state. The energy features are represented by E b , where b is the number of the characteristic frequency band;

[0101] S24, extract the energy feature E b Convert to feature vector F b , and perform standardization to reduce the differences between different signal sources. The standardized feature vector is expressed as Fs ;

[0102] S25, the standardized feature vector F s As input data, it is input into the long short-term memory network model to perform time series analysis and identify the user's emotional state.

[0103] In this implementation, S3 specifically includes:

[0104] S3, using time-frequency analysis technology to process S2 EEG signals and extract time-frequency features related to emotional state;

[0105] S31, extract the feature vector F through time-frequency analysis s Input into the pre-trained long short-term memory network model, use the long short-term memory network model to process the time series data, and generate the emotion prediction sequence Y(t), where Y(t) represents the predicted value of the emotion state at time t;

[0106] S32, in the input layer of the long short-term memory network model, gradually process the input feature vector F s , generate the hidden state vector h t , where h t Represents the intermediate emotional state at time t:

[0107] h t =σ(W h ·F s +U h ·h t-1 +b h );

[0108] Among them, h t represents the intermediate emotional state at the current time t, h t-1 represents the intermediate emotional state at the previous moment, W h and U h The input feature vector F s and the previous moment's intermediate emotional state h t-1 Convert to the current intermediate emotional state h t The weight parameter, b h To adjust the bias of the model output, σ is the activation function to better fit the complex emotional patterns;

[0109] S33, through the output layer of the long short-term memory network model, the implicit state vector h t Converted to the final sentiment prediction value Y(t):

[0110] Y(t)=σ(W y ·h t +b y );

[0111] Among them, Y(t) represents the predicted value of the emotional state at time t, and W y is the weight parameter that maps the hidden state vector h t to the predicted value of the emotion, and b y is the bias that adjusts the output of the emotion prediction. σ is the activation function of the output layer, which makes the predicted value of the output emotion change within a reasonable range;

[0112] S34. Post-process the output of the long short-term memory network model, including smoothing and normalization processes, to obtain the final predicted value Y s (t) of the emotional state. Y s (t) represents the predicted value of the emotional state after standardization at time t.

[0113] In this embodiment, S4 specifically includes:

[0114] S4. Input the extracted time-frequency features into the trained long short-term memory network model for time series analysis to identify the user's emotional state;

[0115] S41. Based on the predicted value Y(t) of the emotional state generated in step S3, obtain the current emotional state of the user at time t;

[0116] S42. Retrieve the user's historical emotional data stored in the system, denoted as H(t), where H(t) is the emotional state of the user at the previous time t′ < t;

[0117] S43. Compare the current predicted value Y(t) of the emotional state with the historical emotional data H(t) to determine the current trend of emotional change, denoted as T(t):

[0118] T(t) = Y(t) - H(t);

[0119] Among them, T(t) represents the change amount of the emotional state at time t, reflecting the change trend of the user's current emotion relative to the historical emotion;

[0120] S44. According to the emotional change trend T(t) and the user's individual characteristic data P u including age, gender, and emotional sensitivity, dynamically adjust the emotion regulation strategy C(t), where C(t) represents the regulation strategy at time t:

[0121] C(t) = f(T(t), P u );

[0122] Among them, the function f represents the process of optimizing the regulation strategy according to the emotional change trend and the user characteristics;

[0123] S45. Emotion regulation strategy C(t) includes adjusting the following parameters: neurofeedback parameter N f Indicates the intensity and frequency of neurofeedback, brain stimulation parameter S b represents the current intensity and stimulation pattern of brain stimulation, sensory feedback parameter S f Indicates the type and intensity of sensory feedback;

[0124] S46. Apply the adjusted emotion regulation strategy C(t) to the user, with the goal of guiding the user's emotional state to the target emotional state G(t), where G(t) represents the expected emotional state of the user at time t:

[0125] G(t)=Y(t)+ΔC(t);

[0126] Among them, ΔC(t) is the change in emotional state after applying the regulation strategy, which adjusts the user's current emotional state Y(t) to the target emotional state G(t).

[0127] In this implementation, S5 specifically includes:

[0128] S5, dynamically adjust the emotion regulation strategy based on the emotion state identified by S4 and the user's historical emotion data and individual characteristics;

[0129] S51, obtain the current emotional state Y(t) and historical emotional data H(t) identified in step S4, and combine the user's individual characteristic parameter P u , where P u Represents the user's personalized characteristics, including age, gender, emotional sensitivity, and emotional response patterns;

[0130] S52. Calculate the emotion change rate R(t) based on the current emotion state Y(t) and the historical emotion data H(t):

[0131]

[0132] in, represents the instantaneous rate of change of emotional state over time, μH is the mean of historical emotional data, α, β, γ are constant parameters for adjusting the rate of change of emotion, and |H(t)| is the absolute value of historical emotional data;

[0133] S53, according to the emotion change rate R(t) and individual characteristic parameter P u , the emotion regulation strategy C(t) is dynamically adjusted by nonlinear combination, and the adjustment formula is:

[0134]

[0135] Among them, W uis the weight factor set according to the individual characteristics of the user, indicating the degree of influence of different characteristics on the regulation strategy, λ and θ are the slope and threshold of the regulation strategy respectively, η is the fine-tuning parameter, ∈ is a small value to prevent the denominator from being zero, and the overall expression reflects the complex relationship between emotional changes and individual characteristics of users;

[0136] S54. In the emotion regulation strategy C(t), based on the adjusted parameters, the neural feedback N f , Brain Stimulation b , sensory feedback f The control measures are updated nonlinearly:

[0137]

[0138] Among them, σ is the activation function, tanh is the hyperbolic tangent function, ω is the frequency parameter, κ, ζ, δ, v are the parameters for controlling nonlinear changes, and N f , S b , S f are the updated parameters of neurofeedback, brain stimulation, and sensory feedback, respectively;

[0139] S55. Apply the dynamically adjusted emotion regulation strategy C(t) to the user, regulate the user's emotional state in real time, and continuously monitor changes in the emotional state to transform the user's emotional state to the target emotional state G(t).

[0140] In this implementation, S6 specifically includes:

[0141] S6. Through multimodal data fusion technology, EEG signals are fused with other biological signals including heart rate and skin electrical response to comprehensively evaluate the user's emotional state.

[0142] S61, obtaining a user's brain electrical signal E(t) through an electroencephalogram (EEG) sensor, where E(t) represents brain electrical activity at time t;

[0143] S62, obtaining a heart rate signal HR(t) of the user through a heart rate sensor, where HR(t) represents a change in heart rate over time t, and the heart rate signal is pre-processed and filtered to remove noise interference;

[0144] S63, obtaining a skin conductance signal GSR(t) of the user through a galvanic skin response (GSR) sensor, wherein GSR(t) represents a change in skin conductance at time t, and the skin conductance signal is preprocessed to remove baseline drift;

[0145] S64, perform time synchronization processing on the EEG signal E(t), the heart rate signal HR(t) and the skin conductance signal GSR(t), and the time synchronized signals are represented as E(t), HR(t), and GSR(t). s (t), HRs (t) and GSR s (t);

[0146] S65, the synchronized EEG signal E s (t), heart rate signal HR s (t) and skin conductance signal GSR s (t) Perform feature extraction, and the extracted features are represented as F E (t), F HR (t) and F GSR (t):

[0147]

[0148]

[0149] Among them, W Ei 、w HRj 、w GSRk are the characteristic weight coefficients of EEG signal, heart rate signal and skin electrical signal respectively, and n, m and p are the dimensions of the characteristic vector of each signal respectively;

[0150] S66, extract the multimodal features F E (t), F HR (t) and F GSR (t) Perform weighted fusion through the fusion algorithm to generate a comprehensive emotion assessment feature vector F fusion (t), the fusion formula is:

[0151] F fusion (t) = α·F E (t)+β·F HR (t)+γ·F GSR (t);

[0152] Among them, α, β, and γ are weighted coefficients of multimodal feature fusion, reflecting the relative contribution of each modal signal to emotion assessment;

[0153] S67, the comprehensive emotion evaluation feature vector F fusion (t) Input the emotion recognition model to generate the user’s comprehensive emotional state evaluation result Y fusion (t), where Y fusion (t) represents the emotional state after comprehensive evaluation at time t.

[0154] In this embodiment, the real-time emotion recognition and regulation device based on brain-computer interface includes the following modules:

[0155] The EEG signal acquisition module collects the user's EEG signal E(t) in real time through the EEG sensor;

[0156] The bio-signal acquisition module includes a heart rate sensor and a skin galvanic response sensor, which respectively collect the user's heart rate signal HR(t) and skin conductance signal GSR(t);

[0157] The signal preprocessing module preprocesses the EEG signal, heart rate signal and skin conductance signal to remove noise and artifacts and generate the preprocessed signal E s (t), HR s (t) and GSR s (t);

[0158] Feature extraction module, extracts emotion-related feature vectors F from the preprocessed signal E (t), (F HR (t) and F GSR (t), the extraction process refers to step S65 in claim 7;

[0159] The multimodal data fusion module extracts the multimodal features F E (t), F HR (t) and F GSR (t) Perform weighted fusion to generate a comprehensive emotion assessment feature vector F fusion (t), the fusion process refers to step S66 in claim 7;

[0160] The emotion recognition module receives the comprehensive emotion evaluation feature vector F fusion (t) and generate a comprehensive emotional state assessment result Y fusion (t);

[0161] The historical emotion database stores the user's historical emotion data H(t), where H(t) is the user's emotional state before time t;

[0162] The emotion regulation strategy generation module is based on the current emotion state prediction value Y(t), historical emotion data H(t) and individual characteristic parameters P u Dynamically adjust the emotion regulation strategy C(t), the adjustment process refers to step S53 in claim 6, and generates the regulation strategy C(t);

[0163] Emotion regulation execution module, which adjusts the neural feedback N according to the emotion regulation strategy C(t) f , Brain Stimulation b and sensory feedback f , execute emotion regulation measures, and guide the user's emotional state to transform to the target state G(t).

[0164] Embodiment 1:

[0165] In March 2023, a two-week stress management training experiment was conducted in a psychology laboratory of a university in Beijing. The experiment aims to verify the effectiveness of the real-time emotion recognition and regulation method and device based on brain-computer interface in emotion management under high-pressure environment. The experimental participants included 20 volunteers, aged between 22 and 30, all of whom were healthy adults. There were 10 people in the experimental group and 10 people in the control group. The experimental group used the system of the present invention for emotion management training, while the control group used traditional meditation training methods.

[0166] The experimental scene was set in a simulated virtual reality (VR) environment, and participants were asked to give public speeches in a virtual conference room. Each speech lasted 10 minutes, during which the system collected participants' EEG signals in real time through EEG sensors, while recording heart rate and skin electrical response data. To ensure the accuracy of the data, all signals were preprocessed to remove artifacts and noise. The experiment was conducted from 9 to 11 a.m. every day to avoid interference from physiological fluctuations in the experimental results.

[0167] On the first day of the experiment, participants in the experimental group entered the VR environment and began their first speech training. The system monitored in real time that when the participants began to speak, their heart rate rose from an average of 72 beats per minute at rest to 94 beats per minute, and the skin electrical response value increased from 1.2 microsiemens per second to 3.8 microsiemens. EEG signal analysis showed that the energy of the beta wave was significantly enhanced and the energy of the alpha wave was significantly weakened, indicating that the participants were in a highly tense state. Based on these signal features, the LSTM model identified the participants' emotional state as "high-pressure anxiety" in real time.

[0168] According to the system's emotional regulation strategy, the system first combined the participant's historical data to find that the participant was more sensitive to sensory feedback in similar stressful scenarios, so the system decided to intervene emotionally through music and light adjustment. The system selected soothing classical music and reduced the light intensity in the virtual environment to 50% of the original value. At the same time, the brain stimulation device started working, applying a 1.2 mA microcurrent to the participant's frontal lobe area for 45 seconds.

[0169] After 45 seconds of emotional intervention, the system collected and analyzed the participants' physiological data again. The results showed that the participants' heart rate gradually dropped to 78 beats / minute, the skin galvanic response value decreased to 2.1 micro-Siemens, the alpha wave energy in the EEG signal recovered, and the beta wave energy decreased. The system updated the participant's emotional state to "mild anxiety" and continued to play relaxing music to consolidate the regulation effect. The entire intervention process lasted about 2 minutes, and eventually the participant's emotional state stabilized, showing obvious signs of relaxation.

[0170] On the 7th day of the experiment, the experimental and control groups were evaluated for a one-week mid-term. The average heart rate of the participants in the experimental group dropped from the initial 85 beats per minute to 72 beats, the skin galvanic response value dropped from 3.2 microsiemens per second to 1.8 microsiemens, and the alpha wave energy in the EEG signal increased by about 30%. In contrast, after meditation training, the heart rate of the participants in the control group only dropped from 83 beats per minute to 78 beats, the skin galvanic response value dropped from 3.1 microsiemens per second to 2.6 microsiemens, and the changes in the EEG signal were not significant.

[0171] Participants in the experimental group generally reported that the systematic intervention significantly reduced their anxiety during speeches, that they were able to enter a state of relaxation more quickly, and that their performance during speeches improved. Participants in the control group said that traditional meditation training had limited effects on alleviating short-term anxiety and that it was difficult to quickly adjust emotions under high-pressure situations.

[0172] On the last day of the experiment (Day 14), the system conducted the last emotion management training. When the participants in the experimental group entered the VR environment to give a speech, the system monitored a peak heart rate of 80 beats / minute and a peak skin electrical response value of 2.0 microSiemens, both of which were significantly lower than the values ​​at the beginning of the experiment. In the EEG signal, the average increase in alpha wave energy reached 35%, while the beta wave energy decreased by about 28%. In the end, the average anxiety score of the participants in the experimental group dropped from an initial 7.4 points to 3.9 points (out of 10 points) within two weeks, while the control group only dropped from 7.3 points to 5.8 points.

[0173] In addition to subjective feedback, the objective data of the system also showed significant differences in effect. The average error rate of the experimental group in the virtual speech task dropped from the initial 16% to 4%, while the error rate of the control group dropped from 15% to 9%. These data show that the use of the real-time emotion recognition and regulation system based on the brain-computer interface of the present invention can not only significantly improve the accuracy of emotion recognition, but also effectively help users achieve emotional self-regulation in a high-pressure environment and significantly improve task performance.

[0174] Table 1 Changes in physiological data of the experimental group and the control group

[0175]

[0176]

[0177] Table 2 Anxiety scores and task performance of the experimental group and the control group

[0178]

[0179] Table 3 Effects of applying emotion regulation strategies

[0180]

[0181] With reference to the above table, the feasibility and effectiveness of the present invention in practical applications are verified. Through multimodal data fusion and dynamic control strategy, the system achieves high accuracy of emotion recognition and high efficiency of control, significantly improves the user's emotion management ability, overcomes the shortcomings of the existing technology, and has broad application prospects.

[0182] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method and device for real-time emotion recognition and regulation combined with a brain-computer interface, characterized in that: It includes the following steps: S1. Real-time collect the electroencephalogram (EEG) signals of the user through an EEG sensor to obtain the original EEG data; S2. Preprocess the original EEG data to remove artifacts and noise, and obtain the processed EEG signals; S3. Use time-frequency analysis technology to process the EEG signals in S2, and extract the time-frequency features related to the emotional state; S4. Input the extracted time-frequency features into a trained long short-term memory network model for time series analysis to identify the user's emotional state; S5. According to the emotional state identified in S4, combined with the user's historical emotional data and individual characteristics, dynamically adjust the emotional regulation strategy; S6. Through multi-modal data fusion technology, fuse the EEG signals with other biological signals including heart rate and skin conductance response to comprehensively evaluate the user's emotional state.

2. The method and device for real-time emotion recognition and regulation combined with a brain-computer interface according to claim 1, characterized in that: The specific content of S1 includes: S11. Real-time collect multi-channel EEG signals through a multi-channel EEG sensor array installed on the user's head. The multi-channel EEG signals are represented as X(t), where t is the time variable and X(t) is the EEG potential at each moment; S12, bandpass filter the collected multi-channel EEG signals, set the filter range to 05Hz to 45Hz, remove power frequency interference and low-frequency drift, and the filtered signal is represented by X f (t); S13, the EEG signal X after filtering f (t) Gain adjustment is performed to unify the signal amplitude. The adjusted signal is expressed as X a (t); S14, the gain-adjusted signal X a (t) Perform artifact removal processing, including electrooculographic artifacts and electromyographic artifacts. The signal after artifact removal is represented by X c (t), that is, the final preprocessed EEG signal is obtained.

3. The method and device for real-time emotion recognition and regulation combined with a brain-computer interface according to claim 1, characterized in that: The specific content of S2 includes: S21, the preprocessed EEG signal X c (t) is divided into multiple time windows of fixed length, and the length of each time window is T w , capture the short-term variation characteristics of EEG signals; S22, for each EEG signal X in the time window c (t) Perform fast Fourier transform to obtain the spectrum information in the time window. The spectrum is expressed as P(ω), where ω is the frequency component; S23, perform feature selection on the spectrum P(ω) to extract the energy features in the characteristic frequency bands related to the emotional state. The energy features are represented by E b , where b is the number of the characteristic frequency band; S24, extract the energy feature E b Convert to feature vector F b , and perform standardization to reduce the differences between different signal sources. The standardized feature vector is expressed as F s ; S25, the standardized feature vector F s As input data, it is input into the long short-term memory network model to perform time series analysis and identify the user's emotional state.

4. The method and device for real-time emotion recognition and regulation combined with a brain-computer interface according to claim 1, characterized in that: The specific content of S3 includes: S31, extract the feature vector F through time-frequency analysis s Input into the pre-trained long short-term memory network model, use the long short-term memory network model to process the time series data, and generate the emotion prediction sequence Y(t), where Y(t) represents the predicted value of the emotion state at time t; S32, in the input layer of the long short-term memory network model, gradually process the input feature vector F s , generate the hidden state vector h t , where h t Represents the intermediate emotional state at time t: h t =σ(W h ·F s +U h ·h t-1 +b h ); Among them, h t represents the intermediate emotional state at the current time t, h t-1 represents the intermediate emotional state at the previous moment, W h and U h The input feature vector F s and the previous moment's intermediate emotional state h t-1 Convert to the current intermediate emotional state h t The weight parameter, b h To adjust the bias of the model output, σ is the activation function to better fit the complex emotional patterns; S33, through the output layer of the long short-term memory network model, the implicit state vector h t Converted to the final sentiment prediction value Y(t): Y(t)=σ(W y ·h t +b y ); Where Y(t) represents the predicted value of the emotional state at time t, W y The implicit state vector h t The weight parameter mapped to the sentiment prediction value, b y is the bias for adjusting the output of emotion prediction, and σ is the activation function of the output layer, so that the output emotion prediction value changes within a reasonable range; S34, post-processing the output of the long short-term memory network model, including smoothing and normalization, to obtain the final emotional state prediction value Y s (t), Y s (t) represents the predicted value of the emotional state after standardization at time t.

5. The method and device for real-time emotion recognition and regulation combined with a brain-computer interface according to claim 1, characterized in that: The specific content of S4 includes: S41. Based on the predicted value of the emotional state Y(t) generated in step S3, obtain the user's current emotional state at time t; S42. Retrieve the user's historical emotional data stored in the system, represented as H(t), where H(t) is the emotional state of the user at the previous moment t'<t; S43. Compare the current predicted value of the emotional state Y(t) with the historical emotional data H(t) to determine the trend of the current emotional change, represented as T(t): T(t) = Y(t) - H(t); Among them, T(t) represents the change in the emotional state at time t, reflecting the change trend of the user's current emotion relative to the historical emotion; S44, based on the emotion change trend T(t) and the user's individual characteristic data P u Including age, gender, and emotional sensitivity, dynamically adjust the emotion regulation strategy C(t), where C(t) represents the regulation strategy at time t: C(t)=f(T(t),P u ); Among them, the function f represents the process of optimizing the regulation strategy according to the emotional change trend and user characteristics; S45. Emotion regulation strategy C(t) includes adjusting the following parameters: neurofeedback parameter N f Indicates the intensity and frequency of neurofeedback, brain stimulation parameter S b represents the current intensity and stimulation pattern of brain stimulation, sensory feedback parameter S f Indicates the type and intensity of sensory feedback; S46. Apply the adjusted emotional regulation strategy C(t) to the user, aiming to guide the user's emotional state to the target emotional state G(t), where G(t) represents the expected emotional state of the user at time t: G(t) = Y(t) + ΔC(t); Among them, ΔC(t) is the change in the emotional state after applying the regulation strategy, and adjusts the user's current emotional state Y(t) to the target emotional state G(t).

6. The method and device for real-time emotion recognition and regulation combined with a brain-computer interface according to claim 1, characterized in that: The specific content of S5 includes: S51, obtain the current emotional state Y(t) and historical emotional data H(t) identified in step S4, and combine the user's individual characteristic parameter P u , where P u Represents the user's personalized characteristics, including age, gender, emotional sensitivity, and emotional response patterns; S52. Based on the current emotional state Y(t) and historical emotional data H(t), calculate the emotional change rate R(t): in, represents the instantaneous rate of change of emotional state over time, μH is the mean of historical emotional data, α, β, γ are constant parameters for adjusting the rate of change of emotion, and |H(t)| is the absolute value of historical emotional data; S53, according to the emotion change rate R(t) and individual characteristic parameter P u , the emotion regulation strategy C(t) is dynamically adjusted by nonlinear combination, and the adjustment formula is: Among them, W u is the weight factor set according to the individual characteristics of the user, indicating the degree of influence of different characteristics on the regulation strategy, λ and θ are the slope and threshold of the regulation strategy respectively, η is the fine-tuning parameter, ∈ is a small value to prevent the denominator from being zero, and the overall expression reflects the complex relationship between emotional changes and individual characteristics of users; S54. In the emotion regulation strategy C(t), based on the adjusted parameters, the neural feedback N f , Brain Stimulation b , sensory feedback f The control measures are updated nonlinearly: Among them, σ is the activation function, tanh is the hyperbolic tangent function, ω is the frequency parameter, κ, ζ, δ, v are the parameters for controlling nonlinear changes, and N f , S b , S f are the updated parameters of neurofeedback, brain stimulation, and sensory feedback, respectively; S55. Apply the dynamically adjusted emotional regulation strategy C(t) to the user, real-time regulate the user's emotional state, and continuously monitor the change of the emotional state, so that the user's emotional state changes to the target emotional state G(t).

7. The method and device for real-time emotion recognition and regulation combined with a brain-computer interface according to claim 1, characterized in that: The specific content of S6 includes: S61. Obtain the user's EEG signal E(t) through an electroencephalogram (EEG) sensor, where E(t) represents the EEG activity at time t; S62, obtaining a heart rate signal HR(t) of the user through a heart rate sensor, where HR(t) represents a change in heart rate over time t, and the heart rate signal is pre-processed and filtered to remove noise interference; S63, obtaining a skin conductance signal GSR(t) of the user through a galvanic skin response (GSR) sensor, wherein GSR(t) represents a change in skin conductance at time t, and the skin conductance signal is preprocessed to remove baseline drift; S64, perform time synchronization processing on the EEG signal E(t), the heart rate signal HR(t) and the skin conductance signal GSR(t), and the time synchronized signals are represented as E(t), HR(t), and GSR(t). s (t), HR s (t) and GSR s (t); S65, the synchronized EEG signal E s (t), heart rate signal HR s (t) and skin conductance signal GSR s (t) Perform feature extraction, and the extracted features are represented as F E (t), F HR (t) and F GSR (t): Among them, W Ei , W HRi 、w GSRk are the characteristic weight coefficients of EEG signal, heart rate signal and skin electrical signal respectively, and n, m and p are the dimensions of the characteristic vector of each signal respectively; S66, extract the multimodal features F E (t), F HR (t) and F GSR (t) Perform weighted fusion through the fusion algorithm to generate a comprehensive emotion assessment feature vector F fusion (t), the fusion formula is: F fusion (t)=α·F E (t)+β·F HR (t)+γ·F GSR (t); Among them, α, β, and γ are weighted coefficients of multimodal feature fusion, reflecting the relative contribution of each modal signal to emotion assessment; S67, the comprehensive emotion evaluation feature vector F fusion (t) Input the emotion recognition model to generate the user’s comprehensive emotional state evaluation result Y fusion (t), where Y fusion (t) represents the emotional state after comprehensive evaluation at time t.

8. A real-time emotion recognition and regulation device based on a brain-computer interface, characterized in that: The device includes the following modules: The EEG signal acquisition module collects the user's EEG signal E(t) in real time through the EEG sensor; The bio-signal acquisition module includes a heart rate sensor and a skin galvanic response sensor, which respectively collect the user's heart rate signal HR(t) and skin conductance signal GSR(t); The signal preprocessing module preprocesses the EEG signal, heart rate signal and skin conductance signal to remove noise and artifacts and generate the preprocessed signal E s (t), HR s (t) and GSR s (t); Feature extraction module, extracts emotion-related feature vectors F from the preprocessed signal E (t), (F HR (t) and F GSR (t), the extraction process refers to step S65 in claim 7; The multimodal data fusion module extracts the multimodal features F E (t), F HR (t) and F GSR (t) Perform weighted fusion to generate a comprehensive emotion assessment feature vector F fusion (t), the fusion process refers to step S66 in claim 7; The emotion recognition module receives the comprehensive emotion evaluation feature vector F fusion (t) and generate a comprehensive emotional state assessment result Y fusion (t); The historical emotion database stores the user's historical emotion data H(t), where H(t) is the user's emotional state before time t; The emotion regulation strategy generation module is based on the current emotion state prediction value Y(t), historical emotion data H(t) and individual characteristic parameters P u Dynamically adjust the emotion regulation strategy C(t), the adjustment process refers to step S53 in claim 6 to generate the regulation strategy C(t).

9. Emotional regulation execution module, adjusts the neural feedback N according to the emotional regulation strategy C(t) f , Brain Stimulation b and sensory feedback f , execute emotion regulation measures, and guide the user's emotional state to transform to the target state G(t).

Images