Human-computer interaction real-time emotion recognition and feedback method and system based on brain-computer interface

Abstract

The invention provides a human-computer interaction real-time emotion recognition and feedback method and system based on a brain-computer interface, and the method comprises the steps: S1, obtaining an electroencephalogram signal, carrying out the preprocessing and feature extraction, and obtaining a first emotion feature set; s2, based on the first emotion feature set, generating a first emotion state tag through an individualized emotion classification model; s3, acquiring a corresponding multi-sensory stimulation parameter combination from a preset emotion-atmosphere mapping database according to the first emotion state label, and generating a first stimulation parameter set; s4, performing multi-modal synchronization based on the first stimulation parameter set to generate a multi-modal collaborative stimulation signal; s5, generating a closed-loop control signal according to the multi-mode collaborative stimulation signal, and sending the closed-loop control signal to the feedback adjustment device; s6, extracting real-time change trend data based on the electroencephalogram signals, and dynamically updating the first stimulation parameter set according to the real-time change trend data to generate a second stimulation parameter set. According to the method, personalized man-machine emotion interaction can be realized, and the user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the field of human-computer interaction, and in particular relates to a human-computer interaction real-time emotion recognition and feedback method and system based on a brain-computer interface. Background Art

[0002] Real-time emotion recognition and feedback in brain-computer interface (BCI) human-computer interaction is a cutting-edge field at the intersection of neuroscience and artificial intelligence, crucial for enhancing the intelligence and personalization of human-computer interaction. By directly analyzing a user's EEG signals, it enables real-time perception and dynamic feedback of their emotional state, demonstrating significant value in areas such as emotional healing and intelligent work environments. Current technologies primarily combine EEG signal analysis with machine learning models for emotion recognition and attempt to modulate user emotions through single-modal feedback, such as sound and light. However, existing methods suffer from significant limitations in the accuracy and adaptability of emotion recognition and feedback. First, emotion recognition often relies on generalized models that struggle to adapt to individual EEG differences, resulting in inaccurate capture of deeper subjective emotions (such as implicit emotions and ambivalence). Second, feedback mechanisms are limited to single sensory stimulation outputs, lacking the coordinated optimization of multiple modalities, such as sound, light, and touch. Furthermore, environmental parameter adjustments and emotional states struggle to form a real-time closed-loop linkage, weakening the immersive and dynamic adaptability of creating an emotional atmosphere. However, current traditional methods suffer from poor individual adaptability of emotion recognition, weak multimodal feedback synergy, and insufficient real-time closed-loop environmental control.

[0003] However, the current brain-computer interface human-computer interaction methods have problems such as poor individual adaptability of emotion recognition, weak multimodal feedback coordination, and insufficient real-time closed-loop environmental control. Summary of the Invention

[0004] Based on this, it is necessary to provide a real-time emotion recognition and feedback method and system for human-computer interaction based on brain-computer interface to address the above technical problems. It can adapt to the differences in user EEG characteristics through individualized emotion classification models, improve the accuracy of emotion recognition, and realize real-time matching of multi-sensory feedback such as sound, light, and touch with environmental parameters, thereby enhancing the immersion and adaptability of emotional state regulation and improving user experience.

[0005] In a first aspect, the present application provides a method for real-time emotion recognition and feedback in human-computer interaction based on a brain-computer interface, comprising:

[0006] S1. Obtaining an EEG signal and performing preprocessing and feature extraction to obtain a first emotion feature set. The EEG signal is obtained in real time through an EEG device worn by the user.

[0007] S2. Based on the first emotion feature set, determine the user's current emotion state through an individualized emotion classification model and generate a first emotion state label. The individualized emotion classification model is used to adapt the emotion classification weight according to the user's historical EEG signals;

[0008] S3. According to the first emotional state label, obtaining a corresponding multi-sensory stimulation parameter combination from a preset emotion-atmosphere mapping database to generate a first stimulation parameter set. The emotion-atmosphere mapping database is used to store mapping relationships between emotional states and sound, light, and touch.

[0009] S4. Performing multimodal synchronization based on the first stimulation parameter set to generate a multimodal co-stimulation signal;

[0010] S5. generating a closed-loop control signal according to the multimodal co-stimulation signal and sending the closed-loop control signal to a feedback regulation device;

[0011] S6. Extract real-time change trend data based on the EEG signal, and dynamically update the first stimulation parameter set according to the real-time change trend data to generate a second stimulation parameter set, which is used to iteratively adjust the closed-loop control signal until the user's emotional state matches the preset target.

[0012] In a possible embodiment, S2 includes:

[0013] S21, normalizing the first emotion feature set to obtain a standardized emotion feature vector;

[0014] S22, inputting the standardized emotion feature vector into a pre-trained individualized emotion classification model, and outputting a probability distribution of the user's emotion state;

[0015] S23. If the highest probability value in the probability distribution is greater than a preset confidence threshold, the corresponding emotion category is marked as the first emotion state label; otherwise, a re-collection instruction is generated, and the re-collection instruction is used to instruct to return to execute S1.

[0016] In a possible embodiment, S4 includes:

[0017] S41, generating a control signal according to the first stimulation parameter set, where the control signal includes a sound frequency modulation signal, a light color temperature adjustment signal, and a tactile vibration intensity control signal;

[0018] S42, performing timing synchronization calibration on the control signal to generate a first multi-sensory output sequence;

[0019] S43: Acquire and dynamically adjust the signal strength of the first multi-sensory output sequence based on the user physiological response data to generate a multimodal synergistic stimulation signal.

[0020] In a possible embodiment, after S4, the method further includes:

[0021] S44. Obtaining initial temperature, humidity, and light intensity reference values from a preset environmental parameter template according to the first emotional state tag;

[0022] S45, calculating the target environmental parameter value through a weighted fusion algorithm based on the user's historical preference data and the current environmental sensor data;

[0023] S46 , performing protocol conversion processing on the target environmental parameter value to generate an environmental adjustment parameter set, which is used to generate a multimodal co-stimulation signal.

[0024] In a possible embodiment, S6 includes:

[0025] S61. Calculate the deviation of the real-time change trend data within a preset time window using the following formula:

[0026]

[0027] Where D(t) is the deviation within the time window t, ΔE i,t is the difference between the energy of the i-th frequency band and the target value, is the historical energy standard deviation, is the average energy change rate, λ is the weight coefficient;

[0028] S62: if the deviation exceeds a preset threshold, generating a parameter update instruction, the parameter update instruction being used to instruct generation of a second stimulation parameter set based on the current environmental state data and the multi-sensory output sequence;

[0029] S63. Generate a closed-loop control signal according to the second stimulation parameter set and send it to the feedback adjustment device until the deviation converges to the target range.

[0030] In a possible embodiment, S1 includes:

[0031] S11. Remove the electrooculogram (EOG) noise and myoelectrical (EMG) noise from the EEG signal through independent component analysis to obtain a pure EEG signal.

[0032] S12, performing wavelet transform on the pure EEG signal to extract energy features of multiple frequency bands, including Alpha frequency band, Beta frequency band and Gamma frequency band;

[0033] S13. Dynamically smooth the energy features based on a sliding window mechanism to generate a first emotion feature set.

[0034] In a possible embodiment, performing timing synchronization calibration on the control signal to generate a first multi-sensory output sequence includes:

[0035] Establish a timestamp alignment mechanism for sound, light, and tactile signals to set the start time deviation of multimodal signals within a preset threshold;

[0036] generating an intensity weight matrix according to the first emotional state label, wherein the intensity weight matrix is used to allocate an output ratio of each modal signal;

[0037] The control signals are cross-modally fused based on the intensity weight matrix to generate a first multisensory output sequence.

[0038] Secondly, this application also provides a real-time emotion recognition and feedback system for human-computer interaction based on a brain-computer interface, comprising:

[0039] A data preprocessing module is used to obtain EEG signals and perform preprocessing and feature extraction to obtain a first emotion feature set. The EEG signals are obtained in real time through an EEG device worn by the user;

[0040] An emotion recognition module is configured to determine the user's current emotional state based on the first emotion feature set and generate a first emotional state label using an individualized emotion classification model, wherein the individualized emotion classification model is configured to adapt the emotion classification weight according to the user's historical EEG signals;

[0041] An emotion mapping module is configured to obtain, based on the first emotion state label, a corresponding multi-sensory stimulation parameter combination from a preset emotion-atmosphere mapping database to generate a first stimulation parameter set. The emotion-atmosphere mapping database is configured to store mapping relationships between emotion states and sound, light, and touch.

[0042] an interactive signal generation module, configured to generate a multimodal collaborative stimulation signal by performing multimodal synchronization based on a first stimulation parameter set;

[0043] A signal control module, configured to generate a closed-loop control signal based on the multimodal co-stimulation signal and send the closed-loop control signal to a feedback regulation device;

[0044] The interactive feedback module is used to dynamically update the first stimulation parameter set based on the real-time change trend data extracted from the EEG signal to generate a second stimulation parameter set. The second stimulation parameter set is used to iteratively adjust the closed-loop control signal until the user's emotional state matches the preset target.

[0045] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned human-computer interaction real-time emotion recognition and feedback method based on brain-computer interface is implemented.

[0046] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned human-computer interaction real-time emotion recognition and feedback method based on brain-computer interface is implemented.

[0047] The above-mentioned real-time emotion recognition and feedback method and system for human-computer interaction based on brain-computer interface obtains the EEG signals obtained in real time by the user wearing the EEG device and performs preprocessing and feature extraction to obtain a first emotion feature set; based on the emotion feature set, the user's current emotional state is determined by using an individualized emotion classification model that adapts the emotion classification weight according to the user's historical EEG signals to generate a first emotion state label; according to the first emotion state label, the corresponding multi-sensory stimulation parameter combination is obtained from the emotion-atmosphere mapping database that stores the mapping relationship between the emotional state and sound, light, and touch to generate a first stimulation parameter set, and then multi-modal synchronization is performed based on the first stimulation parameter set to generate a multi-modal collaborative stimulation signal; a closed-loop control signal is generated based on the signal and sent to the feedback adjustment device; real-time change trend data is extracted based on the EEG signal, and the first stimulation parameter set is dynamically updated according to the real-time change trend data to generate a second stimulation parameter set for iteratively adjusting the closed-loop control signal until the user's emotional state matches the preset target. The above technical solution adapts to the differences in user EEG characteristics through an individualized emotion classification model to improve the accuracy of emotion recognition, and dynamically updates stimulation parameters and generates closed-loop control signals based on real-time EEG signal change trend data, forming a real-time matching closed loop between emotional state and environmental parameters, thereby improving the adaptive ability of emotion regulation feedback and the immersive user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A flowchart of a method for real-time emotion recognition and feedback in human-computer interaction based on a brain-computer interface provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart of a confidence threshold setting and closed-loop iteration strategy method in an optional embodiment of the present invention;

[0051] Figure 3 A schematic structural diagram of a real-time emotion recognition and feedback system for human-computer interaction based on a brain-computer interface provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] First, a brief introduction is given to the terms involved in the embodiments of this application.

[0054] A brain-computer interface (BCI) is an advanced technology that establishes direct communication between the brain and external devices. It collects and analyzes electrical signals generated by brain neurons, translating the brain's intentions into commands that can be recognized and executed by external devices, thereby enabling direct interaction between the human brain and machines. This technology not only has enormous potential in medical rehabilitation, such as helping paralyzed patients control prosthetic limbs or wheelchairs, but also demonstrates broad application prospects in virtual reality, intelligent interaction, and military fields, offering new possibilities for the deep integration of humans and machines.

[0055] Electroencephalogram (EEG) is the bioelectric activity generated by the synchronous discharge of clusters of neurons in the cerebral cortex. It is a time-series voltage signal formed by non-invasive electrode collection and amplification. Its frequency domain characteristics (such as the energy distribution of Alpha waves, Beta waves, and Gamma waves) can reflect the user's cognitive states such as attention and emotional arousal, while its time domain characteristics (such as the P300 and N400 components in event-related potentials (ERPs)) are highly correlated with emotional cognition and decision-making. In emotion recognition, EEG signals have millisecond-level time resolution and can capture the instantaneous dynamic characteristics of emotional changes. However, it is necessary to overcome challenges such as individual physiological differences and environmental noise interference. Through signal preprocessing, feature fusion and machine learning, high-precision emotion modeling can be achieved, providing a core data foundation for real-time brain-computer interaction.

[0056] Multimodal refers to a technical system that integrates multiple perception or interaction channels (such as sound, vision, touch, physiological signals, etc.) to achieve collaborative information processing and feedback. In the field of brain-computer interface and affective computing, multimodal technology integrates EEG signals with other physiological data (such as heart rate, skin conductance) or environmental parameters (such as light, temperature) to build a multi-dimensional feature fusion model to improve the robustness and accuracy of emotional state recognition; at the same time, it combines the coordinated output of sensory stimuli such as sound, light, and touch to achieve an immersive feedback experience that matches the user's emotional state. The core challenge lies in the efficient fusion and spatiotemporal synchronization of cross-modal data, but compared with a single modality, multimodal technology can break through the limitations of information dimensions, more comprehensively capture user intentions and emotional dynamics, and provide a more natural solution for intelligent human-computer interaction.

[0057] Based on the above explanation of terms, the implementation environment of the real-time emotion recognition and feedback method for human-computer interaction based on brain-computer interface provided in the embodiment of the present application is explained. Schematically, the implementation environment includes: a sensor group, a feedback adjustment device, a terminal, and a processor. The terminal is connected to the sensor group and processor signals via a network; the sensor group includes but is not limited to EEG devices, physiological sensors, environmental sensors, tactile sensors, etc.; the feedback adjustment device can be a temperature and humidity controller, smart furniture, display device, or projection device; and the processor can be a central processing unit, a neural network processor, or a multi-core processor, which is not limited here.

[0058] In combination with the above explanations of terms and implementation environments, the application scenarios of the embodiments of this application are explained. The real-time emotion recognition and feedback method for human-computer interaction based on brain-computer interface provided in the embodiments of this application can be applied to, but not limited to, the following scenarios:

[0059] In psychological counseling or depression treatment scenarios, users wear EEG devices for emotional monitoring. The above-mentioned technical methods can be used to identify negative emotional states such as anxiety and depression in real time, and automatically trigger multi-sensory collaborative intervention: adjust the lighting in the treatment room to soft warm tones, play soothing natural sounds, and provide gentle massage through the built-in tactile vibration device in the seat. The intervention strategy can be adjusted according to real-time emotional data, forming a closed loop of "emotion recognition-environmental adjustment-effect evaluation", helping patients relieve emotional stress and improve treatment efficiency.

[0060] In classrooms or online learning scenarios, students wearing a lightweight EEG headband can monitor their attention levels in real time. When distraction is detected (e.g., a decrease in beta wave power), the system automatically adjusts the classroom lighting to focus mode (cool white light), issues a gentle vibration reminder via the desktop haptic feedback device, and simultaneously pushes a short meditation audio message to the student's device to help them quickly regain focus. Teachers can optimize the teaching rhythm based on group emotional data to provide personalized learning support.

[0061] In high-pressure work environments, employees wearing EEG devices and smart wristbands can use the aforementioned technology to analyze their stress levels and mood swings in real time. When stress thresholds are detected, the system automatically adjusts the office environment: raising the air conditioning temperature to a comfortable level, switching workstation lighting to eye-protection mode, and playing white noise through directional speakers to block out external distractions. Furthermore, smart chairs provide a wave-like back massage based on a set of tactile parameters, helping employees quickly relax and improve their work efficiency and mental well-being.

[0062] Illustratively, the real-time emotion recognition and feedback method for human-computer interaction based on brain-computer interface provided in the embodiment of the present application can also be applied to other application scenarios. It is only used as an example here and is not limited to the specific application scenario.

[0063] In one embodiment, Figure 1 As shown, a method for real-time emotion recognition and feedback in human-computer interaction based on a brain-computer interface is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The method includes:

[0064] S1. Obtain an EEG signal and perform preprocessing and feature extraction to obtain a first emotion feature set. The EEG signal is obtained in real time through an EEG device worn by the user.

[0065] Specifically, the EEG signal can be acquired in real time through a high-precision EEG acquisition device worn by the user, such as a dry electrode EEG cap, and its power frequency interference and myoelectric noise can be removed through a filtering algorithm. For example, feature extraction methods such as wavelet transform can be used to extract emotion-related features from the EEG signal to form a first emotion feature set.

[0066] S2. Based on the first emotion feature set, determine the user's current emotion state through an individualized emotion classification model and generate a first emotion state label. The individualized emotion classification model is used to adapt the emotion classification weight according to the user's historical EEG signals.

[0067] Specifically, based on the first emotional feature set, a pre-trained individualized emotion classification model is called. This model can initialize the general emotion classification network through the transfer learning framework and use the user's historical EEG data for incremental learning to optimize the classification weights. The standardized emotional feature vector is input into the model, and the probability distribution of the user's emotional state is output. If the highest probability value exceeds the preset confidence threshold (such as 0.8), the corresponding emotional category is marked as the first emotional state label (such as "joyful" or "anxious"). This step significantly improves the accuracy of emotional state discrimination by dynamically adapting to individual EEG feature differences.

[0068] S3. According to the first emotional state label, obtain the corresponding multi-sensory stimulation parameter combination from a preset emotion-atmosphere mapping database to generate a first stimulation parameter set. The emotion-atmosphere mapping database is used to store the mapping relationship between emotional state and sound, light, and touch.

[0069] Specifically, based on the first emotional state tag, a preset emotion-atmosphere mapping database is accessed. This database stores mappings between emotional states and multi-sensory stimulation parameters (e.g., "joy" corresponds to high-frequency sound waves, warm-colored light, and low-frequency tactile vibrations). The database is queried to obtain the sound frequency range, light color temperature, and tactile intensity parameters that match the current emotional state, generating a first stimulation parameter set. This mapping can be constructed based on psychological experimental data to ensure that the stimulation parameters are consistent with the physiological responses of the emotional state.

[0070] S4. Perform multimodal synchronization based on the first stimulation parameter set to generate a multimodal co-stimulation signal.

[0071] Specifically, based on the first stimulation parameter set, a collaborative optimization algorithm is used to synchronously calibrate the output timing of sound, light, and tactile signals (for example, controlling the start time deviation of multimodal signals to ≤10ms). The output ratio of each modal signal is adjusted based on a preset intensity weight matrix to generate a multimodal collaborative stimulation signal. This collaborative optimization algorithm ensures the spatiotemporal consistency of multisensory stimulation, avoids sensory conflict, and enhances the immersive experience.

[0072] S5. Generate a closed-loop control signal according to the multimodal co-stimulation signal and send it to a feedback regulation device.

[0073] Specifically, the multimodal collaborative stimulation signal is encapsulated as a closed-loop control instruction (such as a JSON format data packet) and sent to the feedback regulation device through a network protocol (such as MQTT). The closed-loop control signal drives the device to output the corresponding multi-sensory stimulation, ensuring the real-time and accuracy of the stimulation signal, thereby achieving effective feedback on the emotional state.

[0074] S6. Extract real-time change trend data based on the EEG signal, and dynamically update the first stimulation parameter set according to the real-time change trend data to generate a second stimulation parameter set, which is used to iteratively adjust the closed-loop control signal until the user's emotional state matches the preset target.

[0075] Specifically, by continuously monitoring the user's EEG signals, real-time trend data is extracted. Based on this data, the first stimulation parameter set is dynamically updated to generate a second stimulation parameter set. By iteratively adjusting the closed-loop control signal, this method optimizes the stimulation parameters in real time until the user's emotional state reaches a preset target, such as adjusting from "anxiety" to "calmness." This adaptive emotional regulation is achieved through continuous iterative optimization.

[0076] The above-mentioned real-time emotion recognition and feedback method and system for human-computer interaction based on brain-computer interface obtains the EEG signals obtained in real time by the user wearing the EEG device and performs preprocessing and feature extraction to obtain a first emotion feature set; based on the emotion feature set, the user's current emotional state is determined by using an individualized emotion classification model that adapts the emotion classification weight according to the user's historical EEG signals to generate a first emotion state label; according to the first emotion state label, the corresponding multi-sensory stimulation parameter combination is obtained from the emotion-atmosphere mapping database that stores the mapping relationship between the emotional state and sound, light, and touch to generate a first stimulation parameter set, and then multi-modal synchronization is performed based on the first stimulation parameter set to generate a multi-modal collaborative stimulation signal; a closed-loop control signal is generated based on the signal and sent to the feedback adjustment device; real-time change trend data is extracted based on the EEG signal, and the first stimulation parameter set is dynamically updated according to the real-time change trend data to generate a second stimulation parameter set for iteratively adjusting the closed-loop control signal until the user's emotional state matches the preset target. The above technical solution adapts to the differences in user EEG characteristics through an individualized emotion classification model to improve the accuracy of emotion recognition, and dynamically updates stimulation parameters and generates closed-loop control signals based on real-time EEG signal change trend data, forming a real-time matching closed loop between emotional state and environmental parameters, thereby improving the adaptive ability of emotion regulation feedback and the immersive user experience.

[0077] like Figure 2 As shown, in a possible embodiment, S2 may include:

[0078] S21. Normalize the first emotion feature set to obtain a standardized emotion feature vector.

[0079] Specifically, the first emotion feature set is subjected to maximum-minimum normalization processing, and EEG features of different dimensions (such as Alpha wave energy and event-related potential amplitude) are linearly mapped to the [0,1] interval to generate a standardized emotion feature vector. The normalization processing eliminates the interference of feature dimension differences on the classification model, thereby improving the model convergence speed and classification stability.

[0080] S22. Input the standardized emotion feature vector into the pre-trained individualized emotion classification model and output the probability distribution of the user's emotion state.

[0081] Specifically, the standardized emotional feature vector is input into a pre-trained individualized emotion classification model. This model is based on a hybrid architecture of convolutional neural networks (CNN) and long short-term memory networks (LSTM). It extracts spatiotemporal features and outputs the probability distribution of the user belonging to each preset emotion category (such as happiness, anxiety, and calmness). The probability distribution represents the credibility quantification result of the user's current emotional state, providing a basis for subsequent decision-making.

[0082] S23. If the highest probability value in the probability distribution is greater than a preset confidence threshold, the corresponding emotion category is marked as the first emotion state label; otherwise, a re-collection instruction is generated, and the re-collection instruction is used to instruct to return to execute S1.

[0083] Specifically, a confidence threshold of 0.7 can be set. If the highest probability value in the probability distribution exceeds this threshold, the corresponding emotion category is marked as the first emotion state label. Otherwise, a re-collection instruction is generated, which triggers the EEG device to re-collect EEG signals and return to step S1. This step filters low-confidence classification results using the confidence threshold to avoid interference with feedback control caused by misjudgments, while also improving the robustness of emotion recognition through closed-loop iteration.

[0084] In a possible embodiment, S4 includes:

[0085] S41 . Generate a control signal according to the first stimulation parameter set, where the control signal includes a sound frequency modulation signal, a light color temperature adjustment signal, and a tactile vibration intensity control signal.

[0086] Specifically, a control signal is generated based on the first stimulation parameter set. The control signal includes a sound frequency modulation signal, a light color temperature adjustment signal, and a tactile vibration intensity control signal. The control signals correspond to input instructions from the sound device, light device, and tactile feedback device, respectively, and are used to drive these devices to output corresponding multi-sensory stimulation. For example, the sound frequency modulation signal is used to control a speaker to play music of a specific frequency, the light color temperature adjustment signal is used to adjust the color temperature of the smart lamp, and the tactile vibration intensity control signal is used to control the vibration intensity of the vibration motor. This achieves the physical signal conversion of multi-sensory stimulation parameters and ensures the physical accuracy of multi-sensory feedback.

[0087] S42: Perform timing synchronization calibration on the control signal to generate a first multi-sensory output sequence.

[0088] Specifically, the generated control signals are calibrated for timing synchronization. Through precise time synchronization technology, the sound, light, and tactile stimuli are output in coordination at the same time point or in a preset time sequence, generating a first multi-sensory output sequence, allowing users to experience a highly consistent multi-sensory experience. For example, when the system needs to output soft music, warm light, and gentle tactile vibrations at the same time, timing synchronization calibration ensures that these stimulus signals are precisely synchronized, eliminating sensory conflicts caused by cross-modal signal transmission delays and enhancing the immersive feeling of multi-sensory collaborative feedback.

[0089] S43: Acquire and dynamically adjust the signal strength of the first multi-sensory output sequence based on the user physiological response data to generate a multimodal synergistic stimulation signal.

[0090] Specifically, physiological response data of the user (such as heart rate variability and skin conductance level) are collected in real time through physiological sensors (such as heart rate monitors and skin conductance sensors). The deviation between the physiological data and the preset target response model is analyzed based on a dynamic weight adjustment algorithm (such as fuzzy PID control). The intensity ratio of each modal signal in the first multi-sensory output sequence is dynamically adjusted (such as enhancing the tactile vibration intensity and reducing the light brightness) to generate an optimized multimodal collaborative stimulation signal. The above method realizes adaptive adjustment of feedback intensity to ensure that the stimulation parameters are accurately matched with the user's real-time physiological state.

[0091] In a possible embodiment, after S4, the method further includes:

[0092] S44. Obtain initial temperature, humidity, and light intensity reference values from a preset environmental parameter template according to the first emotional state label.

[0093] Specifically, the preset environmental parameter template is queried according to the first emotional state label (such as "relaxation" or "anxiety"), which stores the mapping relationship between the emotional state and temperature, humidity, and light intensity; the initial temperature reference value, humidity reference value, and light intensity reference value that match the current emotional state are called through the database interface to generate an initial environmental parameter set. Through the initial parameter matching driven by the emotional state, the precise correspondence between the environmental adjustment and the user's emotional needs is ensured, thereby improving physiological comfort.

[0094] S45. Calculate the target environmental parameter value through a weighted fusion algorithm based on the user's historical preference data and the current environmental sensor data.

[0095] Specifically, it can retrieve historical user preference data from local storage (such as user-set temperature preference ranges and light color temperature preferences) and collect real-time environmental sensor data (such as current room temperature, hygrometer readings, and light sensor data). Using a weighted fusion algorithm, it calculates target temperature, humidity, and light intensity values, generating dynamically corrected target environmental parameter values. This algorithm balances user personalized needs with the real-time state of the environment, improving the rationality and comfort of parameter adjustments.

[0096] S46 , performing protocol conversion processing on the target environmental parameter value to generate an environmental adjustment parameter set, which is used to generate a multimodal co-stimulation signal.

[0097] Specifically, the target environmental parameter values are converted to a protocol to generate an environmental adjustment parameter set, achieving efficient integration of environmental adjustment devices and multimodal co-stimulation signals. This collaborative control mechanism ensures the synchronous output of environmental adjustment and multi-sensory stimulation, providing users with a consistent and immersive experience and further enhancing the effectiveness of emotional regulation.

[0098] In a possible embodiment, S6 may include:

[0099] S61. Calculate the deviation of the real-time trend data within the preset time window using the following formula:

[0100]

[0101] Where D(t) is the deviation within the time window t, ΔE i,t is the difference between the energy of the i-th frequency band and the target value, is the historical energy standard deviation, is the average energy change rate, λ is the weight coefficient;

[0102] Specifically, by extracting the dynamic deviation between the EEG signal frequency band energy (such as the mean Alpha wave energy) and the target emotional state in the real-time change trend data within a preset time window (such as 60 seconds), the dynamic fluctuation of the emotional state is quantified, providing a basis for parameter updating.

[0103] S62: If the deviation exceeds a preset threshold, a parameter update instruction is generated, where the parameter update instruction is used to instruct generation of a second stimulation parameter set based on the current environmental state data and the multi-sensory output sequence.

[0104] Specifically, when the deviation exceeds a preset threshold, a parameter update instruction is generated, generating a second set of stimulation parameters based on the current environmental state data and the multi-sensory output sequence. This ensures that the stimulation parameter update comprehensively considers the user's real-time emotional state and environmental conditions, thereby achieving precise emotional regulation. By dynamically adjusting the stimulation parameters, this calculation method can more effectively guide the user's emotional state towards the target state.

[0105] S63. Generate a closed-loop control signal according to the second stimulation parameter set and send it to the feedback adjustment device until the deviation converges to the target range.

[0106] Specifically, a closed-loop control signal is generated based on the second stimulation parameter set and sent to the feedback adjustment device until the deviation characteristics converge to the target range. The stability and effectiveness of emotional state regulation are ensured through the closed-loop control mechanism. By continuously monitoring the deviation characteristics and dynamically adjusting the stimulation parameters, accurate matching of emotional states is achieved, thereby improving user experience.

[0107] In a possible embodiment, S1 may include:

[0108] S11. Remove the electrooculogram (EOG) noise and myoelectrical (EMG) noise from the EEG signal through independent component analysis to obtain a pure EEG signal.

[0109] Specifically, the original EEG signal is blindly separated through the independent component analysis algorithm, and the independent components corresponding to electro-oculogram (EOG) and electromyography (EMG) noise are identified and removed, the neural electrical signals related to cerebral cortical activity are retained, and a pure EEG signal is generated. Furthermore, the independent component analysis algorithm is implemented using FastICA, which separates the noise source by maximizing the non-Gaussianity of the signal, effectively eliminating the interference components introduced by blinking and facial muscle movements, and improving the signal quality.

[0110] S12. Perform wavelet transform on the pure EEG signal to extract energy features of multiple frequency bands, including Alpha frequency band, Beta frequency band and Gamma frequency band.

[0111] Specifically, the pure EEG signal is subjected to a wavelet transform to extract energy features in multiple frequency bands. Wavelet transform is a method that can analyze signals simultaneously in the time and frequency domains. It is suitable for processing non-stationary signals and extracts energy features including the Alpha band (8-13Hz), Beta band (13-30Hz), and Gamma band (30-100Hz). The energy features of these frequency bands are closely related to a person's emotional state. For example, the energy of the Alpha band is generally associated with a state of relaxation, while the energy of the Beta band is associated with a state of alertness. Time-frequency energy analysis based on wavelet transform comprehensively captures the emotional association features of the Alpha, Beta, and Gamma bands, enhancing the information richness of emotional state discrimination.

[0112] S13. Dynamically smooth the energy features based on a sliding window mechanism to generate a first emotion feature set.

[0113] Specifically, the extracted energy features are dynamically smoothed using a sliding window mechanism. This mechanism smoothes the data within a fixed-length window by sliding it across the time series, thereby reducing the impact of noise and preserving the signal's key features. After smoothing, a first set of emotion features is generated, providing a stable and reliable feature input for subsequent emotion recognition.

[0114] In a possible embodiment, performing timing synchronization calibration on the control signal to generate the first multi-sensory output sequence may include:

[0115] A timestamp alignment mechanism for sound, light, and tactile signals is established to set the starting time deviation of multimodal signals within a preset threshold.

[0116] Specifically, a hardware clock synchronization module (such as GPS-PPS or IEEE 1588) can be used to assign unified timestamps to sound, light, and tactile signals. This, combined with the Network Time Protocol (NTP), can be used to calibrate the local clocks of each modal device, keeping the start time deviation of multimodal signals within 10ms. This timestamp alignment mechanism dynamically adjusts signal trigger timing by monitoring signal transmission delays (such as audio buffering delay and light response delay) in real time, generating a time-synchronized multi-sensory output sequence to avoid sensory conflicts, eliminate the sense of fragmentation in the sensory experience, and enhance immersion.

[0117] An intensity weight matrix is generated according to the first emotional state label, and the intensity weight matrix is used to allocate the output ratio of each modal signal.

[0118] Specifically, according to the first emotional state label (such as "anxiety" and "joy"), the preset emotion-intensity mapping table is queried. The mapping table defines the intensity ratios of sound, light, and touch under different emotional states (such as "anxiety" corresponds to a tactile intensity weight of 0.7, a sound weight of 0.2, and a light weight of 0.1); the weight ratios are converted into an intensity weight matrix (such as a diagonal matrix form) through a matrix generation algorithm. The matrix is used to dynamically allocate the output intensity of each modal signal. The weight matrix is constructed based on psychological experimental data to ensure that the stimulus intensity matches the physiological response law of the emotional state.

[0119] The control signals are cross-modally fused based on the intensity weight matrix to generate a first multisensory output sequence.

[0120] Specifically, the control signals are cross-modally fused based on an intensity weight matrix to generate a first multi-sensory output sequence, achieving the coordinated output of sound, light, and tactile signals. This cross-modal fusion mechanism provides users with a consistent and immersive multi-sensory experience, enhancing the immersiveness and adaptability of emotional regulation, and further improving the user experience.

[0121] In summary, the real-time emotion recognition and feedback method for human-computer interaction based on brain-computer interface provided in the embodiment of the present application collects and preprocesses the user's EEG signals in real time, removes physiological noise interference based on independent component analysis, and extracts multi-band energy features in combination with wavelet transform to generate a first emotion feature set that represents the emotional state; dynamically analyzes the feature set based on an individualized emotion classification model, adapts the classification weights using the user's historical EEG data, and outputs accurate emotion state labels; matches multi-sensory stimulation parameters from a preset emotion-atmosphere mapping database according to the emotion labels, generates sound, light, and tactile collaborative control signals, and optimizes the multimodal output sequence through time synchronization calibration and intensity weight matrix to ensure the spatiotemporal consistency of cross-sensory feedback; at the same time, dynamically corrects temperature, humidity, and light parameters in combination with environmental parameter templates and real-time sensor data to generate environmental adjustment instructions; drives the multimodal device to execute feedback through closed-loop control signals, and iteratively updates the stimulation parameter set based on EEG change trend data to achieve continuous adaptation of the emotional state to the external environment. The above technical solution can dynamically adapt to the differences in users' EEG characteristics through individualized emotion classification models, solve the defect of traditional methods' insufficient adaptability to individual physiological differences, and significantly improve the accuracy of emotion recognition; combine multimodal collaborative signal generation technology with the dynamic optimization mechanism of environmental parameters to achieve real-time matching of sensory feedback such as sound, light, and touch with environmental parameters, breaking through the limitations of single modality feedback; based on the closed-loop iterative adjustment strategy of EEG change trends, form a complete closed loop of "perception-analysis-feedback-optimization", enhance the adaptive ability and immersive experience of emotional state regulation, and ultimately achieve precise and personalized human-computer emotional interaction.

[0122] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0123] Based on the same inventive concept, the embodiment of the present application also provides a brain-computer interface-based human-computer interaction real-time emotion recognition and feedback system for realizing the above-mentioned brain-computer interface-based human-computer interaction real-time emotion recognition and feedback method. The implementation solution provided by this system is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations in one or more embodiments of the brain-computer interface-based human-computer interaction real-time emotion recognition and feedback system provided below can be found in the above-mentioned limitations on the brain-computer interface-based human-computer interaction real-time emotion recognition and feedback method, and will not be repeated here.

[0124] In an exemplary embodiment, Figure 3 As shown, a human-computer interaction real-time emotion recognition and feedback system 10 based on a brain-computer interface is provided, comprising:

[0125] A data preprocessing module 11 is used to obtain EEG signals and perform preprocessing and feature extraction to obtain a first emotion feature set. The EEG signals are obtained in real time through an EEG device worn by the user;

[0126] An emotion recognition module 12 is configured to determine the user's current emotional state based on the first emotion feature set and generate a first emotional state label using an individualized emotion classification model, wherein the individualized emotion classification model is configured to adapt an emotion classification weight based on the user's historical EEG signals;

[0127] An emotion mapping module 13 is configured to obtain a corresponding multi-sensory stimulation parameter combination from a preset emotion-atmosphere mapping database based on the first emotion state label to generate a first stimulation parameter set. The emotion-atmosphere mapping database is configured to store mapping relationships between emotion states and sound, light, and touch.

[0128] An interactive signal generating module 14 is configured to generate a multimodal collaborative stimulation signal by performing multimodal synchronization based on the first stimulation parameter set;

[0129] A signal control module 15 is configured to generate a closed-loop control signal based on the multimodal co-stimulation signal and send the closed-loop control signal to the feedback regulation device;

[0130] The interactive feedback module 16 is used to dynamically update the first stimulation parameter set based on the real-time change trend data extracted from the EEG signal, and generate a second stimulation parameter set. The second stimulation parameter set is used to iteratively adjust the closed-loop control signal until the user's emotional state matches the preset target.

[0131] In a possible embodiment, the emotion recognition module 12 may include:

[0132] The normalization processing unit is used to perform normalization processing on the first emotion feature set to obtain a standardized emotion feature vector.

[0133] The classification model unit is used to input the standardized emotion feature vector into the pre-trained individualized emotion classification model and output the probability distribution of the user's emotion state.

[0134] The label generation unit is used to mark the corresponding emotion category as the first emotion state label if the highest probability value in the probability distribution is greater than a preset confidence threshold; otherwise, a re-collection instruction is generated, and the re-collection instruction is used to instruct the call of the data preprocessing module 11.

[0135] In a possible embodiment, the interaction signal generating module 14 includes:

[0136] The control signal generating unit is used to generate a control signal according to the first stimulation parameter set, wherein the control signal includes a sound frequency modulation signal, a light color temperature adjustment signal and a tactile vibration intensity control signal.

[0137] The timing synchronization unit is used to perform timing synchronization calibration on the control signal to generate a first multi-sensory output sequence.

[0138] The dynamic adjustment unit is used to obtain and dynamically adjust the signal strength of the first multi-sensory output sequence based on the user's physiological response data to generate a multimodal co-stimulation signal.

[0139] In a possible embodiment, the interaction signal generating module 14 may include:

[0140] The environmental parameter acquisition unit is used to obtain initial temperature, humidity and light intensity reference values from a preset environmental parameter template according to the first emotional state tag.

[0141] The dynamic optimization unit is used to calculate the target environmental parameter value through a weighted fusion algorithm based on the user's historical preference data and current environmental sensor data.

[0142] The protocol conversion unit is used to perform protocol conversion processing on the target environmental parameter value to generate an environmental adjustment parameter set, which is used to generate a multimodal co-stimulation signal.

[0143] In a possible embodiment, the interactive feedback module 16 includes:

[0144] The deviation calculation unit is used to calculate the deviation of the real-time change trend data within a preset time window using the following formula:

[0145]

[0146] Where D(t) is the deviation within the time window t, ΔE i,t is the difference between the energy of the i-th frequency band and the target value, is the historical energy standard deviation, is the average energy change rate, and λ is the weight coefficient.

[0147] The parameter updating unit is used to generate a parameter updating instruction if the deviation exceeds a preset threshold, and the parameter updating instruction is used to instruct to generate a second stimulation parameter set according to the current environmental state data and the multi-sensory output sequence.

[0148] The iterative adjustment unit is used to generate a closed-loop control signal according to the second stimulation parameter set and send it to the feedback adjustment device until the deviation converges to the target range.

[0149] In a possible embodiment, the data preprocessing module 11 includes:

[0150] The data denoising unit is used to remove the electrooculogram noise and electromyographic noise in the EEG signal through independent component analysis to obtain a pure EEG signal.

[0151] The frequency band extraction unit is used to perform wavelet transform on the pure EEG signal and extract the energy characteristics of multiple frequency bands, including the Alpha band, Beta band and Gamma band.

[0152] The smoothing processing unit is used to perform dynamic smoothing processing on the energy features based on a sliding window mechanism to generate a first emotion feature set.

[0153] In a possible embodiment, performing timing synchronization calibration on the control signal to generate a first multi-sensory output sequence includes:

[0154] The timestamp alignment unit is used to establish a timestamp alignment mechanism for sound, light, and tactile signals, and to set the starting time deviation of the multimodal signals within a preset threshold.

[0155] The weight distribution unit is used to generate an intensity weight matrix according to the first emotional state label, and the intensity weight matrix is used to distribute the output ratio of each modal signal.

[0156] The cross-modal fusion unit is used to perform cross-modal fusion on the control signal based on the intensity weight matrix to generate a first multi-sensory output sequence.

[0157] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned method for real-time emotion recognition and feedback of human-computer interaction based on brain-computer interface are implemented.

[0158] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0159] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0160] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A method for real-time emotion recognition and feedback in human-computer interaction based on brain-computer interface, characterized in that: The method comprises: S1. Obtaining an EEG signal and performing preprocessing and feature extraction to obtain a first emotion feature set, wherein the EEG signal is obtained in real time by an EEG device worn by a user; S2. Based on the first emotion feature set, determine the user's current emotion state through an individualized emotion classification model and generate a first emotion state label, wherein the individualized emotion classification model is used to adapt the emotion classification weight according to the user's historical EEG signals; S3. Based on the first emotional state label, obtaining a corresponding multi-sensory stimulation parameter combination from a preset emotion-atmosphere mapping database to generate a first stimulation parameter set, wherein the emotion-atmosphere mapping database is used to store mapping relationships between emotional states and sound, light, and touch; S4. Performing multimodal synchronization based on the first stimulation parameter set to generate a multimodal co-stimulation signal; S5. Generate a closed-loop control signal according to the multimodal co-stimulation signal and send the closed-loop control signal to a feedback regulation device; S6. Extract real-time change trend data based on the EEG signal, and dynamically update the first stimulation parameter set according to the real-time change trend data to generate a second stimulation parameter set, wherein the second stimulation parameter set is used to iteratively adjust the closed-loop control signal until the user's emotional state matches a preset target.

2. The method according to claim 1, characterized in that The S2 includes: S21, normalizing the first emotion feature set to obtain a standardized emotion feature vector; S22: input the standardized emotion feature vector into the pre-trained individualized emotion classification model, and output a probability distribution of the user's emotional state; S23. If the highest probability value in the probability distribution is greater than a preset confidence threshold, the corresponding emotion category is marked as the first emotion state label; otherwise, a re-collection instruction is generated, where the re-collection instruction is used to instruct to return to execute S1.

3. The method according to claim 1, characterized in that The S4 includes: S41. Generate a control signal according to the first stimulation parameter set, where the control signal includes a sound frequency modulation signal, a light color temperature adjustment signal, and a tactile vibration intensity control signal; S42, performing timing synchronization calibration on the control signal to generate a first multi-sensory output sequence; S43: Acquire and dynamically adjust the signal strength of the first multi-sensory output sequence based on the user physiological response data to generate the multimodal synergistic stimulation signal.

4. The method according to claim 1, wherein After S4, it also includes: S44. Obtaining initial temperature, humidity, and light intensity reference values from a preset environmental parameter template according to the first emotional state tag; S45, calculating the target environmental parameter value through a weighted fusion algorithm based on the user's historical preference data and the current environmental sensor data; S46 . Perform protocol conversion processing on the target environmental parameter value to generate an environmental adjustment parameter set, where the environmental adjustment parameter set is used to generate the multimodal co-stimulation signal.

5. The method according to claim 1, wherein The S6 includes: S61. Calculate the deviation of the real-time change trend data within a preset time window using the following formula: Where D(t) is the deviation within the time window t, ΔE i,t is the difference between the energy of the i-th frequency band and the target value, is the historical energy standard deviation, is the average energy change rate, λ is the weight coefficient; S62: if the deviation exceeds a preset threshold, generating a parameter update instruction, wherein the parameter update instruction is used to instruct to generate the second stimulation parameter set according to the current environmental state data and the multi-sensory output sequence; S63. Generate the closed-loop control signal according to the second stimulation parameter set and send it to the feedback adjustment device until the deviation converges to a target range.

6. The method according to claim 1, characterized in that Said S1 comprises: S11, removing electrooculogram noise and electromyography noise from the EEG signal by independent component analysis to obtain a pure EEG signal; S12, performing wavelet transform on the pure EEG signal to extract multi-band energy features, where the energy features include Alpha band, Beta band, and Gamma band; S13. Dynamically smooth the energy features based on a sliding window mechanism to generate the first emotion feature set.

7. The method according to claim 3, characterized in that The step of performing timing synchronization calibration on the control signal to generate a first multi-sensory output sequence includes: Establish a timestamp alignment mechanism for sound, light, and tactile signals to set the start time deviation of multimodal signals within a preset threshold; generating an intensity weight matrix according to the first emotional state label, wherein the intensity weight matrix is used to allocate an output ratio of each modal signal; The control signal is cross-modally fused based on the intensity weight matrix to generate the first multi-sensory output sequence.

8. A real-time emotion recognition and feedback system for human-computer interaction based on brain-computer interface, characterized in that: The system comprises: a data preprocessing module, configured to obtain, preprocess, and extract features of EEG signals to obtain a first emotion feature set, wherein the EEG signals are obtained in real time via an EEG device worn by a user; an emotion recognition module, configured to determine the user's current emotional state based on the first emotion feature set and generate a first emotional state label using an individualized emotion classification model, wherein the individualized emotion classification model is configured to adapt an emotion classification weight based on the user's historical EEG signals; an emotion mapping module, configured to obtain, based on the first emotion state label, a corresponding multi-sensory stimulation parameter combination from a preset emotion-atmosphere mapping database to generate a first stimulation parameter set, wherein the emotion-atmosphere mapping database is configured to store mapping relationships between emotion states and sound, light, and touch; an interactive signal generating module, configured to generate a multimodal collaborative stimulation signal by performing multimodal synchronization based on the first stimulation parameter set; a signal control module, configured to generate a closed-loop control signal according to the multimodal co-stimulation signal and send the closed-loop control signal to a feedback regulation device; An interactive feedback module is used to dynamically update the first stimulation parameter set based on the real-time change trend data extracted from the EEG signal to generate a second stimulation parameter set, wherein the second stimulation parameter set is used to iteratively adjust the closed-loop control signal until the user's emotional state matches the preset target.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images