A multi-modal emotion regulation neurofeedback training system and method

Abstract

The application discloses a multi-modal emotion regulation neural feedback training system and method, and relates to the field of computer-aided technology.The system comprises a pre-training subsystem and an online feedback subsystem; the pre-training subsystem is used for screening and extracting signal feature information from neural activity signals in an emotion regulation stage and for training a classifier for two-class activity mode classification; the online feedback subsystem is used for collecting neural activity data during the execution of emotion regulation activities and for real-time analysis of the category of current neural activity, while feedback recognition data and feedback training effects are presented in real time.The application extracts relevant complex feature patterns through multivariate pattern analysis, improves the training scheme related to individual emotion regulation neural activity through accurate targeting, and overcomes the limitations of traditional neural feedback technology in processing complex cognitive tasks, thereby improving the effectiveness of emotion regulation training and significantly improving the accuracy and personalization of feedback.

Description

Technical Field

[0001] This invention relates to the field of computer-aided technology, and in particular to a multimodal emotion regulation neurofeedback training system and method. Background Technology

[0002] Neurofeedback technology, a special type of biofeedback method, primarily uses equipment such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS) to collect brain signals from subjects. These signals are then fed back to the subject in various ways, such as through visual or auditory means, for example, by adjusting the pitch of a sound or by visually altering the temperature display. This real-time feedback helps subjects adjust their brain activity to a desired pattern, potentially influencing related cognitive functions and behaviors.

[0003] Specifically, the impact of functional magnetic resonance imaging (fMRI)-based neurofeedback on brain activation and behavior has been extensively studied. In recent years, existing technologies have begun to focus on neurofeedback training based on electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, as an electrophysiological technique, records changes in brain activity resulting from the sum of postsynaptic potentials occurring synchronously in a large number of neurons, reflecting the electrophysiological activity of brain neurons in the cerebral cortex or scalp surface. It has high temporal resolution and low cost, and can more specifically modulate and improve specific cognitive activities by combining event-related potentials (ERPs) and event-related oscillations (ERSPs). fNIRS, as a non-invasive functional brain imaging technique, reflects brain activity by measuring changes in blood oxygen concentration in the cerebral cortex. It has high spatial resolution and low sensitivity to motion artifacts, making it suitable for various experimental environments and subject populations. fNIRS can provide information about local hemodynamic responses in the brain, thus complementing EEG and making neurofeedback training more comprehensive and precise. Therefore, neurofeedback based on fNIRS and EEG has broad prospects for practical applications. These technologies have shown potential not only in cognitive and behavioral regulation but also in applications such as neurorehabilitation, psychotherapy, and brain-computer interfaces. With advancements in technology and a deeper understanding of brain function, neurofeedback training based on fNIRS and EEG is expected to play a greater role in scientific research and clinical practice, providing more personalized and effective treatment and training methods.

[0004] However, existing neurofeedback training systems often focus only on analyzing and modulating single or a few simple EEG frequencies or specific brain region activities. This approach neglects the complexity of brain activity, especially the dynamic interactions between brain regions when complex cognitive tasks are involved. This limitation makes it difficult for traditional neurofeedback to fully capture and modulate complex neural networks involving the coordinated work of multiple regions, thus limiting its potential application in treating and improving higher cognitive impairments. Furthermore, existing systems use the same training protocol across multiple feedback training sessions, failing to adjust the training protocol according to changes in individual cognitive activity, resulting in low training efficiency and increased training time costs. Finally, current feedback systems offer only simplistic feedback methods, presenting feedback information only through changes in temperature bars or the height of stones.

[0005] Therefore, existing neurofeedback training systems suffer from technical problems such as relying on a single brain signal for feedback, using a single feedback training scheme, and employing a single feedback method. These systems typically depend on only a single neural activity feature for feedback, simplifying the complexity of brain activity, especially when dealing with complex cognitive tasks involving multi-regional coordination. Feedback based on a single feature may ignore the interactions between different brain regions and the overall network dynamics, weakening the reinforcing effect of feedback on brain activity and reducing the effectiveness of neurofeedback in improving higher cognitive functions and handling complex psychological states. Furthermore, the singular and static feedback method may also reduce user engagement and training motivation, further limiting the application potential and effectiveness of neurofeedback technology. Summary of the Invention

[0006] The purpose of this invention is to propose a multimodal emotion regulation neurofeedback training system and method, aiming to overcome the limitations of traditional technologies. This system and method integrate information from multiple brain regions using multi-source data acquisition, employ multivariate pattern analysis (MMA) technology to comprehensively analyze and identify neural activity patterns across the entire brain network, and provide real-time online feedback.

[0007] On the one hand, the present invention provides a multimodal emotion regulation neurofeedback training system, which includes a pre-training subsystem and an online feedback subsystem;

[0008] The pre-training subsystem is used to filter and extract signal feature information from neural activity signals to distinguish different emotion regulation activity patterns during the execution of specific cognitive activities, and to complete the classifier training for binary classification activity patterns.

[0009] The online feedback subsystem is used to collect users' neural activity data during emotion regulation activities, analyze the current neural activity pattern category in real time based on the collected data, and present the neural activity feedback data in real time.

[0010] Furthermore, the pre-training subsystem includes an adjustment testing module, a first data acquisition module, and a first data analysis module;

[0011] The regulation testing module is used to perform near-infrared and electroencephalographic stimulation operations for specific cognitive activities based on a preset emotion regulation paradigm; the first data acquisition module is used to acquire first signal data for the specific cognitive activity stage; the first data analysis module is used to preprocess and extract features from the first signal data, and use the extracted feature data for classifier training of binary classification activity mode and selection of emotion regulation objects for the emotion regulation activity stage.

[0012] Furthermore, the online feedback subsystem includes a second data acquisition module, a second data analysis module, and an online feedback presentation module;

[0013] The second data acquisition module is used to acquire second signal data during the emotion regulation activity phase; the second data analysis module is used to perform real-time analysis on the second signal data and use a binary activity pattern classifier to identify the neural activity pattern category to which the current neural activity belongs; the neural feedback presentation module 306 is used to convert the recognition results and training effects into intuitive visual feedback, thereby displaying the instantaneous changes in neural activity.

[0014] The first signal data includes a first electroencephalogram (EEG) signal and a first blood oxygenation signal, and the second signal data includes a second EEG signal and a second blood oxygenation signal.

[0015] The classifier for the binary activity pattern includes two categories of neural activity patterns: efficient emotion regulation pattern and inefficient emotion regulation pattern.

[0016] On the other hand, the present invention also provides a multimodal emotion regulation neurofeedback training method based on the aforementioned multimodal emotion regulation neurofeedback training system, the method comprising:

[0017] S1, setting the parameters of the pre-trained subsystem;

[0018] S2, set the emotion regulation paradigm of the regulation test module, and perform specific cognitive activities based on the preset emotion regulation paradigm. At the same time, use the first data acquisition module to collect the first signal data corresponding to the execution of the specific cognitive activities to obtain the corresponding emotion regulation performance test data.

[0019] S3, the first data analysis module is used to preprocess and extract features from the collected first signal data, and the extracted features are used to train a classifier for binary classification activity patterns. The image category with the lowest test score is used as the category of the emotion regulation object in the emotion regulation activity stage.

[0020] S4, setting the parameters of the online feedback subsystem;

[0021] S5, select an emotion regulation object according to the category of emotion regulation object determined in step S3, and perform emotion regulation activities based on the selected emotion regulation object. At the same time, use the second data acquisition module to collect the second signal data corresponding to the execution of the emotion regulation activities to obtain the corresponding emotion regulation activity data.

[0022] S6. The second data analysis module is used to preprocess and extract features from the collected second signal data, and the extracted features are used to identify the neural activity pattern category through a pre-trained binary activity pattern classifier.

[0023] S7 uses the neurofeedback presentation module to provide online real-time feedback on the recognition results of neural activity pattern categories and the effects of neurofeedback training.

[0024] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0025] The improved multimodal emotion regulation neurofeedback training system and method of this invention can capture and analyze the user's brain electrical activity in real time. Through multivariate pattern analysis, it extracts complex feature patterns related to specific psychological states or task conditions, effectively enhancing the relevance and personalization of training. Personalized neurofeedback training programs provide targeted feedback training based on negative events that prevent each subject from effectively regulating their emotions, thus promoting training effectiveness and cognitive function improvement more specifically and effectively. Furthermore, by providing feedback on complex brain electrical signal data in an intuitive visual form, it helps to understand brain activity patterns and facilitates learning to regulate these patterns. By precisely targeting and improving emotion regulation brain activity and related neural network functions, it overcomes the limitations of traditional neurofeedback technology in handling complex cognitive tasks, improving the effectiveness of emotion regulation training and significantly enhancing the accuracy and personalization of feedback. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort, wherein:

[0027] Figure 1 This is a schematic diagram of the structure of the multimodal emotion regulation neurofeedback training system provided in an embodiment of the present invention;

[0028] Figure 2A schematic diagram of the EEG electrodes (right) and near-infrared photoelectric electrodes (left) provided in an embodiment of the present invention;

[0029] Figure 3 A flowchart of a multimodal emotion regulation neurofeedback training method provided in an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of the emotion regulation testing module provided in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of brain activation provided in an embodiment of the present invention.

[0032] Figure 6 This is a schematic diagram of brain connectivity strength provided for an embodiment of the present invention.

[0033] Figure 7 This is a schematic diagram of the power spectral density provided for an embodiment of the present invention.

[0034] Figure 8 A schematic diagram illustrating the final model accuracy of the classification model provided in this embodiment of the invention;

[0035] Figure 9 This is a schematic diagram illustrating the test feedback of sunlight provided in an embodiment of the present invention.

[0036] Figure 10 This is a schematic diagram of the training effect trend curve provided in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described embodiments are merely some embodiments of the invention, and not all embodiments. The components of the embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0038] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0039] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0040] Example 1

[0041] like Figure 1 As shown, this embodiment provides a multimodal emotion regulation neurofeedback training system 100, including a pre-training subsystem 201 and an online feedback subsystem 202.

[0042] The pre-training subsystem 201 is used to filter and construct key information from a wide range of cognitive activity data during a user's performance of a specific cognitive activity, which helps to distinguish different emotion regulation activity patterns. In a preferred embodiment, the pre-training subsystem 201 effectively extracts brain neural activity information by precisely selecting EEG and blood oxygenation signals closely related to the target cognitive state.

[0043] The online feedback subsystem 202 is used to collect neural activity data from users during specific cognitive activities and analyze the collected data in real time to identify activity patterns with different characteristics. Simultaneously, the online feedback subsystem 202 provides visual feedback of the activity patterns to the user, allowing the user to intuitively see the real-time state and changes in brain activity.

[0044] Furthermore, the pre-training subsystem 201 includes an adjustment test module 301, a first data acquisition module 302, and a first data analysis module 303.

[0045] The regulation test module 301 is used to perform near-infrared and electroencephalographic stimulation operations based on a preset emotion regulation paradigm.

[0046] The first data acquisition module 302 is used to acquire the subject's behavior, near-infrared and electroencephalogram (EEG) data, and monitor and record the neural activity signals of the user during the performance of a specific cognitive activity. In a preferred embodiment, the neural activity signals are omnidirectional electroencephalogram (EEG) signals and blood oxygen saturation (HBO) signals acquired using a relevant signal acquisition device. In the pre-training subsystem 201, the signals acquired by the first data acquisition module 302 are the EEG and HBO signals during the user's performance of a specific cognitive activity based on a preset emotion regulation paradigm. To facilitate differentiation from other signals, this invention uniformly describes the first data acquisition module 302 as acquiring first signal data containing a first EEG signal and a first HBO signal.

[0047] The first data analysis module 303 is used to preprocess and extract features from the collected first signal data, and for further data analysis and training. The first data analysis module 303 is divided into a first preprocessing unit 401, a first feature extraction unit 402, and a pattern training unit 403.

[0048] The first preprocessing unit 401 preprocesses the acquired first EEG signal and first blood oxygen signal, and sends the high-quality signal obtained after preprocessing to the subsequent processing unit. The first preprocessing unit 401 preprocesses the acquired first signal data, including steps such as filtering, noise reduction, signal amplification, and data standardization, to improve the quality of the data and the accuracy of data analysis.

[0049] The first feature extraction unit 402 employs a multivariate pattern analysis method to extract complex feature patterns related to specific cognitive activity states or task conditions from the preprocessed first signal data. This design enables the system to not only identify single or simple signal differences, but also to analyze and utilize the overall patterns of cognitive activity, thereby gaining a more comprehensive understanding of how the brain operates under different psychological states.

[0050] The pattern training unit 403 is used to train a classifier for binary classification activity patterns based on the extracted features. The binary classification activity patterns include two categories: efficient emotion regulation pattern (negative emotions are reduced significantly after emotion regulation) and inefficient emotion regulation pattern (negative emotions are reduced less after emotion regulation).

[0051] Furthermore, the online feedback subsystem 202 includes a second data acquisition module 304, a second data analysis module 305, and an online feedback presentation module 306.

[0052] The second data acquisition module 304 is similar to the first data acquisition module 302, acquiring relevant activity signals of the user. The difference is that the second data acquisition module 304 acquires the user's behavior, blood oxygenation signals, and electroencephalogram (EEG) signals during the emotion regulation phase. Correspondingly, this invention is uniformly described as the second data acquisition module 304 acquiring second signal data including second EEG signals and second blood oxygenation signals.

[0053] The second data analysis module 305 is used to perform real-time analysis on the collected second signal data to determine the activity mode category under the current cognitive state. Further, the second data analysis module 305 includes a second preprocessing unit 404, a second feature extraction unit 405, and a state classification unit 406.

[0054] Specifically, the second preprocessing unit 404 uses signal processing techniques to preprocess the acquired second signal data, including filtering, noise reduction, signal amplification, and data standardization, to improve data quality and the accuracy of analysis. The second feature extraction unit 405 is responsible for extracting complex feature patterns related to specific cognitive activity states or task conditions from the preprocessed second signal data using multivariate pattern analysis techniques. The state classification unit 406 inputs the extracted features into a pre-trained binary activity pattern classifier to classify the current emotion regulation state.

[0055] The online feedback presentation module 306 is used to transform the classified feedback data analyzed in real time into intuitive visual real-time feedback. It displays the instantaneous changes in neural activity through dynamic graphics and an interactive interface, helping to understand the correlation between the current activity pattern and specific psychological state or task performance. In addition, the online feedback presentation module 306 is also used to provide visual feedback on the training effect of neurofeedback training to the subject. By providing trend curves and / or text output of training effect, it intuitively presents the cumulative changes and stability of training effect, helping to identify effective strategies and optimize training direction.

[0056] Example 2

[0057] This embodiment of the invention provides a more detailed description of the data acquisition mechanisms of the first data acquisition module 302 and the second data acquisition module 304.

[0058] In a preferred embodiment, the two data acquisition modules are wearable head-mounted data acquisition devices. Further, the wearable head-mounted data acquisition device includes at least a wireless EEG acquisition system for acquiring electrical activity signals of brain neurons and a multi-channel near-infrared spectroscopy (fNIRS) system for acquiring light intensity data. The two data acquisition modules transmit the acquired blood oxygenation signals, representing brain blood oxygenation activity, and EEG signals, representing brain neural electrical activity, to a server for data analysis or feedback presentation.

[0059] The near-infrared spectroscopy (fNIRS) reflects brain activity by measuring changes in the concentrations of oxyhemoglobin and deoxyhemoglobin in the blood of the cerebral cortex. Specifically, the fNIRS system emits near-infrared light that passes through the scalp and skull into the brain. When the infrared light is absorbed by hemoglobin, the detector in the fNIRS system receives the reflected light and calculates the change in blood oxygen concentration.

[0060] The wireless EEG acquisition system primarily collects electroencephalographic signals from the brain. Its principle is that when brain neurons are active, they transmit electrical signals through synapses. The wireless EEG acquisition system captures minute changes in electrical current through electrodes in contact with the scalp, thereby indirectly reflecting the activity of specific areas of the brain under various cognitive and emotional states.

[0061] The wearable head-mounted data acquisition device and the online feedback server can be connected via wired or wireless means. In a preferred embodiment, the electrode placement of the wearable head-mounted data acquisition device is based on the international EEG 10-10 system, with near-infrared electrodes distributed in the frontal, parietal, and occipital lobes, such as... Figure 2 As shown.

[0062] Example 3

[0063] like Figure 3 As shown in the illustration, this invention provides a detailed description of a method for neurofeedback training based on the aforementioned multimodal emotion regulation neurofeedback training system 100. The neurofeedback training method includes the following steps:

[0064] S1 sets the parameters of the pre-training subsystem 201, thereby controlling the pre-training subsystem 201.

[0065] In a preferred embodiment, the parameters of the pre-training subsystem 201 include at least: 1. Content selection; in a preferred embodiment, the content includes social pain or general negative emotions. 2. Number of test blocks (block num), i.e., the number of blocks that each test block will contain; for example, the default parameter value for the number of test blocks is 30, meaning that 30 test blocks will be run. 3. Block duration, i.e., the duration that each test block will last; for example, the default parameter value for the block duration is 30, meaning that rest and test phases will be interleaved, and each phase will last 30 seconds. 4. Model selection, i.e., the classifier model used by the pre-training subsystem 201, which can be either Regularized Linear Discriminant Analysis (LDA) or Support Vector Machine (SVM), with Regularized Linear Discriminant Analysis (LDA) as the default.

[0066] S2, set the emotion regulation paradigm of the regulation test module 301, and perform specific cognitive activities based on the preset emotion regulation paradigm. At the same time, use the first data acquisition module 302 to detect the first signal data corresponding to the stage of performing specific cognitive activities to obtain the user's emotion regulation performance test data.

[0067] S2.1, Set the emotion regulation paradigm of the regulation test module 301, and perform specific cognitive activities based on the preset emotion regulation paradigm.

[0068] After the relevant parameters of the pre-training subsystem 201 are set, the pre-training subsystem 201 can be started. To obtain reference data for training, the user's emotional regulation performance is first tested using the adjustment test module 301. Figure 4 Taking the classic emotion regulation paradigm shown as an example, the regulation test module 301 conducts an emotion regulation performance test on the user based on the set emotion regulation paradigm and the parameters of the pre-training subsystem 201: The process of a complete test block includes 8 seconds of blank display, 6 seconds of viewing negative images, 5 seconds of negative emotion scoring, 6 seconds of emotion regulation, and another 5 seconds of negative emotion scoring.

[0069] In a preferred embodiment, the negative images selected by the emotion regulation paradigm are mainly divided into three categories: traumatic and crisis events, interpersonal conflict events, and social environmental stressors. The same number of negative images are selected for each category.

[0070] For example, with the default parameters of 30 test blocks and 30 block durations, the emotion regulation task will consist of 30 blocks, each lasting 30 seconds. The process for each block is as follows: First, the user views a negative image, naturally experiencing the feelings evoked by the image and initially rating their negative emotions (1 point represents the lowest intensity of negative emotion, 5 points represent the moderate intensity of negative emotion, and 9 points represent the highest intensity of negative emotion); then, the image is used to regulate the user's emotions; finally, the user's negative emotions are rated again (1 point represents very easy to regulate emotions, 5 points represent moderate difficulty in regulating emotions, and 9 points represent very difficult to regulate emotions).

[0071] The test data was processed in two ways, and the two processed data were used for subsequent classifier training and emotion regulation object selection, respectively.

[0072] Specifically, on the one hand, task scores are calculated based on the negative emotion scores of 30 blocks. The calculation method is to subtract the negative emotion score after emotion regulation from the initial negative emotion score before emotion regulation. The task scores of the 30 blocks in each round are sorted from high to low, and the top 33% and bottom 33% of blocks are divided into high-efficiency emotion regulation blocks (significant reduction in negative emotion after emotion regulation) and low-efficiency emotion regulation blocks (less reduction in negative emotion after emotion regulation). The selected high-efficiency emotion regulation blocks and low-efficiency emotion regulation data are used as reference data for the two modes of high-efficiency and low-efficiency emotion regulation included in the binary classification activity mode classifier, and are used for classifier training.

[0073] On the other hand, the 30 blocks were divided into three categories: trauma and crisis events, interpersonal conflict events, and social environmental stressors. Task scores were calculated for each category, using the same method: subtracting the subsequent negative emotion score after emotion regulation from the initial negative emotion score before emotion regulation. The task scores of the three categories were sorted from highest to lowest, and the category with the lowest score was selected as the negative image category for the subsequent online feedback subsystem 202 to determine the target of emotion regulation. The lowest task score in a category indicates an inability to effectively regulate emotions related to that type of negative event.

[0074] S2.2, while the user is performing a specific cognitive activity, the first signal data of this stage is collected using the aforementioned first data acquisition module 302, including the first EEG signal and the first blood oxygen signal.

[0075] S3, the first data analysis module 303 is used to preprocess and extract features from the collected first signal data, and the extracted features are used to train a classifier for binary classification activity patterns.

[0076] S3.1 First, the first signal data collected is preprocessed by the first preprocessing unit 401 to obtain the first signal data after noise removal.

[0077] Specifically, the first preprocessing unit 401 preprocesses the acquired first EEG signal to remove noise, such as non-EEG interference signals, to obtain a high-quality EEG signal. The preprocessing steps for the first EEG signal may include:

[0078] S3.1.1, use FIR (Finite Impulse Response) filters to remove high-frequency and low-frequency noise from EEG signals;

[0079] An FIR filter can be implemented as a bandpass filter (FIR bandpass filter) to remove high-frequency and low-frequency noise from EEG signals. An FIR filter can be implemented in hardware or software; this embodiment of the invention does not impose any limitations on its implementation.

[0080] S3.1.2, use the differential algorithm to remove electrooculogram artifacts in EEG signals;

[0081] Because the magnitude of electrooculogram (EOG) signals is much larger than that of electroencephalogram (EEG) signals, EOG signals are the primary source of contamination in EEG signals (especially frontal lobe EEG signals). For single-channel EEG signals, removing EOG artifacts (contamination) is particularly important. Using a differential algorithm module to remove EOG artifacts is one method in EEG signal preprocessing. The differential algorithm module can be implemented in hardware or software; this embodiment of the invention is not limited to either.

[0082] S3.1.3, using motion sensors to remove head movement artifacts from EEG signals.

[0083] Another source of contamination in EEG signals is electromyography (EMG) signals generated by muscle movement, as well as signal drift caused by head movements, collectively known as head movement artifacts. To remove head movement artifacts, a motion sensor built into the EEG recording device can be used to record head movement signals. The convolved muscle movement signals (EMG signals) in the EEG signal can then be stripped to extract a cleaner EEG signal. As a preferred embodiment, the motion sensor can be a multi-helix motion sensor.

[0084] After removing various non-EEG interference signals through the above steps, a high-quality single-channel EEG signal can be obtained. In some optional embodiments, the preprocessing step of the EEG signal can use two reference electrodes as auxiliary electrodes. The signals acquired by the reference electrodes are used as reference signals for the EEG signal, and preprocessing is performed based on these reference signals. For example, the reference signals can be used as reference signal recording points for EEG artifact removal during the processing of the differential calculation module. By using reference signals acquired by the reference electrodes in the preprocessing, the preprocessing effect can be improved.

[0085] On the other hand, preprocessing of the first blood oxygen signal mainly involves removing light scattering noise caused by the scalp and skull to improve the quality of the blood oxygen signal. The steps for preprocessing the first blood oxygen signal may include:

[0086] S3.1.4, convert the collected light intensity data into light density data;

[0087] S3.1.5, the time derivative distribution repair method is used to correct artifacts caused by user head movements;

[0088] S3.1.6 uses an FIR (Finite Impulse Response) filter to remove high-frequency and low-frequency noise from the optical density signal data.

[0089] S3.1.7, Based on the modified Beer-Lambert law, convert the optical density data into relative change data of oxyhemoglobin and deoxyhemoglobin concentrations.

[0090] It should be noted that the preprocessing of the first EEG signal and the preprocessing of the first blood oxygen signal can be performed simultaneously or asynchronously. The aforementioned order of steps is only used to more clearly describe the processing method.

[0091] S3.2, the first feature extraction unit 402 performs signal feature extraction calculations on the received second EEG signal and second blood oxygen signal, which are collected and preprocessed under a certain emotion regulation paradigm.

[0092] In this embodiment of the disclosure, the feature indicators for signal extraction may include activation of various brain regions, brain connectivity strength, ERP components, and power spectrum characteristics. In a preferred embodiment, the feature indicators may include one or more of the following features: the strength of intra- and extra-brain functional connectivity and / or blood oxygenation activation levels in the right frontal lobe / left frontal lobe / right parietal lobe / left parietal lobe, and the energy magnitudes of delta (1-4Hz) / theta (4-8Hz) / alpha (8-12Hz) / beta (12-30Hz) / gamma (30-55Hz).

[0093] The following is a brief introduction to the above-mentioned characteristic indicators:

[0094] Regarding blood oxygenation signals, the strength of intracranial and extracranial functional connectivity and the level of blood oxygenation activation in different brain regions vary significantly under different activity patterns. For example, the right and left frontal lobes are primarily responsible for higher cognitive functions and emotional processing, exhibiting specific activation and connectivity changes during task execution and emotional responses. The right and left occipital lobes are mainly responsible for visual information processing, and these areas are significantly activated when subjected to visual stimuli. The right and left parietal lobes are involved in spatial localization and attention processing, and the activation and connectivity strength of these areas change during attention-focusing tasks.

[0095] To quantify brain activation levels and connectivity strength, embodiments of the present invention employ the following formula to represent the relationship between blood oxygenation changes and brain activation:

[0096] The formula for calculating changes in blood oxygen activation level is as follows:

[0097]

[0098] Among them, S t It is the spectral signal intensity (usually measured by fNIRS) at a specific time point, S baseline t represents the baseline signal strength, and t represents time (the same applies below).

[0099] The connectivity strength of brain regions can also be analyzed using a similar method, assessing changes in brain network activity by calculating the functional connectivity strength between different brain regions. Extracted brain activation features include... Figure 5 As shown, the extracted brain connectivity strength features are as follows: Figure 6 As shown. By extracting and analyzing these features, we can gain a deeper understanding of the functional performance of different brain regions in different tasks and the interactions between them.

[0100] Different brainwave frequencies are associated with specific cognitive functions and brain activity. Specific areas of the brain respond to these frequencies during task performance, especially when processing visual and auditory information. The activity levels of these frequencies change as the task difficulty increases. Therefore, extracting features from different frequency bands is crucial for understanding brain activity. Common frequency bands include: Delta waves (1-4 Hz), associated with deep sleep and rest; Theta waves (4-8 Hz), associated with learning and memory; Alpha waves (8-12 Hz), associated with relaxation and attention; Beta waves (12-30 Hz), associated with concentration and cognitive processing; and Gamma waves (30-55 Hz), associated with higher cognitive functions.

[0101] To extract frequency band features from EEG signals, the time-domain signal is typically converted to the frequency-domain signal. Therefore, the Short-Time Fourier Transform (STFT) method can be used. STFT allows for signal conversion between the time and frequency domains, enabling the calculation of the power spectral density (PSD) for each frequency band. After obtaining the PSD, the energy of specific frequency bands is further calculated, and normalized energy is obtained through normalization. A schematic diagram of the extracted power spectral density is shown below. Figure 7 As shown.

[0102] The short-time Fourier transform formula used is as follows:

[0103]

[0104] Where X(t,f) are discrete samples of EEG signals, w(mt) is a window function, and e -j2πfn It is a complex exponential function, and m represents a sequence.

[0105] The formula for calculating power spectral density (PSD) is as follows:

[0106] P(f) = |X(t,f)| 2 (4)

[0107] Where P(f) represents the spectral density, the result of which is the square of the amplitude of the complex number X(t,f), reflecting the energy distribution of the signal at different frequencies.

[0108] S3.3, after the signal feature extraction calculation is completed, the pre-training unit 403 of the first data analysis module 303 further performs multivariate pattern analysis on the signal features, identifying and classifying the signal features according to two modes: efficient emotion regulation and inefficient emotion regulation, and using them to train a classifier for binary activity patterns. Specifically, this includes the following steps:

[0109] S3.3.1 performs dimensionality reduction and standardization on the extracted signal features.

[0110] Data standardization is necessary before performing PCA dimensionality reduction. Since various features of EEG and blood oxygenation signals may have different units and magnitudes, data standardization is required before PCA. The purpose of standardization is to ensure that each feature has the same scale, thereby avoiding undue influence on the PCA results from certain features having large dimensions.

[0111] The standardized formula used is as follows:

[0112]

[0113] Among them, X normThese are the original data, μ is the mean of the data, and σ is the standard deviation.

[0114] S3.3.2, Calculate the covariance matrix of the standardized data.

[0115] Next, eigenvalue decomposition is performed on the covariance matrix. This decomposition yields eigenvalues ​​and eigenvectors. The eigenvectors represent the new coordinate axis directions (principal components), while the eigenvalues ​​represent the variance of each principal component. A larger variance indicates a stronger explanatory power of the principal component for data variation. The eigenvalues ​​are sorted in descending order, and the principal components with the largest variances are selected. These principal components will contain the most important information about the changes in the data. The number of principal components selected is usually determined by the cumulative variance contribution rate, choosing those that can explain most of the variance.

[0116] As a preferred embodiment, a threshold condition for selecting principal components can be set, for example, selecting principal components with the top 90% cumulative variance contribution rate.

[0117] S3.3.3 Perform multivariate pattern analysis and classification, including dataset partitioning, validation, and model evaluation.

[0118] Multivariate pattern analysis is used to extract patterns related to specific psychological states or tasks from high-dimensional data. In this process, a linear discriminant analysis (LDA) classifier is first used to analyze and classify the extracted features. LDA achieves optimal classification performance by maximizing between-class variance and minimizing within-class variance, and is suitable for linearly separable feature spaces.

[0119] In practice, the preprocessed feature dataset is first divided into training and test sets. The training set is used to train the machine learning model, while the test set is used to evaluate the model's performance. To ensure the model's stability and generalization ability, a 5-fold cross-validation method is used. The cross-validation process randomly divides the dataset into five subsets, selecting four subsets for training each time, and using the remaining subset for testing. This process is repeated five times, and the average performance index is calculated to ensure the model's robustness and effectiveness across different data subsets. After training, the model is validated using the test set to evaluate its classification performance. Optimal model parameters are automatically selected to achieve the best classification results, especially under conditions of efficient and inefficient emotion regulation, validating the model's recognition and classification abilities. This approach ensures that the model maintains efficient classification performance across different psychological states and task contexts. The final model accuracy distribution is shown below. Figure 8 As shown.

[0120] After training is completed in this way, the trained binary activity pattern classifier model is applied to the subsequent prior feedback subsystem 202. This subsystem enables accurate interpretation of EEG and blood oxygenation signals, supporting the identification and analysis of specific psychological states or tasks. Thus, the emotion regulation neurofeedback training system can effectively distinguish and identify two different activity patterns: efficient emotion regulation and inefficient emotion regulation.

[0121] S4 sets the parameters of the online feedback subsystem 202, thereby controlling the entire online feedback subsystem 202.

[0122] The embodiments of the present invention further describe in detail the functions and operation methods implemented based on the aforementioned online feedback subsystem 202.

[0123] The online feedback subsystem 202 is mainly responsible for real-time monitoring and analysis of the user's activity patterns. It mainly extracts key information from the current activity signals collected by the second data acquisition module 304 based on the second data analysis module 305, and calculates the activity patterns describing different psychological states or cognitive tasks. Then, it transforms these activity patterns into intuitive visual feedback, such as displaying them on the user interface through color changes, shape dynamics, or graphic changes, and provides feedback on the training effect.

[0124] In this embodiment of the invention, the user can manipulate the parameters of the online feedback subsystem 202, thereby controlling the entire online feedback subsystem 202. After all parameters are set, the pre-training subsystem can begin operation.

[0125] In a preferred embodiment, the parameters of the online feedback subsystem 202 include: 1. Selection of the training scheme, which refers to the visual scene and visual feedback content used in the neurofeedback training. For example, sunlight or a thermometer can be selected, with sunlight as the default. 2. Number of training blocks (training block num), i.e., the number of blocks that each training block will contain. For example, the default parameter 30 means that 30 training blocks will be run. 3. Training block duration (training block duration), i.e., the duration that each training block will last. For example, the default parameter 60 means that rest and training phases will be interleaved, with each phase lasting 60 seconds.

[0126] S5. Select an emotion regulation target and perform emotion regulation activities based on the selected emotion regulation target. At the same time, use the second data acquisition module 304 to collect the second signal data corresponding to the current emotion regulation activity to obtain the user's real-time emotion regulation activity data.

[0127] In a preferred embodiment, the type of emotion regulation object corresponding to each training block is a negative image representing the event with the lowest task score in the emotion regulation paradigm. In one embodiment, the presentation time of a single negative image during the emotion regulation activity is 10 seconds, and multiple negative images can be used for training within the same training block.

[0128] Once the relevant parameters of the online feedback subsystem 202 are set, the online feedback subsystem 202 can be started to run.

[0129] The second signal data also includes the second electroencephalogram (EEG) signal and the second blood oxygenation signal.

[0130] Since the signals acquired by the second data acquisition module 304 are output to the server in real time, users should minimize physical movement during use to improve the signal-to-noise ratio, thereby ensuring data accuracy and system effectiveness. The server performs real-time online analysis on the output signals.

[0131] S6, the second data analysis module 305 is used to preprocess and extract features from the collected second signal data, and the extracted features are used to identify the activity pattern.

[0132] S6.1 After receiving the second signal data collected by the second data acquisition module 304, the second data analysis module 305 also performs signal preprocessing and feature extraction by the second preprocessing unit 403 and the second feature extraction unit 404. The specific preprocessing method and feature extraction method can be similar to the data preprocessing and feature extraction operation process in the first data analysis module 303 to obtain the features corresponding to the user's real-time activity status.

[0133] S6.2, thereafter, the state classification unit 405 uses the binary activity pattern classifier trained by the pre-trained subsystem 201 to identify the activity pattern to which the current activity state belongs, thereby realizing real-time and accurate interpretation of electroencephalogram (EEG) signals and blood oxygenation (HBO) signals, and supporting the identification and analysis of the current activity pattern.

[0134] The classifier outputting the binary activity pattern classifier uses a ratio metric f as its analytical parameter. This f represents the real-time score after analyzing the current activity pattern, reflecting the degree of matching between the brain's current activity pattern and its emotion regulation pattern. A higher f score indicates a greater similarity between the current activity pattern and an efficient emotion regulation pattern; conversely, a lower f score indicates a greater similarity between the current activity pattern and an inefficient emotion regulation pattern.

[0135] The method for identifying the activity mode to which the current activity state belongs is as follows:

[0136] S6.2.1 First, calculate the joint matrix Σ of the two activity modes.S :

[0137]

[0138]

[0139] Where Category1 represents efficient emotion regulation, Category2 represents inefficient emotion regulation, μ1 and μ2 are the sample means of Category1 and Category2 respectively, n represents the dimension of the data, x represents the feature vector of the input signal, i represents the i-th sample value, and T represents the transpose of the feature vector.

[0140] S6.2.2, Next, calculate the probability density function p for the two activity modes. 1(x) and p 2(x) :

[0141]

[0142] Where, Σ S It is a joint matrix.

[0143] S6.2.3 Finally, to evaluate the similarity between the current activity pattern and the two classification patterns, the following ratio metric f is used as the output:

[0144]

[0145] Where, p 1(x) and p 2(x) These are the probability density functions for category 1 and category 2, respectively.

[0146] Therefore, the embodiments of the present invention can effectively distinguish and identify efficient and inefficient emotion regulation activity patterns, and the multivariate pattern analysis parameters output by the classifier are real-time scores, reflecting the degree of matching between the brain's current activity pattern and the emotion regulation activity pattern. Specifically, the ratio metric f reflects the relative similarity between the current activity pattern and the two pattern states; the higher the f score, the more similar the current activity pattern is to the efficient emotion regulation activity pattern; the lower the f score, the more similar the current activity pattern is to the inefficient emotion regulation pattern.

[0147] S7 uses the online feedback presentation module 306 to provide real-time feedback on the identified activity pattern analysis parameters and feedback training results.

[0148] After calculating and determining the mode category of the current activity mode, the online feedback presentation module 306 further provides real-time feedback display based on the obtained multivariate mode analysis parameters. The online feedback presentation module 306 transforms the obtained multivariate mode analysis parameters into intuitive visual feedback displayed on the user interface, for example, through color changes, dynamic shapes, or graphic changes. In a preferred embodiment, real-time data transmission and analysis can be performed every second, and the feedback presentation can be updated accordingly.

[0149] The online feedback presentation module 306 serves as the user interaction interface, transforming the pattern category information identified by the second data analysis module 305 into visual feedback. This neurofeedback presentation module 306 generates dynamic visual images or sound signals in real-time based on the user's real-time multivariate pattern analysis parameters. This feedback is presented in a gamified or story-like format, increasing the fun of training and user engagement. For example, if the user prefers efficient emotion regulation, the weather on the feedback interface will gradually brighten; if they prefer inefficient emotion regulation, the weather will gradually become cloudy.

[0150] The calculation method for real-time feedback of the identified activity pattern analysis parameters is as follows:

[0151]

[0152] Here, f is the real-time score obtained from the previous classification step. Therefore, the value of the feedback value F is adjusted according to the range of the score f: when the score f is less than or equal to -2, the feedback value F is -2; when the score is between -2 and 2, the feedback value F is the current score f; when the score is greater than or equal to 2, the feedback value F is 2.

[0153] As an example, the correspondence between the feedback value F and the feedback content is as follows: Figure 9 As shown.

[0154] By controlling the intensity and type of feedback in this way, the system can automatically adjust feedback parameters, such as the intensity, type, and frequency of feedback, based on the user's training history and progress, in order to maximize the training effect and ensure that the user can intuitively understand their activity pattern status and adjust their behavior in a timely manner.

[0155] In addition, the system supports a multi-user mode, allowing each user to have their own customized settings in multi-user environments such as homes or training centers.

[0156] In addition, the online feedback presentation module 306 can also calculate the training effect of neurofeedback training and provide real-time feedback.

[0157] The training effect of the subjects was calculated after the neurofeedback training was completed, and the calculation method is as follows:

[0158]

[0159] Among them, Block i Represents the block number. This represents the average of the chunk numbers. Activity i The f value represents the i-th block. β represents the mean of the f-values ​​for all blocks. β represents the training effect during the neurofeedback training process, i.e., the degree of learning of effective emotion regulation patterns.

[0160] As a preferred embodiment, Actvity can be used. i Plot a trend curve, such as Figure 10 As shown in the image. It also provides visual feedback on the trend curve and training results. When the value of β is greater than 0, the feedback text is "Training successful"; when the value of β is less than 0, the feedback text is "Training failed".

[0161] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the corresponding steps in the foregoing operation method embodiments, and will not be repeated here.

[0162] Those skilled in the art will further recognize that the modules or units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] In summary, this invention proposes a multimodal emotion regulation neurofeedback training system and method. Through a real-time, dynamic feedback mechanism and personalized training programs, it enables users to effectively train for specific states or scenarios, significantly improving training efficiency and user experience. This innovative training tool provides strong support for cognitive science and personal health management.

[0164] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be conceived by those skilled in the art within the technical scope disclosed in the present invention without creative effort should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A multimodal emotion regulation neurofeedback training system, characterized in that, The multimodal emotion regulation neurofeedback training system includes a pre-training subsystem (201) and an online feedback subsystem (202). The pre-training subsystem (201) is used to screen and extract signal feature information from neural activity signals to distinguish different emotion regulation activity modes during the execution of specific cognitive activities, and to complete the classifier training for binary classification activity modes. The pre-training subsystem (201) includes a regulation test module (301), a first data acquisition module (302), and a first data analysis module (303); the regulation test module (301) is used to perform near-infrared and EEG stimulation operations on specific cognitive activities based on a preset emotion regulation paradigm to test the user's emotion regulation performance; The first data acquisition module (302) is used to acquire first signal data for the specific cognitive activity stage; the first data analysis module (303) is used to preprocess and extract features from the first signal data, and use the extracted feature data for classifier training of the binary classification activity mode and selection of emotion regulation objects for the emotion regulation activity stage: on the one hand, the task score is calculated based on the negative emotion scores of all blocks, the calculation method is the initial negative emotion score before emotion regulation minus the subsequent negative emotion score after emotion regulation, the task scores of all blocks in each round are sorted from high to low, and the first 33% and the last 33% of blocks are divided into high and low categories. Effective emotion regulation blocks and ineffective emotion regulation blocks are selected, and the selected effective emotion regulation blocks and ineffective emotion regulation data are used as reference data for the two modes of effective emotion regulation and ineffective emotion regulation contained in the classifier of binary activity mode, and used for classifier training; on the other hand, all blocks are divided into three categories, and the task scores under the three categories are calculated respectively. The calculation method is to subtract the negative emotion score after emotion regulation from the initial negative emotion score before emotion regulation. The task scores of the three categories are sorted from high to low, and the category with the lowest score is selected as the negative image category of the selected emotion regulation object in the subsequent online feedback subsystem (202); The online feedback subsystem (202) is used to take the image category with the lowest test score in step S2 as the category of the emotion regulation object in the emotion regulation activity stage, collect the user's neural activity data during the emotion regulation activity, and analyze the current neural activity pattern category in real time based on the collected data, and present the neural activity feedback data in real time. The online feedback subsystem (202) includes a second data acquisition module (304), a second data analysis module (305), and an online feedback presentation module (306). The second data acquisition module (304) is used to acquire second signal data during the emotion regulation activity phase; the second data analysis module (305) is used to perform real-time analysis on the second signal data and use a binary activity pattern classifier to identify the neural activity pattern category to which the current neural activity belongs; the online feedback presentation module (306) is used to convert the recognition results and training effects into intuitive visual feedback, and display the instantaneous changes in neural activity through dynamic graphics and an interactive interface. The first signal data includes a first electroencephalogram (EEG) signal and a first blood oxygenation signal, and the second signal data includes a second EEG signal and a second blood oxygenation signal; the classifier of the binary activity pattern includes two categories of neural activity patterns: efficient emotion regulation pattern and inefficient emotion regulation pattern.

2. A multimodal emotion regulation neurofeedback training method, characterized in that, The method is based on the multimodal emotion regulation neurofeedback training system described in claim 1, and the method includes: S1, Initialization, setting the parameters of the pre-trained subsystem (201); S2, set the emotion regulation paradigm of the regulation test module (301), and perform specific cognitive activities based on the preset emotion regulation paradigm. At the same time, use the first data acquisition module (302) to collect the first signal data corresponding to the specific cognitive activities to obtain the corresponding emotion regulation performance test data. S3, the first data analysis module (303) is used to preprocess and extract features from the collected first signal data, and the extracted features are used to train a classifier for binary classification activity patterns. The image category with the lowest test score in step S2 is used as the category of the emotion regulation object in the emotion regulation activity stage. S4, set the parameters of the online feedback subsystem (202); S5, select an emotion regulation object according to the category of emotion regulation object determined in step S3, and perform emotion regulation activities based on the selected emotion regulation object. At the same time, use the second data acquisition module (304) to collect the second signal data corresponding to the execution of the emotion regulation activities to obtain the corresponding emotion regulation activity data. S6, the second data analysis module (305) is used to preprocess and extract features from the collected second signal data, and the extracted features are used to identify the neural activity pattern category through a pre-trained binary classification activity pattern classifier. S7, the recognition results of neural activity pattern categories and the neural feedback training effect are fed back online in real time using the online feedback presentation module (306).

3. The multimodal emotion regulation neurofeedback training method based on claim 2, characterized in that, Step S6 specifically includes: S6.1, the second preprocessing unit (403) and the second feature extraction unit (404) of the second data analysis module (305) respectively perform signal preprocessing and feature extraction on the received second signal data; S6.2, the state classification unit (405) of the second data analysis module (305) uses a trained binary activity pattern classifier to analyze the extracted signal features, thereby identifying the neural activity pattern category to which the current activity state belongs.

4. The multimodal emotion regulation neurofeedback training method based on claim 3, characterized in that, The method for identifying the neural activity pattern to which the current activity state belongs in step S6.2 is as follows: S6.2.1, Calculate the joint matrix of the two activity modes. : in, Category 1 represents efficient emotion regulation. Category 2 indicates inefficient emotion regulation. and , which are the sample means of category 1 and category 2, respectively; n represents the dimension of the data; x represents the feature vector of the input signal; i represents the i-th sample value; and T represents the transpose of the feature vector. S6.2.2, Calculate the probability density function for the two activity modes. and : in, It is a joint matrix; S6.2.3, Calculate the ratio metric f to assess the similarity between the current activity pattern and the two classification patterns: in, and These are the probability density functions for category 1 and category 2, respectively; The ratio metric f reflects the relative similarity between the current activity pattern and the two pattern categories. The higher the f score, the more similar the current activity pattern is to the efficient emotion regulation activity pattern; the lower the f score, the more similar the current activity pattern is to the inefficient emotion regulation pattern.

5. The multimodal emotion regulation neurofeedback training method based on claim 4, characterized in that, The calculation method for providing online real-time feedback of the neural activity pattern category recognition results in step S7 is as follows: Where f is the ratio metric and F is the feedback value: when the score f is less than or equal to -2, the feedback value F is -2; when the score is between -2 and 2, the feedback value F is the current score f; when the score is greater than or equal to 2, the feedback value F is 2. The calculation method for providing online real-time feedback on the neurofeedback training effect is as follows: in, Represents the block number. The average value of the chunk number. The f value represents the i-th block. β represents the mean of the f values ​​for all blocks, and β represents the training effect during the neurofeedback training process, i.e., the degree of learning of effective emotion regulation patterns.

Images