Multi-modal emotion regulation neural feedback training system and method

Abstract

The invention discloses a multi-mode emotion regulation neural feedback training system and method, and relates to the technical field of computer assistance. The system comprises a pre-training subsystem and an online feedback subsystem. The pre-training subsystem is used for screening and extracting signal feature information from neural activity signals in an emotion regulation execution stage and is used for classifier training of a dichotomy activity mode; and the online feedback subsystem is used for collecting neural activity data and analyzing the category of the current neural activity in real time during execution of the emotion regulation activity, and presenting feedback recognition data and a feedback training effect in real time. According to the method, related complex feature modes are extracted through multivariable mode analysis, and by accurately aiming at and improving a training scheme related to individual emotion regulation neural activity, the limitation of a traditional neural feedback technology in processing a complex cognitive task is overcome, the effectiveness of emotion regulation training is improved, and the training efficiency is improved. And the accuracy and individuation of feedback are obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of computer-aided technology, and in particular to a multimodal emotion regulation neurofeedback training system and method. Background Art

[0002] Neurofeedback, a specialized biofeedback method, primarily uses devices such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS) to collect brain signals from subjects. These signals are fed back to the subject through various means, including visual or auditory feedback, such as adjusting the pitch of a sound or altering the thermometer display through visual feedback. This real-time feedback can help subjects adjust their brain activity to desired patterns, potentially impacting relevant cognitive functions and behaviors.

[0003] Specifically, the effects of functional magnetic resonance imaging (fMRI)-based neurofeedback on brain activation and behavior have been extensively studied. In recent years, current research has focused on neurofeedback training based on electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, as an electrophysiological technique, records changes in brain activity resulting from the summation of postsynaptic potentials occurring synchronously across a large number of neurons, reflecting the electrophysiological activity of neurons in the cerebral cortex or on the scalp surface. It offers high temporal resolution and low cost, and can more specifically modulate and improve specific cognitive activities by combining event-related potentials (ERPs) and event-related oscillations (ERSPs). fNIRS, as a non-invasive functional brain imaging technique, reflects brain activity by measuring changes in blood oxygen concentration in the cerebral cortex. It has high spatial resolution and low sensitivity to motion artifacts, making it suitable for a variety of experimental settings and subject populations. fNIRS can provide information on local hemodynamic responses in the brain, complementing EEG and making neurofeedback training more comprehensive and precise. Therefore, neurofeedback based on fNIRS and EEG holds great promise in practical applications. These technologies not only demonstrate potential in cognitive and behavioral regulation but also have demonstrated application value in areas such as neurorehabilitation, psychotherapy, and brain-computer interfaces. With technological advancements and a deeper understanding of brain function, fNIRS and EEG-based neurofeedback training is expected to play a greater role in scientific research and clinical practice, providing more personalized and effective treatment and training methods.

[0004] However, existing neurofeedback training systems often focus only on analyzing and regulating a single or a few simple brain wave frequencies or specific brain area activities. This approach ignores the complexity of brain activity, especially the dynamic interactions between brain regions when complex cognitive tasks are involved. This limitation makes it difficult for traditional neurofeedback to fully capture and regulate complex neural networks involving the collaborative work of multiple regions, thereby limiting its application potential in treating and improving higher-level cognitive dysfunction. In addition, the existing system uses the same training program during multiple feedback training sessions and is unable to adjust the training program according to changes in individual cognitive activities, resulting in low training efficiency and increased training time costs. Finally, the current feedback system has a single information feedback method and can only present feedback information through changes in temperature bars or changes in stone height.

[0005] Therefore, existing neurofeedback training systems have technical problems such as a single feedback brain signal, a single multiple feedback training program, and a single feedback method. These systems usually rely on a single neural activity feature for feedback, which simplifies the complexity of brain activity, especially when dealing with complex cognitive tasks that require multi-region coordination. Feedback from a single feature may ignore the interactions between different brain regions and the overall network dynamics, weaken the reinforcing effect of feedback on brain activity, and reduce the effectiveness of neurofeedback in improving higher-level cognitive functions and processing complex psychological states. In addition, the single and static feedback method may also reduce user participation and training motivation, further limiting the application potential and effectiveness of neurofeedback technology. Summary of the Invention

[0006] The present invention aims to propose a multimodal emotion regulation neurofeedback training system and method that overcomes the limitations of conventional techniques. This system and method utilizes multi-source data collection and integration of information from multiple brain regions, employing multivariate pattern analysis techniques to comprehensively analyze and identify neural activity patterns across the entire brain network, and provide real-time online feedback.

[0007] In one aspect, the present invention provides a multimodal emotion regulation neurofeedback training system, the multimodal emotion regulation neurofeedback training system comprising a pre-training subsystem and an online feedback subsystem;

[0008] The pre-training subsystem is used to screen and extract signal feature information that distinguishes different emotion regulation activity patterns from neural activity signals during the execution of specific cognitive activities, and complete classifier training for binary classification activity patterns;

[0009] The online feedback subsystem is used to collect the user's neural activity data during the execution of emotion regulation activities, and analyze the pattern category of the current neural activity in real time based on the collected data, and present the neural activity feedback data in real time.

[0010] Furthermore, the pre-training subsystem includes an adjustment test module, a first data acquisition module, and a first data analysis module;

[0011] The regulation test module is used to perform near-infrared and electroencephalographic stimulation operations of specific cognitive activities based on a preset emotion regulation paradigm; the first data acquisition module is used to collect first signal data of the stage of performing specific cognitive activities; the first data analysis module is used to preprocess and extract features of the first signal data, and use the extracted feature data for classifier training of binary activity patterns and selection of emotion regulation objects in the stage of performing emotion regulation activities.

[0012] Furthermore, the online feedback subsystem includes a second data acquisition module, a second data analysis module and an online feedback presentation module;

[0013] The second data acquisition module is used to collect the second signal data of the emotion regulation activity stage; the second data analysis module is used to perform real-time analysis on the second signal data, and use a binary activity pattern classifier to identify the neural activity pattern category to which the current neural activity belongs; the neural feedback presentation module 306 is used to convert the recognition results and training effects into intuitive visual feedback, thereby displaying the immediate changes in neural activity.

[0014] The first signal data includes a first EEG signal and a first blood oxygen signal, and the second signal data includes a second EEG signal and a second blood oxygen signal.

[0015] The binary activity pattern classifier includes two categories of neural activity patterns: efficient emotion regulation pattern and inefficient emotion regulation pattern.

[0016] On the other hand, the present invention also provides a multimodal emotion regulation neurofeedback training method based on the aforementioned multimodal emotion regulation neurofeedback training system, the method comprising:

[0017] S1, set the parameters of the pre-training subsystem;

[0018] S2, setting an emotion regulation paradigm for the regulation test module, and performing a specific cognitive activity based on the preset emotion regulation paradigm, while simultaneously using the first data acquisition module to collect first signal data corresponding to the specific cognitive activity, to obtain corresponding emotion regulation performance test data;

[0019] S3, using the first data analysis module to preprocess and extract features from the collected first signal data, and using the extracted features to train a classifier for binary classification activity patterns, and using the image category with the lowest test score as the category of the emotion regulation object in the emotion regulation activity stage;

[0020] S4, setting the parameters of the online feedback subsystem;

[0021] S5, selecting an emotion regulation object according to the category of the emotion regulation object determined in step S3, and performing an emotion regulation activity based on the selected emotion regulation object, while simultaneously using a second data acquisition module to collect second signal data corresponding to the execution of the emotion regulation activity to obtain corresponding emotion regulation activity data;

[0022] S6, using the second data analysis module to preprocess and extract features from the collected second signal data, and using the extracted features through a trained binary activity pattern classifier to identify the neural activity pattern category.

[0023] S7, the recognition results of the neural activity pattern categories and the neurofeedback training effects are fed back online in real time using the neurofeedback presentation module.

[0024] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0025] The improved multimodal emotion regulation neurofeedback training system and method of the present invention can capture and analyze the user's EEG activity in real time, and extract complex feature patterns related to specific psychological states or task conditions through multivariate pattern analysis, effectively enhancing the relevance and personalization of training. The personalized neurofeedback training program conducts targeted feedback training based on the negative events in which each subject is unable to effectively regulate emotions, so as to promote training effects and cognitive function improvements in a more targeted and effective manner. In addition, by feeding back complex EEG signal data in an intuitive visual form, it can help understand brain activity patterns and help learn to regulate these patterns. By accurately targeting and improving emotion regulation brain activity and its related neural network functions, it overcomes the limitations of traditional neurofeedback technology in handling complex cognitive tasks, improves the effectiveness of emotion regulation training, and significantly improves the accuracy and personalization of feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort, among which:

[0027] Figure 1 A schematic diagram of the structure of a multimodal emotion regulation neurofeedback training system provided by an embodiment of the present invention;

[0028] Figure 2Schematic diagram of an EEG electrode (right) and a near-infrared optode (left) provided in an embodiment of the present invention;

[0029] Figure 3 A flowchart of a multimodal emotion regulation neurofeedback training method provided by an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of an emotion regulation test module provided in an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of brain activation provided by an embodiment of the present invention.

[0032] Figure 6 A schematic diagram of brain connection strength provided by an embodiment of the present invention.

[0033] Figure 7 A schematic diagram of power spectrum density provided by an embodiment of the present invention.

[0034] Figure 8 A schematic diagram of the final model accuracy of the classification model provided by an embodiment of the present invention;

[0035] Figure 9 A schematic diagram of a test feedback in which the feedback content provided in an embodiment of the present invention is sunlight;

[0036] Figure 10 A schematic diagram of a training effect trend curve provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention. That is, the embodiments described herein are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in various different configurations.

[0038] It should be noted that relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "includes..." do not exclude the presence of other identical elements in the process, method, article, or device that includes the elements.

[0039] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0040] Example 1

[0041] like Figure 1 As shown, this embodiment provides a multimodal emotion regulation neurofeedback training system 100 , including a pre-training subsystem 201 and an online feedback subsystem 202 .

[0042] The pre-training subsystem 201 is used to filter and construct key information from a wide range of cognitive activity data during a user's performance of a specific cognitive activity, helping to distinguish different emotion regulation activity patterns. In a preferred embodiment, the pre-training subsystem 201 effectively extracts brain neural activity information by precisely selecting EEG signals and blood oxygenation signals that are closely related to the target cognitive state.

[0043] The online feedback subsystem 202 is used to collect neural activity data from the user during specific cognitive activities and analyze the collected data in real time for activity patterns with different characteristics. Simultaneously, the online feedback subsystem 202 provides visual feedback to the user on the activity patterns, allowing the user to intuitively observe the real-time status and changes of brain activity.

[0044] Furthermore, the pre-training subsystem 201 includes an adjustment test module 301 , a first data acquisition module 302 , and a first data analysis module 303 .

[0045] The regulation test module 301 is used to perform near infrared and electrical brain stimulation operations based on a preset emotion regulation paradigm.

[0046] The first data acquisition module 302 is used to collect behavioral, near-infrared, and EEG data from the subject, monitoring and recording the user's neural activity signals while performing specific cognitive activities. In a preferred embodiment, the neural activity signals are omnidirectional EEG signals and blood oxygenation signals (HBO) acquired using relevant signal acquisition devices. In the pre-training subsystem 201, the signals collected by the first data acquisition module 302 are the EEG signals and HBO signals during the user's specific cognitive activity phase based on a preset emotion regulation paradigm. To distinguish them from other signals, the present invention uniformly describes the signals collected by the first data acquisition module 302 as first signal data comprising a first EEG signal and a first HBO signal.

[0047] The first data analysis module 303 is used to pre-process and extract features from the collected first signal data, and to use it for further data analysis and training. The first data analysis module 303 is divided into a first pre-processing unit 401 , a first feature extraction unit 402 and a pattern training unit 403 .

[0048] The first preprocessing unit 401 preprocesses the collected first EEG signal and first blood oxygen signal and sends the high-quality signals obtained after preprocessing to the subsequent processing unit. The first preprocessing unit 401 preprocesses the collected first signal data, including filtering, denoising, signal amplification, and data normalization, to improve the quality of the data and the accuracy of the data analysis.

[0049] The first feature extraction unit 402 uses a multivariate pattern analysis method to extract complex feature patterns related to specific cognitive activity states or task conditions from the preprocessed first signal data. This design enables the system to go beyond identifying single or simple signal differences and instead analyze and utilize holistic patterns of cognitive activity, thereby providing a more comprehensive understanding of how the brain functions under different psychological states.

[0050] The pattern training unit 403 is used to train a classifier for binary activity patterns using the extracted features. The binary activity patterns include two categories: efficient emotion regulation patterns (negative emotions are greatly reduced after emotion regulation) and inefficient emotion regulation patterns (negative emotions are less reduced after emotion regulation).

[0051] Furthermore, the online feedback subsystem 202 includes a second data collection module 304 , a second data analysis module 305 and an online feedback presentation module 306 .

[0052] The second data acquisition module 304, similar to the first data acquisition module 302, collects signals related to the user's activities. However, the second data acquisition module 304 collects the user's behavior, blood oxygenation signal, and electroencephalogram (EEG) signal during the emotion regulation phase. Accordingly, the present invention generally describes the second data acquisition module 304 as collecting second signal data including the second EEG signal and the second blood oxygenation signal.

[0053] The second data analysis module 305 is used to perform real-time analysis on the collected second signal data to determine the activity mode category of the current cognitive state. Further, the second data analysis module 305 includes a second pre-processing unit 404, a second feature extraction unit 405 and a state classification unit 406.

[0054] Specifically, the second preprocessing unit 404 preprocesses the collected second signal data using signal processing techniques, including filtering, denoising, signal amplification, and data normalization, to improve data quality and analysis accuracy. The second feature extraction unit 405 is responsible for extracting complex feature patterns associated with specific cognitive activity states or task conditions from the preprocessed second signal data using multivariate pattern analysis techniques. The state classification unit 406 is used to input the extracted features into a trained binary activity pattern classifier to complete the classification of the current emotion regulation state.

[0055] The online feedback presentation module 306 is used to convert the classified feedback data that has undergone real-time analysis into intuitive visual real-time feedback, and to display the immediate changes in neural activity through dynamic graphics and an interactive interface, thereby helping to understand the relationship between the current activity pattern and a specific psychological state or task performance. In addition, the online feedback presentation module 306 is also used to provide visual feedback of the training effects of the neurofeedback training to the subjects, by providing trend curves and / or text output of the training effects, intuitively presenting the cumulative changes, stability, and other effects of the training effects, thereby helping to identify effective strategies and optimize the training direction.

[0056] Example 2

[0057] The embodiment of the present invention further describes in detail the data acquisition mechanisms of the first data acquisition module 302 and the second data acquisition module 304 .

[0058] In a preferred embodiment, the two data acquisition modules are head-wearable data acquisition devices. Furthermore, the head-wearable data acquisition device includes at least a wireless EEG acquisition system for collecting brain neuronal electrical activity signals and a multi-channel near-infrared spectroscopy (fNIRS) system for collecting light intensity data. The two data acquisition modules transmit the collected blood oxygenation signals representing brain blood oxygenation activity and EEG signals representing brain neural electrical activity to a server for data analysis or feedback presentation.

[0059] The fNIRS system reflects brain activity by measuring changes in the concentrations of oxyhemoglobin and deoxyhemoglobin in the cerebral cortex. Specifically, the fNIRS system transmits near-infrared light through the scalp and skull into the brain. When the infrared light is absorbed by hemoglobin, a detector in the fNIRS system receives the reflected light and calculates changes in blood oxygen concentration.

[0060] The wireless EEG acquisition system primarily collects electrophysiological signals from the brain. The principle is that when brain neurons activate, electrical signals are transmitted through synapses. The wireless EEG acquisition system captures tiny current changes through electrodes in contact with the scalp, indirectly reflecting the activity of specific brain regions in various cognitive and emotional states.

[0061] The head wearable data acquisition device and the online feedback server can be connected via wired or wireless means. As a preferred embodiment, the electrode placement of the head wearable data acquisition device refers to the international EEG 10-10 system, and the near-infrared electrodes are distributed in the frontal lobe, parietal lobe and occipital lobe. Figure 2 shown.

[0062] Example 3

[0063] like Figure 3 As shown, the embodiment of the present invention describes in detail a method for performing neurofeedback training based on the aforementioned multimodal emotion regulation neurofeedback training system 100. The neurofeedback training method includes the following steps:

[0064] S1, setting parameters of the pre-training subsystem 201, thereby controlling the pre-training subsystem 201.

[0065] As a preferred embodiment, the parameters of the pre-training subsystem 201 include at least: 1. Content selection; as a preferred embodiment, the content includes social pain or general negative emotions. 2. The number of test blocks (block num), that is, the number of blocks that each test block will contain; for example, the default parameter value of the test block number is 30, which means that 30 test blocks will be run. 3. Test block duration (block duration), that is, the time each test block will last, for example, the default parameter value of the test block duration is 30, which means that the rest and test phases will be carried out alternately, and each phase will last for 30 seconds. 4. Model selection, that is, the classifier model used by the pre-training subsystem 201, which can be regularized linear discriminant analysis (LDA) or support vector machine (SVM), and the default is regularized linear discriminant analysis LDA.

[0066] S2, setting the emotion regulation paradigm of the regulation test module 301, and performing specific cognitive activities based on the preset emotion regulation paradigm, while using the first data acquisition module 302 to detect the first signal data corresponding to the stage of performing the specific cognitive activity, to obtain the user's emotion regulation performance test data.

[0067] S2.1, setting the emotion regulation paradigm of the regulation test module 301, and performing specific cognitive activities based on the preset emotion regulation paradigm.

[0068] After the relevant parameters of the pre-training subsystem 201 are set, the pre-training subsystem 201 can be started. In order to obtain reference data for training, the user's emotion regulation performance is first tested using the regulation test module 301. Figure 4 Taking the classic emotion regulation paradigm shown as an example, the regulation test module 301 tests the user's emotion regulation performance according to the set emotion regulation paradigm and the parameters of the pre-training subsystem 201: the testing process of a complete test block includes 8 seconds of blank display, 6 seconds of negative picture viewing, 5 seconds of negative emotion scoring, 6 seconds of emotion regulation, and another 5 seconds of negative emotion scoring.

[0069] In a preferred embodiment, the negative images used in the emotion regulation paradigm are mainly divided into three categories: trauma and crisis events, interpersonal conflict events, and social and environmental stress events. The same number of negative images are selected for each category.

[0070] For example, if the default parameters are 30 blocks and 30 seconds, the emotion regulation task will be conducted in 30 blocks, each lasting 30 seconds. The process of each block is as follows: first, the user views a negative picture, naturally experiences the feelings brought about by the picture, and initially rates their negative emotions (1 points for the lowest intensity of negative emotions, 5 points for moderate intensity of negative emotions, and 9 points for the highest intensity of negative emotions); then, the user regulates the picture to change their emotions; and finally, the user rates their negative emotions again (1 points for very easy emotion regulation, 5 points for moderate difficulty in emotion regulation, and 9 points for very difficult emotion regulation).

[0071] The test data were processed in two ways, and the two processed data were used for subsequent classifier training and emotion regulation subject selection respectively.

[0072] Specifically, on the one hand, the task score was calculated based on the negative emotion scores of the 30 blocks. The calculation method was to subtract the negative emotion score after emotion regulation from the initial negative emotion score before emotion regulation. The task scores of the 30 blocks in each round were sorted from high to low, and the first 33% and the last 33% of the blocks were divided into efficient emotion regulation blocks (negative emotions were greatly reduced after emotion regulation) and inefficient emotion regulation blocks (negative emotions were less reduced after emotion regulation). The screened efficient emotion regulation blocks and inefficient emotion regulation data were used as reference data for the two modes of efficient emotion regulation and inefficient emotion regulation contained in the binary classification activity pattern classifier, and were used for classifier training.

[0073] On the other hand, the 30 blocks were divided into three categories: trauma and crisis events, interpersonal conflict events, and social and environmental stress events. Task scores were calculated for each category using the same calculation method: the initial negative emotion score before emotion regulation minus the negative emotion score after emotion regulation. The task scores for the three categories were sorted from high to low, and the category with the lowest score was selected as the negative image category for the subsequent emotion regulation target in the online feedback subsystem 202. The lowest task score for a category indicates that effective emotion regulation was not possible for that negative event.

[0074] S2.2, while the user performs a specific cognitive activity, the first data acquisition module 302 is used to acquire the first signal data of this stage, including the first EEG signal and the first blood oxygen signal.

[0075] S3 , using the first data analysis module 303 to pre-process and extract features from the collected first signal data, and using the extracted features to train a classifier for binary classification of activity patterns.

[0076] S3.1, first preprocessing the collected first signal data by the first preprocessing unit 401 to obtain the first signal data after noise removal.

[0077] Specifically, the first preprocessing unit 401 preprocesses the collected first EEG signal to remove noise such as non-EEG interference signals contained therein to obtain a high-quality EEG signal. The steps of preprocessing the first EEG signal may include:

[0078] S3.1.1, use FIR (Finite Impulse Response) filter to remove high-frequency noise and low-frequency noise from EEG signals;

[0079] The FIR filter can be implemented as a bandpass filter (FIR bandpass filter) to remove high-frequency noise and low-frequency noise in the EEG signal. The FIR filter can be implemented in hardware or software, which is not limited in the embodiments of the present invention.

[0080] S3.1.2, using a differential algorithm to remove eye artifacts from EEG signals;

[0081] Since the order of magnitude of the electrooculogram (EOG) signal is much greater than that of the electroencephalogram (EEG), it is the most important source of contamination in EEG signals (especially frontal lobe EEG signals). For single-channel EEG signals, removing EOG artifacts (contamination) is particularly important. Using a differential algorithm module to remove EOG artifacts is one method of EEG signal preprocessing. The differential algorithm module can be implemented in hardware or software, which is not limited in the embodiments of the present invention.

[0082] S3.1.3, use motion sensors to remove head movement artifacts from EEG signals.

[0083] Another source of contamination in EEG signals is the myoelectric signals generated by muscle movement and signal drift caused by head movement, collectively referred to as head motion artifacts. To remove head motion artifacts, the motion sensor built into the EEG recording device can be used to record the head motion signal. The muscle movement signals (myoelectric signals) convoluted within the EEG signal can then be stripped away to extract a cleaner EEG signal. As a preferred embodiment, the motion sensor can be a multi-spiral motion sensor.

[0084] After removing various non-EEG interference signals through the above steps, a high-quality single-channel EEG signal can be obtained. In some optional embodiments, the preprocessing step of the EEG signal can be assisted by two left and right reference electrodes, by obtaining the signal collected by the reference electrode as the reference signal of the EEG signal, and preprocessing the EEG signal based on the reference signal. Exemplarily, the reference signal can be used as a reference signal recording point for removing electrooculogram artifacts during the processing of the differential calculation module. By using the reference signal collected by the reference electrode in the preprocessing, the effect of the preprocessing can be improved.

[0085] On the other hand, the preprocessing of the first blood oxygen signal is mainly to remove the light scattering noise caused by the scalp and skull to improve the quality of the blood oxygen signal. The steps of preprocessing the first blood oxygen signal may include:

[0086] S3.1.4, convert the collected light intensity data into optical density data;

[0087] S3.1.5, uses the time derivative distribution restoration method to correct artifacts caused by user head motion;

[0088] S3.1.6, use FIR (Finite Impulse Response) filter to filter out high-frequency noise and low-frequency noise in the optical density signal data.

[0089] S3.1.7. Convert the optical density data to relative changes in oxyhemoglobin and deoxyhemoglobin concentrations according to the modified Beer-Lambert law.

[0090] It should be noted that the preprocessing of the first EEG signal and the preprocessing of the first blood oxygen signal can be performed synchronously or asynchronously, and the aforementioned sequence of steps is only used to more clearly describe the processing method.

[0091] S3.2, the first feature extraction unit 402 performs signal feature extraction calculation on the received second EEG signal and the second blood oxygen signal collected and preprocessed under a certain emotion regulation paradigm.

[0092] In the disclosed embodiments, the characteristic indicators for signal extraction may include activation of various brain regions, brain connectivity strength, ERP components, and power spectrum characteristics. In a preferred embodiment, the characteristic indicators include one or more of the following features: the strength of the internal and external brain functional connectivity of the right frontal lobe / left frontal lobe / right parietal lobe / left parietal lobe and / or the blood oxygen activation level, and the energy of delta (1-4 Hz), theta (4-8 Hz), alpha (8-12 Hz), beta (12-30 Hz), and gamma (30-55 Hz).

[0093] The following is a brief introduction to the above characteristic indicators:

[0094] For blood oxygen signals, the strength of functional connections between the brain and the outside world and the blood oxygen activation levels of each brain region will change significantly under different activity patterns. For example, the right and left frontal lobes are mainly responsible for high-level cognitive functions and emotional processing, and they show specific activation and connection changes during task execution and emotional reactions. The right and left occipital lobes are mainly responsible for visual information processing, and these areas will be significantly activated when exposed to visual stimuli. The right and left parietal lobes are involved in spatial positioning and attention processing. When performing attention-focusing tasks, the activation and connection strength of these areas will change.

[0095] To quantify brain activation levels and connection strength, the present invention uses the following formula to express the relationship between blood oxygen changes and brain activation:

[0096] The formula for calculating the change in blood oxygen activation level is as follows:

[0097]

[0098] Among them, S t is the spectral signal intensity at a specific time point (usually measured by fNIRS), S baseline is the baseline signal intensity, and t represents time (the same below).

[0099] The connection strength of brain regions can also be analyzed by similar methods, and the activity changes of brain networks can be evaluated by calculating the functional connection strength between different brain regions. Figure 5 As shown, the extracted brain connection strength features are as follows Figure 6 By extracting and analyzing these features, we can gain a deeper understanding of the functional performance of each brain region in different tasks and the interactions between them.

[0100] Brainwaves in different frequency bands are associated with specific cognitive functions and brain activity. When performing tasks, specific areas of the brain respond to these frequency bands, especially when processing visual and auditory information. As the difficulty of the task increases, the activity level of these frequency bands also changes. Therefore, extracting the characteristics of different frequency bands is crucial for understanding brain activity. Common frequency bands include: Delta waves (1-4Hz), which are associated with deep sleep and rest; Theta waves (4-8Hz), which are associated with learning and memory; Alpha waves (8-12Hz), which are associated with relaxation and attention; Beta waves (12-30Hz), which are associated with concentration and cognitive processing; Gamma waves (30-55Hz), which are associated with higher-level cognitive functions.

[0101] In order to extract the frequency band characteristics of EEG signals, time domain signals are usually converted into frequency domain signals. Therefore, the short-time Fourier transform (STFT) method can be used. Through STFT, the signal conversion between time domain and frequency domain can be realized, thereby calculating the power spectrum density (PSD) of each frequency band. After obtaining the spectral density, the energy of a specific frequency band is further calculated, and the standardized energy is obtained through normalization. The schematic diagram of the extracted power spectrum density is shown in the figure below. Figure 7 shown.

[0102] The short-time Fourier transform formula used is as follows:

[0103]

[0104] Where X(t,f) is the discrete sample of EEG signal, w(mt) is the window function, e -j2πfn is the complex exponential function, and m represents the sequence.

[0105] The power spectral density (PSD) is calculated as follows:

[0106] P(f)=|X(t,f)| 2 (4)

[0107] Where P(f) represents the spectral density, which is the square of the amplitude of the complex number X(t,f), reflecting the energy distribution of the signal at different frequencies.

[0108] S3.3. After the signal feature extraction and calculation are completed, the pre-training unit 403 of the first data analysis module 303 further performs multivariate pattern analysis on the signal features, identifies and classifies the signal features into two modes: efficient emotion regulation and inefficient emotion regulation, and uses them to train a classifier for binary activity patterns. Specifically, the steps include:

[0109] S3.3.1, perform dimensionality reduction and normalization on the extracted signal features.

[0110] Before PCA dimensionality reduction, data normalization is performed. Because the various features of EEG and blood oxygen signal data may have different units and magnitudes, the data must be normalized before PCA. The purpose of normalization is to ensure that each feature has the same scale, thus preventing some features with larger dimensions from unduly affecting the PCA results.

[0111] The normalization formula used is the following:

[0112]

[0113] Among them, X normis the original data, μ is the mean of the data, and σ is the standard deviation.

[0114] S3.3.2, Calculate the covariance matrix of the standardized data.

[0115] Next, perform eigenvalue decomposition on the covariance matrix. By decomposing the covariance matrix, eigenvalues and eigenvectors can be obtained. The eigenvectors represent the new coordinate axis directions (principal components), while the eigenvalues represent the variance of each principal component. The larger the variance, the stronger the principal component's ability to explain data changes. By sorting the eigenvalues in descending order, the first few principal components with the largest variance are selected. These principal components will contain the most important information about changes in the data. The number of principal components selected is usually determined by the cumulative variance contribution rate, and the principal components that can explain most of the variance are selected.

[0116] As a preferred embodiment, a condition threshold for selecting the principal component may be set, for example, the principal component with the top 90% cumulative variance contribution rate may be selected.

[0117] S3.3.3, perform multivariate pattern analysis and classification, including data set partitioning, validation, and model evaluation.

[0118] Multivariate pattern analysis is used to extract patterns associated with specific psychological states or tasks from high-dimensional data. During this process, the extracted features are first analyzed and classified using a linear discriminant analysis (LDA) classifier. LDA achieves optimal classification by maximizing between-class variance and minimizing within-class variance, and is suitable for linearly separable feature spaces.

[0119] In the specific operation, the preprocessed feature dataset is first divided into a training set and a test set. The training set is used to train the machine learning model, while the test set is used to evaluate the performance of the model. In order to ensure the stability and generalization ability of the model, a 5-fold cross-validation method is used for verification. The cross-validation process randomly divides the dataset into five subsets, selects four subsets for training each time, and the remaining subset is used for testing. The process is repeated five times, and finally the average performance index is calculated to ensure the robustness and effectiveness of the model on different data subsets. After the training is completed, the test set is used to verify the model and evaluate its classification performance. The optimal model parameters are automatically selected to achieve the best classification effect, especially under the conditions of efficient emotion regulation and inefficient emotion regulation, to verify the recognition and classification capabilities of the model. In this way, it is ensured that the model can maintain efficient classification performance in different psychological states and task situations. The calculated final model accuracy distribution is as follows Figure 8 shown.

[0120] After training is complete, the trained binary activity pattern classifier model is applied to the subsequent prior feedback subsystem 202. This subsystem enables accurate interpretation of EEG and blood oxygen signals, supporting the identification and analysis of specific psychological states or tasks. In this way, the emotion regulation neurofeedback training system can effectively distinguish and identify two distinct activity patterns: effective and ineffective emotion regulation.

[0121] S4, setting parameters of the online feedback subsystem 202, thereby controlling the entire online feedback subsystem 202.

[0122] The embodiment of the present invention further describes in detail the functions and operation methods implemented based on the aforementioned online feedback subsystem 202.

[0123] The online feedback subsystem 202 is mainly responsible for real-time monitoring and analysis of the user's activity patterns, mainly based on the second data analysis module 305 extracting key information from the current activity signal collected by the second data acquisition module 304, and calculating the activity patterns describing different psychological states or cognitive tasks, and then converting these activity patterns into intuitive visual feedback, such as displaying them on the user interface through color changes, shape dynamics or graphic changes, and providing feedback on the training effect.

[0124] In the embodiment of the present invention, the user can operate the parameters of the online feedback subsystem 202, thereby controlling the entire online feedback subsystem 202. After all parameters are set, the pre-training subsystem can start running.

[0125] As a preferred embodiment, the parameters of the online feedback subsystem 202 include: 1. The selection of a training scheme, which refers to the visual scene and visual feedback content used in neurofeedback training. For example, sunlight or a thermometer can be selected, with sunlight being the default. 2. The number of training blocks (training block num), which refers to the number of blocks each training block will contain. For example, the default parameter of 30 means that 30 training blocks will be run. 3. The duration of the training block (training block duration), which refers to the duration of each training block. For example, the default parameter of 60 means that rest and training phases will be interleaved, and each phase will last for 60 seconds.

[0126] S5, selecting an emotion regulation object, and performing an emotion regulation activity based on the selected emotion regulation object. Simultaneously, the second data acquisition module 304 acquires second signal data corresponding to the current emotion regulation activity, and obtains real-time emotion regulation activity data of the user.

[0127] In a preferred embodiment, the emotion regulation target for each training block is a negative image of the type of event that has the lowest task score in the emotion regulation paradigm. In one embodiment, the presentation time of a single negative image during the emotion regulation activity is 10 seconds, and multiple negative images can be used in the same training block.

[0128] After the relevant parameters of the online feedback subsystem 202 are set, the online feedback subsystem 202 can start running.

[0129] The second signal data also includes a second EEG signal and a second blood oxygen signal.

[0130] Since the signals collected by the second data acquisition module 304 are output to the server in real time, the user should try to avoid physical movement during use to improve the signal-to-noise ratio, thereby ensuring the accuracy of the data and the effectiveness of the system. The server performs real-time online analysis on the real-time output signals.

[0131] S6 , using the second data analysis module 305 to pre-process and extract features from the collected second signal data, and using the extracted features to identify the activity pattern.

[0132] S6.1. After receiving the second signal data collected by the second data acquisition module 304, the second data analysis module 305 also performs signal preprocessing and feature extraction by the second preprocessing unit 403 and the second feature extraction unit 404. The specific preprocessing method flow and feature extraction method flow can be similar to the data preprocessing and feature extraction operation process in the first data analysis module 303 to obtain the features corresponding to the user's real-time activity status.

[0133] S6.2, after that, the state classification unit 405 uses the binary activity pattern classifier trained by the pre-training subsystem 201 to identify the activity pattern to which the current activity state belongs, thereby realizing real-time and accurate interpretation of the electroencephalogram (EEG) signal and the blood oxygenation signal (HBO), and supporting the recognition and analysis of the current activity pattern.

[0134] The binary activity pattern classifier outputs a ratio metric, f, which represents a real-time score of the current activity pattern after analysis. This score reflects the degree of match between the current brain activity pattern and the emotion regulation pattern. A higher f-score indicates a greater similarity between the current activity pattern and the effective emotion regulation pattern; a lower f-score indicates a greater similarity between the current activity pattern and the ineffective emotion regulation pattern.

[0135] The method for identifying the activity mode to which the current activity state belongs is specifically as follows:

[0136] S6.2.1, first calculate the joint matrix Σ of the two activity modesS :

[0137]

[0138]

[0139] Among them, Category1 is category 1, which represents efficient emotion regulation, Category2 is category 2, which represents inefficient emotion regulation, μ1 and μ2 are the sample means of category 1 and category 2, respectively, n represents the dimension of the data, x represents the eigenvector of the input signal, i represents the i-th sample value, and T represents the transpose of the eigenvector.

[0140] S6.2.2, Next, calculate the probability density function p of the two activity patterns 1(x) and p 2(x) :

[0141]

[0142] Among them, Σ S is the joint matrix.

[0143] S6.2.3, Finally, to evaluate the similarity between the current active pattern and the two classified patterns, use the following ratio metric f as output:

[0144]

[0145] Among them, p 1(x) and p 2(x) are the probability density functions of category 1 and category 2 respectively.

[0146] Thus, the embodiments of the present invention can effectively distinguish and identify efficient and inefficient emotion regulation activity patterns, and the multivariate pattern analysis parameter output by the classifier is a real-time score that reflects the degree of match between the current brain activity pattern and the emotion regulation activity pattern. Specifically, the ratio metric f reflects the relative similarity between the current activity pattern and the two mode states. A higher f-score indicates a greater similarity between the current activity pattern and the efficient emotion regulation activity pattern; a lower f-score indicates a greater similarity between the current activity pattern and the inefficient emotion regulation pattern.

[0147] S7, the identified activity pattern analysis parameters and feedback training effect are fed back online in real time using the online feedback presentation module 306.

[0148] After calculating and determining the mode category to which the current activity mode belongs, the online feedback presentation module 306 further displays real-time feedback based on the obtained multivariate mode analysis parameters. The online feedback presentation module 306 converts the obtained multivariate mode analysis parameters into intuitive visual feedback and displays it on the user interface, for example, through color changes, dynamic shape changes, or graphical changes. In a preferred embodiment, real-time data transmission and analysis can be performed every second, and the feedback presentation can be updated.

[0149] The online feedback presentation module 306 is a user interface that converts the pattern category information identified by the second data analysis module 305 into visual feedback. This neurofeedback presentation module 306 generates dynamic visual images or sound signals in real time based on the user's real-time multivariate pattern analysis parameters. This feedback is presented in a gamified or storytelling format, increasing the fun of training and user engagement. For example, if the user prefers effective emotion regulation, the weather on the feedback interface will gradually turn sunny; if the user prefers ineffective emotion regulation, the weather will gradually turn cloudy.

[0150] The calculation method for real-time feedback of the activity pattern analysis parameters after recognition is as follows:

[0151]

[0152] Among them, f is the real-time score obtained by the previous classification step. Therefore, the feedback value F is adjusted according to the range of the score f: when the score f is less than or equal to -2, the feedback value F is -2; when the score is between -2 and 2, the feedback value F is the current score f; when the score is greater than or equal to 2, the feedback value F is 2.

[0153] As an example, the corresponding relationship between the feedback value F and the feedback content is as follows: Figure 9 shown.

[0154] By controlling the intensity and type of feedback in this way, the system can automatically adjust feedback parameters such as the intensity, type and frequency of feedback based on the user's training history and progress to maximize training effects and ensure that users can intuitively understand the status of their activity patterns and adjust their behavior in a timely manner.

[0155] In addition, the system also supports multi-user mode, so that each user can have their own customized settings in a multi-person environment such as home or training center.

[0156] In addition, the online feedback presentation module 306 can also calculate the training effect of the neurofeedback training and provide real-time feedback.

[0157] After completing the neurofeedback training, the training effect of the subjects was calculated as follows:

[0158]

[0159] Among them, Block i Represents the block number, Represents the mean value of the block number. Activity i represents the f value of the i-th block, represents the mean f-value of all blocks. β represents the training effect during neurofeedback training, that is, the degree of learning of effective emotion regulation patterns.

[0160] As a preferred embodiment, you can use Actvity i Draw a trend curve, such as Figure 10 As shown. Visual feedback can also be provided on the trend curve and training results. When the value of β is greater than 0, the feedback text is "Training Successful"; when the value of β is less than 0, the feedback text is "Training Failed".

[0161] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the corresponding steps in the above-mentioned operating method embodiment, and will not be repeated here.

[0162] Professionals may further appreciate that the modules or units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0163] In summary, this paper proposes a multimodal emotion regulation neurofeedback training system and method. Through real-time, dynamic feedback mechanisms and personalized training plans, users can effectively train for specific states or situations, significantly improving training efficiency and user experience. This innovative training tool provides strong support for cognitive science and personal health management.

[0164] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be conceived by a person skilled in the art within the technical scope disclosed by the present invention without inventive effort should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection defined in the claims.

Claims

1. A multimodal emotion regulation neurofeedback training system, characterized in that: The multimodal emotion regulation neurofeedback training system includes a pre-training subsystem (201) and an online feedback subsystem (202); The pre-training subsystem (201) is used to screen and extract signal feature information that distinguishes different emotion regulation activity patterns from neural activity signals during the execution of a specific cognitive activity phase, and complete classifier training for the binary classification activity pattern; The online feedback subsystem (202) is used to collect the user's neural activity data during the execution of the emotion regulation activity, and to analyze the pattern category of the current neural activity in real time based on the collected data, and to present the neural activity feedback data in real time.

2. A multimodal emotion regulation neurofeedback training system according to claim 1, characterized in that: The pre-training subsystem 201 includes an adjustment test module (301), a first data acquisition module (302), and a first data analysis module (303); The regulation test module (301) is used to perform near-infrared and electroencephalographic stimulation operations of a specific cognitive activity based on a preset emotion regulation paradigm; the first data acquisition module (302) is used to collect first signal data of a specific cognitive activity stage; the first data analysis module (303) is used to preprocess and extract features from the first signal data, and use the extracted feature data for classifier training of a binary activity pattern and selection of emotion regulation subjects in the emotion regulation activity stage.

3. A multimodal emotion regulation neurofeedback training system according to claim 2, characterized in that: The online feedback subsystem (202) includes a second data acquisition module (304), a second data analysis module (305) and an online feedback presentation module (306); The second data acquisition module (304) is used to collect the second signal data of the emotion regulation activity stage; the second data analysis module (305) is used to analyze the second signal data in real time and use a binary activity pattern classifier to identify the neural activity pattern category to which the current neural activity belongs; the neural feedback presentation module (306) is used to convert the recognition results and training effects into intuitive visual feedback, thereby displaying the immediate changes in neural activity.

4. A multimodal emotion regulation neurofeedback training system according to claim 2 or 3, characterized in that: The first signal data includes a first EEG signal and a first blood oxygen signal, and the second signal data includes a second EEG signal and a second blood oxygen signal.

5. A multimodal emotion regulation neurofeedback training system according to claim 2 or 3, characterized in that: The binary activity pattern classifier includes two categories of neural activity patterns: efficient emotion regulation pattern and inefficient emotion regulation pattern.

6. A multimodal emotion regulation neurofeedback training method, characterized in that: The method is implemented based on a multimodal emotion regulation neurofeedback training system according to any one of claims 1 to 5, and the method comprises: S1, initialization, setting the parameters of the pre-training subsystem (201); S2, setting an emotion regulation paradigm of the regulation test module (301), and performing a specific cognitive activity based on the preset emotion regulation paradigm, while using the first data acquisition module (302) to collect first signal data corresponding to the execution of the specific cognitive activity, to obtain corresponding emotion regulation performance test data; S3, using the first data analysis module (303) to preprocess and extract features from the collected first signal data, using the extracted features to train a classifier for binary activity patterns, and using the image category with the lowest test score as the category of the emotion regulation object in the emotion regulation activity stage; S4, setting parameters of the online feedback subsystem (202); S5, selecting an emotion regulation object according to the category of the emotion regulation object determined in step S3, and performing an emotion regulation activity based on the selected emotion regulation object, while simultaneously using a second data acquisition module (304) to collect second signal data corresponding to the execution of the emotion regulation activity, to obtain corresponding emotion regulation activity data; S6, using the second data analysis module (305) to pre-process and extract features from the collected second signal data, and using the extracted features to identify the neural activity pattern category through a trained binary activity pattern classifier.

7. The multimodal emotion regulation neurofeedback training method according to claim 6, characterized in that: The method further includes S7, providing online real-time feedback of the recognition result of the neural activity pattern category and the neural feedback training effect using a neural feedback presentation module (306).

8. The multimodal emotion regulation neurofeedback training method according to claim 6, characterized in that: The step S6 specifically includes: S6.1, the second preprocessing unit (403) and the second feature extraction unit (404) of the second data analysis module (305) respectively perform signal preprocessing and feature extraction on the received second signal data; S6.2, the state classification unit (405) uses a trained binary activity pattern classifier to analyze the extracted signal features, thereby identifying the neural activity pattern category to which the current activity state belongs.

9. The multimodal emotion regulation neurofeedback training method according to claim 8, characterized in that: The method for identifying the neural activity pattern to which the current activity state belongs in step S6.2 is specifically as follows: S6.2.1, calculate the joint matrix Σ of the two activity patterns S : Where Category1 is Category 1, which indicates efficient emotion regulation; Category2 is Category 2, which indicates inefficient emotion regulation; μ1 and μ2 are the sample means of Category 1 and Category 2, respectively; n represents the dimension of the data; x represents the eigenvector of the input signal; i represents the i-th sample value; and T represents the transpose of the eigenvector. S6.2.2, calculate the probability density function p of the two activity patterns 1(x) and p 2(x) : Among them, Σ S is the joint matrix; S6.2.3, calculate a ratio metric f that assesses the similarity between the current activity pattern and the two classification patterns: Among them, p 1(x) and p 2(x) are the probability density functions of category 1 and category 2 respectively; The ratio metric f reflects the relative similarity between the current activity pattern and the two pattern categories. The higher the f-value score, the more similar the current activity pattern is to the efficient emotion regulation activity pattern; the lower the f-value score, the more similar the current activity pattern is to the inefficient emotion regulation pattern.

10. The multimodal emotion regulation neurofeedback training method according to claim 7, characterized in that: The calculation method for providing online real-time feedback of the recognition result of the neural activity pattern category in step S7 is as follows: Where f is the ratio metric and F is the feedback value: when the score f is less than or equal to -2, the feedback value F is -2; when the score is between -2 and 2, the feedback value F is the current score f; when the score is greater than or equal to 2, the feedback value F is 2; The calculation method for online real-time feedback of neurofeedback training effect is as follows: Among them, Block i Represents the block number, Represents the mean value of the block number, Activity i represents the f value of the i-th block, represents the mean f-value of all blocks, and β represents the training effect during neurofeedback training, that is, the degree of learning of effective emotion regulation patterns.

Images