EEG-fNIRS coupling brain network construction method and system

Abstract

The invention discloses an EEG-fNIRS coupling brain network construction method and system, and relates to the technical field of emotion recognition, and the method comprises the steps: obtaining an EEG signal and an fNIRS signal; respectively preprocessing the EEG signal and the fNIRS signal to obtain a preprocessed EEG signal and a preprocessed fNIRS signal; the EEG-fNIRS coupling strength is calculated, and an EEG-fNIRS coupling strength matrix is obtained; according to the EEG-fNIRS coupling intensity matrix, an EEG-fNIRS coupling brain network is constructed and obtained; the EEG-fNIRS coupling brain network serves as a new emotion recognition feature and is used for being input into an emotion recognition model to obtain an emotion recognition result. According to the method, the multi-modal signal brain network suitable for the emotion induction experiment normal form can be effectively constructed, and the emotion recognition performance can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition technology, and in particular to a method and system for constructing an EEG-fNIRS coupled brain network. Background Art

[0002] Emotion is the result of the coordinated activities of the cerebral cortex and subcortical nervous system triggered by external stimuli. Among the physiological signals related to brain activity, EEG (Electroencephalogram) and fNIRS (functional Near-Infrared Spectroscopy) signals are widely used in emotion recognition research due to their advantages of low acquisition cost, good mobility, and relatively simple testing process. EEG records the voltage fluctuations generated by brain activity through electrodes placed on the surface of the scalp, directly reflecting the electrophysiological activities of neurons in the cerebral cortex, and has the characteristics of high temporal resolution, non-invasiveness, and low cost. fNIRS uses the characteristics of different absorption rates of near-infrared light of 650nm-950nm wavelength in local brain tissues, and measures the changes in the concentrations of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) related to brain activity through optical couplers (transmitters and receivers) placed on the surface of the scalp, reflecting the hemodynamic activities of the cerebral cortex, with high spatial resolution and good anti-interference. EEG and fNIRS are not only complementary in temporal and spatial resolution, but the two signals also reflect different aspects related to brain neuronal activity and can provide complementary information. Therefore, making full use of their complementary advantages is of great significance to further improve emotion recognition performance and reveal the working mechanism of brain emotion cognition.

[0003] With the in-depth integration of complex networks and neuroscience, brain networks constructed based on multimodal physiological signals can describe the interaction and coordination mechanism between brain regions and study the cognitive function of the brain from a macroscopic level. Given the complementary advantages of EEG and fNIRS signals in reflecting brain neuronal activity, they can effectively overcome the limitations of single-modal signals. Therefore, exploring a new brain network construction method based on synchronous EEG-fNIRS (Electroencephalogram-functional Near-Infrared Spectroscopy) dual-modal signals is of great value and significance for further improving emotion recognition performance. Due to the large differences between EEG and fNIRS signals in terms of sampling rate, signal type, dimension, etc., the existing method of constructing brain networks based on EEG single-modal signals cannot be used to construct multimodal signal brain networks.

[0004] For synchronously collected EEG-fNIRS signals, the coupling strength calculation methods of existing studies are only applicable to experimental paradigms in which the experimental stimulation is short in duration and constant in intensity. For example, existing methods calculate the coupling strength of EEG-fNIRS collected during left-right hand motor imagery tasks. In tasks such as motor imagery, the experimental stimulation appears for a short time and the stimulation intensity remains constant. However, in the emotion-induced experimental paradigm, the experimental stimulation is usually presented in the form of audio or video clips, which lasts for a relatively long time and the content is constantly changing. Therefore, the existing methods are not applicable to the emotion-induced experimental paradigm and cannot guarantee the performance of emotion recognition.

[0005] Based on this, how to provide an experimental paradigm suitable for emotion induction and a method to effectively construct a multimodal signal brain network to effectively improve emotion recognition performance has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0006] The purpose of this application is to provide an EEG-fNIRS coupled brain network construction method and system, which can effectively construct a multimodal signal brain network suitable for the experimental paradigm of emotion induction, and can effectively improve the emotion recognition performance.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a method for constructing an EEG-fNIRS coupled brain network, the method comprising:

[0009] Acquire EEG signals and fNIRS signals; the EEG signals and the fNIRS signals are signals synchronously acquired by multiple acquisition channels.

[0010] The EEG signal and the fNIRS signal are preprocessed respectively to obtain a preprocessed EEG signal and a preprocessed fNIRS signal.

[0011] According to the preprocessed EEG signal and the preprocessed fNIRS signal, the EEG-fNIRS coupling strength is calculated to obtain an EEG-fNIRS coupling strength matrix.

[0012] According to the EEG-fNIRS coupling strength matrix, an EEG-fNIRS coupled brain network is constructed; the EEG-fNIRS coupled brain network is used as a new emotion recognition feature and is input into an emotion recognition model to obtain an emotion recognition result.

[0013] Optionally, preprocessing the EEG signal and the fNIRS signal respectively to obtain a preprocessed EEG signal and a preprocessed fNIRS signal specifically includes:

[0014] The EEG signal is subjected to re-reference processing, down-sampling processing, artifact removal processing and filtering processing in sequence to obtain the pre-processed EEG signal.

[0015] The fNIRS signal is filtered and artifacts are removed in sequence to obtain a preliminarily processed fNIRS signal.

[0016] According to the modified Beer-Lambert law, the preliminarily processed fNIRS signal is converted into an HbO concentration change signal, and upsampled to the current sampling frequency of the EEG signal to obtain the preprocessed fNIRS signal; the HbO concentration change signal is used to characterize the degree of activation of a local brain area.

[0017] Optionally, calculating the EEG-fNIRS coupling strength according to the preprocessed EEG signal and the preprocessed fNIRS signal to obtain an EEG-fNIRS coupling strength matrix specifically includes:

[0018] Perform short-time Fourier transform on each of the preprocessed EEG signals to obtain a time-frequency power spectrum of each of the preprocessed EEG signals.

[0019] The time-frequency power spectrum of each of the preprocessed EEG signals is power-superimposed in the frequency dimension to obtain the time-varying power of each of the preprocessed EEG signals.

[0020] The time-varying power of each pre-processed EEG signal is normalized and convolved with the HRF function to obtain the predicted fNIRS signals under several experimental stimuli.

[0021] Rectangular windows were used to segment samples of the preprocessed measured fNIRS signals and the predicted fNIRS signals under several experimental stimuli.

[0022] For each sample after sample segmentation, the predicted fNIRS signals under several experimental stimulations were used as the design matrix of the general linear model, and the preprocessed measured fNIRS signals were fitted to obtain the regression coefficient matrix.

[0023] The absolute value of the regression coefficient matrix is ​​calculated to obtain the EEG-fNIRS coupling strength matrix.

[0024] Optionally, the EEG-fNIRS coupling strength matrix is ​​represented by the following formula:

[0025]

[0026] Wherein, CS represents the EEG-fNIRS coupling strength matrix, cs nm represents the coupling strength between the nth EEG signal and the mth fNIRS signal.

[0027] Optionally, constructing an EEG-fNIRS coupled brain network according to the EEG-fNIRS coupling strength matrix specifically includes:

[0028] The i-th EEG signal and the j-th fNIRS signal in the EEG-fNIRS coupling strength matrix are respectively used as nodes, and the coupling strength cs corresponding to the i-th EEG signal and the j-th fNIRS signal is ij As an edge connecting two nodes, the EEG-fNIRS coupling strength matrix is ​​used as an adjacency matrix to construct the EEG-fNIRS coupled brain network.

[0029] Optionally, after the step of constructing an EEG-fNIRS coupled brain network according to the EEG-fNIRS coupling strength matrix, the EEG-fNIRS coupled brain network construction method further comprises:

[0030] The EEG-fNIRS coupled brain network is used to perform emotion recognition and obtain an emotion recognition result.

[0031] Optionally, using the EEG-fNIRS coupled brain network to perform emotion recognition to obtain an emotion recognition result specifically includes:

[0032] The EEG-fNIRS coupling strength matrix corresponding to the EEG-fNIRS coupled brain network is expanded row by row or column by column to obtain a feature vector for emotion recognition.

[0033] The feature vector is input into the emotion recognition model to obtain the emotion recognition result.

[0034] Optionally, the emotion recognition model is a SVM model.

[0035] In a second aspect, the present application provides an EEG-fNIRS coupled brain network construction system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the EEG-fNIRS coupled brain network construction method described in the first aspect.

[0036] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0037] The present application provides a method and system for constructing an EEG-fNIRS coupled brain network, based on synchronously collected EEG signals and fNIRS signals, using the characteristics of neurovascular coupling between EEG signals and fNIRS signals, modeling experimental stimuli under the emotion induction paradigm through EEG signals, and realizing the calculation of the coupling strength between EEG signals and fNIRS signals under the emotion induction paradigm, thereby forming an EEG-fNIRS coupling strength matrix based on EEG signals and fNIRS signals, and further constructing an EEG-fNIRS coupled brain network using the coupling strength between EEG signals and fNIRS signals, thereby achieving the purpose of constructing a multimodal signal brain network suitable for the experimental paradigm of emotion induction. In addition, the EEG-fNIRS coupled brain network can be used as a new emotion recognition feature in emotion recognition scenarios, and the corresponding emotion recognition results can be obtained by inputting it into any model with emotion recognition classification performance, which can effectively improve the emotion recognition performance and obtain more accurate and reliable emotion recognition classification results, which is of great significance for in-depth understanding of the brain neural activity mechanism under emotion tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a method for constructing an EEG-fNIRS coupled brain network provided in one embodiment of the present application.

[0040] Figure 2 This is a principle flow chart of a method for constructing an EEG-fNIRS coupled brain network provided in one embodiment of the present application.

[0041] Figure 3 A flowchart of EEG-fNIRS coupling strength matrix calculation provided in one embodiment of the present application.

[0042] Figure 4 A schematic diagram of the structure of an EEG-fNIRS coupled brain network constructed based on two EEG signals and two fNIRS signals provided in one embodiment of the present application.

[0043] Figure 5 A flowchart of an emotion recognition process provided for an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] At present, no method has been found in the relevant technology to construct a coupled brain network based on synchronously collected EEG and fNIRS signals under the experimental paradigm of emotion induction. The existing coupling strength calculation method is only applicable to the experimental paradigm with short stimulation duration and constant intensity. The specific implementation process is as follows: (1) Select 8 EEG channel signals that are highly relevant to the experimental task, average the channels, and obtain an averaged EEG signal. (2) Use short-term Fourier transform (STFT) to calculate the time-frequency power spectrum of the above average EEG signal, and perform power superposition at each time point from 0 to 2s to obtain the power change function of the signal in the first 2s. (3) Extract the peak and latency information in the above power change function, and use the square pulse signal carrying this information to model the experimental stimulus, which is called the Boxcar (rectangular window) function. (4) Convolve the Boxcar function with the HRF (Hemodynamic Response Function) function to obtain the predicted blood oxygen concentration change under the experimental stimulus, that is, the predicted fNIRS signal. (5) The predicted fNIRS signal obtained is used as the design matrix of the general linear model (General Linear Model, GLM) to fit the measured fNIRS signal. The fitting coefficient obtained is the coupling strength, which is used as a feature and sent to the classifier for classification. This application mainly uses the neurovascular coupling characteristics between EEG signals and fNIRS signals to construct an EEG-fNIRS coupled brain network. The so-called neurovascular coupling refers to the fact that brain neural activity is inevitably accompanied by fluctuations in cerebral blood flow. Specifically, when neurons are activated, blood flows to the area to meet the increased demand for glucose and oxygen, resulting in fluctuations in hemoglobin concentration, which can be detected by fNIRS signals. The above physiological process is called neurovascular coupling, which reflects the correlation between neuronal activity and hemodynamic changes in the brain, thereby laying the theoretical foundation for EEG-fNIRS joint analysis and playing a key role in the construction of coupled brain networks.

[0046] The main technical problem to be solved in this application is how to utilize the neurovascular coupling characteristics between EEG and fNIRS signals, study the calculation method of the coupling strength between EEG-fNIRS signals under the emotion induction experimental paradigm, and construct the corresponding coupled brain network to effectively improve the emotion recognition performance.

[0047] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0048] like Figure 1 As shown, this embodiment proposes a method for constructing an EEG-fNIRS coupled brain network, which is mainly used in emotion recognition scenarios to construct an EEG-fNIRS coupled brain network, and the EEG-fNIRS coupled brain network can be used as a new emotion recognition feature to help obtain more accurate emotion recognition results. The EEG-fNIRS coupled brain network construction method mainly includes the following steps:

[0049] Step S1, obtaining synchronously collected EEG signals and fNIRS signals.

[0050] In this embodiment, the user's brain is induced to produce EEG signals and fNIRS signals corresponding to different emotions by means of emotional video induction. At this time, the ESI NeuroScan (high-precision EEG signal acquisition and analysis) system is used to collect EEG signals, and the Nir Smart (portable near-infrared brain functional imaging) system is used to synchronously collect fNIRS signals. The above two signal acquisition systems are multi-channel acquisition devices, and the EEG signal and fNIRS signal are two signals synchronously collected by multiple acquisition channels, so that signal data under multiple acquisition channels can be collected.

[0051] Step S2: preprocess the EEG signal and the fNIRS signal respectively to obtain a preprocessed EEG signal and a preprocessed fNIRS signal.

[0052] In this embodiment, step S2 preprocesses the EEG signal and the fNIRS signal respectively to obtain a preprocessed EEG signal and a preprocessed fNIRS signal, which specifically includes the following steps:

[0053] Step S21, performing re-reference processing, down-sampling processing, artifact removal processing and filtering processing on the EEG signal in sequence to obtain the pre-processed EEG signal.

[0054] Step S22: performing filtering processing and artifact removal processing on the fNIRS signal in sequence to obtain a preliminarily processed fNIRS signal.

[0055] Step S23: According to the modified Beer-Lambert law, the preliminarily processed fNIRS signal is converted into an HbO concentration change signal, and upsampled to the current sampling frequency of the EEG signal to obtain the preprocessed fNIRS signal; the HbO concentration change signal is used to characterize the activation degree of the local brain area.

[0056] like Figure 2 As shown, in this embodiment, after collecting EEG signals and fNIRS signals, the collected EEG signals and fNIRS signals must first be preprocessed respectively. In this embodiment, the original EEG signal is processed in the following order: re-reference processing, downsampling to 200Hz, artifact removal processing, and 4-45Hz bandpass filtering processing. For the collected fNIRS signal, it is first processed by 0.01-0.2Hz bandpass filtering and artifact removal, and then converted into a HbO concentration change signal according to the modified Beer-Lambert law, and upsampled to the current sampling frequency of the EEG signal of 200Hz. The concentration change reflects the blood flow fluctuation in the local area of ​​the brain, which can characterize the activation degree of the corresponding brain area.

[0057] Step S3: Calculate the EEG-fNIRS coupling strength according to the preprocessed EEG signal and the preprocessed fNIRS signal to obtain an EEG-fNIRS coupling strength matrix.

[0058] In this embodiment, step S3 calculates the EEG-fNIRS coupling strength according to the preprocessed EEG signal and the preprocessed fNIRS signal to obtain an EEG-fNIRS coupling strength matrix, which specifically includes the following steps:

[0059] Step S31, performing short-time Fourier transform on each of the preprocessed EEG signals to obtain a time-frequency power spectrum of each of the preprocessed EEG signals.

[0060] Step S32: performing power superposition on the time-frequency power spectrum of each of the preprocessed EEG signals in the frequency dimension to obtain the time-varying power of each of the preprocessed EEG signals.

[0061] Step S33: normalize the time-varying power of each pre-processed EEG signal and convolve it with the HRF function to obtain fNIRS signals predicted under several experimental stimuli.

[0062] Step S34: using a rectangular window to perform sample segmentation on the preprocessed measured fNIRS signal and the fNIRS signal predicted under several experimental stimuli.

[0063] Step S35: for each sample after sample segmentation, the predicted fNIRS signals under several experimental stimulations are used as the design matrix of the general linear model, and the pre-processed measured fNIRS signals are fitted to obtain a regression coefficient matrix.

[0064] Step S36, calculating the absolute value of the regression coefficient matrix to obtain the EEG-fNIRS coupling strength matrix.

[0065] In this embodiment, for each experiment, the EEG signal and fNIRS signals Where l is the signal length, n is the number of EEG signal channels, and m is the number of fNIRS signal channels. The process of calculating the coupling strength matrix is ​​as follows: Figure 3 As shown, the calculation process mainly includes the following:

[0066] (1) For each EEG signal Where l is the length of the EEG signal, n is the number of EEG signal channels, and the time-frequency power spectrum is calculated by short-time Fourier transform. The calculation formula is as follows:

[0067]

[0068] Among them, P i (t,f) represents the time-frequency power spectrum, t is time, f is frequency, h(τ-t) is the analysis window function, and τ is the integral variable of the time dimension.

[0069] (2) The time-frequency power spectrum of each EEG signal is superimposed in the frequency dimension to obtain the time-varying power of each EEG signal. The time-varying power here refers to the change of power over time. The calculation formula is as follows:

[0070]

[0071] Among them, P i (t) represents the time-varying power, f max and f min They are the upper and lower limits of the EEG signal frequency range, respectively.

[0072] (3) Each time-varying power is normalized and convolved with the HRF function to obtain the predicted fNIRS signal under experimental stimulation, which is expressed as:

[0073]

[0074] in, is the predicted fNIRS signal, is the normalized time-varying power, HRF represents the HRF function, Represents a convolution operation.

[0075] (4) Use a rectangular window to perform sample segmentation on the measured fNIRS signal and the predicted fNIRS signal.

[0076] (5) For each sample, use the predicted fNIRS signal Constructing the design matrix for the general linear model It is expressed as:

[0077]

[0078] (6) Use the general linear model to analyze the measured fNIRS signals The fitting is performed, where l is the length of the fNIRS signal, m is the number of fNIRS signal channels, and the measured fNIRS signal refers to the fNIRS signal collected in step S1 and preprocessed in step S2.

[0079] In this embodiment, the measured fNIRS signal of the first channel is Take as an example to illustrate the above fitting process, which is expressed as:

[0080]

[0081] in, represents the measured fNIRS signal of the first channel, is the design matrix, represents the regression coefficient obtained after fitting the measured fNIRS signal y1(t) of the first channel, Represents the fitting error. Similarly, by fitting the measured fNIRS signal Y of all m channels, the regression coefficient matrix can be obtained: It is expressed as:

[0082] Y=Gβ+ε (6).

[0083] (7) The absolute value of the regression coefficient matrix β is obtained, which is the EEG-fNIRS coupling strength matrix: CS = |β|, where the element cs i,j represents the coupling strength between the i-th EEG signal and the j-th fNIRS signal, expressed as:

[0084]

[0085] Among them, CS (Coupling Strength) represents the EEG-fNIRS coupling strength matrix, cs nm represents the coupling strength between the nth EEG signal and the mth fNIRS signal.

[0086] Step S4, constructing an EEG-fNIRS coupled brain network according to the EEG-fNIRS coupling strength matrix. The EEG-fNIRS coupled brain network is used as a new emotion recognition feature to be input into an emotion recognition model to obtain an emotion recognition result. The emotion recognition model refers to any model with emotion recognition and classification capabilities, which can be a SVM (Support Vector Machine) model or other classification models.

[0087] In this embodiment, step S4 constructs an EEG-fNIRS coupled brain network according to the EEG-fNIRS coupling strength matrix, which specifically includes the following steps:

[0088] The i-th EEG signal and the j-th fNIRS signal in the EEG-fNIRS coupling strength matrix are respectively used as nodes, and the coupling strength cs corresponding to the i-th EEG signal and the j-th fNIRS signal is ij As an edge connecting two nodes, the EEG-fNIRS coupling strength matrix is ​​used as an adjacency matrix to construct the EEG-fNIRS coupled brain network.

[0089] In this embodiment, two EEG signals and two fNIRS signals are used as examples to illustrate the process of constructing an EEG-fNIRS coupled brain network. CS in formula (8) is the EEG-fNIRS coupling strength matrix calculated based on two EEG signals and two fNIRS signals. Figure 4 is the EEG-fNIRS coupled brain network constructed according to the EEG-fNIRS coupling strength matrix.

[0090]

[0091] In this embodiment, after the step of constructing an EEG-fNIRS coupled brain network according to the EEG-fNIRS coupling strength matrix in step S4, the EEG-fNIRS coupled brain network construction method further includes the following steps:

[0092] Step S5, using the EEG-fNIRS coupled brain network to perform emotion recognition and obtain an emotion recognition result. Specifically, the following steps are included:

[0093] Step S51: Expand the EEG-fNIRS coupling strength matrix corresponding to the EEG-fNIRS coupled brain network by rows or columns to obtain a feature vector for emotion recognition.

[0094] Step S52: input the feature vector into the emotion recognition model to obtain the emotion recognition result.

[0095] From the above analysis, we can know that as long as the EEG-fNIRS coupling strength matrix is ​​obtained, the EEG-fNIRS coupled brain network can be constructed. Therefore, an EEG-fNIRS coupling strength matrix can represent an EEG-fNIRS coupled brain network. In the process of emotion recognition, an EEG-fNIRS coupling strength matrix is ​​vectorized, that is, expanded by rows or columns, to obtain a feature vector of a sample. For example, after the EEG-fNIRS coupling strength matrix calculated based on 2 EEG signals and 2 fNIRS signals is vectorized, the corresponding feature vector fea is obtained, which is expressed as:

[0096] fea=[cs 11 cs 12 cs 21 cs 22 ] (9).

[0097] Then, the feature vectors of all q samples are combined to form the feature set FS for emotion recognition, which is expressed as:

[0098]

[0099] Among them, fea q Represents the feature vector of the qth sample.

[0100] In this embodiment, when performing emotion classification, 80% of the feature vectors in the feature set FS are divided as a training set, and the remaining 20% ​​are divided as a test set, wherein the training set is used to train the recognition SVM model, and the test set is used to evaluate the classification effect of the model recognition. The flowchart of the above emotion recognition process is shown in FIG. Figure 5 shown.

[0101] This embodiment is based on the synchronously collected emotional EEG-fNIRS signals (including EEG signals and fNIRS signals), and uses the characteristics of neurovascular coupling between EEG signals and fNIRS signals to directly use the time-varying power of EEG signals to model the experimental stimulation under the emotion induction paradigm, and uses the energy information of EEG signals at different times to reflect the real-time response intensity of the subjects to the experimental stimulation. In addition, the EEG signals are no longer channel-averaged, but the energy information of all EEG channel signals is used to reflect the different responses of different brain regions to the same experimental stimulation. The method for calculating the coupling strength between EEG and fNIRS under the emotion induction paradigm in this embodiment uses the coupling strength to construct an EEG-fNIRS coupled brain network. For each sample's EEG-fNIRS coupling strength matrix, the i-th EEG signal and the j-th fNIRS signal are used as nodes, and the coupling strength cs ijThat is, it is the edge connecting two nodes. In this way, the EEG-fNIRS coupling strength matrix can be used as an adjacency matrix to construct a coupled brain network and use it as a feature for emotion recognition.

[0102] This embodiment comprehensively utilizes the change of EEG signal energy over time in all brain regions to model emotional experimental stimuli with long duration and changing intensity. For the synchronously collected EEG signals and fNIRS signals, the neurovascular coupling characteristics between EEG-fNIRS are utilized to propose a new EEG-fNIRS coupling strength calculation method under the emotion induced experimental paradigm. Based on the synchronously collected EEG signals and fNIRS signals, the EEG-fNIRS coupling strength matrix is ​​used as the adjacency matrix to construct a brain network based on multimodal signals, and the EEG-fNIRS coupled brain network is obtained. It is used as a new feature for emotion recognition, which can effectively improve the accuracy and performance of emotion recognition classification.

[0103] This embodiment is based on the EEG signals and fNIRS signals synchronously collected by emotion video induction, and utilizes the neurovascular coupling characteristics and complementary advantages of the two to propose a technical solution for calculating the coupling strength between EEG-fNIRS signals and constructing an EEG-fNIRS coupled brain network under an emotion induction experimental paradigm. By constructing the EEG-fNIRS coupled brain network, it can not only be used as a new feature for emotion recognition, but also can be used to explore the information interaction of EEG-fNIRS signals between brain areas under emotion induction, which is of great significance for in-depth understanding of the brain neural activity mechanism under emotion tasks.

[0104] At present, in tasks such as motor imagery, experimental stimuli often appear for a very short time. When the subject performs the corresponding action according to the experimental stimulus requirements, the intensity of a certain experimental stimulus received by the subject often remains unchanged. At this time, the experimental stimulus can be modeled using a square pulse function. However, in the emotion induction experimental paradigm, the experimental stimulus is usually presented in the form of audio or video clips, which lasts for a relatively long time (1-2 minutes) and the content is constantly changing, so that during the task, the response of the subject's brain neurons to the stimulus also fluctuates over time, and the response intensity of different cortical areas to the same stimulus is also different. Therefore, the experimental stimulus cannot be modeled using a simple square pulse function, and the existing method cannot be used to calculate the coupling strength between EEG-fNIRS. In response to the above problems, this embodiment is based on the EEG signals and fNIRS signals synchronously collected by emotion video induction, utilizes the neurovascular coupling characteristics of the two, and proposes a method for calculating the coupling strength between EEG and fNIRS under the emotion induction experimental paradigm. The EEG signals and fNIRS signals are used as nodes, and the coupling strength between the two is used as an edge to construct an EEG-fNIRS coupled brain network, which is sent as a feature into the recognition model for emotion classification. This method is of great significance for revealing the working mechanism of brain emotion cognition.

[0105] In an exemplary embodiment, an EEG-fNIRS coupled brain network construction system is provided, the system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the above-mentioned EEG-fNIRS coupled brain network construction method.

[0106] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for constructing an EEG-fNIRS coupled brain network, characterized in that: The EEG-fNIRS coupled brain network construction method comprises: Acquire EEG signals and fNIRS signals; the EEG signals and the fNIRS signals are signals synchronously acquired by multiple acquisition channels; Preprocessing the EEG signal and the fNIRS signal respectively to obtain a preprocessed EEG signal and a preprocessed fNIRS signal; Calculating the EEG-fNIRS coupling strength according to the preprocessed EEG signal and the preprocessed fNIRS signal to obtain an EEG-fNIRS coupling strength matrix; According to the EEG-fNIRS coupling strength matrix, an EEG-fNIRS coupled brain network is constructed; the EEG-fNIRS coupled brain network is used as a new emotion recognition feature and is input into an emotion recognition model to obtain an emotion recognition result.

2. The EEG-fNIRS coupled brain network construction method according to claim 1, characterized in that: Preprocessing the EEG signal and the fNIRS signal respectively to obtain a preprocessed EEG signal and a preprocessed fNIRS signal, specifically comprising: The EEG signal is sequentially subjected to re-reference processing, down-sampling processing, artifact removal processing and filtering processing to obtain the pre-processed EEG signal; The fNIRS signal is subjected to filtering processing and artifact removal processing in sequence to obtain a preliminarily processed fNIRS signal; According to the modified Beer-Lambert law, the preliminarily processed fNIRS signal is converted into an HbO concentration change signal, and upsampled to the current sampling frequency of the EEG signal to obtain the preprocessed fNIRS signal; the HbO concentration change signal is used to characterize the degree of activation of a local brain area.

3. The EEG-fNIRS coupled brain network construction method according to claim 1, characterized in that: According to the preprocessed EEG signal and the preprocessed fNIRS signal, the EEG-fNIRS coupling strength is calculated to obtain an EEG-fNIRS coupling strength matrix, which specifically includes: Performing short-time Fourier transform on each of the preprocessed EEG signals to obtain a time-frequency power spectrum of each of the preprocessed EEG signals; Performing power superposition of the time-frequency power spectrum of each of the preprocessed EEG signals in the frequency dimension to obtain the time-varying power of each of the preprocessed EEG signals; Normalizing the time-varying power of each preprocessed EEG signal and convolving it with the HRF function to obtain fNIRS signals predicted under several experimental stimuli; Rectangular windows were used to segment samples of the preprocessed measured fNIRS signals and the predicted fNIRS signals under several experimental stimuli. For each sample after sample segmentation, the predicted fNIRS signals under several experimental stimulations are used as the design matrix of the general linear model, and the pre-processed measured fNIRS signals are fitted to obtain the regression coefficient matrix; The absolute value of the regression coefficient matrix is ​​calculated to obtain the EEG-fNIRS coupling strength matrix.

4. The EEG-fNIRS coupled brain network construction method according to claim 3, characterized in that: The EEG-fNIRS coupling strength matrix is ​​expressed as follows: Wherein, CS represents the EEG-fNIRS coupling strength matrix, cs nm represents the coupling strength between the nth EEG signal and the mth fNIRS signal.

5. The EEG-fNIRS coupled brain network construction method according to claim 1, characterized in that: According to the EEG-fNIRS coupling strength matrix, an EEG-fNIRS coupled brain network is constructed, which specifically includes: The i-th EEG signal and the j-th fNIRS signal in the EEG-fNIRS coupling strength matrix are respectively used as nodes, and the coupling strength cs corresponding to the i-th EEG signal and the j-th fNIRS signal is ij As an edge connecting two nodes, the EEG-fNIRS coupling strength matrix is ​​used as an adjacency matrix to construct the EEG-fNIRS coupled brain network.

6. The EEG-fNIRS coupled brain network construction method according to claim 1, characterized in that: After the step of constructing an EEG-fNIRS coupled brain network according to the EEG-fNIRS coupling strength matrix, the EEG-fNIRS coupled brain network construction method further comprises: The EEG-fNIRS coupled brain network is used to perform emotion recognition and obtain an emotion recognition result.

7. The EEG-fNIRS coupled brain network construction method according to claim 6, characterized in that: The EEG-fNIRS coupled brain network is used to perform emotion recognition to obtain emotion recognition results, which specifically include: Expanding the EEG-fNIRS coupling strength matrix corresponding to the EEG-fNIRS coupled brain network by rows or columns to obtain a feature vector for emotion recognition; The feature vector is input into the emotion recognition model to obtain the emotion recognition result.

8. The EEG-fNIRS coupled brain network construction method according to claim 7, characterized in that: The emotion recognition model is a SVM model.

9. An EEG-fNIRS coupled brain network construction system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the EEG-fNIRS coupled brain network construction method according to any one of claims 1 to 8.

Images