Cognitive state classification method based on EEG-fNIRS space-time fusion features

Abstract

The invention relates to a cognitive state classification method based on EEG-fNIRS spatio-temporal fusion features, which comprises the following steps of: setting an experiment according to a mental calculation experiment normal form, and synchronously acquiring EEG data and fNIRS data of a tested mental calculation task; preprocessing the two collected data to obtain electroencephalogram signal data and hemoglobin concentration change data; eEG signal data and hemoglobin concentration change data are taken, data enhancement is carried out through a time window, then the data are sent to a space-time fusion network, EEG features, oxyhemoglobin HBO features and deoxyhemoglobin HBR features are obtained, fusion features are obtained through fusion, and an EEG classification result, an fNIRS classification result and a fusion feature classification result are obtained. According to the method, the characteristics of the EEG signal and the fNIRS signal are effectively combined, the overfitting problem in modal deep learning is relieved, and cognitive state classification can be accurately carried out.

Description

TECHNICAL FIELD

[0001] The application relates to a cognitive state classification method based on EEG-fNIRS spatio-temporal fusion features, and belongs to the technical field of neural system cognitive states. BACKGROUND

[0002] Cognitive state research helps us understand the basic principles and mechanisms of human cognitive function. By studying brain activity during cognitive processes, we can gain a deeper understanding of the activity patterns of the brain in different cognitive tasks and reveal the neural basis of cognitive function. At the same time, cognitive function research plays an important role in the diagnosis of brain diseases such as depression, but identifying mental states through brain activity measurement has always been a key challenge in neuroscience.

[0003] EEG is usually used for cognitive state research, and Lawhern et al. proposed EEGNet, which performs well under multiple paradigms and has a small number of parameters, suitable for data-limited scenarios. Song et al. proposed EEG Conformer, which combines the advantages of convolutional neural networks (CNN) and Transformers, captures long and short time dependencies, and proposes a visualization method to enhance model interpretability. However, due to the low spatial resolution of EEG and the presence of severe volume effects, the cognitive state decoding expression ability of single-modality EEG is limited. fNIRS provides excellent spatial resolution and robustness, and Wang et al. introduced the delayed hemodynamic response as domain knowledge into fNIRS classification and proposed a simple and effective fNIRSNet model, but the time resolution of fNIRS is poor and there is a hemodynamic response delay, which hinders real-time BCI applications.

[0004] EEG has high time resolution, while fNIRS has rich spatial resolution and optical information, and does not interact with physiological signals, which makes the time and spatial features of the two complementary to some extent. Therefore, more and more researchers have begun to study the feature fusion of EEG-fNIRS multi-modal. Shin et al. improved BCI through EEG and fNIRS fusion, showing that hybrid BCI can improve classification performance. Recently, Jiang et al. proposed an independent decision path fusion (IDPF) method for late fusion of dual modalities to distinguish between multiple mental states. Arif et al. proposed EF_Net, which uses a parallel structure to use deep neural networks to extract and fuse features from EEG and fNIRS, and exhibits excellent performance. In addition, Kwak et al. proposed a fNIRS attention-guided network that achieves final prediction through mid-fusion and decision score fusion. On this basis, Liu et al. proposed STA-Net for spatio-temporal alignment of two modalities to achieve feature fusion and achieve high recognition accuracy. SUMMARY

[0005] To overcome the poor fusion effect and weak interpretability of existing EEG-fNIRS integration work, the present invention provides a cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features. The EEG-fNIRS end-to-end deep fusion framework consists of three key components: an EEG spatiotemporal feature extraction network based on multi-scale convolution, an fNIRS network based on a dynamic graph convolutional network, and a multimodal fusion module.

[0006] Multi-scale convolution is used to obtain EEG features at different frequencies, thereby obtaining a more robust feature representation; a dynamic graph convolutional network is used to model the local spatial state of fNIRS to enrich the spatial domain information; finally, the features aligned by regional similarity are fused and spliced ​​using cross-attention, which enhances the temporal expression ability of the EEG signal, improves the accuracy of cognitive state classification, and effectively combines the complementary information of EEG and fNIRS.

[0007] A cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features comprises the following steps:

[0008] Step 1: The experiment was set up according to the mental arithmetic (MA) experimental paradigm, with EEG and fNIRS data collected simultaneously during the mental arithmetic task. Each trial consisted of a 2-second introduction phase, a 10-second task phase, and a 15-17-second rest phase. A 60-second buffer period separated each trial. Mental arithmetic instructions were given during the introduction phase, and participants responded during the task phase. The collected data were labeled "mental arithmetic" and "resting," with the buffer phase ([-62, -2] s), introduction phase ([-2, 0] s), task phase ([0, 10] s), and rest phase ([10, 25] s) of the entire experiment being annotated. Finally, the two collected data sets were preprocessed to generate EEG data and hemoglobin concentration change data.

[0009] Step 2: Take the EEG signal data and hemoglobin concentration change data in step 1, perform data enhancement through the time window, and then send them into the deep fusion network to obtain EEG features, oxyhemoglobin HBO features, deoxyhemoglobin HBR features, and fusion features, and further obtain EEG classification results, fNIRS classification results, and fusion feature classification results.

[0010] In step three, the topological structure learned by graph convolution obtained in step two is aligned, and the regional correlation of the EEG signal and the HBO signal is aligned. The EEG classification results, fNIRS classification results, and fusion feature classification results are trained using the cross entropy loss function.

[0011] The EEG signal preprocessing process in step 1 specifically includes: first removing the two eye movement signals (EOG), downsampling the EEG signal to 200Hz, and then taking the [-5, -2]s data in the buffer stage for baseline correction. The baseline correction is performed by subtracting the average value between -5s and -2s from the timing signal of each EEG channel. Finally, a 1Hz-40Hz bandpass filter is applied for noise reduction. For near-infrared signals, the data is first downsampled to 10Hz, and then the near-infrared signal is converted into oxygenated hemoglobin concentration change and deoxygenated hemoglobin concentration change data using the modified Beer-Lambert law. Finally, baseline correction and 0.01Hz-0.1Hz bandpass filtering are performed.

[0012] During the experiment, 12s of data representing the task phase were used. The sizes of the final EEG and NIR data were (B, C e ,T e ) and (B,C f ,T f ), where B is the size of each batch of input data, C e and C f represents the number of brain signal channels of EEG and fNIRS (including HBO and HBR data), T e and T f Represents a 12s time series.

[0013] The specific steps of step 2 are:

[0014] S2.1: Take the preprocessed data from step 1 and perform data enhancement on the data by performing time window processing with a size of 3 seconds and a step size of 1 second, and finally obtain the EEG and near-infrared input data.

[0015] S2.2: The EEG and NIR input data are subjected to feature extraction using the EEG branch and NIR branch of the spatiotemporal fusion network, followed by spatiotemporal feature fusion, and finally classification to output the classification results.

[0016] S2.2 also includes:

[0017] S2.21: The EEG data is first extracted through the EEG spatiotemporal feature extraction module at different scales to obtain the primary feature E i＝1,2 ,This module consists of a series of spatiotemporal convolution and spatial grouped convolution;,then the features of different scales are summed up through the learnable parameter,a,to obtain the EEG primary features, and the primary features are passed through the,fully connected layer to obtain the EEG classification results;

[0018] S2.22: The near-infrared data is first preliminarily integrated with the local spatial information of HBO and HBR through group convolution. Then, using a learnable adjacency matrix, after Laplace normalization, graph convolution is performed independently on each group of features to extract HBO and HBR features. The adjacency matrices of HBO and HBR are aligned to ensure the same topological structure. The graph convolution calculation process is as follows:

[0019]

[0020] H (l) =σ(BN(LW (l) ))

[0021] Among them, A represents the learnable adjacency matrix, I is the unit matrix, D represents the degree matrix of (A+I), L is the result after Laplace normalization, and H (l) represents the output of the lth graph convolutional layer, σ represents the activation function, BN represents batch normalization, and W (l) Represents the parameters of the lth group of input features.

[0022] The fNIRS features are obtained by summing the features of different scales through the learnable parameter b, and then the fNIRS classification results are obtained through the fully connected layer;

[0023] S2.23: After regional correlation alignment of EEG primary features and HBO features, the HBO features are used as the K of the self-attention block, and the EEG primary features are used as the QV. After the cross-attention block, they are concatenated with the fNIRS features and the classification results are output through the fully connected layer.

[0024] The number of group convolution groups in S2.21 and S2.22 is set to 3, which is obtained by the clustering method and is used to represent the division of brain regions.

[0025] The calculation ideas of topological structure alignment and regional correlation alignment in step 3 are as follows:

[0026] When training HBO and HBR features, the spatial topology of HBR is constrained and the topology alignment loss L is designed. topo , use the Frobenius norm to measure the distance between two adjacency matrices. The calculation formula is as follows:

[0027] L topo =F(L o -L r )

[0028] Where F represents the Frobenius norm of the matrix, L o and L r Represent the HBO and HBR adjacency matrices obtained through training respectively;

[0029] The region correlation alignment calculation idea is as follows:

[0030] After obtaining the EEG primary features and the HBO features, it is expected that the brain region activation of the EEG and the HBO is consistent, and a region alignment loss L is designed cor The Pearson correlation coefficient PCC is used to measure the correlation between regions, and the Frobenius norm is used to measure the distance between region matrices, and the calculation formula is as follows:

[0031]

[0032] L cor =F(P eeg -P hbo )

[0033] Wherein, And Indicate the mean value of a single channel, P eeg And P hbo Are the region correlation matrices obtained from the EEG and the HBO features.

[0034] Compared with the prior art, the beneficial effects of the present application are:

[0035] The core of the present application is to develop an end-to-end EEG-fNIRS spatio-temporal fusion framework, which fully utilizes the complementarity between modalities, and initiatively uses the spatial information of HBO to guide the spatial information of HBR, and then enhances the EEG, which significantly improves the cognitive state classification performance. In order to extract more robust features, a multi-scale convolution extraction module is proposed to capture EEG signals of different frequencies and improve the adaptability of the network to complex data. At the same time, a fNIRS feature extraction network based on dynamic graph convolution network is proposed, which not only retains the original spatial structure features of fNIRS, but also enhances the spatial information, providing a basis for multi-modal fusion. Through cross-attention, the time sequence of the EEG signal is enhanced, and the spliced features can accurately classify the cognitive state. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0037] Figure 1 The EEG signal preprocessing flowchart of the cognitive state classification method based on EEG-fNIRS spatio-temporal fusion features of the present application.

[0038] Figure 2 This is a flowchart of the near-infrared signal preprocessing of the cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features of the present invention.

[0039] Figure 3 This is a framework diagram of the cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] Research has shown that EEG and fNIRS are two complementary brain physiological signals, and that electrophysiological and optical signals do not interfere with each other during synchronous acquisition. Therefore, EEG-fNIRS multimodal fusion research can effectively improve the efficiency of attention state recognition and is of great significance for the study of cognitive brain function. This paper proposes a cognitive state classification method based on EEG-fNIRS spatiotemporal fusion, which mainly includes an EEG spatiotemporal feature extraction network based on multiscale convolution, an fNIRS network based on a dynamic graph convolutional network, and a multimodal fusion module. The implementation of this invention mainly includes three steps: EEG-fNIRS data acquisition and preprocessing; multimodal feature fusion and cognitive state recognition; and state recognition performance optimization.

[0042] The following describes each step in detail.

[0043] Step 1: EEG-fNIRS Data Acquisition and Preprocessing: The experiment was set up according to the mental arithmetic experimental paradigm. EEG signals and fNIRS data were simultaneously collected during the mental arithmetic task. The two collected data were preprocessed to obtain EEG data and hemoglobin concentration change data.

[0044] The subjects were instructed to perform the experiment according to the mental arithmetic experimental paradigm. During the mental arithmetic task, participants performed subtraction operations, such as "subtracting one digit from a three-digit number," based on visual cues and brief beep signals displayed on the screen; in the baseline task, participants were asked to relax by gazing at a black fixation cross on the screen. Each participant completed 30 trials on both tasks. Each trial began with a 2-second visual introduction, followed by a 10-second task period, and ended with a rest period randomly set between 15 and 17 seconds. EEG signals were recorded using 32 electrodes at a sampling rate of 200 Hz, while 36 channels of near-infrared data were recorded using a functional near-infrared spectroscopy device.

[0045] Figure 1 The EEG data is preprocessed, as it is susceptible to interference from cardiac signals, power-frequency electromagnetic waves, and other sources. During the EEG preprocessing stage, the two eye movement channels were first removed, and baseline correction was performed using the [-5, -2] s interval before the task began. Next, a bandpass filter with a frequency range of 1-40 Hz was applied, using only the [-2, 10] s interval during the instruction and task phases for analysis. Therefore, the final EEG dataset had a dimensionality of (N, 60, 30, 2400), representing the number of subjects × the number of trials × the number of channels × the number of sampling points.

[0046] The preprocessing process of near infrared signals is as follows: Figure 2 As shown, the data is first downsampled to 10Hz. Because it has a certain resistance to external interference, the modified Beer-Lambert law is used to convert the optical density ΔA(λ) into the concentration of oxygenated hemoglobin (ΔHbO) and deoxygenated hemoglobin (ΔHbR). The conversion process is described as follows:

[0047]

[0048] where ε HbO (λ1),ε HbO (λ2),ε HbR (λ1) and ε HbR (λ2) represents the absorption coefficient of HbO and HbR at different wavelengths, respectively, and d represents the distance between the signal transmitter and receiver. Dividing by d yields the actual concentration changes Δ[HbO] and Δ[HbR]. Baseline correction was then performed, and the signal was bandpass filtered from 0.01 to 0.1 Hz to preserve relevant physiological information. Consistent with the EEG data, we segmented the data into time periods of [-2, 10] s for analysis. The dimensions of the near-infrared data are (N, 60, 36, 120), representing the number of subjects × number of trials × number of channels × number of sampling points.

[0049] Step 2: Multimodal feature fusion and cognitive state recognition: EEG signal data and hemoglobin concentration change data are taken, enhanced through time windows, and then sent to the spatiotemporal fusion network to obtain EEG features, oxyhemoglobin (HBO) features, deoxyhemoglobin (HBR) features, and fusion features. The fusion features are then fused to obtain EEG classification results, fNIRS classification results, and fusion feature classification results.

[0050] The cognitive classification network based on EEG-fNIRS spatiotemporal fusion involved in the present invention is as follows Figure 3 As shown, the specific implementation steps are:

[0051] S2.1: To enhance the data, the EEG and NIR signals were segmented using a time window of 3s and a step size of 1s, and each experiment was divided into 10 segments.

[0052] S2.2: The EEG and NIR data after time window processing in S2.1 are input into the EEG branch and NIR branch of the fusion network respectively for feature extraction, and then the spatiotemporal feature fusion is performed, and finally the classification is performed to output the classification results.

[0053] S2.2 also includes:

[0054] S2.21: The EEG data is first extracted through the EEG spatiotemporal feature extraction module at different scales to obtain the primary feature E i＝1,2 ,This module consists of a series of spatiotemporal convolution and spatial grouped convolution;,then the features of different scales are summed up through the learnable parameter,a,to obtain the EEG primary features, and the primary features are passed through the,fully connected layer to obtain the EEG classification results;

[0055] S2.22: The near-infrared data is first preliminarily integrated with the local spatial information of HBO and HBR through group convolution. Then, using a learnable adjacency matrix, after Laplace normalization, graph convolution is performed independently on each group of features to extract HBO and HBR features. The adjacency matrices of HBO and HBR are aligned to ensure the same topological structure. The graph convolution calculation process is as follows:

[0056]

[0057] H (l) =σ(BN(LW (l) ))

[0058] Among them, A represents the learnable adjacency matrix, I is the unit matrix, D represents the degree matrix of (A+I), L is the result after Laplace normalization, and H (l) represents the output of the lth group of graph convolutional layers, σ represents the activation function, BN represents batch normalization, and W (l) Represents the parameters of the lth group of input features.

[0059] The fNIRS features are obtained by summing the features of different scales through the learnable parameter b, and then the fNIRS classification results are obtained through the fully connected layer;

[0060] S2.23: After regional correlation alignment of EEG primary features and HBO features, the HBO features are used as the K of the self-attention block, and the EEG primary features are used as the QV. After the cross-attention block, they are concatenated with the fNIRS features and the classification results are output through the fully connected layer.

[0061] Step 3: Optimize state recognition performance: Align the topological structures of HBO and HBR learned by graph convolution, align the regional correlation of EEG signals with HBO signals, and train the EEG classification results, fNIRS classification results, and fusion feature classification results using the cross-entropy loss function.

[0062] When training HBO and HBR features, the spatial topology of HBR is constrained and the topology alignment loss L is designed. topo , use the Frobenius norm to measure the distance between two adjacency matrices. The calculation formula is as follows:

[0063] L topo =F(L o -L r )

[0064] Where F represents the Frobenius norm of the matrix, L o and L r Represent the HBO and HBR adjacency matrices obtained through training respectively;

[0065] The idea of ​​regional correlation alignment calculation is as follows:

[0066] After obtaining the EEG primary features and HBO features, we expect the brain region activations of EEG and HBO to be consistent, and design the regional alignment loss L cor , the Pearson correlation coefficient PCC is used to measure the correlation between regions, and the Frobenius norm is used to measure the distance between regional matrices. The calculation formula is as follows:

[0067]

[0068] L cor =F(P eeg -P hbo )

[0069] in, and represents the mean of a single channel, P eeg and P hbo are the regional correlation matrices obtained from EEG and HBO features, respectively.

[0070] The EEG, fNIRS, and fusion results obtained in step 2 are fed into a cross-entropy loss based on a label smoothing strategy as the main task to improve the model's attention state recognition performance and minimize the topological structure alignment loss and region similarity loss. Finally, the model is optimized by minimizing the following losses:

[0071] L=L fusion +α·(L eeg +L fNIRS)+β·L topo +γ·L cor

[0072] Among them, α, β, γ are weights, L eeg , L fNIRS , L fusion Losses computed with label smoothing for EEG, fNIRS, and fusion branches, respectively.

[0073] Table 1 compares the performance of the proposed EEG-fNIRS spatiotemporal fusion framework in single-modality and multimodal scenarios, where the EEG branch (EEGBranch) represents the EEG feature extraction branch and the fNIRS branch (fNIRSBranch) represents the near-infrared feature extraction branch. The proposed framework achieved an average accuracy improvement of 2.4% compared to single-modality EEG and 13.51% compared to single-modality near-infrared. Twenty-four subjects (approximately 82.8%) showed improved recognition accuracy after fusion.

[0074] Table 1 Comparison of multimodal and unimodal classification performance

[0075]

[0076]

[0077] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features, characterized by: The following steps are involved: Step 1: Set up the experiment according to the mental arithmetic experimental paradigm, synchronously collect EEG data and functional near-infrared spectroscopy (fNIRS) data during the mental arithmetic task, and preprocess the two collected data to obtain EEG data and hemoglobin concentration change data; Step 2: Take the EEG signal data and hemoglobin concentration change data, perform data enhancement through the time window, and then send them into the spatiotemporal fusion network to obtain EEG features, oxyhemoglobin HBO features, deoxyhemoglobin HBR features, and fuse them to obtain fusion features, and obtain EEG classification results, fNIRS classification results, and fusion feature classification results; Step 3: Align the topological structures of HBO and HBR learned by graph convolution, and align the regional correlation of EEG signals with HBO signals. The EEG classification results, fNIRS classification results, and fusion feature classification results are trained using the cross-entropy loss function.

2. The cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features according to claim 1, characterized in that: The step 1 specifically includes setting up an experiment according to the mental arithmetic experimental paradigm: one experiment includes an introduction phase, a task phase, and a rest phase, with a buffer time between each trial, mental arithmetic instructions are given in the introduction phase, and the subjects respond in the task phase; the collected data are labeled as mental arithmetic and rest, and the buffer phase, introduction phase, task phase, and rest phase included in the entire experimental process are marked.

3. The cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features according to claim 1, characterized in that: The preprocessing in step 1 specifically includes: first removing the two eye movement signals, downsampling the EEG signal to 200 Hz, performing baseline correction on the buffered data, and applying a 1 Hz-40 Hz bandpass filter to reduce noise; for the near-infrared signal, downsampling the data to 10 Hz, converting the near-infrared signal into oxyhemoglobin concentration change data and deoxyhemoglobin concentration change data using the modified Beer-Lambert law, and then performing baseline correction and 0.01 Hz-0.1 Hz bandpass filtering; During the experiment, 12s of data representing the task phase were used. The sizes of the final EEG and NIR data were (B, C e ,T e ) and (B,C f ,T f ), where B is the size of each batch of input data, C e and C f represents the number of brain signal channels of EEG and fNIRS, T e and T f Represents a 12s time series.

4. The cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features according to claim 1, characterized in that: The second step specifically includes: S2.1: Take the pre-processed data from step 1 and perform time window processing on the data with a size of 3s and a step size of 1s to obtain EEG and near-infrared input data; S2.2: The EEG and NIR input data are subjected to feature extraction using the EEG branch and NIR branch of the spatiotemporal fusion network, and then spatiotemporal feature fusion is performed to classify and output the classification results.

5. The cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features according to claim 4, characterized in that: Said S2.2 specifically includes: Step S2.21: The EEG data is first passed through the EEG spatiotemporal feature extraction module at different scales to obtain the primary feature E i＝1,2 The module consists of a series of spatiotemporal convolution and spatial grouped convolution. The features of different scales are summed up through the learnable parameter a to obtain the EEG primary features, and the primary features are passed through the fully connected layer to obtain the EEG classification results. Step S2.22: The near-infrared data is first preliminarily integrated with the local spatial information of HBO and HBR through group convolution. Then, using a learnable adjacency matrix and after Laplace normalization, graph convolution is performed independently on each group of features to extract HBO and HBR features. The adjacency matrices of HBO and HBR are aligned to ensure the same topological structure. The graph convolution calculation process is as follows: H (l) =σ(BN(LW (l) )) Among them, A represents the learnable adjacency matrix, I is the unit matrix, D represents the degree matrix of (A+I), L is the result after Laplace normalization, and H (l) represents the output of the lth group of graph convolutional layers, σ represents the activation function, BN represents batch normalization, and W (l) Represents the parameters of the lth group of input features; The fNIRS features are obtained by summing the features of different scales through the learnable parameter b, and then the fNIRS classification results are obtained through the fully connected layer; Step S2.23: After regional correlation alignment of the EEG primary features and HBO features, the HBO features are used as the K of the self-attention block, and the EEG primary features are used as the QV. After the cross-attention block, they are concatenated with the fNIRS features and the classification results are output through the fully connected layer.

6. The cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features according to claim 1, characterized in that: The topology alignment in step 3 specifically includes: When training HBO and HBR features, the spatial topology of HBR is constrained and the topology alignment loss L is designed. topo , use the Frobenius norm to measure the distance between two adjacency matrices. The calculation formula is as follows: L topo =F(L o -L r ) Where F represents the Frobenius norm of the matrix, L o and L r Represent the HBO and HBR adjacency matrices obtained through training respectively; The regional correlation alignment in step 3 specifically includes: After obtaining EEG primary features and HBO features, we design the region alignment loss L cor , the Pearson correlation coefficient PCC is used to measure the correlation between regions, and the Frobenius norm is used to measure the distance between regional matrices. The calculation formula is as follows: L cor =F(P eeg -P hbo ) in, and represents the mean of a single channel, P eeg and P hbo Regional correlation matrices obtained from EEG and HBO features, respectively.

7. The cognitive state classification method based on EEG-fNIRS spatiotemporal fusion features according to claim 1, characterized in that: In step 3, the EEG classification results, fNIRS classification results, and fusion feature classification results are trained using the cross entropy loss function, specifically including: The EEG classification results, fNIRS classification results, and fusion feature classification results obtained in step 2 are respectively input into the cross entropy loss based on the label smoothing strategy as the main task to improve the model's attention state recognition performance and minimize the topological structure alignment loss and regional similarity loss. The model is optimized by minimizing the following losses: L=L fusion +α·(L eeg +L fNIRS )+β·L topo +γ·L cor Among them, α, β, γ are weights, L eeg 、L fNIRS 、L fusion Losses computed with label smoothing for EEG, fNIRS, and fusion branches, respectively.

Images