Cognitive state classification method based on EEG-fNIRS deep fusion of feature decoupling

Abstract

The invention discloses an EEG-fNIRS deep fusion cognitive state classification method based on feature decoupling, and the method comprises the steps: firstly synchronously collecting EEG data and fNIRS data of a tested mental calculation task, and carrying out the preprocessing of the two collected data, so as to obtain EEG data and hemoglobin concentration change data; secondly, electroencephalogram signal data and hemoglobin concentration change data are taken, data enhancement is carried out through a time window, then the data are sent to a deep fusion network, and public features and private features of EEG and primary features of fNIRS are obtained; and an EEG classification result, an fNIRS classification result and an output fusion classification result are obtained. And finally, designing decoupling loss and cross entropy loss based on a label smoothing strategy, and carrying out training. According to the method, the overfitting problem in fNIRS modal deep learning is effectively relieved, and cognitive state classification can be accurately carried out.

Description

Technical Field

[0001] The present invention belongs to the research field of cognitive state of nervous system, and relates to a cognitive state classification method based on EEG-fNIRS deep fusion with multimodal feature decoupling. Background Art

[0002] Attention is crucial for learning, memory and cognitive ability, and directly affects the performance of individuals' daily activities. With the development of science and technology and the continuous expansion of application scenarios, the accurate identification of individual attention states has become an important topic in the field of interdisciplinary research. At present, people's pace of life has significantly accelerated, and the number of people suffering from mental illnesses such as cognitive impairment is increasing year by year. Among them, attention state recognition plays an important role in the early diagnosis and training rehabilitation of cognitive impairment. In addition, in the field of brain-computer interface (BCI), distraction of attention will reduce the performance of BCI. Therefore, rapid and accurate identification of attention states is very important for designing robust and reliable BCI systems. At present, the measurement of attention states can be divided into subjective measurement and objective measurement. Subjective measurement measures people's attention states through questionnaires and psychological tests, but cannot produce real-time, objective and accurate results. Objective measurement is mainly achieved by decoding various physiological signals. This method is clear and reliable in measuring attention states. Electroencephalogram (EEG) and near-infrared spectroscopy (fNIRS) are commonly used physiological signals.

[0003] Studies have shown that there is a correlation between EEG and attention state. Although EEG has the advantages of high temporal resolution and rapid responsiveness, it is susceptible to motion artifacts and electromagnetic noise. fNIRS provides excellent spatial resolution and robustness, but its temporal resolution is poor and there is a delay in hemodynamic response, which hinders the application of real-time BCI. Wang et al. introduced delayed hemodynamic response as domain knowledge into fNIRS classification and proposed a concise and effective fNIRSNet model. However, this method is insensitive to outliers and is prone to severe overfitting. Physiological studies have shown that there is a correlation between EEG rhythm and fNIRS hemodynamic response, indicating that a hybrid BCI system that integrates these two modalities can significantly improve classification performance.

[0004] The differences between EEG and fNIRS in terms of resolution and sensor placement make their integration challenging. Existing methods usually adopt feature-level fusion strategies. Shin et al. extracted common spatial pattern (CSP) features and mean slope features from both signals and used shrinkage linear discriminant (LDA) for classification. However, these methods are limited to manual feature extraction and concatenation, which limits their ability to fully exploit the interaction between EEG and fNIRS. Hybrid EEG-fNIRS models enabled by deep learning can autonomously extract effective representations from each modality and significantly improve classification performance. Sun et al. proposed a tensor decomposition-based EEG-fNIRS fusion method, which used one-dimensional convolutional neural networks (1D CNNs) to learn features in the time domain to achieve end-to-end fusion classification. This method ignores the intrinsic coupling between modalities. He et al. developed an M2NN framework for EEG-fNIRS recognition. Although these methods improve the performance of multimodal recognition, they mainly rely on feature concatenation on fully connected layers and ignore the deeper associations between modalities. Other frameworks, such as FGANet proposed by Kwak et al., consider 3D alignment between modalities, but the alignment process is complex and may generate a lot of redundant data.

[0005] Gu et al. pioneered Mamba, an innovative approach based on state-space model (SSM) that can effectively capture dynamic changes and trends in time series data. This approach has shown great potential in EEG feature decoding. Summary of the invention

[0006] In order to improve the accuracy of cognitive attention state classification, effectively combine the complementary information of EEG-fNIRS, and reduce redundant calculations through feature decoupling methods, and improve the accuracy of attention state recognition, the problem to be solved in the present invention is to provide a cognitive state classification method based on feature decoupling EEG-fNIRS deep fusion, and an EEG-fNIRS end-to-end deep fusion framework. The framework consists of three key components: a network designed for EEG (EMCNet) combining Mamba and cross-attention technology, an fNIRS network (PfNIRSNet) based on spatial strategy reinforcement, and a deep fusion block (DFBlock). It uses fNIRS features to decompose two high-order EEG features into common features and private features. Finally, the features from the two modalities are merged and deeply fused through the cross-attention module.

[0007] To achieve the above object, the method of the present invention mainly comprises the following steps:

[0008] Step S1, set up the experiment according to the mental arithmetic (MA) experimental paradigm, synchronously collect the EEG signal and fNIRS data of the subjects under the mental arithmetic task. One trial includes a 2s introduction phase, a 10s task phase and a 15s-17s rest phase. There is a 60s buffer time between each trial. The mental arithmetic instructions are given in the introduction phase, and the subjects respond in the task phase; the collected data are marked as "mental arithmetic" and "resting", and the buffer phase [-62, -2]s, introduction phase [-2, 0]s, task phase [0, 10]s and rest phase [10, 25]s included in the whole experimental process are marked. Finally, the two collected data are preprocessed to obtain EEG signal data and hemoglobin concentration change data.

[0009] Step S2, take the EEG signal data and hemoglobin concentration change data in S1, perform data enhancement through the time window and send them to the deep fusion network to obtain the public features and private features of EEG, the primary features of fNIRS; as well as the EEG classification results, fNIRS classification results and the output fusion classification results.

[0010] In step S3, the EEG public features and private features obtained in S2 and the fNIRS primary features are input into the decoupling loss, aiming to decouple the EEG high-level features into inter-modality public features and EEG private features through the fNIRS modality to reduce redundant calculations. The EEG classification results, fNIRS classification results, and fusion results are respectively input into the cross entropy loss based on the label smoothing strategy for training, which is used as the main task to improve the model's attention state recognition performance.

[0011] A further improvement of the technical solution of the present invention is that: in the EEG signal preprocessing process of step S1, firstly, two eye movement signals (EOG) are removed, the EEG signal is downsampled to 200Hz, and then the [-5, -2]s data of the buffer stage is taken for baseline correction, and the baseline correction is performed by subtracting the average value between -5s and -2s from the timing signal of each EEG channel. Finally, a 1Hz-40Hz bandpass filter is applied for noise reduction. For near-infrared signals, the data is first downsampled to 10Hz, and then the near-infrared signal is converted into oxygenated hemoglobin concentration change and deoxygenated hemoglobin concentration change data by the modified Beer-Lambert law, and finally baseline correction and 0.01Hz-0.1Hz bandpass filtering are performed.

[0012] The oxygenated hemoglobin and deoxygenated hemoglobin data were spliced ​​in the spatial dimension. During the experiment, only the 12s data representing the introduction phase and the task phase were taken. The sizes of the final EEG and near-infrared data were (N, K, C e ,T e ) and (N,K,C f ,Tf ), where N is the number of subjects, K is the number of experiments, and C e and C f represents the number of brain signal channels of EEG and fNIRS, T e and T f Represents a 12s time series.

[0013] A further improvement of the technical solution of the present invention is that the specific steps of step S2 are:

[0014] Step S21, taking the preprocessed data of step S1, for data enhancement, the data is processed with a time window of 3s and a step of 1s, and finally the EEG and near-infrared data input matrix is ​​obtained.

[0015] In step S22, the EEG and near-infrared data input matrices after the time window processing in step S21 are respectively input into the EEG branch and near-infrared branch of the deep fusion network for feature extraction, and finally classified and output the classification results.

[0016] Step S221, the EEG input matrix is ​​firstly processed through a convolution block to obtain the primary feature E b , the convolution block consists of a series of temporal convolution, spatial convolution and average pooling layers, and the sizes of temporal convolution and spatial convolution are (1,25) and (30,1) respectively. Then, private features and public features are obtained through two parallel Mamba structures, and the two features are input into the deep fusion module to obtain the final classification result. The deep fusion module consists of a cross attention block and a series of fully connected layers. The public features and private features are input as Q and KV of the self-attention block respectively. The Mamba calculation process is as follows:

[0017] M1=SSM(σ(DWConv(Linear(E b ))))

[0018] M2=σ(Linear(E b ))

[0019] E p,c =Linear(M1⊙M2)

[0020] Where ⊙ represents dot product, σ represents activation function, SiLU is used here, DWConv represents depthwise separable convolution, Linear represents linear layer, and SSM layer is implemented based on selective state space sequence model, which is used to encode and process EEG features. First, the patch sequence is expanded by selective scanning in multiple different directions on the two-dimensional feature map, and the resulting sequence is processed by state space modeling, and then merged back. In this process, the discretized state space model is constructed:

[0021] ht =Ah t-1 +Bu t

[0022] y t =Ch t +Du t

[0023] where h t represents the hidden state, u t and t denote the input and output sequences respectively, They are the state matrix, input matrix, output matrix and direct transfer matrix, whose parameters can be learned, and N represents the dimension of state representation.

[0024] In step S222, the near-infrared input matrix first obtains primary features through a convolution block, wherein the convolution block is composed of a series of temporal convolution, spatial convolution, and point convolution, and the primary features are passed through a fully connected layer to obtain a near-infrared classification result.

[0025] In step S223, the deep fusion between modalities is completed by the deep fusion module, which includes a self-attention block and a fully connected layer. The public features of the EEG data and the primary features of the near-infrared data fNIRS are concatenated as the Q of the self-attention block, and the primary features and private features of the EEG data are concatenated as KV. After passing through the self-attention block, the classification result is output through the fully connected layer.

[0026] A further improvement of the technical solution of the present invention is that the convolution kernels of the temporal convolution and the spatial convolution in step S222 are both strip convolutions, and the step length of the spatial convolution is (2, 1), wherein the spatial convolution retains the spatial structural characteristics of the near-infrared data to a certain extent.

[0027] A further improvement of the technical solution of the present invention is that the decoupling loss calculation idea in step S3 is as follows:

[0028] After extracting features from EEG and fNIRS, a disentanglement loss L is designed dis , cosine embedding loss is used to measure the similarity between different modalities. The calculation formula is as follows:

[0029]

[0030]

[0031]

[0032] Where ε is a positive number used to prevent the denominator from being zero, and cosloss is the cosine embedding loss. c and E pRepresent the public and private features of EEG, respectively, and F b Represents fNIRS features.

[0033] A further improvement of the technical solution of the present invention is that the idea of ​​calculating the cross entropy based on the label smoothing strategy in step S3 is as follows:

[0034] The idea of ​​label smoothing is introduced to adjust the cross entropy loss function by introducing a smoothing parameter. Specifically, the actual label y is adjusted from 0 and 1 to y' i :

[0035]

[0036] Where δ is the smoothing parameter, K is the number of categories, and the final cross entropy loss based on the label smoothing strategy is written as:

[0037]

[0038] in, For the prediction results.

[0039] Compared with the existing EEG-fNIRS multimodal fusion method, the present invention has the following advantages:

[0040] In order to solve the problems of feature redundancy and insufficient fusion in the existing EEG-fNIRS integration work, the present invention develops an end-to-end feature decomposition-based deep fusion framework for EEG-fNIRS. This framework makes full use of the complementarity between modalities and pioneeringly applies deep fusion strategies to EEG and fNIRS, significantly improving the performance of attention state classification. In order to effectively filter out redundant information in EEG signals, EMCNet is proposed, which explores the role of Mamba in EEG feature extraction and organically combines it with the Transformer architecture. At the same time, a new improved strategy based on fNIRSNet is proposed, which retains the original spatial structure characteristics of fNIRS. It effectively alleviates the overfitting problem in deep learning of fNIRS modality and can accurately classify cognitive states. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the flow chart of EEG signal preprocessing;

[0042] Figure 2 This is a flowchart of near infrared signal preprocessing;

[0043] Figure 3 The EEG-fNIRS deep fusion network based on feature decoupling proposed in the present invention;

[0044] Figure 4This is a structural diagram of Mamba and self-attention involved in the present invention. DETAILED DESCRIPTION

[0045] The following is a detailed description of an embodiment of the cognitive state classification method based on feature decoupling and EEG-fNIRS deep fusion of the present invention in conjunction with the accompanying drawings: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation plan and specific operation process are given.

[0046] Studies have shown that EEG and fNIRS are two complementary brain physiological signals, and electrophysiological signals and optical signals will not interfere with each other during synchronous acquisition. Therefore, EEG-fNIRS multimodal fusion research can effectively improve the efficiency of attention state recognition, which is of great significance for cognitive brain function research. This paper proposes an EEG-fNIRS deep fusion method based on feature decoupling, which mainly involves EFDFNet, including EMCNet, PfNIRSNet and DFBlock. The implementation of the present invention mainly includes three steps: (1) EEG-fNIRS data acquisition and preprocessing; (2) multimodal feature extraction and attention state recognition; (3) attention state recognition performance optimization and feature decoupling.

[0047] The following is a detailed description of each step.

[0048] Step S1: EEG-fNIRS data acquisition and preprocessing

[0049] The subjects were instructed to conduct the experiment according to the mental arithmetic experimental paradigm. When performing the mental arithmetic task, the participants performed subtraction operations, such as "subtracting one from a three-digit number", according to the visual cues and short beep signals displayed on the screen; in the baseline task, the participants were asked to relax by looking at the black fixation cross on the screen. Each participant completed 30 experiments on both tasks. Each experiment started with a 2-second visual introduction, followed by a 10-second task period, and finally a rest period randomly set at 15-17 seconds. EEG signals were recorded using 32 electrodes at a sampling rate of 200Hz, while 36 channels of near-infrared data were recorded using a functional near-infrared spectroscopy device.

[0050] Figure 1This is the preprocessing process of EEG data. EEG signals are easily interfered by ECG signals, power frequency electromagnetic waves, and other sources. In the EEG preprocessing stage, the two eye movement channels are first removed, and the [-5,-2]s interval before the task starts is used for baseline correction. Next, a bandpass filter with a frequency range of 1-40Hz is applied, and only the [-2,10]s interval of the instruction and task phases is used for analysis during the implementation. Therefore, the dimension of the final EEG dataset is (N,60,30,2400), which represents the number of subjects × number of trials × number of channels × number of sampling points.

[0051] The preprocessing process of near infrared signals is as follows: Figure 2 As shown, the data is first downsampled to 10Hz. Since it has a certain resistance to external interference, the optical density ΔA(λ) is converted into the concentration of oxygenated hemoglobin (ΔHbO) and deoxygenated hemoglobin (ΔHbR) using the modified Beer-Lambert law. The conversion process is described as follows:

[0052]

[0053] where ε HbO (λ1), ε HbO (λ2), ε HbR (λ1) and ε HbR (λ2) represents the absorption coefficient of HbO and HbR at different wavelengths, and d represents the distance between the signal transmitter and the receiver. Dividing by d gives the actual concentration changes Δ[HbO] and Δ[HbR]. Baseline correction was then performed and the signal was bandpass filtered from 0.01 to 0.1 Hz to retain relevant physiological information. Consistent with the EEG data, we segmented the time period of [-2,10] s for data analysis. The dimension of the near-infrared data is (N,60,72,120), which means the number of subjects × number of trials × number of channels × number of sampling points.

[0054] Step S2: Multimodal feature extraction and attention state recognition

[0055] The EEG-fNIRS deep fusion network based on feature decoupling involved in the present invention is as follows Figure 3 As shown, the specific implementation steps are:

[0056] Step S21, in order to enhance the data, the EEG and near-infrared signals are segmented by a time window with a size of 3s and a step length of 1s, and each experiment is divided into 10 segments.

[0057] Step S22, the EEG and near infrared matrices after the time window processing in step S21 are respectively input into the EEG branch and near infrared branch of the deep fusion network for feature extraction, and finally classified and output the classification results.

[0058] Step S221, the EEG input matrix is ​​firstly processed through a convolution block to obtain the primary feature E b , the convolution block contains time convolution, space convolution and average pooling layer, where the time convolution kernel size is (1,25) and the space convolution kernel size is (30,1). Then, two Mamba structures are used to obtain private features and public features. The two features are input into the deep fusion module to obtain the final classification result. The deep fusion module consists of a self-attention structure and a fully connected layer. The public features and private features are respectively input as Q and KV of the self-attention block. The Mamba and self-attention structures are shown in the figure below. Figure 4 As shown, the Mamba calculation process is as follows:

[0059] M1=SSM(σ(DWConv(Linear(E b ))))

[0060] M2=σ(Linear(E b ))

[0061] E p,c =Linear(M1⊙M2)

[0062] Among them, ⊙ represents the dot product, σ represents the activation function, SiLU is used here, and the SSM layer is implemented based on the selective state space sequence model to encode and process EEG features. The calculation of the cross attention block is as follows:

[0063]

[0064] Where k is the scaling factor, which is equal to the length of the token.

[0065] In step S222, the near-infrared input matrix first obtains primary features through a convolution block, wherein the convolution block includes temporal convolution, spatial convolution and point convolution, the convolution kernel sizes of temporal convolution and spatial convolution are (1,30) and (14,1) respectively, and the spatial convolution step size is (2,1), and the primary features are passed through a fully connected layer to obtain a near-infrared classification result.

[0066] In step S223, the deep fusion between modalities is completed by the deep fusion module, which includes a self-attention block and a fully connected layer. The public features of the EEG data and the primary features of the near-infrared data are concatenated as the Q of the self-attention block, and the primary features and private features of the EEG data are concatenated as KV, which are input into the deep fusion block to obtain the final classification result.

[0067] Step S3: Attention state recognition performance optimization and feature decoupling

[0068] After extracting features from EEG and fNIRS, their respective temporal-spatial features are obtained. In order to further separate the EEG features, a disentanglement loss L is designed. dis , whose purpose is to make the public features of EEG closer to the fNIRS features in the feature space, while keeping the private features as far away as possible. This separation aims to clearly distinguish between private and public features in EEG. The cosine embedding loss is used to measure the similarity between different modalities. The calculation formula is as follows:

[0069]

[0070]

[0071]

[0072] Where ε is a positive number used to prevent the denominator from being 0, which is set to 1.01 here, and cosloss is the cosine embedding loss. c and E p Represent the public and private features of EEG, respectively, and F b Represents fNIRS features.

[0073] A further improvement of the technical solution of the present invention is that the idea of ​​calculating the cross entropy based on the label smoothing strategy in step S3 is as follows:

[0074] The idea of ​​label smoothing is introduced. In order to reduce the model's overconfidence in the label, a smoothing parameter is introduced to adjust the cross entropy loss function. Specifically, the actual label y is adjusted from 0 and 1 to y' i :

[0075]

[0076] Where δ is the smoothing parameter, which defaults to 0.1, and K is the number of categories. When the model predicts incorrectly, the actual label will be close to the probability of other categories, thereby reducing the model's penalty for incorrect predictions. The final cross entropy loss based on the label smoothing strategy is written as:

[0077]

[0078] in, For the prediction results.

[0079] The EEG high-level features and fNIRS primary features obtained by S2 are input into the decoupling loss, aiming to decouple the EEG high-level features into common features between modalities and EEG private features through the fNIRS modality to reduce redundant calculations. EEG, fNIRS and fusion results are respectively input into the cross entropy loss based on the label smoothing strategy as the main task to improve the model's attention state recognition performance. Finally, the model is optimized by minimizing the following loss:

[0080] Loss = L task +αL dis

[0081] L task =L eeg +L fNIRS +L fusion

[0082] Among them, α is the weight, L eeg , L fNIRS , L fusion Losses computed with label smoothing for EEG, fNIRS, and fusion branches, respectively.

[0083] Table 1 shows the performance comparison of the proposed deep fusion framework in single modality and multimodality, where EMCNet is the EEG feature extraction branch and PfNIRSNet is the near-infrared feature extraction branch. The average accuracy of the present invention is improved by 1.2% compared with the EEG single modality and 13.37% compared with the near-infrared single modality. Among them, 21 subjects (about 72.4%) have improved recognition accuracy after fusion.

[0084] Table 1 Comparison of multimodal and unimodal classification performance

[0085]

[0086]

[0087] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A cognitive state classification method based on feature decoupling EEG-fNIRS deep fusion, characterized in that: The following steps are involved: Step S1, setting up an experiment according to the mental arithmetic experimental paradigm, synchronously collecting EEG data and fNIRS data of the subjects under the mental arithmetic task, and preprocessing the two collected data to obtain EEG data and hemoglobin concentration change data; Step S2, taking EEG signal data and hemoglobin concentration change data, performing data enhancement through a time window and then sending them into a deep fusion network to obtain public and private features of EEG, primary features of fNIRS; as well as EEG classification results, fNIRS classification results and output fusion classification results; In step S3, the EEG public features, private features, and fNIRS primary features are input into the decoupling loss, and the EEG classification results, fNIRS classification results, and fusion results are respectively input into the cross entropy loss based on the label smoothing strategy for training.

2. The cognitive state classification method based on feature decoupling and EEG-fNIRS deep fusion according to claim 1 is characterized in that: The experiment in step S1 is set up according to the mental arithmetic experimental paradigm, and one trial includes a 2-s introduction phase, a 10-s task phase, and a 15-17-s rest phase. There is a 60-s buffer time between each trial. Mental arithmetic instructions are given in the introduction phase, and the subjects respond in the task phase; the collected data are labeled as mental arithmetic and rest, and the buffer phase [-62, -2] s, introduction phase [-2, 0] s, task phase [0, 10] s, and rest phase [10, 25] s included in the entire experimental process are marked.

3. The cognitive state classification method based on feature decoupling and EEG-fNIRS deep fusion according to claim 2 is characterized in that: The preprocessing of step S1 is specifically as follows: firstly, the two eye movement signals are removed, the EEG signal is downsampled to 200 Hz, then the [-5, -2] s data in the buffer stage is taken for baseline correction, and finally a 1 Hz-40 Hz bandpass filter is applied for noise reduction; for near-infrared signals, the data is first downsampled to 10 Hz, and then the near-infrared signals are converted into oxygenated hemoglobin concentration change data and deoxygenated hemoglobin concentration change data by using the modified Beer-Lambert law, and finally baseline correction and a 0.01 Hz-0.1 Hz bandpass filter are performed; The oxygenated hemoglobin and deoxygenated hemoglobin data were spliced ​​in the spatial dimension. During the experiment, only the 12s data representing the introduction phase and the task phase were taken. The sizes of the final EEG and near-infrared data were (N, K, C e ,T e ) and (N,K,C f ,T f ), where N is the number of subjects, K is the number of experiments, and C e and C f represents the number of brain signal channels of EEG and fNIRS, T e and T f Represents a 12s time series.

4. The cognitive state of EEG-fNIRS deep fusion based on feature decoupling according to claim 3 A classification method, characterized in that The specific implementation process of step S2 is as follows: Step S21, taking the preprocessed data of step S1, performing time window processing on the data with a size of 3s and a step length of 1s, and obtaining an EEG and near-infrared data input matrix; Step S22, the EEG and near-infrared data are input into a matrix, respectively input into the EEG branch and near-infrared branch of the deep fusion network for feature extraction, and finally classified and output the classification results.

5. The cognitive state classification method based on feature decoupling and EEG-fNIRS deep fusion according to claim 4 is characterized in that: The specific implementation process of S22 is as follows: Step S221, the EEG input matrix is ​​firstly processed through a convolution block to obtain the primary feature E b , the convolution block consists of a series of temporal convolution, spatial convolution and average pooling layers; then private features and public features are obtained through two parallel Mamba structures, and the two features are input into the deep fusion module to obtain the final classification result, where the deep fusion module consists of a cross attention block and a series of fully connected layers, and the public features and private features are respectively input as Q and KV of the self-attention block; the Mamba calculation process is as follows: M1=SSM(σ(DWConv(Linear(E b )))) M2=σ(Linear(E b )) E p，c =Linear(M1⊙M2) Among them, ⊙ represents dot product, σ represents activation function, SiLU is used here, DWConv represents depthwise separable convolution, Linear represents linear layer, and SSM layer is implemented based on selective state space sequence model to encode and process EEG features. First, the patch sequence is expanded by selective scanning in multiple different directions on the two-dimensional feature map, the resulting sequence is processed by state space modeling, and then merged; Step S222, the near-infrared input matrix first obtains primary features through a convolution block, wherein the convolution block is composed of a series of temporal convolution, spatial convolution and point convolution, and the primary features are passed through a fully connected layer to obtain a near-infrared classification result; In step S223, the public features of EEG and the primary features of fNIRS are concatenated as Q of the self-attention block, and the primary features of EEG and the private features are concatenated as KV. After passing through the self-attention block, the classification results are output through the fully connected layer.

6. The cognitive state classification method based on feature decoupling and EEG-fNIRS deep fusion according to claim 5, characterized in that: The decoupling loss calculation idea in step S3 is as follows: After extracting features from EEG and fNIRS, a disentanglement loss L is designed dis , the cosine embedding loss is used to measure the similarity between different modalities, and the calculation formula is as follows: Where ε is a positive number used to prevent the denominator from being 0, cosloss is the cosine embedding loss; E c and E p Represent the public and private features of EEG, respectively, and F b represents fNIRS features; The idea of ​​calculating the cross entropy based on the label smoothing strategy is as follows: A smoothing parameter is introduced to adjust the cross entropy loss function, adjusting the actual label y from 0 and 1 to y′ i : Where δ is the smoothing parameter, K is the number of categories, and the cross entropy loss based on the label smoothing strategy is written as: in, For the prediction results.

Images