Cross-modal cognitive load assessment method based on dual residual network

Abstract

The application provides a cross-modal cognitive load evaluation method based on a double residual network, which effectively improves the accuracy of cognitive load evaluation. The method first acquires electroencephalogram (EEG) data and functional near-infrared spectroscopy (fNIRS) data, and then performs band-pass filtering, downsampling and other preprocessing operations on the two different data. The preprocessed data are used as inputs to train a space-time feature learning module to obtain feature information. The two types of feature information are fused to obtain high-dimensional space-time features. A double residual network is constructed to extract multi-modal features. Finally, a full connection layer is used to realize cognitive load classification. The application can replace manual feature extraction and fuse cross-modal information features by building a double residual network module to realize cross-modal cognitive load evaluation.

Description

Technical Field

[0001] This invention relates to a cross-modal cognitive load assessment method based on dual residual networks, belonging to the field of cognitive load assessment in brain-computer interface technology. Background Technology

[0002] Brain-computer interface (BCI) is a technology that connects the human brain to other external devices without requiring peripheral nerves or voice commands. It uses brain signals to control these devices and transmit information. BCI has been widely applied in medical rehabilitation, communication, entertainment, and clinical rehabilitation, achieving significant results. With the continuous development and improvement of BCI technology, it will be applied to even more scenarios in the future. With the development of non-invasive neuroimaging technology, many signals have been widely used in the field of BCI, including electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). EEG is an electrophysiological signal that measures the electrical potential of neurons in the brain, while fNIRS primarily measures changes in blood oxygen concentration using optical signals. Meanwhile, EEG and fNIRS signals have advantages such as being non-invasive, portable, low-cost, and easy to use, making them the most popular signals in cognitive workload research.

[0003] Traditional unimodal cognitive load assessment methods fail to achieve satisfactory classification performance due to their limited representation capabilities of EEG or fNIRS signals. Numerous studies have used EEG and fNIRS signals for BCI system analysis, employing power spectral analysis to determine frequency bands, studying their impact, and using Support Vector Machines (SVMs) for assessment. However, handcrafted features rely on prior knowledge, thus limiting classification accuracy. Deep learning techniques represent a promising feature learning approach that can assess cognitive load without prior knowledge.

[0004] Therefore, this invention proposes a cross-modal cognitive load assessment method based on a dual residual network. The model mainly consists of two modules: a spatiotemporal feature learning module and a dual residual network module. First, due to the different characteristics of EEG and fNIRS, this invention utilizes two independent branches to extract temporal and spatial features from EEG and fNIRS signals respectively. Second, to better extract information features from multimodal data, a dual residual network is proposed. Using the dual residual network to extract multimodal features maximizes the utilization rate of hidden features in the residual module. Finally, a fully connected layer completes the final classification task. This invention can replace manual feature extraction, achieving an end-to-end learning approach, effectively compensating for the differences between heterogeneous modal data, improving the accuracy of cognitive load analysis, and providing a new approach and solution for effectively integrating information from two modalities. Summary of the Invention

[0005] The purpose of this invention is to design a cross-modal cognitive load assessment method based on dual residual networks, which effectively improves the accuracy of cognitive load assessment.

[0006] To achieve the above objectives, this method first proposes a cross-modal cognitive load assessment method based on dual residual networks, which mainly includes a spatiotemporal feature learning module and a dual residual network module, to fully learn the temporal and spatial features in EEG and fNIRS.

[0007] This invention includes the following steps:

[0008] (1) Conduct the n-back experimental task, in which the subject judges whether the current stimulus is the same as one of the previous n stimuli, and simultaneously collect the generated EEG data and functional near-infrared spectroscopy fNIRS data.

[0009] (2) The EEG and fNIRS data of the subjects were preprocessed separately. First, due to the different wavelengths, the optical density of the fNIRS data was converted into the change in hemoglobin concentration using the modified Beer-Lambert law, that is, the change in the concentration of hemoglobin and deoxyhemoglobin. Second, because environmental noise and subject blinking and movement during brain experiments can affect data quality, noise removal and bandpass filtering were performed on the EEG and fNIRS data respectively.

[0010] (3) Based on the design of the n-back experimental paradigm, the labels L for the 0-back, 1-back, and 2-back tasks in the EEG and fNIRS data were respectively... EEG and L fNIRS It is represented as {0, 1, 2}, and the data is normalized using the z-score normalization method;

[0011] (4) The spatiotemporal feature learning module is designed with two independent branches based on CNN, which extract feature information X from EEG and fNIRS data from both temporal and spatial dimensions respectively. EEG and X fNIRS We trained the EEG and fNIRS data separately using leave-one-out cross-validation.

[0012] (5) Further, considering the characteristics of both datasets, a dual residual network module is designed, which consists of four residual feature fusion modules (RFF). Each module contains four residual blocks (RB), and the output features of all residuals are concatenated using skip connections. Finally, a convolutional layer is used to fuse the concatenated features to obtain the output of the RFF module.

[0013] (6) Conduct multiple experiments to explore the optimal network parameters and use various evaluation metrics to evaluate the model, such as accuracy, precision and recall.

[0014] (7) Design ablation experiments to explore the problems solved by each module and prove the effectiveness of the module;

[0015] (8) Prove the feasibility and superiority of the method, that is, the method can achieve the fusion of EEG and fNIRS data, and improve the accuracy of the fusion results through this method.

[0016] The beneficial effects of this invention are as follows: It proposes a cross-modal cognitive load assessment method based on a dual residual network, which effectively improves the accuracy of cognitive load assessment. This method first acquires EEG data and fNIRS data. The fNIRS data is converted from optical density to hemoglobin data using the modified Beer-Lambert law. Then, bandpass filtering and downsampling are performed on the two different data types respectively for preprocessing. The preprocessed data are used as inputs, and a spatiotemporal feature learning module is used for training to obtain feature information. The two feature information types are fused to obtain high-dimensional spatiotemporal features. Multimodal features are extracted by constructing a dual residual network, and finally, a fully connected layer is used to achieve cognitive load classification. This invention can replace manual feature extraction. By constructing a dual residual network module to fuse cross-modal information features, it achieves cross-modal cognitive load assessment, providing a new approach and solution for effectively integrating information from two modalities. Brief description of the attached figures

[0017] Figure 1 : Overall flowchart of the cross-modal cognitive load assessment method based on dual residual networks.

[0018] Figure 2 : Schematic diagram of the spatiotemporal feature learning module proposed in this invention.

[0019] Figure 3 : Schematic diagram of the dual residual network module proposed in this invention. Detailed Implementation

[0020] This invention provides a cross-modal cognitive load assessment method based on a dual residual network. This model can fully utilize the complementarity of the temporal and spatial features of the two signals and achieve the fusion of different signals using deep learning. The specific implementation steps are as follows:

[0021] 1. First, the subjects performed n-back tasks under three different load conditions, namely 0-back, 1-back, and 2-back tasks. Each subject performed a total of 180 trials for each task type. The channel positions for synchronous acquisition all conformed to the international 10-5 system: 30 channels of EEG signal acquisition, including right eye upper and lower eye movement (EOG) data acquisition positions, with a sampling frequency of 1000Hz; 36 channels of fNIRS signal acquisition, with channel data acquisition positions evenly distributed on the head, and a sampling frequency of 10.4Hz.

[0022] 2. During brain experiment data collection, environmental noise and factors such as blinking and movement of the subjects can affect data quality. This noise can interfere with the signal features we want to capture; therefore, the acquired EEG and fNIRS data from the subjects must be preprocessed separately.

[0023] 3. EEG data preprocessing: The sampling rate of EEG in the dataset used in this invention is 1000Hz, which needs to be downsampled to 200Hz first. In order to obtain more suitable physiological information, a third-order Butterworth filter is used for bandpass filtering from 1 to 40Hz, and components such as electrooculography are removed using the bbci toolbox.

[0024] 4. fNIRS Data Preprocessing: Since this data is light intensity data, it is first converted into light density data. HbR and HbO are calculated using the modified Beer-Lambert law. A third-order Butterworth filter is applied to HbR and HbO for bandpass filtering in the 0.01-0.2Hz range. The modified Beer-Lambert law includes the following steps: The changes in oxyhemoglobin (HbO) concentration ΔHbO and deoxyhemoglobin (HbR) concentration ΔHbR are derived from it.

[0025]

[0026]

[0027] Where ε1 is the extinction coefficient of oxyhemoglobin, ε2 is the extinction coefficient of deoxyhemoglobin, λ1 represents wavelength, and λ2 represents another wavelength. This represents the extinction coefficient of deoxyhemoglobin at wavelength λ2. This represents the change in absorbance at wavelength λ1. This represents the extinction coefficient of deoxyhemoglobin at wavelength λ1. This represents the change in absorbance at wavelength λ2. This represents the extinction coefficient of deoxyhemoglobin at wavelength λ1. λ is the extinction coefficient of deoxyhemoglobin at wavelength λ2.

[0028] 5. Obtain EEG and fNIRS data:

[0029] (1)R t If the data represents all channels of HbR data at time t, then... i represents the channel number of HbR, with a total of 36 channels; O t If the data represents all channels of HbO data at time t, then... i represents the channel number of HbO, with a total of 36 channels; the data from both are concatenated according to their two-dimensional relative positions to obtain the fNIRS data. m represents the channel number of the fNIRS data;

[0030] (2)E t If the EEG data represents all channels at time t, then... j represents the channel number of the EEG; there are a total of 28 channels after the experimental data preprocessing.

[0031] (3) Perform z-score normalization on the EEG and fNIRS data respectively:

[0032]

[0033] Where X is E t or F t Z is E t or F t The normalized data, μ is E t or F t The average value, σ is E t or F t The standard deviation.

[0034] 6. After obtaining the preprocessed EEG data, a spatiotemporal feature module network model is designed to extract the temporal and spatial features of EEG and fNIRS. This model mainly includes two independent branches based on CNN:

[0035] (1) The EEG branch's modular structure consists of three 2D convolutional modules. The kernel size of the first and third layers is (1, 64), and the number of filters is 8, used to extract temporal features. Each convolutional module provides perceptual information for the next step. The kernel size of the second convolutional module is (channels, 1), where channels refer to the number of channels in the EEG signal. After each convolutional layer, there is a batch normalization layer and an average pooling layer to accelerate convergence, finally obtaining the feature X. EEG ;

[0036] (2) The module structure of the fNIRS branch consists of four 2D convolutional layers, which are encapsulated as convolutional modules. The kernel size of the first layer is (1, 3), and the number of filters is 8. The kernel size of the second layer is (36, 1), and the number of filters is 16. The remaining layers consist of two consecutive two-dimensional convolutions with kernel sizes of (channels, 1) and (1, 64), where channels refer to the number of channels in the fNIRS signal. After each convolutional layer, there is a batch normalization layer and an average pooling layer to accelerate convergence and further preserve the temporal and spatial features, finally obtaining the feature X. fNIRS .

[0037] (3) After (1) and (2) respectively, the output X is obtained. EEG and X fNIRS X is converted using the concat operation. EEG and X fNIRS The output X is obtained by merging and combining. Input ;

[0038] 7. Place X Input As input, the signal is fed into a dual residual network module. This module employs a dual structure consisting of a backbone network and an inner network to extract hidden features between signals. The backbone network primarily uses four Residual Feature Fusion (RFF) modules to concatenate all features via skip connections. The inner network primarily uses four Residual Block (RB) modules to concatenate all features via skip connections. First, the X obtained in the previous step... EEG and X fNIRS Perform concat to get X Input X InputAs input, the data is passed to the Regression Block (RB). The RB first passes through two 3×3 convolutional neural networks. After the first convolutional neural network, a ReLU activation function is used to enhance the model's feature representation capabilities. Each RB process is as follows:

[0039] 1)X Input =Concat([X EEG X fNIRS ]);

[0040] 2) S1 = Conv(X) Input );

[0041] 3) S2 = σ(S1);

[0042] 4) S3 = Conv(S2);

[0043] 5)X Output-i =S3+X Input ;

[0044] The features extracted from the first three RBs are then directly passed to the end of the last RB module, and the output of the last module is concatenated: X Output =Concat([X Output-1 X Output-1 , ..., X Output-4 ]);

[0045] Among them, X Input Let X be the input feature, σ(·) be the ReLU activation function, S1, S2, and S3 be the features after convolution and activation, and X be the input feature. Output-i For each feature output by RB, X Output The features obtained after concatenation;

[0046] Finally, a 1×1 convolution is used to obtain the final feature information. In the backbone network, a process similar to that of the internal network structure is adopted, and spatiotemporal features after EEG-near-infrared fusion are collected to the maximum extent through skip connections.

[0047] 8. The features extracted from the model training are processed through a fully connected layer for three-class classification, thus realizing an end-to-end deep learning method.

[0048] 9. Accuracy, precision, recall, and F1 score were used as evaluation metrics to demonstrate the superiority of this method in cognitive load analysis. The formulas for these metrics are as follows:

[0049]

[0050]

[0051]

[0052]

[0053] In this context, TP (True Positive) represents the number of samples correctly predicted as positive, FN (False Negative) represents the number of samples correctly predicted as negative, FP (False Positive) represents the number of samples incorrectly predicted as positive, and TN (True Negative) represents the number of samples incorrectly predicted as negative. These represent different prediction outcomes. Precision reflects the model's accuracy, while recall reflects its comprehensiveness. The F1-score is related to both precision and recall and is the harmonic mean of the two. Higher values ​​for these metrics generally indicate better classification performance.

[0054] It should be noted that the above descriptions are merely embodiments of the present invention and are only intended to explain the invention, and are not intended to limit the scope of the patent. Modifications that are obvious from the inventive concept are also within the scope of protection of the present invention.

Claims

1. A cross-modal cognitive load assessment method based on dual residual networks, characterized in that, Includes the following steps: (1) The subjects were given an n-back experimental task, in which they judged whether the current stimulus was similar to the previous one. n One of the stimuli is the same, and the electroencephalogram (EEG) data and functional near-infrared spectroscopy (fNIRS) data generated by the subjects are collected simultaneously. (2) Preprocess the EEG data and fNIRS data respectively; (3) Based on the n-back experimental task, label the 0-back, 1-back and 2-back tasks in the EEG data and fNIRS data respectively. and Represented as The data is then normalized using the z-score normalization method. (4) Design a spatiotemporal feature learning module to extract feature information from two different signal features, EEG and fNIRS. and Cross-validation was used to train the EEG and fNIRS data respectively; (5) Design a dual residual network module to convert the residual obtained in the previous step into a dual residual network module. and Performing the concat operation yields The input to the dual residual network structure is used as the input; after extracting multimodal features through the dual residual network module, a fully connected layer is used to achieve cognitive load classification. The dual residual network module employs a dual structure consisting of a backbone network and an internal network to extract hidden features between signals. The backbone network primarily uses four Residual Feature Fusion (RFF) modules that perform skip connections to concatenate all features. The internal network primarily uses four Residual Block (RB) modules that perform skip connections to concatenate all features. The process for each RB is as follows: 1） ； 2） ； 3） ； 4） ； 5） ； The features extracted from the first three RBs are then directly passed to the end of the last RB module, and the output of the last module is fused and skipped. ； in, As input features, For ReLU activation functions, The features are those obtained after convolution and activation functions. For each RB output feature, The features obtained after concatenation; Finally, 1×1 convolution was used to extract features; in the backbone network, a process similar to that of the internal network structure was adopted, and spatiotemporal features after EEG-near-infrared fusion were collected to the maximum extent through skip connections.

2. The cross-modal cognitive load assessment method based on dual residual networks as described in claim 1, characterized in that, In step (1), the n-back experimental task is performed, and the generated electroencephalogram (EEG) data and functional near-infrared spectroscopy (fNIRS) data are collected simultaneously. The specific method is as follows: (1) First, the subjects performed n-back tasks under three different loads, namely 0-back, 1-back and 2-back tasks. Each subject performed a total of 180 trials for each task type. (2) EEG signal acquisition of 30 channels, including two right eye upper and lower eye EOG channels in addition to the head channel, with a sampling frequency of 1000Hz. fNIRS signal acquisition of 36 channels, with the channel data acquisition positions evenly distributed on the head, and a sampling frequency of 10.4Hz.

3. The cross-modal cognitive load assessment method based on dual residual networks as described in claim 1, characterized in that, The preprocessing of EEG and fNIRS data in step (2) is performed as follows: (1) The EEG signal preprocessing steps include rereference, bandpass filtering, interpolation bad derivative, artifact removal and downsampling. Among them, 0.1-40Hz bandpass filtering is used to remove environmental noise during EEG data acquisition, and two EOG channels in the channel data are deleted and downsampled to 200Hz. (2) fNIRS data preprocessing first converts optical density into hemoglobin concentration change using the modified Beer-Lambert law, then uses a 0.2Hz low-pass filter to remove interference signals, and finally downsamples it to 10Hz. The change in oxyhemoglobin (HbO) concentration is derived based on the modified Beer-Lambert law. : (1) Derive the change in deoxygenated hemoglobin (HbR) concentration. : (2) in, and These are the extinction coefficients of oxyhemoglobin and deoxyhemoglobin, respectively. and Representing two different wavelengths, For deoxyhemoglobin Extinction coefficient of wavelength, For deoxyhemoglobin Extinction coefficient of wavelength, for The change in wavelength absorbance for The change in wavelength absorbance, DPF is the path difference factor, and d is the distance the light travels in the medium.

4. The cross-modal cognitive load assessment method based on dual residual networks as described in claim 1, characterized in that, The data normalization process using the z-score normalization method in step (3) is as follows: (1) HbR data at time t All channel data, then ,1≤ i ≤36, i The channel number represents HbR; there are 36 channels in total. HbO data at time t All channel data, then ,1≤ i ≤36, i The channel numbers represent HbO, with a total of 36 channels; the data from both are concatenated according to their two-dimensional relative positions to obtain the fNIRS data. ,1≤ m ≤72, m The channel number representing the fNIRS data; (2) It is EEG data at any time t All channel data, then ,1≤ j ≤28, j The channel number represents the EEG data; there are 28 channels in total after preprocessing of the experimental data. (3) Perform z-score normalization on the EEG and fNIRS data respectively: ,in for or , for or Normalized data, for or The average value, for or The standard deviation.

5. The cross-modal cognitive load assessment method based on dual residual networks as described in claim 1, characterized in that, The spatiotemporal feature module in step (4) extracts feature information from two different signal features: EEG and fNIRS. and The specific method is as follows: A spatiotemporal feature module network model is designed to extract the temporal and spatial features of EEG and fNIRS. This model mainly includes two independent branches based on CNN: (1) The module structure of the EEG branch consists of three 2D convolutional modules. The kernel size of the first and third layers is (1, 64), and the number of filters is 8. These are used to extract the temporal features on a single channel. Each convolutional module provides perceptual information for the next step. The kernel size of the second convolutional module is (channels, 1), where channels refer to the number of channels in the EEG signal. This is used to extract the spatial features on multiple channels. After each convolutional layer, there is a batch normalization layer and an average pooling layer to speed up convergence. Finally, the features are obtained. ; (2) The module structure of the fNIRS branch consists of four 2D convolutional modules. The first layer has a kernel size of (1,3) and 8 filters to extract temporal features on a single channel. The second layer has a kernel size of (36,1) and 16 filters to extract spatial features on multiple channels. The remaining layers consist of two consecutive two-dimensional convolutions with kernel sizes of (channels,1) and (1,64), where channels refer to the number of channels in the fNIRS signal. Each convolutional layer is followed by a batch normalization layer and an average pooling layer to accelerate convergence, and finally, the features are obtained. .

Images