A method for analyzing brain cognitive function rehabilitation status based on multimodal data

Abstract

This invention belongs to the field of functional near-infrared spectroscopy and electroencephalogram (EEG) image processing technology. Specifically, it provides a method for analyzing the state of brain cognitive function rehabilitation based on multimodal data. The method collects resting fMRI data from patients, as well as fNIRS and EEG data from assessment tasks. The fMRI data is then input into an fMRI-FC-CNN network to determine the probability of classification as normal. The fNIRS and EEG data are then preprocessed and input into a multimodal feature fusion classification network to obtain the probability of classification as normal. This data is then plotted as a probability-visit number line graph, and the Mann-Kendall test is used to analyze the trend of the line graph. This method utilizes complementary data from two modalities to provide the network model with different information sources, thereby enhancing the learned features. Furthermore, the method utilizes the Inception module and residual structure to improve the LSTM model, enhancing the network's representational capabilities.

Description

Technical Field

[0001] The present invention belongs to the technical field of functional near-infrared spectroscopy and electroencephalogram image processing, and specifically is a method for analyzing the state of brain cognitive function rehabilitation based on multimodal data. Background Art

[0002] The brain is the body's most complex and efficient information processing system. Exploring the generation of human intelligence and the brain's information processing processes is a core area of ​​research in brain cognition. Brain cognition examines the fundamental expression of perception and information processing mechanisms at various levels, understanding how the brain forms perceptions of the external world and how it enables higher-level cognitive functions such as learning, memory, language, thinking, emotion, and consciousness.

[0003] Normal human work, study and life cannot be separated from perfect brain cognitive function, and some people suffer from cognitive dysfunction due to genetic or acquired external stimulation and damage to the brain. Common brain cognitive disorder-related diseases usually include: schizophrenia, depression, Alzheimer's disease, children's attention disorder, etc. A series of studies have been or are being carried out on these diseases at home and abroad, and some preliminary research results have been achieved. At present, the most commonly used analysis method in clinical practice is based on high-resolution functional magnetic resonance imaging data fMRI analysis, but functional magnetic resonance imaging data fMRI has the disadvantages of being difficult to collect, high cost, and some patients do not have the conditions for collection due to physical conditions. Other physiological data that are more convenient to collect, such as functional near-infrared spectroscopy fNIRS and electroencephalogram EEG, are needed as substitutes. However, in existing literature, it is usually a simple analysis after the extraction of single-modal data features, and the extraction of physiological characteristics of the human body is not comprehensive. For example, the literature 【1】 In order to calculate the slope and mean of functional near-infrared spectroscopy (fNIRS) signals as features and input them into the linear discriminant analysis (LDA) model for classification, the literature 【2】 The functional near-infrared spectroscopy (fNIRS) is also fed into the cascade random forest for classification. 【3】 The EEG signal is processed by fast Fourier transform, and then the spectrum is sent to PCANet for classification.

[0004] Therefore, a method that can improve the above problems is needed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for analyzing the state of brain cognitive function rehabilitation based on multimodal data. By constructing a classification model based on multimodal data fusion of functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG), and combining it with the Mann-Kendall trend analysis method, it is used for intelligent evaluation of the effect of brain cognitive function rehabilitation.

[0006] In order to solve the above technical problems, the present invention provides a method for analyzing the state of brain cognitive function rehabilitation based on multimodal data, comprising the following steps:

[0007] The patient's resting-state functional magnetic resonance imaging (fMRI) data, as well as functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data during the assessment task test, were collected. The functional magnetic resonance imaging (fMRI) data were then input into the fMRI-FC-CNN network to obtain the probability of being classified as normal. Where n is the patient number, k is the number of visits, and the functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data are preprocessed and input into the multimodal feature fusion classification network to obtain the probability of classification as normal. Will The results were plotted as a probability-number of visits line graph, and the Mann-Kendall test was used for trend analysis of the line graph.

[0008] As an improvement of the brain cognitive function rehabilitation status analysis method based on multimodal data of the present invention:

[0009] The multimodal feature fusion classification network includes three consecutive convolution + batch normalization modules, which are then divided into LSTM branches and Inception branches. The outputs of the LSTM branches and Inception branches are spliced ​​and then passed through two fully connected layers.

[0010] The LSTM branch includes 2 consecutive LSTM modules;

[0011] The Inception branch includes 6 consecutive Inception modules and 3 Inception modules as a group, and each group adopts a residual structure.

[0012] As a further improvement of the brain cognitive function rehabilitation status analysis method based on multimodal data of the present invention:

[0013] The fMRI-FC-CNN network includes first obtaining a time series using an anatomical automatic labeling template, then constructing a functional connection matrix using non-oscillatory connections and then feeding it into an AlexNet network; the AlexNet network includes five convolutional layers and three fully connected layers.

[0014] As a further improvement of the brain cognitive function rehabilitation status analysis method based on multimodal data of the present invention:

[0015] The training and testing process of the multimodal feature fusion classification network and fMRI-FC-CNN network is as follows:

[0016] Resting-state functional magnetic resonance imaging (fMRI) data, as well as functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data during task testing, were collected and divided into training and testing sets. The training set included an EEG-fNIRS training set and an fMRI training set, while the testing set included an EEG-fNIRS testing set and an fMRI testing set. Both the EEG-fNIRS training set and the EEG-fNIRS testing set were then preprocessed.

[0017] The pre-processed EEG-fNIRS training set is input into the multimodal feature fusion classification network for training, and the fMRI training set is input into the fMRI-FC-CNN network for training; the trained multimodal feature fusion classification network is verified on the pre-processed EEG-fNIRS test set to obtain the accuracy acc A And the classification results on M samples The trained fMRI-FC-CNN network is verified on the fMRI test set to obtain the accuracy acc B And the classification results on M samples Calculate the probability P and accuracy difference Acc of the classification results based on the multimodal feature fusion classification network and the fMRI-FC-CNN network on the same sample data diff :

[0018]

[0019] Acc diff =|acc A -acc B | (2)

[0020] If P+Acc diff <λ, verification ends, where λ is 0.1.

[0021] As a further improvement of the brain cognitive function rehabilitation status analysis method based on multimodal data of the present invention:

[0022] The pre-processing process is as follows:

[0023] (1) Statistically obtain the shortest length l of the functional near-infrared spectroscopy fNIRS fNIRS-min and the shortest length of EEG l EEG-min ; Then intercept the first l in each functional near infrared spectroscopy fNIRS data fNIRS-min The length of the time dimension is the data, intercepting the first l in each EEG data EEG-min The data of the time dimension length is used to obtain the functional near-infrared spectroscopy fNIRS data of N samples with the dimension (c1, w) Where c1 is the number of channels of functional near-infrared spectroscopy (fNIRS) data, w is the length of the time dimension, and the EEG data is of dimension (c2, kw) Where c2 is the number of channels of EEG data, and k is a constant;

[0024] (2) Data Perform downsampling with a sampling frequency of S to obtain a new data set Data Perform downsampling with a sampling frequency of k*S to obtain a new data set

[0025] (3) For samples from the same source, the dimension is of and latitude is of Splicing is performed on the channel dimension to obtain S dimensions: The fused data And the dimension is The fused dataset

[0026] The beneficial effects of the present invention are mainly reflected in:

[0027] 1. The present invention expands the training data set by downsampling and splices functional near-infrared spectroscopy (fNIRS) data (reflecting brain blood oxygenation) and EEG data (reflecting brain electrical signals) in the channel dimension as the input of the network model of the present invention. The complementary data of the two modalities can provide different information sources for the network model, making the features learned by the model more complete. At the same time, the classic time series signal processing model LSTM is improved by using the Inception module that can extract features at different scales and the residual module that prevents overfitting, thereby enhancing the network's representation ability.

[0028] 2. The present invention uses functional magnetic resonance imaging (fMRI) data, which has higher resolution and is more frequently used in clinical practice, to verify the analytical effect of fNIRS-EEG, making it more convincing. Simultaneously collecting fMRI data from patients makes the prediction results more complete and accurate. However, fMRI data is not required, thus avoiding the disadvantages of high cost and the physical condition of some patients that may not be able to collect data.

[0029] 3. The present invention uses the Mann-Kendall test to perform a probability-visit number line graph analysis on the predicted classification probabilities of functional magnetic resonance imaging data, near-infrared spectroscopy fNIRS data, and EEG data reflecting brain electrical signals during multiple visits, thereby obtaining more accurate analysis results of the growth or decline trend of the process data.

[0030] It should be emphasized that the present invention is intended to obtain more accurate analysis results of the growth or decline trend of process data, and therefore the present invention does not constitute a method for diagnosing a disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0032] Figure 1 Schematic diagram of a method for analyzing brain cognitive function rehabilitation status based on multimodal data according to the present invention;

[0033] Figure 2 Schematic diagram of the process of downsampling and feature fusion of fNIRS and EEG data during preprocessing;

[0034] Figure 3 This is a schematic diagram of the structures of the three basic networks of the present invention: LSTM module, Inception module, and residual module;

[0035] Figure 4 Schematic diagram of the structure of the multimodal feature fusion classification (LSTM-Inception-NN) network of the present invention;

[0036] Figure 5 Schematic diagram of the Mann-Kendall test analysis process used in the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described below with reference to specific embodiments, but the protection scope of the present invention is not limited thereto:

[0038] Example 1: A method for analyzing the state of brain cognitive function rehabilitation based on multimodal data, such as Figure 1-4 As shown, the method includes the following steps:

[0039] S1. Children with attention deficit disorder and normal children in the control group underwent resting-state functional magnetic resonance imaging (fMRI) data collection before performing the prescribed task. Functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data were collected during the prescribed task. The dataset was constructed and divided into training and test sets in a 4:1 ratio:

[0040] S101. Sixty children with attention deficit disorder and sixty children without attention deficit disorder in the control group were recruited from cooperative hospitals, and resting-state functional magnetic resonance imaging (fMRI) data were collected from them.

[0041] S102. The 120 participants in step S101 were assigned a Go / NoGo paradigm assessment task, requiring participants to respond to one stimulus (Go) and not respond to the other (NoGo). The Go stimulus consisted of the number "1" displayed on the screen, and the NoGo stimulus consisted of the number "2" displayed on the screen. Three seconds after the task began, a cross appeared in the center of the screen to indicate the imminent stimulus. This lasted for seven seconds. Afterward, a random stimulus was presented every one second, lasting one second. Forty sets of stimuli were presented, with a Go to NoGo ratio of 3:1. During the task, participants wore an EEG cap and attached a near-infrared acquisition probe and transmitter to their foreheads to collect physiological data, including functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG).

[0042] S103. The functional magnetic resonance imaging (fMRI) data collected in step S101 and the functional near-infrared spectroscopy (fNIRS) data and electroencephalogram (EEG) data collected in step S102 are archived according to the subjects and labeled with 1 for the patient group and 0 for the control group. Each subject is associated with one piece of functional magnetic resonance imaging (fMRI) data, one piece of functional near-infrared spectroscopy (fNIRS) data, and one piece of electroencephalogram (EEG) data. The data are then divided into a training set and a test set according to the subject granularity at a ratio of 4:1. The training set contained 48 patients and 48 controls. The functional near-infrared spectroscopy (fNIRS) data and electroencephalogram (EEG) data of the 96 subjects in the training set constituted the EEG-fNIRS training set, and the functional magnetic resonance imaging (fMRI) data constituted the fMRI training set. The test set group contained 12 patients and 12 controls. The functional near-infrared spectroscopy (fNIRS) data and electroencephalogram (EEG) data of the 24 subjects in the test set group constituted the EEG-fNIRS test set, and the functional magnetic resonance imaging (fMRI) data constituted the fMRI test set.

[0043] S104. Preprocessing of EEG-fNIRS training and test data

[0044] (1) Since the collected functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data have some differences in the time dimension, the dimensions need to be cropped and aligned: the collected functional near-infrared spectroscopy (fNIRS) data and electroencephalogram (EEG) data are statistically analyzed for their time lengths, and the shortest length of fNIRS data, l, is obtained. fNIRS-min and the shortest length l of EEG EEG-min For each functional near infrared spectroscopy fNIRS data, the first l fNIRS-min For each EEG data, the first l EEG-minThe data of the time dimension length is used to obtain the functional near-infrared spectroscopy fNIRS data of N test subjects with the dimension (c1, w). Where c1 is the number of channels of functional near-infrared spectroscopy (fNIRS) data, w is the length of the time dimension, and N is the EEG data of the subject with the dimension (c2, kw). Where c2 is the number of channels of EEG data and k is a constant.

[0045] (2) In order to prevent the model from overfitting during training, the dataset needs to be enhanced. Perform downsampling with a sampling frequency of S to obtain a new data set with the number of samples increased by S times For the dataset Perform downsampling with a sampling frequency of k*S to obtain a new data set with the number of samples expanded by k*S times

[0046] (3) In order to integrate the information of fNIRS and EEG to improve the robustness of the model, the downsampled dataset X fNIRS-ds and X EEG-ds Perform feature pre-fusion, such as Figure 2 As shown, for each patient's sample, the dimension is fNIRS data and latitude is EEG data Splicing is performed on the channel dimension to obtain S dimensions: The fused data The resulting dimension is The fused dataset

[0047] S2. Design a multimodal feature fusion classification (LSTM-Inception-NN) network based on the functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data collected in step S1. Build a commonly used fMRI-FC-CNN network for higher-resolution functional magnetic resonance imaging (fMRI) data:

[0048] S21. Build a multimodal feature fusion classification (LSTM-Inception-NN) network

[0049] Based on multimodal feature fusion classification (LSTM-Inception-NN) network, such as Figure 4 As shown, the data set obtained after the preprocessing in step 104 is As input, it first passes through three consecutive convolution + batch normalization modules, and then passes through the LSTM branch and Inception branch respectively. The LSTM branch includes two consecutive LSTM modules, and the Inception branch includes six consecutive Inception modules. Three Inception modules form an Inception group. Each Inception group module adopts a residual structure. The input of the first Inception module and the output of the last Inception module are added together. The features extracted by the LSTM branch and the Inception branch are spliced. The spliced ​​feature vector is then input into two fully connected layers for feature integration to obtain the final output. Specifically:

[0050] (1) To extract the features on the data channel dimension, the dataset On the channel dimension, two convolution operations with padding = "same" and a convolution kernel size of (m, 1) and one convolution operation with padding = "valid" and a convolution kernel size of ((c1+k*c2), 1) are performed in sequence to obtain a convolution kernel with a dimension of The feature map F1, where batch_size is the batch size fed into the network;

[0051] (2) Use Figure 3 The LSTM module, Inception module, and residual module shown in the figure build the network for feature extraction in the time dimension. The feature map F1 is input into two consecutive LSTM modules to integrate the context information of the long-term signal, and the output dimension is Feature map F2 = LSTM(LSTM(F1));

[0052] (3) Input the feature map F1 into three consecutive Inception modules to extract the multi-scale features of the signal, where the output of the previous Inception module is used as the input of the next Inception module. Referring to the residual module, the input of the first Inception module and the output of the last Inception module are added together to output the feature map

[0053] (4) The feature map F3 is input into three consecutive Inception modules to extract features. The output of the previous Inception module is used as the input of the next Inception module. The input of the first Inception module and the output of the last Inception module are added together to output the feature map.

[0054] (5) Concatenate the feature maps F2 and F4 in the time dimension to obtain the feature map F5 = concate(F2, F4). The feature map F5 is sequentially input into two fully connected layers with n and 2 neurons respectively to synthesize the feature vectors, and the final output is used as the classification result.

[0055] S22. Build fMRI-FC-CNN network

[0056] For the functional magnetic resonance imaging (fMRI) data collected in step S1, the anatomical automatic labeling (AAL) template is used to functionally partition the brain, and a time series containing the activities of 116 brain regions is obtained. The functional connection (FC) matrix is ​​constructed for the extracted time series using non-oscillatory connections, and then the functional connection matrix is ​​sent to the AlexNet network, which consists of five convolutional layers and three fully connected layers, to complete the construction of the fMRI-FC-CNN network.

[0057] S3. Train the multimodal feature fusion classification (LSTM-Inception-NN) network and the fMRI-FC-CNN network built in step S2. Use the fMRI-FC-CNN network to evaluate the effect of the multimodal feature fusion classification (LSTM-Inception-NN) network:

[0058] S301. Training environment configuration

[0059] Training was performed on a CentOS server, using a Tesla P4 GPU to accelerate the experiment, and the development environment was based on the TensorFlow deep learning framework. The specific hardware and software configuration is shown in Table 1:

[0060] Table 1

[0061] name Environment Configuration operating system CentOS 7.3.1611 processor 12*E5-2609v3@1.9GHz,15M Cache Graphics card Tesla P4 8GB (384.81) Memory 125GB Development Environment Python 3.7 TensorFlow 1.15.0

[0062] S302. Set hyperparameters: batch size to 8, sampling frequency S of the multimodal feature fusion classification (LSTM-Inception-NN) network to 3, channel dimension convolution kernel size m to 11, and number of neurons in the first layer of fully connected network n to 64; set the training loss function to cross entropy, optimizer to Adam; and training epochs to 50.

[0063] S303: Input the EEG-fNIRS training set preprocessed in step S104 into the multimodal feature fusion classification (LSTM-Inception-NN) network constructed in step S2, input the fMRI training set obtained in step S103 (without the preprocessing of step 104) into the fMRI-FC-CNN network, start training, and save the trained multimodal feature fusion classification (LSTM-Inception-NN) network and fMRI-FC-CNN network.

[0064] S304: The trained multimodal feature fusion classification (LSTM-Inception-NN) network obtained in step S303 is verified on the EEG-fNIRS test set obtained in step S104, and the accuracy is obtained. And the classification results on M samples

[0065] The fMRI-FC-CNN network trained in step S303 is verified on the fMRI test set obtained in step S103 to obtain the accuracy acc B And the classification results on M samples

[0066] S305. Calculate the probability and accuracy difference Acc of the classification results of the multimodal feature fusion classification model (LSTM-Inception-NN) and the fMRI-FC-CNN network on the same patient's data. diff :

[0067]

[0068] Acc diff =|acc A -acc B | (2)

[0069] If P+Acc diff <λ, it is proved that the multimodal feature fusion classification (LSTM-Inception-NN) network can replace the fMRI-FC-CNN network. The multimodal feature fusion classification (LSTM-Inception-NN) network and fMRI-FC-CNN network are verified and can be used in practice. The error between the multimodal feature fusion classification (LSTM-Inception-NN) network and the fMRI-FC-CNN network is controlled within λ, where λ is 0.1. If P+Acc diff ≥λ, then return to step S303 to retrain the model.

[0070] S4, the actual use stage, for each visit of the patient during the rehabilitation process, resting state functional magnetic resonance imaging (fMRI) data collection (optional), as well as functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) collection during the Go / NoGo paradigm assessment task test, the collected functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data are pre-processed according to step S104, and then input into the practical multimodal feature fusion classification (LSTM-Inception-NN) network and fMRI-FC-CNN network obtained in step S3 for analysis, and the prediction results of multiple data evaluations during the rehabilitation process are recorded. The specific process is as follows: Figure 1 As shown:

[0071] S401. For patient n's kth visit (including the first visit), if resting-state fMRI data are collected, the fMRI-FC-CNN network obtained in step S3 is input to analyze the collected fMRI data to obtain the probability of classification as normal.

[0072] S402: When patient n undergoes the GO / NoGO task test at the kth visit (including the initial visit), functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) are collected and preprocessed according to step S104. The data are then input into the practical multimodal feature fusion classification (LSTM-Inception-NN) network obtained in step S3 for analysis to obtain the probability of classification as normal.

[0073] S5. For each patient's multiple classification results in step S4, a line graph is drawn and a mathematical model is used to evaluate the multiple classification results. The process is as follows: Figure 5 As shown:

[0074] S501, the sequence of analysis results of the functional magnetic resonance imaging (fMRI) of patient n in step S4 is: The analysis results of functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) are as follows:

[0075] S502, for step S501 P n-fMRI and P n-fNIRS-EEGWe plotted the probability of fMRI and fNIRS-EEG data versus the number of visits and used the Mann-Kendall test to analyze the trend of the line graphs. The Mann-Kendall test is suitable for analyzing time series data with a continuously increasing or decreasing trend (monotonic trend). This nonparametric test is applicable to all distributions (i.e., the data do not need to meet the assumption of normality), but the data should be free of serial correlation.

[0076] experiment:

[0077] This invention proposes a method for analyzing the state of brain cognitive function rehabilitation based on multimodal data. This method is primarily implemented using a multimodal feature fusion classification (LSTM-Inception-NN) network. The effectiveness of the Mann-Kendall test also depends on the accuracy of the classification model. The following two experiments demonstrate the advanced nature of this model. The datasets used are the same fNIRS-EEG training and test sets preprocessed in step S104 of Example 1.

[0078] The experiment will evaluate the performance of each method based on four indicators: Accuracy, Precision, Recall, and F1 Score. These indicators can be represented by the elements in the confusion matrix shown in Table 2 below:

[0079] Table 2 Confusion matrix

[0080]

[0081] TP (True Positive) is a true positive, meaning both the sample's actual result and the model's prediction are positive. FN (False Negative) is a false negative, meaning the sample's actual result is positive but the model's prediction is negative. FP (False Positive) is a false positive, meaning the sample's actual result is negative but the model's prediction is positive. TN (True Negative) is a true negative, meaning both the sample's actual result and the model's prediction are negative.

[0082] Accuracy refers to the ratio of the number of samples correctly predicted by the model to the total number of samples, which can be expressed as:

[0083]

[0084] Precision and recall are both for positive samples. Precision refers to the proportion of positive samples in the data predicted by the model, while recall refers to the proportion of positive samples in the data that are actually positive. The two can be expressed as:

[0085]

[0086]

[0087] The F1 Score is a comprehensive analysis of precision and recall, which can be expressed as:

[0088]

[0089] (1): Ablation experiment

[0090] In this experiment, fNIRS and EEG were separately input into a multimodal feature fusion classification (LSTM-Inception-NN) network. A unimodal classification model for fNIRS and a unimodal classification model for EEG were obtained by training on the training set. The models were then tested and compared with the multimodal feature fusion classification (LSTM-Inception-NN) network obtained by training in step S303 of Example 1 on the test set. The model indicators are shown in Table 3 below:

[0091] Table 3. Comparison of ablation experiment performance

[0092]

[0093] As can be seen from Table 3, the performance indicators of the multimodal feature fusion classification based on the present invention are all higher than those of the two single-modal classification models. It can be seen that the multimodality fills in the information that may be missing from each other, making the model more robust.

[0094] (2): Comparison with existing technologies

[0095] This experiment compares the proposed method with the existing literature. The literature was replicated on the training set and tested on the training set. The performance comparison is shown in Table 4 below:

[0096] Table 4 Literature comparison

[0097]

[0098] As can be seen from Table 4, compared with other methods, the present invention achieves full characterization and fusion of multimodal data features by splicing the channel dimension, extracting channel dimension features, and extracting time dimension features after downsampling fNIRS and EEG. Its accuracy, precision, recall rate, and F1 value are all higher than those of the reference methods.

[0099] In summary, the classification algorithm based on multimodal feature fusion proposed in the present invention integrates the brain blood oxygen information reflected by fNIRS and the brain electrical information reflected by EEG through feature pre-fusion, extracts features through channel convolution, LSTM modules, and Inception modules, and integrates features through fully connected layers. The performance of the entire classification model is significantly superior, and it also ensures the accuracy of the trend assessment method based on the Mann-Kendall test established on this classification model.

[0100] literature 【1】 :Hong KS, Naseer N, Kim Y H. Classification of prefrontal and motor cortex signals for three-class fNIRS–BCI[J]. Neuroscience letters, 2015, 587:87-92.

[0101] literature 【2】 :Chen C, Wen Y, Cui S, et al. Amultichannel fNIRS system forprefrontal mental taskclassification with dual-level excitation and deepforest algorithm[J]. Journal of Sensors, 2020, 2020.

[0102] literature 【3】 :Li M, Chen W.FFT-based deep feature learning method for EEGclassification[J]. Biomedical Signal Processing and Control, 2021,66:102492.

[0103] Finally, it should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples and is subject to numerous variations. All variations that can be directly derived or conceived by a person of ordinary skill in the art from the disclosure of the present invention are considered to be within the scope of protection of the present invention.

Claims

1. A method for analyzing the state of brain cognitive function rehabilitation based on multimodal data, characterized in that: The following steps are involved: The patient's resting-state functional magnetic resonance imaging (fMRI) data, as well as functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data during the assessment task test, were collected. The functional magnetic resonance imaging (fMRI) data were then input into the fMRI-FC-CNN network to obtain the probability of being classified as normal. Where n is the patient number, k is the number of visits, and the functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data are preprocessed and input into the multimodal feature fusion classification network to obtain the probability of classification as normal. Will The results were plotted as a probability-number of visits line graph, and the Mann-Kendall test was used for trend analysis of the line graph; The multimodal feature fusion classification network includes three consecutive convolution + batch normalization modules, which are then divided into LSTM branches and Inception branches. The outputs of the LSTM branches and Inception branches are spliced ​​and then passed through two fully connected layers. The fMRI-FC-CNN network includes first obtaining a time series using an anatomical automatic labeling template, then constructing a functional connection matrix using non-oscillatory connections and then feeding it into an AlexNet network; the AlexNet network includes five convolutional layers and three fully connected layers; The process of the pretreatment is: (1) Statistically obtain the shortest length l of the functional near-infrared spectroscopy fNIRS fNIRS-min and the shortest length of EEG l EEG-min ; Then intercept the first l in each functional near infrared spectroscopy fNIRS data fNIRS-min The length of the time dimension is captured by taking the first l of each EEG data. EEG-min The data of the time dimension length is used to obtain the functional near-infrared spectroscopy fNIRS data of N samples with the dimension (c1, w) Where c1 is the number of channels of functional near-infrared spectroscopy (fNIRS) data, w is the length of the time dimension, and the EEG data of dimension (c2, kw) Where c2 is the number of channels of EEG data, and k is a constant; (2) Data Perform downsampling with a sampling frequency of S to obtain a new data set Data Perform downsampling with a sampling frequency of k*S to obtain a new data set (3) For samples from the same source, the dimension is of and dimensions are of Splicing is performed on the channel dimension to obtain S dimensions: The fused data And the dimension is The fused dataset 2. The method for analyzing brain cognitive function rehabilitation status based on multimodal data according to claim 1, characterized in that: The LSTM branch includes 2 consecutive LSTM modules; The Inception branch includes 6 consecutive Inception modules and 3 Inception modules as a group, and each group adopts a residual structure.

3. The method for analyzing brain cognitive function rehabilitation status based on multimodal data according to claim 2, characterized in that: The training and testing process of the multimodal feature fusion classification network and fMRI-FC-CNN network is as follows: Resting-state functional magnetic resonance imaging (fMRI) data, as well as functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) data during task testing, were collected and divided into training and testing sets. The training set included an EEG-fNIRS training set and an fMRI training set, while the testing set included an EEG-fNIRS testing set and an fMRI testing set. Both the EEG-fNIRS training set and the EEG-fNIRS testing set were then preprocessed. The pre-processed EEG-fNIRS training set is input into the multimodal feature fusion classification network for training, and the fMRI training set is input into the fMRI-FC-CNN network for training; the trained multimodal feature fusion classification network is verified on the pre-processed EEG-fNIRS test set to obtain the accuracy acc A And the classification results on M samples The trained fMRI-FC-CNN network is verified on the fMRI test set to obtain the accuracy acc B And the classification results on M samples Calculate the probability P and accuracy difference Acc of the classification results based on the multimodal feature fusion classification network and the fMRI-FC-CNN network on the same sample data diff : Acc diff =|acc A -acc B | (2) If P+Acc diff <λ, verification ends, where λ is 0.1.

Images