English learner cognitive interference residue detection method fusing machine learning and near infrared spectrum data

Abstract

The invention discloses a data acquisition, classification and verification method based on a machine learning algorithm, and aims to effectively and reasonably divide data into two classes through a machine learning method. According to the method, brain oxyhemoglobin concentration data of a student under a specific task is collected by using an fNIRS technology, and data processing, classified training and testing and medical verification are carried out through a series of steps. And performing classification training, verification and analysis on the data by using three machine learning algorithms: 1D-CNN, GRU and GBDT. And finally, based on the algorithm, key feature analysis and medical verification are carried out, and the similarity between the classification process of the algorithm and the medical classification mode is verified. Compared with the prior art, the brain activity pattern classification method has the advantages that machine learning and fNIRS technologies are combined, and the accuracy of brain activity pattern classification is remarkably improved. Through key feature analysis and medical verification, the accuracy and medical correlation of the classification result are verified. The method is expected to play an important role in the fields of neuroscience research and education.

Description

Technical Field

[0001] The present invention relates to a method for classifying and verifying data samples collected by functional near infrared spectroscopy (fNIRS) of the brain by using a machine learning algorithm, and more particularly to classifying brain activity patterns by using a machine learning algorithm, and verifying the accuracy and medical relevance of the classification results by key feature analysis and medical verification. Background Art

[0002] In modern cognitive neuroscience research, the relationship between brain activity and specific tasks has become a topic of widespread concern. Researchers have gradually realized that different tasks and cognitive processes lead to specific activity patterns in brain regions, which makes it increasingly important to understand brain activity when learning, thinking, and performing tasks. Functional near-infrared spectroscopy (fNIRS) has become a powerful tool for studying brain activity because of its non-invasive, real-time, and applicability to different ages and populations.

[0003] At the same time, with the rapid development of machine learning technology, its application in large-scale data analysis, pattern recognition and classification has gradually shown great potential. Machine learning algorithms can learn regularities and patterns from data, so that they can classify and predict complex problems. Combining machine learning with the study of brain activity patterns can not only provide new perspectives for cognitive neuroscience, but also provide useful information for research in education, medicine and other fields.

[0004] Currently, few studies have used machine learning algorithms to train and classify fNIRS data, and explained the classification process of the algorithm through feature analysis and medical verification to verify the accuracy and medical relevance of the algorithm classification results.

[0005] In previous patent applications, classification is usually performed by manual analysis and comparison of data. For example, in the invention of Chinese patent application number 202110600796.4, entitled A method for detecting cognitive function in schizophrenia, data is collected using an fNIRS instrument. The invention compares the patient's blood oxygen index, resting EEG data, and cerebral blood oxygen data measured twice to obtain changes in the patient's cognitive function. However, manually comparing the data alone may ignore many subtle features in the data, resulting in an incomplete and in-depth analysis of the data. Summary of the invention

[0006] The present invention aims to provide a data collection, classification and verification method based on machine learning algorithm, which is used to study the brain activity patterns of students under different tasks.

[0007] To achieve the above purpose, the present invention adopts the following technologies:

[0008] The detection method comprises the following steps:

[0009] Step 1: Using fNIRS technology and instruments (Huichuang Medical Company), collect brain oxygenated hemoglobin concentration data when students complete specific tasks;

[0010] 1.1) The fNIRS instrument has 49 detection points, dividing the brain into 49 areas. Therefore, a single data sample collected in the present invention has 49 features.

[0011] Step 2: Perform preliminary processing on the collected data samples;

[0012] 2.1) Eliminate outliers in the data sample: data that is affected by external factors during data collection and data that is seriously inconsistent with the actual situation;

[0013] 2.2) Label the data: a single English spelling task is regarded as category 0, and different math tasks set before English spelling as interference are regarded as category 1. The math interference tasks set include algebra, geometry, and scale.

[0014] Step 3: Apply three machine learning algorithms to the processed data set for classification training and testing. These algorithms are gradient boosted decision tree (GBDT), one-dimensional convolutional neural network (1D-CNN), and gated recurrent unit (GRU).

[0015] Step 4: Compare the classification effects of the three algorithms.

[0016] 4.1) All three algorithms can achieve very good classification results, which are significantly stronger than the traditional weak classifiers MLP and KNN. The accuracy of the three algorithms on the test set is: 1D-CNN: 0.998; GRU: 0.993; GBDT: 0.978;

[0017] 4.2) Among the three algorithms, the 1D-CNN algorithm performs better in classifying brain activity patterns under different tasks, with higher accuracy and good convergence on the test set;

[0018] Step 5: In order to further explore the key features that affect data labels, the present invention performs grid search based on the GBDT algorithm. After determining the optimal parameter combination, the 49 features of the sample are critically ranked based on this classification model. The first N (10-20) features are selected, and the data samples are classified and trained and tested using these N features. It is found that compared with using all features for classification training, the classification effect of selecting the first N features is not ideal.

[0019] Step 6: Based on step 5, the present invention further analyzes the key feature items;

[0020] 6.1) By analyzing the vertical box plots and vertical violin plots under four categories of labels for 49 channels, several key features were found. The four categories of labels are 0 (no mathematical interference), 1 (algebraic interference), 2 (geometric interference), and 3 (scale interference), as follows:

[0021] (1) The feature channels with the largest distribution differences are S6-D5, which contribute more to classification;

[0022] (2) S2-D1 and S4-D4 are the largest brain regions in category 0, which are interpreted as the two regions with the least mathematical interference, and the brain regions where mathematics obviously produces control. The S2-D1 channel is the largest and significantly higher than 1, 2, and 3 in category 0, and the brain region with the least mathematical influence;

[0023] (3) S3-D5 is the largest and significantly higher than 0 for categories 1, 2, and 3, which is mathematically interpreted as the area where activation is most obvious;

[0024] (4) S10-D6 is the area with the smallest difference between class 0 and classes 1, 2, and 3. The feature distribution is relatively consistent and there is little difference in classification.

[0025] 6.2) Then, by comparing the 49 channel values ​​under the two-category and four-category label tasks, the average value was used as the comparative analysis indicator of the activation degree, and the ratio of the number of brain areas activated by each type of label was compared. It was concluded that: overall, the effects of scale, geometry, and algebra on brain activation are decreasing; then 14 channels with 0 greater than 1, 2, and 3 were found, and then based on these 14 channels, the channels with the largest mean value in category 0 were found: S2-D1 and S4-D4, which are the ones least affected by mathematics.

[0026] The two methods confirmed that after engaging in mathematical tasks, the S3-D5 area showed the most obvious activation. In contrast, the S4-D5 area was less affected after engaging in mathematical tasks, and the difference in the S6-D10 area was the most significant.

[0027] Step 7: The present invention further verifies the similarity between these classification results and the existing brain activity pattern classification methods in medicine. By comparing the correlation between the feature items with higher weights selected by the machine learning algorithm in the classification process and the medical classification method, it is verified whether the machine learning algorithm classification method in the present invention is consistent with the medical classification method. The analysis results show that during the classification process, the activity patterns of some specific brain regions show significant similarities with the medical classification method.

[0028] The 1D-CNN network model in step 3 includes an input layer, a convolutional layer, a maximum pooling layer, a flatten layer, a fully connected layer and an output layer. The specific structure is as follows:

[0029] (1) Before inputting into the network model, the collected data is preprocessed, including standardization and noise reduction. Each sample includes 49 channels of time series data. The processed data is used as the input of the network. The shape of the input data is (49, 1), where '1' represents the number of channels, which is represented by x here. i (t), where i represents the channel number and t represents the time step;

[0030] (2) Perform convolution operation. The model contains a convolution layer, which uses a one-dimensional convolution kernel for feature extraction. It has 64 convolution kernels (filters), each of which has a size of 3 and an activation function of ReLU. The convolution kernel extracts features from different positions of the data and generates a new feature map.

[0031] (3) Pooling is applied to the feature map of each channel after convolution. The model contains a pooling layer that uses maximum pooling to reduce the latitude and extract the main features. The window size used in the pooling operation is 2.

[0032] (4) After the pooling layer, there is a flatten layer and a fully connected layer. The flatten layer converts multi-dimensional data into one dimension so that it can be input into the fully connected layer. The fully connected layer connects the pooled feature sequence into a vector and then passes it to one or more fully connected layers for classification tasks. In this model, the fully connected layer has 128 neurons and uses ReLU as the activation function. Each fully connected layer has a weight matrix W and a bias b, and uses an activation function (such as ReLU) for nonlinear transformation.

[0033] (5) The last one is the output layer, which is a fully connected layer with the number of output categories. It uses softmax as the activation function and is used to predict classification tasks. The softmax activation function is used to convert the output of the fully connected layer into category probability.

[0034] The GRU algorithm in step 3 is a type of recurrent neural network (RNN). The specific implementation of the GRU-based classifier in the present invention is as follows:

[0035] (1) First, use the Keras library to build a GRU model. The model consists of three layers: a GRU layer, a Dropout layer, and a fully connected layer.

[0036] (2) The GRU layer has 32 neurons, and the shape of the input data is (1, 49), which means that each sample is a vector of length 49. The output of the GRU layer is passed to the next Dropout layer with a dropout rate of 0.5. The output of the Dropout layer is passed to a fully connected layer, which has one neuron and uses a sigmoid activation function;

[0037] (3) Use binary cross entropy as the loss function, Adam as the optimizer, and accuracy as the evaluation metric. Finally, evaluate the performance of the model on the test data.

[0038] The GBDT algorithm in step 3 is an iterative decision tree algorithm that constructs a set of weak learners (trees) and accumulates the results of multiple decision trees as the final prediction output. The specific implementation method is as follows:

[0039] (1) First, preprocess the data to divide the features and labels;

[0040] (2) Apply grid search technology to optimize the hyperparameters of the classifier. By adjusting parameters such as n_estimators (the number of trees) and max_depth (the depth of the tree), we can find the best hyperparameter combination to improve the performance of the classification model.

[0041] (3) Using the optimized hyperparameters, a gradient boosted tree-based classifier is trained, which gradually improves the prediction performance by gradually generating base classifiers;

[0042] (4) After training, obtain the importance score of each feature. Use the model's 'feature_importance_' attribute to obtain the importance of each feature for classification;

[0043] (5) Sort the importance scores of the features, which will reveal the features that have the most significant impact on the classification results.

[0044] Compared with the prior art, the present invention has the following beneficial technical effects:

[0045] 1. The present invention uses three strong classifiers, 1D-CNN, GRU and GBDT, to classify and analyze the data, combining the machine learning algorithm with fNIRS. The accuracy of fNIRS data set classification is much higher than that of weak classifiers such as MLP and traditional methods of manually analyzing fNIRS data sets;

[0046] 2. This invention verifies the correlation between the algorithm's classification method and the medical classification method through analysis of key features of classification and medical verification, and explains the rationality of the algorithm's classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A complete flow chart of the present invention is shown;

[0048] Figure 2 Experimental flow chart for data collection;

[0049] Figure 3 The network structure diagrams of three different algorithms;

[0050] Figure 4 ROC curves for five classifiers;

[0051] Figure 5 This is the training process of 1D-CNN;

[0052] Figure 6 Rank the importance of features obtained by grid search;

[0053] Figure 7 Vertical box plots and vertical violin plots for key features. Specific implementation plan

[0054] In order to further explain the actual operation process of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0055] The main goal of the present invention is to classify and analyze the collected brain imaging data through machine learning algorithms. A functional near-infrared spectroscopy (fNIRS) instrument is used to obtain data sets, and these data are classified, trained and tested through machine learning algorithms. During the data collection process, the subjects are assigned tasks to activate different brain regions, and then three machine learning algorithms are used to classify, train and test the brain imaging data. The core of this process is that the present invention aims to effectively and reasonably classify the data into two categories through machine learning methods, in order to provide strong support for future applications in other fields such as medicine.

[0056] The present invention used an fNIRS instrument (manufactured by Hui Chuang Medical Company) in the early data collection experiments. The detection head cap has 49 detection points for detecting 49 areas of the brain, that is, the collected data samples have 49 features. Medical research has shown that different activity tasks will trigger activity in different areas of the brain, which provides a basis for the research of the present invention.

[0057] exist Figure 2In the data collection experiment, the overall process is shown. The subjects are divided into two groups. Group A receives an English spelling task without interference, while Group B is assigned a math task as interference before the English spelling task. The set math tasks include scale, algebra, and geometry (in the subsequent feature analysis, the data set will be analyzed as a four-category feature based on the type of math task). The collected data is preliminarily processed. First, the outliers in the data set are removed, such as excluding data that was interfered with during the collection process and data that is obviously abnormal. Then the data is marked, and the group A data set is marked as category 0 and the group B data set is marked as category 1.

[0058] Next, three machine learning algorithms are used to classify and train the data set. The three algorithms are gradient boosted decision tree (GBDT), one-dimensional convolutional neural network (1D-CNN) and gated recurrent unit (GRU). The network structure diagrams of the three algorithms are shown in the following figure. Figure 3 shown.

[0059] The specific implementation of the 1DCNN network is as follows:

[0060] 1.1DCNN network structure:

[0061] Before inputting into the network model, the collected data is preprocessed, including standardization and noise reduction. Each sample includes 49 channels of time series data. The processed data is used as the input of the network. The shape of the input data is (49, 1), where '1' represents the number of channels, which is represented by the symbol x here. i (t), where i represents the channel number and t represents the time step;

[0062] The model contains a convolution layer, which uses a one-dimensional convolution kernel for feature extraction. It has 64 convolution kernels (filters), each of which has a size of 3 and an activation function of ReLU. The convolution kernel extracts features from different locations of the data and generates a new feature map. The convolution operation can be expressed as:

[0063]

[0064] Among them, w k is the weight of the convolution kernel, and k is the size of the convolution kernel. In this way, each channel will get a new time series feature;

[0065] A pooling operation is applied to the feature map of each channel after convolution. The model contains a pooling layer, which uses maximum pooling to reduce the latitude and extract the main features. The window size used in the pooling operation is 2. The maximum pooling can be expressed as:

[0066]

[0067] 4) After the pooling layer, there is a flatten layer and a fully connected layer. The flatten layer converts multi-dimensional data into one dimension so that it can be input into the fully connected layer. The fully connected layer connects the pooled feature sequence into a vector and then passes it to one or more fully connected layers for classification tasks. In this model, the fully connected layer has 128 neurons and uses ReLU as the activation function. Each fully connected layer has a weight matrix W and a bias b, and uses an activation function (such as ReLU) for nonlinear transformation. This step can be expressed as:

[0068] FC(z)=σ(W·z+b)

[0069] 5) The last one is the output layer, which is a fully connected layer with the number of output categories. It uses softmax as the activation function for prediction of classification tasks. The softmax activation function is used to convert the output of the fully connected layer into category probability. The softmax function can be expressed as:

[0070]

[0071] Where C represents the number of categories.

[0072] After forward propagation through the above network, including convolution, pooling and full connection layers, a classification result is obtained. The Sparse Categorical Crossentropy loss function is used to measure the difference between the predicted value and the true label. The gradient of the network parameters is calculated through the back-propagation algorithm, and the Adam optimizer is used to update the network parameters to gradually reduce the loss. The model is trained through multiple iterations to gradually optimize the model.

[0073] Validation and testing: Finally, an independent validation set is used to verify the performance of the model.

[0074] The specific implementation of the GRU algorithm is as follows:

[0075] 1) First, use the Keras library to build a GRU model, which consists of three layers: a GRU layer, a Dropout layer, and a fully connected layer;

[0076] 2) The GRU layer has 32 neurons and the shape of the input data is (1, 49), which means that each sample is a vector of length 49. The output of the GRU layer is passed to the next Dropout layer with a dropout rate of 0.5. The output of the Dropout layer is passed to a fully connected layer with one neuron and a sigmoid activation function;

[0077] 3) Use binary cross entropy as the loss function, Adam as the optimizer, and accuracy as the evaluation metric. Finally, evaluate the performance of the model on the test data;

[0078] GRU mainly includes update gate and reset gate. The calculation formula of update gate is as follows:

[0079] z t =σ(W z x t +U z h t-1 +b z )

[0080] where z t represents the value of the update gate, σ represents the sigmoid function, W z , U z and b z are model parameters, x t Represents the input data at the current moment, h t-1 Indicates the hidden state at the previous moment;

[0081] The reset gate is calculated as follows:

[0082] r t =σ(W r x t +U r h t-1 +b r )

[0083] where r t Represents the value of the reset gate, and other symbols are the same as the calculation formula of the update gate.

[0084] The specific implementation of the GBDT algorithm is as follows:

[0085] (1) First, preprocess the data to divide the features and labels;

[0086] (2) Apply grid search technology to optimize the hyperparameters of the classifier. By adjusting parameters such as n_estimators (the number of trees) and max_depth (the depth of the tree), we can find the best hyperparameter combination to improve the performance of the classification model.

[0087] (3) Using the optimized hyperparameters, a gradient boosted tree-based classifier is trained, which gradually improves the prediction performance by gradually generating base classifiers;

[0088] (4) After training, obtain the importance score of each feature. Use the model's 'feature_importance_' attribute to obtain the importance of each feature for classification;

[0089] (5) Sort the importance scores of the features, which will reveal the features that have the most significant impact on the classification results.

[0090] The three algorithms all performed well on the validation set, with accuracy rates of 1D-CNN: 0.998, GRU: 0.993, GBDT: 0.978. The classification accuracy is significantly higher than that of weak classifiers such as MLP and KNN. The specific classification effects of each classifier are as follows: Figure 4 As shown in the ROC curve (the five curves from top to bottom in the figure are the ROC curves of 1D-CNN, GRU, GBDT, MLP and KNN), 1D-CNN has the best effect, the highest classification accuracy and good convergence. Its training process is as follows Figure 5 As shown in the figure, 1D-CNN can converge quickly in a smaller epoch, and can achieve higher accuracy and lower error rate on both the training set and the validation set.

[0091] After obtaining a good classification result, in order to further explore the key features that affect the data label, the present invention performs grid search based on the GBDT algorithm. After determining the optimal parameters of the model, the key ranking of the 49 features by the GBDT algorithm is determined. The ranking results are as follows: Figure 6 As shown, next, in order to further study the impact of key feature items on classification, a supervised algorithm is used to perform classification training and prediction based on the first N (10-20) features. However, experiments have shown that using only the first N (10-20) features for classification cannot achieve ideal results, which is inferior to using all features.

[0092] Based on this discovery, the present invention conducts a more in-depth analysis of the key features:

[0093] 1) First, by analyzing the vertical box plots and vertical violin plots of 49 channels under four categories of labels, several key features were found, such as Figure 7 The four labels are 0 (no mathematical interference), 1 (algebraic interference), 2 (geometric interference), and 3 (scale interference), as follows:

[0094] (1) The characteristic channels with the largest distribution differences are S6-D5, such as Figure 7 As shown in (a), it contributes more to the classification;

[0095] (2) S2-D1 and S4-D4 are the largest brain regions in category 0, e.g. Figure 7 (b) shows the two areas interpreted as the least mathematical interference, the brain area where mathematics obviously produces control, the S2-D1 channel is the largest in category 0, which is significantly higher than 1, 2, and 3, and the brain area with the least mathematical influence;

[0096] (3) S3-D5 is the largest and significantly higher than 0 for categories 1, 2, and 3, e.g. Figure 7 (c) shows the area where the mathematical interpretation makes the activation most significant;

[0097] (4) S10-D6 is the area with the smallest difference between class 0 and classes 1, 2, and 3. The feature distribution is relatively consistent and the classification difference is not significant, e.g. Figure 7 (d) as shown.

[0098] 2) The average values ​​of the two types of labels on 49 channels were compared, and the average values ​​were used as a comparative indicator of the degree of activation. The results showed that the average value of label 0 was greater than the average value of label 1 on 28 channels, and less than the average value of label 1 on 21 channels. This shows that after performing mathematical tasks, the proportion of overall brain active areas increased, proving the activation effect of mathematical tasks. The data set was further divided into four categories to compare the average values ​​of the channels, namely 0 (no interference), 1 (algebra), 2 (geometry) and 3 (scale), and the effects of different tasks on brain area activation were compared. The results showed that after mathematical tasks, algebraic operations led to a decrease in brain active areas, geometric operations increased active areas, and scale tasks were most obvious in activating brain areas.

[0099] Then, the features with label 0 are found to be greater than the features with labels 1, 2, and 3. Specifically, the present invention experimentally determines 14 channels, such as ['S1-D1', 'S2-D1', 'S2-D2', 'S2-D4', 'S4-D1', 'S4-D4', 'S5-D4', 'S12-D8', 'S17-D14', 'S19-D15', 'S20-D16', 'S22-D14', 'S22-D15', 'S22-D16'], which show an inhibitory effect after the math task, or it can be understood that when math exercises are not performed, these channels are more easily activated and less disturbed. Among these 14 channels, the means of S2-D1 and S4-D4 channels were the largest, and the numerical differences of these two channels explained the activation differences of S3-D5 and S4-D5 after the math task, and the differences of S6-D10 between the groups were the most significant.

[0100] The two methods confirmed that: S3-D5 is the most activated area after engaging in mathematics; S4-D5 is the area least affected by mathematics after engaging in mathematics; S6-D10 has the greatest difference.

[0101] The present invention has carried out medical verification on the classification results of the machine learning algorithm and the existing classification methods of brain activity patterns in medicine. By comparing the feature items with higher weights selected in the classification process and their association with the medical classification method, the degree of consistency between the classification method of the machine learning algorithm and the medical classification method is verified. The results show that the activity patterns of some specific brain regions show similarities in the classification process. For example, the S6-D5 channel shows huge differences in classification, while the S6-D5 region is medically interpreted as the fusiform gyrus region of the right brain, which is a typical activated brain region for Chinese students reading English. This result verifies the rationality of the algorithm classification.

[0102] The research of this invention provides a feasible method for machine learning classification of brain imaging data, and further verifies the accuracy and reliability of the algorithm through verification with medical classification methods. This method not only contributes to the development of cognitive neuroscience, but also has broad application prospects.

Claims

1. A method for classifying and verifying data samples collected by functional near infrared spectroscopy (fNIRS) of the brain by machine learning algorithm, characterized in that: The following steps are involved: 1.1 By using fNIRS technology and instruments to collect brain oxygenated hemoglobin concentration data when students complete specific tasks; 1.2 Carry out preliminary processing on the collected data samples; 1.3 Apply the processed data set to three machine learning algorithms for classification training and testing, and compare their classification results; 1.4 Further analysis of key features; 1.5 Further verify the similarity between the algorithm's classification method and the brain activity patterns already known in medicine.

2. According to the method of claim 1, the step 1.1 of collecting brain map data under a specific task is characterized in that: Three mathematical tasks with obvious differences in the degree of brain area activation were designed: algebra, scale and geometry, so as to facilitate comparative analysis of various channel values ​​after classification training.

3. The method according to claim 1, wherein the data is classified and trained in step 1.2, characterized in that: The machine learning algorithm was combined with fNIRS technology, and three machine strong classifier models, namely gradient boosting decision tree (GBDT), one-dimensional convolutional neural network (1DCNN) and gated recurrent unit (GRU), were used to achieve classification and feature analysis of brain map data through the strong classifier model.

4. The method according to claim 1, wherein the key features are further analyzed in step 1.4, characterized in that: (1) Based on GBDT, grid search is performed to determine the optimal parameter combination. Then, based on the GBDT classifier model, the model's 'feature_importance_' attribute is used to rank the features according to their importance during classification. The top N (10-20) features are selected, and classification training and testing of data samples are performed based on these top N features. The results are then compared and analyzed with the results of classification training using all features. (2) By analyzing the vertical box plots and vertical violin plots of the feature channels under four types of labels (no mathematical interference, algebraic interference, geometric interference, and scale interference), several key features were found. Then, by comparing the 49 channel values ​​under two types of labels and four types of labels, the average value was used as a comparative analysis indicator of the degree of activation, and the number of brain regions activated by each type of label was compared. It was confirmed in two ways: S3-D5 was the most obviously activated after engaging in mathematics; S4-D5 was the area least affected by mathematics after engaging in mathematics; and S6-D10 had the largest difference.

5. The method according to claim 1, wherein the step 1.5 further verifies the similarity between the classification method of the algorithm and the brain activity pattern already known in medicine, characterized in that: By comparing the correlation between the features with higher weights selected by the machine learning algorithm during the classification process and the medical classification method, it is verified whether the machine learning algorithm classification method in the present invention is consistent with the medical classification method. The analysis results show that during the classification process, the activity patterns of some specific brain regions show significant similarities with the medical classification method.

Images