Classification method of multi-physiological signal fusion for cognitive state detection

Abstract

The present invention discloses a classification method for multi-physiological signal fusion for cognitive state detection. First, the EMD algorithm is used to decompose the signal in the data set, and several components are obtained after the signal decomposition. Some of the components of different signals are exchanged, and new signals are obtained after reorganization. By this method, the data set is expanded to achieve the purpose of data enhancement. Afterwards, different physiological signal data are placed in the data matrix according to the acquisition position, so as to achieve the purpose of multi-physiological signal fusion. Finally, the processed data matrix is ​​placed in the designed deep learning network for feature extraction and classification to obtain the cognitive state to which the signal belongs. The present invention adopts a data enhancement method to avoid the overfitting phenomenon caused by too little physiological signal data, and adopts the fusion of multiple physiological signals to obtain a higher classification accuracy than that of a single physiological signal, so that the effect of cognitive state detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of biomedical state detection, and in particular to a classification method for multi-physiological signal fusion for cognitive state detection. Background Art

[0002] With the rapid development of machine learning and deep learning, the use of artificial intelligence algorithms to process biomedical signals and images has become increasingly common. Cognitive state, as one of the contents in the medical field, includes many aspects such as concentration, load, fatigue, and stress. Electroencephalogram (EEG) is one of the most common physiological signals processed by computers. It observes the process of brain wave activity by placing electrodes on the scalp according to certain rules. Depending on the task to be completed, these electrodes placed on the scalp can be combined in a variety of different ways. In addition to EEG, other common physiological signals include electrocardiogram (ECG), electrooculogram (EOG), etc. Currently, most studies using physiological signals to detect cognitive state use EEG.

[0003] In practice, the detection and identification of a subject's cognitive state is typically accomplished through manual assessment by professionals or by having the subject fill out various cognitive status forms. In reality, due to the diverse scenarios requiring cognitive status assessment, some of which require real-time monitoring, relying on manual assessment or subject-filled forms is unrealistic. Therefore, computer-assisted technology for real-time assessment of collected physiological signals has become a more practical option.

[0004] Currently, a growing number of researchers are leveraging artificial intelligence methods to build more convenient and efficient cognitive state detection systems. Most of these systems use EEG signals as input. Currently, the technical methods for using EEG signals to detect cognitive states are relatively mature, but physiological signals generally have small data volumes, resulting in relatively poor detection results for individual physiological signals, and manual feature selection is required for classification. Existing cognitive state detection methods suffer from relatively low accuracy and the potential for overfitting during training. Therefore, more effective technical solutions are needed for classifying and detecting cognitive states.

[0005] The empirical mode decomposition (EMD) method decomposes signals based on the time scale characteristics of the data itself. It can decompose complex signals into a finite number of components, and the frequency and energy of these components decrease in the order of decomposition. For signals, the signals can be decomposed and reorganized. The reorganized signals contain some features from other signals, and a considerable part of their own features are still retained, so it can be considered that new signal data is obtained. There is a certain connection between various physiological signals of a person, and fusing them is more conducive to improving the detection accuracy. Deep learning methods are a type of machine learning method. Compared with ordinary machine learning methods, deep learning methods do not require manual selection and calculation of features required for classification. Therefore, this patent uses an EMD-based method for data enhancement, uses and fuses multiple physiological signals, and selects deep learning methods for prediction of cognitive states.

[0006] In summary, the present invention addresses the problem that current cognitive state detection only uses a single physiological signal and is prone to overfitting during training, and proposes a classification method for cognitive state detection that integrates multiple physiological signals. Summary of the Invention

[0007] Content of the present invention is:

[0008] (1) A method for data enhancement using the EMD algorithm is proposed;

[0009] (2) A multi-physiological signal fusion method using EEG and EOG signals was proposed;

[0010] (3) Use deep learning networks to classify and identify cognitive states.

[0011] To address the problem of overfitting and low recognition accuracy in cognitive state detection datasets due to the limited data and the use of only a single physiological signal for classification, this paper proposes a classification method for cognitive state detection that uses multi-physiological signal fusion. The technical solution adopted is to perform data enhancement using an EMD algorithm and to use multi-physiological signal (EEG and EOG) fusion for classification. It is characterized by:

[0012] The raw physiological signal data is preprocessed, filtered through a filter, and downsampled (if necessary). Data enhancement is then performed using an EMD-based method. Multiple physiological signals are then fused together to form a mixed signal. Finally, the mixed signal is placed into a deep learning network for training and prediction. Therefore, the present invention has the following advantages:

[0013] (1) Use the existing EMD algorithm to decompose the signal, and only need to reconstruct the signal to complete the data enhancement step.

[0014] (2) The multi-physiological signal fusion process is simple, and only requires arranging the signal data according to the spatial distribution of signal acquisition.

[0015] (3) The deep learning network has a simple structure, fast training and prediction speed, and requires less time. It is conducive to quickly obtaining the cognitive state of the subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of detailed relationship between system modules in an embodiment of the present invention

[0017] Figure 2 Schematic diagram of signal collection location

[0018] Figure 3 Schematic diagram of multi-physiological signal fusion method

[0019] Figure 4 Schematic diagram of deep learning network structure

[0020] Figure 5 Schematic diagram of the convolutional neural network (CNN) structure DETAILED DESCRIPTION

[0021] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. The implementation examples described here are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0022] Please see Figure 1 , the present invention provides a classification method and system for multi-physiological signal fusion for cognitive state detection. The original physiological signal is preprocessed, the noise in the signal is removed by filtering, and the signal types with high sampling rate are downsampled to obtain different types of physiological signals with the same sampling rate. The signal is divided according to the trial to obtain a number of trial segments. The trial segments are divided according to the annotation information, and the divided signals are decomposed and reorganized respectively to obtain an expanded data set. The data is then arranged according to the spatial distribution of the signal acquisition position to obtain a signal matrix; the obtained signal matrix is ​​sorted in chronological order to obtain a signal matrix sequence. Finally, the signal matrix sequence is sent to the deep learning network for network training and prediction.

[0023] The input module is used to read different types of physiological signals and annotation information, and store different types of signals and annotation information into their corresponding arrays;

[0024] The preprocessing module is used to preprocess the physiological signal array and remove noise from the signal by filtering. If different types of physiological signals have different sampling rates, the sampling rates of all signals need to be reduced to the sampling rate of the signal with the lowest sampling rate. Different types of signals are spliced ​​into an array to obtain a mixed signal, and the mixed signal is divided according to the trial to obtain mixed signals of several trials. According to the labeling information, trials of the same category are stored in the same array to obtain several arrays, and the number of arrays is the number of categories.

[0025] The data enhancement module uses the EMD algorithm to decompose the signal, recombines the decomposed signal according to certain rules to obtain a new signal sample, and adds the recombined signal to its corresponding category to obtain the expanded data of each category;

[0026] The multi-physiological signal fusion module arranges all the data after data enhancement according to certain rules, so that the signal data originally belonging to different leads are arranged in the same matrix;

[0027] The deep learning module uses a deep learning network. First, the network is trained with training samples. The trained network is used to predict the category of the test samples, and finally the prediction results are displayed.

[0028] The technical solution adopted by the method of the present invention is: a classification method for multi-physiological signal fusion for cognitive state detection, characterized by comprising the following steps:

[0029] Step 1: Read the input cognitive state detection data containing multiple physiological signals (EEG and EOG) and their annotation information, and store different physiological signals and annotation information separately;

[0030] Step 2: Perform preprocessing operations on the signal data stored in step 1, filter according to the frequency range required by different signals (0.5-50Hz for EEG and 0.1-20Hz for EOG) to remove noise signals. If the sampling rates of different physiological signals are different, the signal with the high sampling rate needs to be downsampled to make it consistent with the sampling rate of the signal with the lowest sampling rate. After filtering and downsampling, the signal format is: the number of signal leads × the number of signal sampling points, where the number of sampling points for all signals is the same;

[0031] Step 3: Splice the signals into the same array. The format of the spliced ​​signal is: the sum of all signal leads × the number of signal sampling points. Then divide the signal by trial, so that its format becomes: number of trials × the sum of all signal leads × the number of sampling points per trial.

[0032] Step 4: Read the signal annotation information stored in step 1, and group the signals obtained in step 3 according to the annotation information. Each category is divided into one group. Each trial in the data set has a corresponding category. Therefore, the form of each category of signals is: number of category trials × sum of all signal leads × number of sampling points per trial;

[0033] Step 5: Use the EMD algorithm to decompose the signal and then reconstruct the signal to achieve data enhancement;

[0034] In specific implementation, for a certain category, two trial signals are extracted. If the signals of these two trial signals have not been extracted before, EMD decomposition is performed on each lead of the two trial signals. The result obtained for each trial is: the sum of the number of signal leads × the number of intrinsic mode functions (IMFs) obtained by EMD decomposition × the number of sampling points per trial, and the result obtained for that trial is saved; if the trial used has been extracted before, the decomposed result is directly used. The IMF components of each lead of the two extracted trials are exchanged, and when the exchange is performed, the exchange is performed between the same leads. Trials that have exchanged components with each other will no longer be exchanged, and newly generated trials will not participate in the exchange. Taking the first lead of the two trials as an example, the result of its EMD decomposition is extracted, and the original signals are recorded as X and Y.

[0035]

[0036]

[0037] Among them, n means there are n IMF components after EMD decomposition, IMFx i 、IMFy i Represents the i-th IMF component of signal X and Y respectively, x r 、y r They represent the residuals after EMD decomposition of X and Y, respectively. For X and Y, the third IMF component IMFx3 and IMFy3 are taken out respectively and exchanged with the third IMF component of the other trial. After the exchange is completed, the IMF components are summed to obtain two new signals X' and Y'.

[0038] X'=X-IMFx3+IMFy3 (3)

[0040] Y'=Y-IMFy3+IMFx3 (4)

[0042] At this point, the signal form is consistent with the original signal, and it can be considered that X' and Y' are consistent with the categories to which X and Y belong. The data volume is expanded to about twice the original data set to obtain the final data set.

[0043] Step 6: Fusion of multiple physiological signals;

[0044] For specific implementation, please see Figure 2 、 Figure 3 . Figure 2 The diagram shows the electrode positions for EOG and EEG signal collection, where EOG1 and EOG2 are for the left and right eyes, respectively, and the rest of the positions are for EEG signals collected according to the 10-20 international standard lead system. Each signal lead position can correspond to a unique coordinate and can be mapped to Figure 3 In the matrix shown, the matrix size is 11×11. Figure 3 The matrix shown is denoted as A. The matrix coordinates are counted from 1, and the horizontal and vertical coordinates are positive from left to right and from top to bottom, respectively. When the coordinate (p, q) has no corresponding signal lead, the value of A[p][q] is 0. When the coordinate (p, q) has a corresponding signal lead, the value of A[p][q] is the signal sampling value of the corresponding signal lead.

[0045] according to Figure 4The specific correspondence is as follows: A[1][3] corresponds to lead EOG1, A[1][9] corresponds to lead EOG2, A[2][4] corresponds to lead Fp1, A[2][6] corresponds to lead Fpz, A[2][8] corresponds to lead Fp2, A[3][2] corresponds to lead AF7, A[3][3] corresponds to lead AF5, A[3][4] corresponds to lead AF3, A[3][5] corresponds to lead AF1, A[3][6] corresponds to lead AFz, A[3][7] corresponds to lead AF2, A[3][8] corresponds to lead AF4, A[3][9] corresponds to lead AF6, A[3]

[10] corresponds to lead AF8, A[4][1] corresponds to lead F9, A[4][2] corresponds to lead F 7, A[4][3] corresponds to lead F5, A[4][4] corresponds to lead F3, A[4][5] corresponds to lead F1, A[4][6] corresponds to lead Fz, A[4][7] corresponds to lead F2, A[4][8] corresponds to lead F4, A[4][9] corresponds to lead F6, A[4]

[10] corresponds to lead F8, A[4]

[11] corresponds to lead F10, A[5][1] corresponds to lead FT9, A[5][2] corresponds to lead FT7, A[5][3] corresponds to lead FC5, A[5][4] corresponds to lead FC3, A[5][5] corresponds to lead FC1, A[5][6] corresponds to lead FCz, A[5][7] corresponds to lead FC2, and A[5][8] corresponds to lead FC4 , A[5][9] corresponds to lead FC6, A[5]

[10] corresponds to lead FT8, A[5]

[11] corresponds to lead FT10, A[6][1] corresponds to lead T9, A[6][2] corresponds to lead T7, A[6][3] corresponds to lead C5, A[6][4] corresponds to lead C3, A[6][5] corresponds to lead C1, A[6][6] corresponds to lead Cz, A[6][7] corresponds to lead C2, A[6][8] corresponds to lead C4, A[6][9] corresponds to lead C6, A[6]

[10] corresponds to lead T8, A[6]

[11] corresponds to lead T10, A[7][1] corresponds to lead TP9, A[7][2] corresponds to lead TP7, and A[7][3] corresponds to lead CP5 , A[7][4] corresponds to lead CP3, A[7][5] corresponds to lead CP1, A[7][6] corresponds to lead CPz, A[7][7] corresponds to lead CP2, A[7][8] corresponds to lead CP4, A[7][9] corresponds to lead CP6, A[7]

[10] corresponds to lead TP8, A[7]

[11] corresponds to lead TP10, A[8][1] corresponds to lead P9, A[8][2] corresponds to lead P7, A[8][3] corresponds to lead P5, A[8][4] corresponds to lead P3, A[8][5] corresponds to lead P1, A[8][6] corresponds to lead Pz, A[8][7] corresponds to lead P2, A[8][8] corresponds to lead P4, and A[8][9] corresponds to lead P6,A[8]

[10] corresponds to lead P8, A[8]

[11] corresponds to lead P10, A[9][1] corresponds to lead PO9, A[9][2] corresponds to lead PO7, A[9][3] corresponds to lead PO5, A[9][4] corresponds to lead PO3, A[9][5] corresponds to lead PO1, A[9][6] corresponds to lead POz, A[9][7] corresponds to lead PO2, A[9][8] corresponds to lead PO4, A [9][9] corresponds to lead PO6, A[9]

[10] corresponds to lead PO8, A[9]

[11] corresponds to lead PO10, A

[10] [4] corresponds to lead O1, A

[10] [6] corresponds to lead Oz, A

[10] [8] corresponds to lead O2, A

[11] [4] corresponds to lead I1, A

[11] [6] corresponds to lead Iz, A

[11] [8] corresponds to lead I2, and other unmentioned positions have no corresponding leads.

[0046] For each trial, each sampling point in the trial can get a Figure 3 The signal sampling value matrix shown in the figure, then the number of matrices that can be obtained for each trial is the number of sampling points of the trial, and these matrices form a matrix sequence in the order of sampling points; after all the trials obtained in step 5 are implemented according to this step, the matrix sequence of all trials can be obtained;

[0047] Step 7: Training of deep learning network;

[0048] For specific implementation, please see Figure 4 . Figure 4 Middle t1-t m The matrix represents the sampling point matrix sequence of a single trial in step 6, where m is the number of sampling points in a single trial. In the figure, CNN is first used to extract the features of each sampling point. The structure of CNN is convolution layer + pooling layer + convolution layer + pooling layer + fully connected layer. The pooling layer uses the maximum pooling method. For specific implementation, please see Figure 5For a single CNN, the input size is 11×11. After passing through a convolutional layer with two 3×3 convolution kernels and a stride of 1, the shape becomes 2×9×9. After passing through a pooling layer with a kernel size of 2×2 and a stride of 2, the shape becomes 2×5×5. After passing through a convolutional layer with a 3×3 convolution kernel and a stride of 1, the shape becomes 3×3. After passing through a pooling layer with a kernel size of 2×2 and a stride of 2, the shape becomes 2×2. After passing through a fully connected layer, the output size is 2. Then, a long short-term memory network (LSTM) is used to extract the features of the CNN output sequence. The sequence length of the LSTM is m. The input size of each LSTM unit is 2 and the output size is 1. Finally, a fully connected layer is used to further integrate the extracted features. The input size of the fully connected layer is m and the output size is the number of categories. The SoftMax function is used for the final classification. During training, the matrix sequence obtained from one trial is a training sample, and all trials of a single subject are fed into Figure 4 The network shown in the figure can be used to obtain a trained model. During training, the Adam optimizer was used as the model optimizer, and all Adam parameters were consistent with their default parameters. The cross entropy loss function was used as the model loss function. The training and validation sets accounted for 60% and 20% of the total number of test samples, respectively. The batch size of each training and validation data was 32, and the training process lasted for 40 rounds.

[0049] Step 8: Prediction of test samples;

[0050] In the specific implementation, the test data of a single subject (accounting for 20% of the total number of samples of the subject) is sent to Figure 4 The network shown in the figure uses the network parameters trained in step 7 to calculate the predicted category of the test sample and then calculate the prediction accuracy; the test data batch size is 32.

[0051] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0052] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A classification method for multi-physiological signal fusion for cognitive state detection, characterized by The details are as follows: Input: multiple physiological signals and their annotation information Output: Category of the test sample and classification accuracy Step 1: Read multiple input physiological signals and their annotation information, and store the various physiological signals and annotation information separately; Step 2: Preprocess the input physiological signal, including filtering and downsampling; Step 3: Splice different physiological signals into the same array and divide the signals according to trial; Step 4: Divide the signal into categories based on the labeled information; Step 5: Data enhancement: Use the EMD algorithm to decompose the signal, exchange the IMF components and reorganize the signal to obtain an expanded data set; data enhancement is specifically as follows: For a certain category, two trial signals are taken out. If the signals of these two trials have not been taken out before, each lead of the two trial signals is subjected to EMD decomposition. The result of each trial is: the sum of the number of signal leads × the number of intrinsic mode functions (IMFs) obtained by EMD decomposition × the number of sampling points per trial, and the result of the trial is saved; if the trial used has been taken out before, the decomposed result is used directly; the IMF components of each lead of the two trials are exchanged, and when the exchange is performed, the exchange is performed between the same leads; trials that have exchanged components with each other will no longer be exchanged, and newly generated trials will not participate in the exchange; For the first lead of the two trials, extract the result of EMD decomposition and record the original signal as X and Y; Among them, n means there are n IMF components after EMD decomposition, IMFx i 、IMFy i Represents the i-th IMF component of signal X and Y respectively, x r 、y r Represent the residues of X and Y after EMD decomposition respectively; for X and Y, the third IMF component IMFx3 and IMFy3 are taken out respectively, and exchanged with the third IMF component of the other trial. After the exchange is completed, the IMF components are summed to obtain two new signals X' and Y'; X'=X-IMFx3+IMFy3 (3) Y'=Y-IMFy3+IMFx3 (4) At this point, the signal form is consistent with the original signal, and X' and Y' are considered to be consistent with the categories to which X and Y belong. The data volume is expanded to twice the original data set to obtain the final data set. Step 6: Fusion of multiple physiological signals; Step 7: Training of deep learning network; Step 8: Prediction of the category to which the test sample belongs and calculation of classification accuracy; Step 9: End.

2. The classification method according to claim 1, wherein: When performing data enhancement, the signal decomposed by the EMD algorithm can be expressed as IMF i represents the i-th IMF component decomposed, n represents the total number of n IMF components after EMD decomposition, and r represents the residual obtained by EMD decomposition; when two different trials exchange IMF components, the number i of the exchanged IMF components should be consistent, and the value of i should be between 1 and 10; after a series of exchange and recombination operations, the final total number of trials for a single subject should be between 1 and 10 times the original number of trials, and the number of each category should be basically the same.

3. The classification method according to claim 1, wherein: The data matrix is ​​filled according to the distribution of the acquisition positions of different physiological signals. One acquisition position corresponds to the position coordinate of one matrix, and the signal values ​​of all signal acquisition positions are filled into the corresponding coordinates of the same two-dimensional matrix.

4. The classification method according to claim 1, wherein: The deep learning network structure constructed is: CNN+LSTM+outer fully connected layer+SoftMax, among which the specific structure of CNN is: convolution layer+pooling layer+convolution layer+pooling layer+inner fully connected layer.

5. The classification method according to claim 1, wherein: The Python language was chosen to implement the classification method of multi-physiological signal fusion for cognitive state detection; therefore, when processing signals, the scipy library and the numpy library were used for data reading, signal filtering, and splicing operations; when performing data enhancement, the Python PyEMD library was used to implement EMD decomposition; when constructing deep learning networks and using deep learning networks for training and prediction, the Python torch library was used.

6. The classification method according to claim 1, wherein: Step 1: Read the input cognitive state detection data including EEG and EOG and their annotation information, and store different physiological signals and annotation information separately; Step 2: Preprocess the signal data stored in step 1 by filtering according to the frequency range required by different signals to remove noise signals; EEG is 0.5-50Hz, and EOG is 0.1-20Hz. If different physiological signals have different sampling rates, the signal with the higher sampling rate needs to be downsampled to match the sampling rate of the signal with the lowest sampling rate. After filtering and downsampling, the signal format is: the number of signal leads × the number of signal sampling points, where the number of sampling points is the same for all signals. Step 3: Splice the signals into the same array. The format of the spliced ​​signal is: the sum of all signal leads × the number of signal sampling points. Then divide the signal by trial, so that its format becomes: number of trials × the sum of all signal leads × the number of sampling points per trial. Step 4: Read the signal annotation information stored in step 1, and group the signals obtained in step 3 according to the annotation information. Each category is divided into one group. Each trial in the data set has a corresponding category. Therefore, the form of each category of signals is: number of category trials × sum of all signal leads × number of sampling points per trial; Step 5: Use the EMD algorithm to decompose the signal and then reconstruct the signal to achieve data enhancement; Step 6: Fusion of multiple physiological signals; According to the signal acquisition position, the sampling value of the signal at each position at a certain moment is mapped to the matrix of the corresponding position; for each trial, each sampling point in the trial can obtain such a signal sampling value matrix, and the number of matrices that can be obtained for each trial is the number of sampling points in the trial. These matrices form a matrix sequence in the order of the sampling points; after all the trials obtained in step 5 are implemented according to this step, the matrix sequence of all trials can be obtained; Step 7: Training of deep learning network; First, CNN is used to extract the features of each sampling point. The structure of CNN is convolution layer + pooling layer + convolution layer + pooling layer + fully connected layer, where the pooling layer uses the maximum pooling method. For a single CNN, the input size is 11×11. After passing through a convolutional layer with two 3×3 convolution kernels and a stride of 1, the shape becomes 2×9×9. After passing through a pooling layer with a kernel size of 2×2 and a stride of 2, the shape becomes 2×5×5. After passing through a convolutional layer with a 3×3 convolution kernel and a stride of 1, the shape becomes 3×3. After passing through a pooling layer with a kernel size of 2×2 and a stride of 2, the shape becomes 2×2. After passing through a fully connected layer, the output size is 2. Then, a long short-term memory network LSTM is used to extract the features of the CNN output sequence. The sequence length of the LSTM is m, the input size of each LSTM unit is 2, and the output size is 1. Finally, a fully connected layer is used to further integrate the extracted features. The input size of the fully connected layer is m, and the output size is the number of categories. The SoftMax function is used for the final classification. During training, the matrix sequence obtained from one trial is considered a training sample. All trials of a single subject are fed into the network to obtain a trained model. During training, the Adam optimizer is used as the model optimizer, and the cross entropy loss function is used as the model loss function. The training process is a total of 40 rounds. Step 8: Prediction of test samples; In specific implementation, the test data of a single subject is fed into the network, and the network parameters trained in step 7 are used for calculation to obtain the predicted category of the test sample, and then the prediction accuracy is calculated.

Images