Multi-modal cognitive screening method and device based on brain-computer interface and expression recognition

Abstract

The invention discloses a multi-modal cognitive screening method and device based on a brain-computer interface and expression recognition. The device comprises a brain-computer interface module, an image acquisition module, a signal feature extraction module and a cognitive evaluation module. The brain-computer interface module is used for acquiring an electroencephalogram signal sequence when a user performs a cognitive test; the image acquisition module is used for acquiring a facial expression image set when a user performs a cognitive test; the signal feature extraction module is used for performing feature recognition extraction processing on the acquired electroencephalogram signal sequence and the facial expression image set to obtain an electroencephalogram feature sequence and an expression category sequence; and the cognitive assessment module is used for performing fusion assessment processing on the electroencephalogram feature sequence, the expression category sequence, the facial expression image set and the electroencephalogram signal sequence to obtain a cognitive impairment assessment result.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface and brain cognition, and in particular to a multimodal cognitive screening method and device based on brain-computer interface and expression recognition. Background Art

[0002] Cognitive dysfunction seriously affects an individual's daily life and work ability, including but not limited to dementia, mild cognitive impairment, etc. Early and accurate cognitive screening is of great significance for its intervention and early recovery.

[0003] At present, common cognitive screening methods mainly rely on questionnaires, neuropsychological tests, etc. However, these methods have certain limitations. Neuropsychological tests rely on the patient's subjective cooperation and expression ability, and are easily affected by factors such as emotions and fatigue; questionnaires have problems such as low accuracy and cumbersome operation.

[0004] With the development of brain-computer interface and computer vision technology, new approaches have been provided for cognitive screening. Brain-computer interface can directly obtain the brain's neural activity signals to reflect cognitive status; computer vision technology can capture the patient's emotions and reactions during cognitive tasks. However, in the existing technology, there is no multimodal cognitive screening method and device that combines brain-computer interface and computer vision technology.

[0005] How to organically integrate brain-computer interface and computer vision technology to construct multimodal cognitive screening methods and devices, thereby effectively improving the accuracy and comprehensiveness of cognitive screening, is a problem that urgently needs to be solved. Summary of the invention

[0006] The present invention mainly solves the problem of how to organically integrate brain-computer interface and computer vision technology to construct a multimodal cognitive screening method and device, thereby effectively improving the accuracy and comprehensiveness of cognitive screening. The present invention discloses a multimodal cognitive screening method and device based on brain-computer interface and expression recognition.

[0007] In a first aspect of the embodiment of the present application, a multimodal cognitive screening device based on brain-computer interface and expression recognition is disclosed, comprising: a brain-computer interface module, an image acquisition module, a signal feature extraction module, and a cognitive assessment module;

[0008] The brain-computer interface module is used to collect and obtain the EEG signal sequence of the user when performing cognitive tests;

[0009] The image acquisition module is used to acquire a set of facial expression images of the user when taking a cognitive test;

[0010] The signal feature extraction module is connected to the brain-computer interface module, the image acquisition module and the cognitive assessment module respectively, and is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence;

[0011] The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence, expression category sequence, facial expression image set and EEG signal sequence to obtain a cognitive impairment assessment result.

[0012] The signal feature extraction module is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence, including:

[0013] The signal feature extraction module preprocesses the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set;

[0014] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence.

[0015] The signal feature extraction module preprocesses the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set, including:

[0016] The signal feature extraction module performs data cleaning processing on the collected EEG signal sequence and facial expression image set to obtain a cleaned EEG signal sequence and a cleaned facial expression image set;

[0017] The cleaned EEG signal sequence and the cleaned facial expression image set are respectively subjected to data normalization processing to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

[0018] In a second aspect of an embodiment of the present invention, a multimodal cognitive screening method based on a brain-computer interface and facial expression recognition is disclosed, which is implemented using the multimodal cognitive screening device based on a brain-computer interface and facial expression recognition, and includes:

[0019] Using the brain-computer interface module, the brain electrical signal sequence of the user when taking a cognitive test is collected;

[0020] Using the image acquisition module, a set of facial expression images of the user when taking a cognitive test is acquired;

[0021] Using the signal feature extraction module, the collected EEG signal sequence and facial expression image set are processed for feature recognition and extraction to obtain an EEG feature sequence and an expression category sequence;

[0022] The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence, expression category sequence, facial expression image set and EEG signal sequence to obtain a cognitive impairment assessment result.

[0023] The signal feature extraction module is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence, including:

[0024] Using the signal feature extraction module, preprocessing the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set;

[0025] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence.

[0026] The method of using the signal feature extraction module to preprocess the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set includes:

[0027] Using the signal feature extraction module, respectively performing data cleaning processing on the collected EEG signal sequence and facial expression image set to obtain a cleaned EEG signal sequence and a cleaned facial expression image set;

[0028] The cleaned EEG signal sequence and the cleaned facial expression image set are respectively subjected to data normalization processing to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

[0029] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence, including:

[0030] Obtaining a standard EEG signal and a standard expression image set; the standard expression image set includes a facial expression image and a corresponding expression category value;

[0031] Performing difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence;

[0032] The preset expression recognition model is used to calculate and process the standard expression image set and the preprocessed facial expression image set to obtain an expression category sequence; the expression category sequence includes expression category prediction values ​​corresponding to all standard expression images in the standard expression image set.

[0033] The step of performing difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence includes:

[0034] The pre-processed EEG signal sequence includes a plurality of pre-processed EEG signals;

[0035] Subtract each preprocessed EEG signal from the standard EEG signal to obtain a corresponding difference signal;

[0036] Using all the difference signals, an EEG abnormality matrix is ​​constructed; the row vector of the EEG abnormality matrix is ​​the difference signal;

[0037] The difference signal is used as sample data to calculate the sample covariance matrix; the calculation expression of the sample covariance matrix A is:

[0038]

[0039] Among them, E represents the mean value, X represents the EEG abnormality matrix, represents the mean of all difference signals;

[0040] Performing a first eigenvector extraction process on the abnormal EEG matrix to obtain a first eigenvector;

[0041] The expression of the first feature vector extraction process is:

[0042]

[0043] Among them, m and n represent the row and column dimensions of the EEG abnormality matrix, z ij represents the element of the ith row and jth column of the EEG abnormality matrix, represents the i-th element of the first eigenvector, z + represents the first eigenvector;

[0044] Performing a second eigenvector extraction process on the abnormal EEG matrix to obtain a second eigenvector;

[0045] The expression of the second feature vector extraction process is:

[0046]

[0047] in, represents the i-th element of the second eigenvector, z - represents the second eigenvector;

[0048] The first eigenvector, the second eigenvector and the sample covariance matrix are subjected to feature calculation processing to obtain an EEG feature sequence.

[0049] The expression of the feature calculation process is:

[0050]

[0051] Among them, A ij is the element of the i-th row and j-th column of the sample covariance matrix, A ii is the element of the i-th row and i-th column of the sample covariance matrix, p i is the element of the i-th row of the EEG feature sequence.

[0052] The method uses a preset expression recognition model to calculate and process the standard expression image set and the pre-processed facial expression image set to obtain an expression category sequence, including:

[0053] Each pre-processed facial expression image in the pre-processed facial expression image set is represented as a corresponding matrix to be identified;

[0054] Representing the standard expression image corresponding to each expression category in the standard expression image set as a standard matrix corresponding to the expression category;

[0055] Subtracting the matrix to be identified from the standard matrix corresponding to each expression category respectively to obtain a difference matrix set; the difference matrices in the difference matrix set have corresponding expression categories;

[0056] For each difference matrix, solve to obtain the eigenvalue set and trace value, calculate the absolute value of the difference between each eigenvalue and the trace value in the eigenvalue set, and determine the minimum value of all the absolute values ​​of the difference, which is the characteristic value of the difference matrix;

[0057] Determine the expression category corresponding to the difference matrix with the minimum characteristic quality, which is the expression category prediction value of the matrix to be identified;

[0058] According to the expression category prediction value of the matrix to be identified, the expression category prediction value of the preprocessed facial expression image corresponding to the matrix to be identified is determined.

[0059] The beneficial effects of the present invention are:

[0060] The present invention combines the data of two modalities, brain-computer interface and facial expression recognition, to more comprehensively reflect the cognitive state of the subject and improve the accuracy of screening. The present invention achieves fusion at both the feature level and the signal level, improves the fusion effect, and ensures the accuracy of cognitive impairment screening.

[0061] The present invention adopts a non-invasive detection method, is non-invasive to the tested person, and is easy for users to accept and use.

[0062] Before evaluating the EEG signal, the present invention first performs difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence; based on the EEG feature sequence, a sample covariance matrix, a first eigenvector and a second eigenvector are constructed, and the EEG feature sequence is obtained by performing feature calculation processing on the first eigenvector, the second eigenvector and the sample covariance matrix, thereby achieving suppression of interference and noise in the EEG signal; in the process of recognizing the expression image, the present invention calculates the difference matrix for each expression type, obtains the eigenvalue set and the trace value by solving, and achieves the extraction of abnormal expression features.

[0063] After obtaining the EEG feature sequence and the expression category sequence, the present invention obtains the cognitive impairment assessment result by performing fusion evaluation processing on the two types of sequence numbers, thereby ensuring the completeness of the fusion result. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a composition diagram of the device of the present invention;

[0065] Figure 2 It is a flow chart for implementing the method of the present invention. DETAILED DESCRIPTION

[0066] In order to better understand the content of the present invention, an embodiment is given here.

[0067] Figure 1 It is a composition diagram of the device of the present invention; Figure 2 It is a flow chart for implementing the method of the present invention.

[0068] In a first aspect of the embodiment of the present application, a multimodal cognitive screening device based on brain-computer interface and expression recognition is disclosed, comprising: a brain-computer interface module, an image acquisition module, a signal feature extraction module, and a cognitive assessment module;

[0069] The brain-computer interface module is used to collect and obtain the EEG signal sequence of the user when performing cognitive tests;

[0070] The image acquisition module is used to acquire a set of facial expression images of the user when taking a cognitive test;

[0071] The signal feature extraction module is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence;

[0072] The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence and the expression category sequence to obtain a cognitive impairment assessment result;

[0073] The signal feature extraction module is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence, including:

[0074] The signal feature extraction module preprocesses the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set;

[0075] Performing feature extraction processing on the preprocessed EEG signal sequence and the preprocessed facial expression image set respectively to obtain an EEG feature sequence and an expression category sequence;

[0076] The signal feature extraction module preprocesses the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set, including:

[0077] The signal feature extraction module performs data cleaning processing on the collected EEG signal sequence and facial expression image set to obtain a cleaned EEG signal sequence and a cleaned facial expression image set;

[0078] Performing data normalization processing on the cleaned EEG signal sequence and the cleaned facial expression image set respectively to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set;

[0079] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence, including:

[0080] Obtaining a standard EEG signal and a standard expression image set; the standard expression image set includes a facial expression image and a corresponding expression category value;

[0081] Performing difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence;

[0082] Using a preset expression recognition model, the standard expression image set and the pre-processed facial expression image set are subjected to computational processing to obtain an expression category sequence; the expression category sequence includes expression category prediction values ​​corresponding to all standard expression images in the standard expression image set;

[0083] The step of performing difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence includes:

[0084] The pre-processed EEG signal sequence includes a plurality of pre-processed EEG signals;

[0085] Subtract each preprocessed EEG signal from the standard EEG signal to obtain a corresponding difference signal;

[0086] Using all the difference signals, an EEG abnormality matrix is ​​constructed; the row vector of the EEG abnormality matrix is ​​the difference signal;

[0087] The difference signal is used as sample data to calculate the sample covariance matrix; the calculation expression of the sample covariance matrix is:

[0088]

[0089] Among them, E represents the mean value, X represents the EEG abnormality matrix, represents the mean of all difference signals;

[0090] Performing a first eigenvector extraction process on the abnormal EEG matrix to obtain a first eigenvector;

[0091] The expression of the first feature vector extraction process is:

[0092]

[0093] Among them, m and n represent the row and column dimensions of the EEG abnormality matrix, z ij represents the element of the ith row and jth column of the EEG abnormality matrix, represents the i-th element of the first eigenvector, z + represents the first eigenvector;

[0094] Performing a second eigenvector extraction process on the abnormal EEG matrix to obtain a second eigenvector;

[0095] The expression of the second feature vector extraction process is:

[0096]

[0097] in, represents the i-th element of the second eigenvector, z - represents the second eigenvector;

[0098] Performing feature calculation processing on the first eigenvector, the second eigenvector and the sample covariance matrix to obtain an EEG feature sequence;

[0099] The expression of the feature calculation process is:

[0100]

[0101] Among them, A ij is the element of the i-th row and j-th column of the sample covariance matrix, A iiis the element of the i-th row and i-th column of the sample covariance matrix, p i is the element of the i-th row of the EEG feature sequence.

[0102] The method uses a preset expression recognition model to calculate and process the standard expression image set and the pre-processed facial expression image set to obtain an expression category sequence, including:

[0103] Each pre-processed facial expression image in the pre-processed facial expression image set is represented as a corresponding matrix to be identified;

[0104] Representing the standard expression image corresponding to each expression category in the standard expression image set as a standard matrix corresponding to the expression category;

[0105] Subtracting the matrix to be identified from the standard matrix corresponding to each expression category respectively to obtain a difference matrix set; the difference matrices in the difference matrix set have corresponding expression categories;

[0106] For each difference matrix, solve to obtain the eigenvalue set and trace value, calculate the absolute value of the difference between each eigenvalue and the trace value in the eigenvalue set, and determine the minimum value of all the absolute values ​​of the difference, which is the characteristic value of the difference matrix;

[0107] Determine the expression category corresponding to the difference matrix with the minimum characteristic quality, which is the expression category prediction value of the matrix to be identified;

[0108] Determine the predicted value of the expression category of the preprocessed facial expression image corresponding to the matrix to be identified according to the predicted value of the expression category of the matrix to be identified;

[0109] The step of fusing and evaluating the EEG feature sequence and the expression category sequence to obtain a cognitive impairment evaluation result includes:

[0110] Obtain standard expression category sequences for cognitive testing;

[0111] Performing expression difference calculation processing on the expression category sequence and the standard expression category sequence to obtain an expression evaluation value;

[0112] The expression for the expression difference calculation process is:

[0113]

[0114] Among them, qe represents the expression evaluation value, aq i Represents the i-th element of the expression category sequence, ae i represents the i-th element of the standard expression category sequence, and N is the number of elements in the expression category sequence;

[0115] Performing electroencephalogram evaluation processing on the electroencephalogram feature sequence to obtain an electroencephalogram evaluation value;

[0116] The expression of the EEG evaluation process is:

[0117]

[0118] Among them, pe is the EEG evaluation value;

[0119] Performing weighted sum processing on the facial expression evaluation value and the EEG evaluation value to obtain a cognitive impairment evaluation result;

[0120] The cognitive impairment assessment result is used to assess the severity of the user's cognitive impairment, and a larger value indicates a more severe cognitive impairment of the user.

[0121] The weighted summation process is performed on the expression evaluation value and the EEG evaluation value, and the weight values ​​used may be 0.3 and 0.7.

[0122] The value of the expression category is the digital code value of each type of expression. For example, the code value of a calm expression is 1, the code value of an angry expression is 2, the code value of a frowning expression is 3, the code value of a smiling expression is 4, and the code value of an excited expression is 5.

[0123] The cognitive test may be a memory test, an attention test, a digital memory, a graphic matching test, a logical reasoning test, etc.

[0124] The brain-computer interface module can be implemented using a brain-computer interface device.

[0125] The data cleaning process includes filling missing values, smoothing noise data, and smoothing or deleting outliers; the smoothed noise data is obtained by first determining the noise data, and then smoothing the noise data according to the data before and after the noise data; the noise data is a value whose value is less than the detection sensitivity of the sensor of the observed data, or greater than the measurement upper limit of the sensor of the observed data. The outlier point can be determined by the Kalman filter method. The filling value of the missing value can be determined by averaging the measured values ​​within a certain sampling interval before and after the missing value.

[0126] The data normalization process is to map data of different value ranges into a specified value range, and its mathematical expression is:

[0127]

[0128] Among them, t is a data value in the data set, t max is the maximum value of all data in the data set, t min is the minimum value of all data in the data set, t 归一化is the value of data t after normalization;

[0129] In a second aspect of the embodiment of the present application, a multimodal cognitive screening method based on a brain-computer interface and expression recognition is disclosed, which is implemented using the multimodal cognitive screening device based on a brain-computer interface and expression recognition, including:

[0130] Using the brain-computer interface module, the brain electrical signal sequence of the user when taking a cognitive test is collected;

[0131] Using the image acquisition module, a set of facial expression images of the user when taking a cognitive test is acquired;

[0132] Using the signal feature extraction module, the collected EEG signal sequence and facial expression image set are processed for feature recognition and extraction to obtain an EEG feature sequence and an expression category sequence;

[0133] The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence, expression category sequence, facial expression image set and EEG signal sequence to obtain a cognitive impairment assessment result.

[0134] The signal feature extraction module is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence, including:

[0135] Using the signal feature extraction module, preprocessing the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set;

[0136] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence.

[0137] The method of using the signal feature extraction module to preprocess the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set includes:

[0138] Using the signal feature extraction module, respectively performing data cleaning processing on the collected EEG signal sequence and facial expression image set to obtain a cleaned EEG signal sequence and a cleaned facial expression image set;

[0139] The cleaned EEG signal sequence and the cleaned facial expression image set are respectively subjected to data normalization processing to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

[0140] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence, including:

[0141] Obtaining a standard EEG signal and a standard expression image set; the standard expression image set includes a facial expression image and a corresponding expression category value;

[0142] Performing difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence;

[0143] The preset expression recognition model is used to calculate and process the standard expression image set and the preprocessed facial expression image set to obtain an expression category sequence; the expression category sequence includes expression category prediction values ​​corresponding to all standard expression images in the standard expression image set.

[0144] The preset expression recognition model may be a YOLO series image recognition model, or a deep learning model for expression image recognition.

[0145] The step of performing difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence includes:

[0146] The pre-processed EEG signal sequence includes a plurality of pre-processed EEG signals;

[0147] Subtract each preprocessed EEG signal from the standard EEG signal to obtain a corresponding difference signal;

[0148] Using all the difference signals, an EEG abnormality matrix is ​​constructed; the row vector of the EEG abnormality matrix is ​​the difference signal;

[0149] The difference signal is used as sample data to calculate the sample covariance matrix; the calculation expression of the sample covariance matrix A is:

[0150]

[0151] Among them, E represents the mean value, X represents the EEG abnormality matrix, represents the mean of all difference signals;

[0152] Performing a first eigenvector extraction process on the abnormal EEG matrix to obtain a first eigenvector;

[0153] The expression of the first feature vector extraction process is:

[0154]

[0155] Among them, m and n represent the row and column dimensions of the EEG abnormality matrix, zij represents the element of the ith row and jth column of the EEG abnormality matrix, represents the i-th element of the first eigenvector, z + represents the first eigenvector;

[0156] Performing a second eigenvector extraction process on the abnormal EEG matrix to obtain a second eigenvector;

[0157] The expression of the second feature vector extraction process is:

[0158]

[0159] in, represents the i-th element of the second eigenvector, z - represents the second eigenvector;

[0160] The first eigenvector, the second eigenvector and the sample covariance matrix are subjected to feature calculation processing to obtain an EEG feature sequence.

[0161] The expression of the feature calculation process is:

[0162]

[0163] Among them, A ij is the element of the i-th row and j-th column of the sample covariance matrix, A ii is the element of the i-th row and i-th column of the sample covariance matrix, p i is the element of the i-th row of the EEG feature sequence.

[0164] The method uses a preset expression recognition model to calculate and process the standard expression image set and the pre-processed facial expression image set to obtain an expression category sequence, including:

[0165] Each pre-processed facial expression image in the pre-processed facial expression image set is represented as a corresponding matrix to be identified;

[0166] Representing the standard expression image corresponding to each expression category in the standard expression image set as a standard matrix corresponding to the expression category;

[0167] Subtracting the matrix to be identified from the standard matrix corresponding to each expression category respectively to obtain a difference matrix set; the difference matrices in the difference matrix set have corresponding expression categories;

[0168] For each difference matrix, solve to obtain the eigenvalue set and trace value, calculate the absolute value of the difference between each eigenvalue and the trace value in the eigenvalue set, and determine the minimum value of all the absolute values ​​of the difference, which is the characteristic value of the difference matrix;

[0169] Determine the expression category corresponding to the difference matrix with the minimum characteristic quality, which is the expression category prediction value of the matrix to be identified;

[0170] According to the expression category prediction value of the matrix to be identified, the expression category prediction value of the preprocessed facial expression image corresponding to the matrix to be identified is determined.

[0171] The step of fusing and evaluating the EEG feature sequence and the expression category sequence to obtain a cognitive impairment evaluation result includes:

[0172] Obtain standard expression category sequences for cognitive testing;

[0173] Performing expression difference calculation processing on the expression category sequence and the standard expression category sequence to obtain an expression evaluation value;

[0174] The expression for the expression difference calculation process is:

[0175]

[0176] Among them, qe represents the expression evaluation value, aq i Represents the i-th element of the expression category sequence, ae i represents the i-th element of the standard expression category sequence, and N is the number of elements in the expression category sequence;

[0177] Performing electroencephalogram evaluation processing on the electroencephalogram feature sequence to obtain an electroencephalogram evaluation value;

[0178] The expression of the EEG evaluation process is:

[0179]

[0180] Among them, pe is the EEG evaluation value;

[0181] Performing weighted sum processing on the facial expression evaluation value and the EEG evaluation value to obtain a cognitive impairment evaluation result;

[0182] The cognitive impairment assessment result is used to assess the severity of the user's cognitive impairment, and a larger value indicates a more severe cognitive impairment of the user.

[0183] The weighted summation process is performed on the expression evaluation value and the EEG evaluation value, and the weight values ​​used may be 0.3 and 0.7.

[0184] The value of the expression category is the digital code value of each type of expression. For example, the code value of a calm expression is 1, the code value of an angry expression is 2, the code value of a frowning expression is 3, the code value of a smiling expression is 4, and the code value of an excited expression is 5.

[0185] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multimodal cognitive screening device based on brain-computer interface and expression recognition, characterized in that: include: Brain-computer interface module, image acquisition module, signal feature extraction module, cognitive assessment module; The brain-computer interface module is used to collect and obtain the EEG signal sequence of the user when performing cognitive testing; The image acquisition module is used to acquire a set of facial expression images of the user when taking a cognitive test; The signal feature extraction module is connected to the brain-computer interface module, the image acquisition module and the cognitive assessment module respectively, and is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence; The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence, expression category sequence, facial expression image set and EEG signal sequence to obtain a cognitive impairment assessment result.

2. The multimodal cognitive screening device based on brain-computer interface and expression recognition as claimed in claim 1, characterized in that: The signal feature extraction module is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence, including: The signal feature extraction module preprocesses the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set; The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence.

3. The multimodal cognitive screening device based on brain-computer interface and expression recognition as claimed in claim 2, characterized in that: The signal feature extraction module preprocesses the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set, including: The signal feature extraction module performs data cleaning processing on the collected EEG signal sequence and facial expression image set to obtain a cleaned EEG signal sequence and a cleaned facial expression image set; The cleaned EEG signal sequence and the cleaned facial expression image set are respectively subjected to data normalization processing to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

4. A multimodal cognitive screening method based on brain-computer interface and expression recognition, characterized in that: The method is implemented by using a multimodal cognitive screening device based on a brain-computer interface and expression recognition as described in any one of claims 1 to 3, comprising: Using the brain-computer interface module, the brain electrical signal sequence of the user when taking a cognitive test is collected; Using the image acquisition module, a set of facial expression images of the user when taking a cognitive test is acquired; Using the signal feature extraction module, the collected EEG signal sequence and facial expression image set are processed for feature recognition and extraction to obtain an EEG feature sequence and an expression category sequence; The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence, expression category sequence, facial expression image set and EEG signal sequence to obtain a cognitive impairment assessment result.

5. The multimodal cognitive screening method based on brain-computer interface and expression recognition as claimed in claim 4, characterized in that: The signal feature extraction module is used to perform feature recognition and extraction processing on the collected EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence, including: Using the signal feature extraction module, preprocessing the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set; The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence.

6. The multimodal cognitive screening method based on brain-computer interface and expression recognition as claimed in claim 5, characterized in that: The method of using the signal feature extraction module to preprocess the collected EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set includes: Using the signal feature extraction module, respectively performing data cleaning processing on the collected EEG signal sequence and facial expression image set to obtain a cleaned EEG signal sequence and a cleaned facial expression image set; The cleaned EEG signal sequence and the cleaned facial expression image set are respectively subjected to data normalization processing to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

7. The multimodal cognitive screening method based on brain-computer interface and expression recognition as claimed in claim 6, characterized in that: The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing respectively to obtain an EEG feature sequence and an expression category sequence, including: Obtaining a standard EEG signal and a standard expression image set; the standard expression image set includes a facial expression image and a corresponding expression category value; Performing difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence; The preset expression recognition model is used to calculate and process the standard expression image set and the preprocessed facial expression image set to obtain an expression category sequence; the expression category sequence includes expression category prediction values ​​corresponding to all standard expression images in the standard expression image set.

8. The multimodal cognitive screening method based on brain-computer interface and expression recognition as claimed in claim 7, characterized in that: The step of performing difference feature extraction processing on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence includes: The pre-processed EEG signal sequence includes a plurality of pre-processed EEG signals; Subtract each preprocessed EEG signal from the standard EEG signal to obtain a corresponding difference signal; Using all the difference signals, an EEG abnormality matrix is ​​constructed; the row vector of the EEG abnormality matrix is ​​the difference signal; The difference signal is used as sample data to calculate the sample covariance matrix; the calculation expression of the sample covariance matrix A is: Among them, E represents the mean value, X represents the EEG abnormality matrix, represents the mean of all difference signals; Performing a first eigenvector extraction process on the abnormal EEG matrix to obtain a first eigenvector; The expression of the first feature vector extraction process is: Among them, m and n represent the row and column dimensions of the EEG abnormality matrix, z ij represents the element of the ith row and jth column of the EEG abnormality matrix, represents the i-th element of the first eigenvector, z + represents the first eigenvector; Performing a second eigenvector extraction process on the abnormal EEG matrix to obtain a second eigenvector; The expression of the second feature vector extraction process is: in, represents the i-th element of the second eigenvector, z - represents the second eigenvector; The first eigenvector, the second eigenvector and the sample covariance matrix are subjected to feature calculation processing to obtain an EEG feature sequence.

9. The multimodal cognitive screening method based on brain-computer interface and expression recognition as claimed in claim 8, characterized in that: The expression of the feature calculation process is: Among them, A ij is the element of the i-th row and j-th column of the sample covariance matrix, A ii is the element of the i-th row and i-th column of the sample covariance matrix, p i is the element of the i-th row of the EEG feature sequence.

10. The multimodal cognitive screening method based on brain-computer interface and expression recognition as claimed in claim 7, characterized in that: The method uses a preset expression recognition model to calculate and process the standard expression image set and the pre-processed facial expression image set to obtain an expression category sequence, including: Each pre-processed facial expression image in the pre-processed facial expression image set is represented as a corresponding matrix to be identified; Representing the standard expression image corresponding to each expression category in the standard expression image set as a standard matrix corresponding to the expression category; Subtracting the matrix to be identified from the standard matrix corresponding to each expression category respectively to obtain a difference matrix set; the difference matrices in the difference matrix set have corresponding expression categories; For each difference matrix, solve to obtain the eigenvalue set and trace value, calculate the absolute value of the difference between each eigenvalue and the trace value in the eigenvalue set, and determine the minimum value of all the absolute values ​​of the difference, which is the characteristic value of the difference matrix; Determine the expression category corresponding to the difference matrix with the minimum characteristic quality, which is the expression category prediction value of the matrix to be identified; According to the expression category prediction value of the matrix to be identified, the expression category prediction value of the preprocessed facial expression image corresponding to the matrix to be identified is determined.

Images