Multimodal cognitive screening method and apparatus based on brain-computer interface and expression recognition

Abstract

This invention discloses a multimodal cognitive screening method and apparatus based on brain-computer interface and facial expression recognition. The apparatus includes: a brain-computer interface module, an image acquisition module, a signal feature extraction module, and a cognitive assessment module. The brain-computer interface module is used to acquire the electroencephalogram (EEG) signal sequence of a user during a cognitive test. The image acquisition module is used to acquire a set of facial expression images of the user during the cognitive test. The signal feature extraction module is used to perform feature recognition and extraction processing on the acquired EEG signal sequence and facial expression image set to obtain an EEG feature sequence and an expression category sequence. The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence, expression category sequence, facial expression image set, and EEG signal sequence to obtain a cognitive impairment assessment result.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interfaces and brain cognition, and specifically to a multimodal cognitive screening method and apparatus based on brain-computer interfaces and facial expression recognition. Background Technology

[0002] Cognitive impairment, including but not limited to dementia and mild cognitive impairment, severely affects an individual's daily life and work abilities. Early and accurate cognitive screening is crucial for intervention and early rehabilitation.

[0003] Currently, common cognitive screening methods mainly rely on questionnaires and neuropsychological tests. However, these methods have certain limitations. Neuropsychological tests depend on the patient's subjective cooperation and expressive ability, and are easily affected by factors such as emotions and fatigue; questionnaires, on the other hand, suffer from low accuracy and cumbersome procedures.

[0004] The development of brain-computer interfaces (BCIs) and computer vision technologies has provided new avenues for cognitive screening. BCIs can directly acquire neural activity signals from the brain, reflecting cognitive states; computer vision technology can capture patients' emotions and responses during cognitive tasks. However, current technologies lack multimodal cognitive screening methods and devices that combine BCIs and computer vision technologies.

[0005] How to organically integrate brain-computer interfaces and computer vision technology to construct multimodal cognitive screening methods and devices, thereby effectively improving the accuracy and comprehensiveness of cognitive screening, is an urgent problem to be solved. Summary of the Invention

[0006] This invention primarily addresses the problem of how to organically integrate brain-computer interfaces and computer vision technology to construct a multimodal cognitive screening method and device, thereby effectively improving the accuracy and comprehensiveness of cognitive screening. This invention discloses a multimodal cognitive screening method and device based on brain-computer interfaces and facial expression recognition.

[0007] In a first aspect, this application discloses a multimodal cognitive screening device based on brain-computer interface and facial expression recognition, comprising: a brain-computer interface module, an image acquisition module, a signal feature extraction module, and a cognitive assessment module;

[0008] The brain-computer interface module is used to acquire the EEG signal sequence of the user during cognitive testing;

[0009] The image acquisition module is used to acquire a set of facial expression images of the user during the cognitive test;

[0010] The signal feature extraction module is connected to the brain-computer interface module, the image acquisition module and the cognitive assessment module respectively, and is used to perform feature recognition and extraction processing on the acquired EEG signal sequence and facial expression image set to obtain EEG feature sequence and expression category sequence.

[0011] The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence, expression category sequence, facial expression image set and EEG signal sequence to obtain cognitive impairment assessment results.

[0012] The signal feature extraction module is used to perform feature recognition and extraction processing on the acquired EEG signal sequences and facial expression image sets to obtain EEG feature sequences and expression category sequences, including:

[0013] The signal feature extraction module preprocesses the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

[0014] Feature extraction processing is performed on the preprocessed EEG signal sequence and the preprocessed facial expression image set to obtain EEG feature sequence and expression category sequence.

[0015] The signal feature extraction module preprocesses the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set, including:

[0016] The signal feature extraction module performs data cleaning processing on the acquired EEG signal sequence and facial expression image set to obtain cleaned EEG signal sequence and cleaned facial expression image set.

[0017] Data normalization was performed on the cleaned EEG signal sequence and the cleaned facial expression image set to obtain the preprocessed EEG signal sequence and the preprocessed facial expression image set.

[0018] A second aspect of this invention discloses a multimodal cognitive screening method based on brain-computer interface and facial expression recognition, implemented using the aforementioned multimodal cognitive screening device based on brain-computer interface and facial expression recognition, comprising:

[0019] Using the brain-computer interface module, the EEG signal sequence of the user during cognitive testing is acquired;

[0020] Using the image acquisition module, a set of facial expression images of the user during the cognitive test is acquired;

[0021] Using the signal feature extraction module, feature recognition and extraction processing is performed on the acquired EEG signal sequence and facial expression image set to obtain EEG feature sequence and expression category sequence;

[0022] Using the cognitive assessment module, the EEG feature sequence, expression category sequence, facial expression image set, and EEG signal sequence are fused and assessed to obtain the cognitive impairment assessment result.

[0023] The signal feature extraction module is used to perform feature recognition and extraction processing on the acquired EEG signal sequence and facial expression image set to obtain EEG feature sequence and expression category sequence, including:

[0024] Using the signal feature extraction module, the acquired EEG signal sequence and facial expression image set are preprocessed to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

[0025] Feature extraction processing is performed on the preprocessed EEG signal sequence and the preprocessed facial expression image set to obtain EEG feature sequence and expression category sequence.

[0026] The process of using the signal feature extraction module to preprocess the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set includes:

[0027] Using the signal feature extraction module, the acquired EEG signal sequence and facial expression image set are respectively cleaned to obtain cleaned EEG signal sequence and cleaned facial expression image set;

[0028] Data normalization was performed on the cleaned EEG signal sequence and the cleaned facial expression image set to obtain the preprocessed EEG signal sequence and the preprocessed facial expression image set.

[0029] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing to obtain EEG feature sequences and expression category sequences, including:

[0030] Obtain a set of standard electroencephalogram (EEG) signals and a set of standard facial expression images; the set of standard facial expression images includes facial expression images and corresponding expression category values;

[0031] The preprocessed EEG signal sequence and the standard EEG signal are subjected to differential feature extraction processing to obtain the EEG feature sequence;

[0032] Using a preset facial expression recognition model, the standard facial expression image set and the preprocessed facial expression image set are processed to obtain an expression category sequence; the expression category sequence includes the predicted expression category values ​​corresponding to all standard facial expression images in the standard facial expression image set.

[0033] The step of extracting differential features from the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence includes:

[0034] The preprocessed EEG signal sequence includes several preprocessed EEG signals;

[0035] Each preprocessed EEG signal was subtracted from the standard EEG signal to obtain the corresponding difference signal.

[0036] Using all the differential signals, an EEG abnormality matrix is ​​constructed; the row vectors of the EEG abnormality matrix are the differential signals.

[0037] Using the difference signals as sample data, the sample covariance matrix is ​​calculated; the expression for calculating the sample covariance matrix A is:

[0038]

[0039] Where E represents the mean, and X represents the EEG abnormality matrix. This represents the mean of all differential signals;

[0040] The first feature vector is obtained by performing a first feature vector extraction process on the EEG abnormality matrix;

[0041] The expression for the first feature vector extraction process is:

[0042]

[0043] Where m and n represent the row and column dimensions of the EEG abnormality matrix, z ij This represents the element in the i-th row and j-th column of the EEG abnormality matrix. Let z represent the i-th element of the first eigenvector. + This represents the first eigenvector;

[0044] The second feature vector is obtained by performing a second feature vector extraction process on the EEG abnormality matrix;

[0045] The expression for the second feature vector extraction process is:

[0046]

[0047] in, Let z represent the i-th element of the second eigenvector. - This represents the second eigenvector;

[0048] The first feature vector, the second feature vector, and the sample covariance matrix are processed to obtain the EEG feature sequence.

[0049] The expression for the feature calculation process is:

[0050]

[0051] Among them, A ij Let A be the element in the i-th row and j-th column of the sample covariance matrix. ii p is the element in the i-th row and i-th column of the sample covariance matrix. i is the element of the i-th row of the EEG feature sequence.

[0052] The process involves using a pre-defined facial expression recognition model to perform calculations on the standard facial expression image set and the pre-processed facial expression image set to obtain an expression category sequence, including:

[0053] Each preprocessed facial expression image in the preprocessed facial expression image set is represented as a corresponding matrix to be recognized;

[0054] Each standard expression image in the set of standard expression images is represented as a standard matrix corresponding to the expression category.

[0055] Subtract the matrix to be identified from the standard matrix corresponding to each expression category to obtain a set of difference matrices; the difference matrices in the set of difference matrices correspond to the expression categories.

[0056] For each difference matrix, the set of eigenvalues ​​and trace values ​​are obtained by solving. The absolute value of the difference between each eigenvalue and the trace value in the eigenvalue set is calculated. The minimum value of all absolute differences is determined as the eigenvalue of the difference matrix.

[0057] The expression category corresponding to the difference matrix with the minimum quality is determined, which is the predicted expression category value of the matrix to be identified;

[0058] Based on the predicted expression category values ​​of the expression category of the matrix to be identified, the predicted expression category values ​​of the preprocessed facial expression image corresponding to the matrix to be identified are determined.

[0059] The beneficial effects of this invention are as follows:

[0060] This invention combines data from both brain-computer interfaces and facial expression recognition, enabling a more comprehensive reflection of the test subject's cognitive state and improving the accuracy of screening. This invention achieves fusion simultaneously at the feature level and the signal level, enhancing the fusion effect and ensuring the accuracy of cognitive impairment screening.

[0061] This invention employs a non-invasive detection method, which is non-invasive to the test subject and easy for users to accept and use.

[0062] Before evaluating EEG signals, this invention first performs differential feature extraction on the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence. Based on the EEG feature sequence, a sample covariance matrix, a first eigenvector, and a second eigenvector are constructed. By performing feature calculation on the first eigenvector, the second eigenvector, and the sample covariance matrix, the EEG feature sequence is obtained, thereby suppressing interference and noise in the EEG signal. In the process of recognizing facial expression images, this invention calculates a difference matrix for each expression type, and obtains a set of eigenvalues ​​and trace values ​​by solving for them, thereby extracting abnormal facial expression features.

[0063] After obtaining the EEG feature sequence and the facial expression category sequence, this invention performs a fusion evaluation on the two types of sequences to obtain the cognitive impairment assessment result, ensuring the completeness of the fusion result. Attached Figure Description

[0064] Figure 1 This is a diagram showing the composition of the device of the present invention;

[0065] Figure 2 This is a flowchart illustrating the implementation of the method of the present invention. Detailed Implementation

[0066] To better understand the content of this invention, an embodiment is provided here.

[0067] Figure 1 This is a diagram showing the composition of the device of the present invention; Figure 2 This is a flowchart illustrating the implementation of the method of the present invention.

[0068] In a first aspect, this application discloses a multimodal cognitive screening device based on brain-computer interface and facial expression recognition, comprising: a brain-computer interface module, an image acquisition module, a signal feature extraction module, and a cognitive assessment module;

[0069] The brain-computer interface module is used to acquire the electroencephalogram (EEG) signal sequence of the user during cognitive testing;

[0070] The image acquisition module is used to acquire a set of facial expression images of the user during the cognitive test;

[0071] The signal feature extraction module is used to perform feature recognition and extraction processing on the acquired EEG signal sequence and facial expression image set to obtain EEG feature sequence and expression category sequence.

[0072] The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence and the facial expression category sequence to obtain the cognitive impairment assessment result.

[0073] The signal feature extraction module is used to perform feature recognition and extraction processing on the acquired EEG signal sequences and facial expression image sets to obtain EEG feature sequences and expression category sequences, including:

[0074] The signal feature extraction module preprocesses the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

[0075] Feature extraction processing is performed on the preprocessed EEG signal sequence and the preprocessed facial expression image set to obtain EEG feature sequence and expression category sequence, respectively;

[0076] The signal feature extraction module preprocesses the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set, including:

[0077] The signal feature extraction module performs data cleaning processing on the acquired EEG signal sequence and facial expression image set to obtain cleaned EEG signal sequence and cleaned facial expression image set.

[0078] Data normalization was performed on the cleaned EEG signal sequence and the cleaned facial expression image set to obtain the preprocessed EEG signal sequence and the preprocessed facial expression image set.

[0079] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing to obtain EEG feature sequences and expression category sequences, including:

[0080] Obtain a set of standard electroencephalogram (EEG) signals and a set of standard facial expression images; the set of standard facial expression images includes facial expression images and corresponding expression category values;

[0081] The preprocessed EEG signal sequence and the standard EEG signal are subjected to differential feature extraction processing to obtain the EEG feature sequence;

[0082] Using a preset facial expression recognition model, the standard facial expression image set and the preprocessed facial expression image set are processed to obtain an expression category sequence; the expression category sequence includes the predicted expression category values ​​corresponding to all standard facial expression images in the standard facial expression image set;

[0083] The step of extracting differential features from the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence includes:

[0084] The preprocessed EEG signal sequence includes several preprocessed EEG signals;

[0085] Each preprocessed EEG signal was subtracted from the standard EEG signal to obtain the corresponding difference signal;

[0086] Using all the differential signals, an EEG abnormality matrix is ​​constructed; the row vectors of the EEG abnormality matrix are the differential signals.

[0087] Using the difference signals as sample data, the sample covariance matrix is ​​calculated; the expression for calculating the sample covariance matrix is ​​as follows:

[0088]

[0089] Where E represents the mean, and X represents the EEG abnormality matrix. This represents the mean of all differential signals;

[0090] The first feature vector is obtained by performing a first feature vector extraction process on the EEG abnormality matrix;

[0091] The expression for the first feature vector extraction process is:

[0092]

[0093] Where m and n represent the row and column dimensions of the EEG abnormality matrix, z ij This represents the element in the i-th row and j-th column of the EEG abnormality matrix. Let z represent the i-th element of the first eigenvector. + This represents the first eigenvector;

[0094] The second feature vector is obtained by performing a second feature vector extraction process on the EEG abnormality matrix;

[0095] The expression for the second feature vector extraction process is:

[0096]

[0097] in, Let z represent the i-th element of the second eigenvector. - This represents the second eigenvector;

[0098] The first feature vector, the second feature vector, and the sample covariance matrix are processed to obtain the EEG feature sequence.

[0099] The expression for the feature calculation process is:

[0100]

[0101] Among them, A ij Let A be the element in the i-th row and j-th column of the sample covariance matrix. iip is the element in the i-th row and i-th column of the sample covariance matrix. i is the element of the i-th row of the EEG feature sequence.

[0102] The process involves using a pre-defined facial expression recognition model to perform calculations on the standard facial expression image set and the pre-processed facial expression image set to obtain an expression category sequence, including:

[0103] Each preprocessed facial expression image in the preprocessed facial expression image set is represented as a corresponding matrix to be recognized;

[0104] Each standard expression image in the set of standard expression images is represented as a standard matrix corresponding to the expression category.

[0105] Subtract the matrix to be identified from the standard matrix corresponding to each expression category to obtain a set of difference matrices; the difference matrices in the set of difference matrices correspond to the expression categories.

[0106] For each difference matrix, the set of eigenvalues ​​and trace values ​​are obtained by solving. The absolute value of the difference between each eigenvalue and the trace value in the eigenvalue set is calculated. The minimum value of all absolute differences is determined as the eigenvalue of the difference matrix.

[0107] The expression category corresponding to the difference matrix with the minimum quality is determined, which is the predicted expression category value of the matrix to be identified;

[0108] Based on the predicted expression category values ​​of the expression category of the matrix to be identified, determine the predicted expression category values ​​of the preprocessed facial expression image corresponding to the matrix to be identified;

[0109] The process of fusing and evaluating the EEG feature sequences and facial expression category sequences to obtain cognitive impairment assessment results includes:

[0110] Obtain the standard facial expression category sequence for cognitive tests;

[0111] The expression category sequence and the standard expression category sequence are processed to calculate the expression difference, and an expression evaluation value is obtained.

[0112] The expression used for calculating and processing facial expression differences is:

[0113]

[0114] Where qe represents the facial expression evaluation value, and aq i Let ae represent the i-th element of the expression category sequence. i This represents the i-th element of the standard expression category sequence, where N is the number of elements in the expression category sequence.

[0115] The EEG feature sequence is subjected to EEG evaluation processing to obtain EEG evaluation values;

[0116] The expression for the EEG assessment processing is:

[0117]

[0118] Where pe represents the electroencephalogram (EEG) assessment value;

[0119] The facial expression assessment value and the electroencephalogram (EEG) assessment value are weighted and summed to obtain the cognitive impairment assessment result.

[0120] The cognitive impairment assessment results are used to evaluate the severity of the user's cognitive impairment; the higher the value, the more severe the user's cognitive impairment.

[0121] The weighted summation of the facial expression assessment value and the EEG assessment value can be performed using weight values ​​of 0.3 and 0.7.

[0122] The value of the expression category is a numerical code value for each expression category. For example, the code value for a calm expression is 1, the code value for an angry expression is 2, the code value for a frowning expression is 3, the code value for a smiling expression is 4, and the code value for an excited expression is 5.

[0123] The cognitive test may be a memory test, attention test, number memory test, image matching test, logical reasoning test, etc.

[0124] The brain-computer interface module can be implemented using a brain-computer interface device.

[0125] The data cleaning process includes filling in missing values, smoothing noisy data, and smoothing or deleting outliers. Smoothing noisy data involves first identifying the noisy data, and then smoothing it based on the data preceding and following it. The noisy data refers to values ​​whose values ​​are less than the sensor's detection sensitivity for the observed data, or greater than the sensor's measurement limit for the observed data. Outlier identification can be performed using Kalman filtering. The filling values ​​for missing values ​​can be determined by averaging the measurements within a certain sampling interval before and after the missing value.

[0126] The data normalization process maps data from different value ranges to a specified value range. Its mathematical expression is:

[0127]

[0128] Where t is a specific data value in the dataset, t max t represents the maximum value of all data in the dataset. min t is the minimum value of all data in the dataset. 归一化The value of data t after normalization;

[0129] A second aspect of this application discloses a multimodal cognitive screening method based on brain-computer interface and facial expression recognition, implemented using the aforementioned multimodal cognitive screening device based on brain-computer interface and facial expression recognition, comprising:

[0130] Using the brain-computer interface module, the EEG signal sequence of the user during cognitive testing is acquired;

[0131] Using the image acquisition module, a set of facial expression images of the user during the cognitive test is acquired;

[0132] Using the signal feature extraction module, feature recognition and extraction processing is performed on the acquired EEG signal sequence and facial expression image set to obtain EEG feature sequence and expression category sequence;

[0133] Using the cognitive assessment module, the EEG feature sequence, expression category sequence, facial expression image set, and EEG signal sequence are fused and assessed to obtain the cognitive impairment assessment result.

[0134] The signal feature extraction module is used to perform feature recognition and extraction processing on the acquired EEG signal sequence and facial expression image set to obtain EEG feature sequence and expression category sequence, including:

[0135] Using the signal feature extraction module, the acquired EEG signal sequence and facial expression image set are preprocessed to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set.

[0136] Feature extraction processing is performed on the preprocessed EEG signal sequence and the preprocessed facial expression image set to obtain EEG feature sequence and expression category sequence.

[0137] The process of using the signal feature extraction module to preprocess the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set includes:

[0138] Using the signal feature extraction module, the acquired EEG signal sequence and facial expression image set are respectively cleaned to obtain cleaned EEG signal sequence and cleaned facial expression image set;

[0139] Data normalization was performed on the cleaned EEG signal sequence and the cleaned facial expression image set to obtain the preprocessed EEG signal sequence and the preprocessed facial expression image set.

[0140] The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing to obtain EEG feature sequences and expression category sequences, including:

[0141] Obtain a set of standard electroencephalogram (EEG) signals and a set of standard facial expression images; the set of standard facial expression images includes facial expression images and corresponding expression category values;

[0142] The preprocessed EEG signal sequence and the standard EEG signal are subjected to differential feature extraction processing to obtain the EEG feature sequence;

[0143] Using a preset facial expression recognition model, the standard facial expression image set and the preprocessed facial expression image set are processed to obtain an expression category sequence; the expression category sequence includes the predicted expression category values ​​corresponding to all standard facial expression images in the standard facial expression image set.

[0144] The preset facial expression recognition model can be a YOLO series image recognition model or a deep learning model for facial expression image recognition.

[0145] The step of extracting differential features from the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence includes:

[0146] The preprocessed EEG signal sequence includes several preprocessed EEG signals;

[0147] Each preprocessed EEG signal was subtracted from the standard EEG signal to obtain the corresponding difference signal.

[0148] Using all the differential signals, an EEG abnormality matrix is ​​constructed; the row vectors of the EEG abnormality matrix are the differential signals.

[0149] Using the difference signals as sample data, the sample covariance matrix is ​​calculated; the expression for calculating the sample covariance matrix A is:

[0150]

[0151] Where E represents the mean, and X represents the EEG abnormality matrix. This represents the mean of all differential signals;

[0152] The first feature vector is obtained by performing a first feature vector extraction process on the EEG abnormality matrix;

[0153] The expression for the first feature vector extraction process is:

[0154]

[0155] Where m and n represent the row and column dimensions of the EEG abnormality matrix, zij This represents the element in the i-th row and j-th column of the EEG abnormality matrix. Let z represent the i-th element of the first eigenvector. + This represents the first eigenvector;

[0156] The second feature vector is obtained by performing a second feature vector extraction process on the EEG abnormality matrix;

[0157] The expression for the second feature vector extraction process is:

[0158]

[0159] in, Let z represent the i-th element of the second eigenvector. - This represents the second eigenvector;

[0160] The first feature vector, the second feature vector, and the sample covariance matrix are processed to obtain the EEG feature sequence.

[0161] The expression for the feature calculation process is:

[0162]

[0163] Among them, A ij Let A be the element in the i-th row and j-th column of the sample covariance matrix. ii p is the element in the i-th row and i-th column of the sample covariance matrix. i is the element of the i-th row of the EEG feature sequence.

[0164] The process involves using a pre-defined facial expression recognition model to perform calculations on the standard facial expression image set and the pre-processed facial expression image set to obtain an expression category sequence, including:

[0165] Each preprocessed facial expression image in the preprocessed facial expression image set is represented as a corresponding matrix to be recognized;

[0166] Each standard expression image in the set of standard expression images is represented as a standard matrix corresponding to the expression category.

[0167] Subtract the matrix to be identified from the standard matrix corresponding to each expression category to obtain a set of difference matrices; the difference matrices in the set of difference matrices correspond to the expression categories.

[0168] For each difference matrix, the set of eigenvalues ​​and trace values ​​are obtained by solving. The absolute value of the difference between each eigenvalue and the trace value in the eigenvalue set is calculated. The minimum value of all absolute differences is determined as the eigenvalue of the difference matrix.

[0169] The expression category corresponding to the difference matrix with the minimum quality is determined, which is the predicted expression category value of the matrix to be identified;

[0170] Based on the predicted expression category values ​​of the expression category of the matrix to be identified, the predicted expression category values ​​of the preprocessed facial expression image corresponding to the matrix to be identified are determined.

[0171] The process of fusing and evaluating the EEG feature sequences and facial expression category sequences to obtain cognitive impairment assessment results includes:

[0172] Obtain the standard facial expression category sequence for cognitive tests;

[0173] The expression category sequence and the standard expression category sequence are processed to calculate the expression difference, and an expression evaluation value is obtained.

[0174] The expression used for calculating and processing facial expression differences is:

[0175]

[0176] Where qe represents the facial expression evaluation value, and aq i Let ae represent the i-th element of the expression category sequence. i This represents the i-th element of the standard expression category sequence, where N is the number of elements in the expression category sequence.

[0177] The EEG feature sequence is subjected to EEG evaluation processing to obtain EEG evaluation values;

[0178] The expression for the EEG assessment processing is:

[0179]

[0180] Where pe represents the electroencephalogram (EEG) assessment value;

[0181] The facial expression assessment value and the electroencephalogram (EEG) assessment value are weighted and summed to obtain the cognitive impairment assessment result.

[0182] The cognitive impairment assessment result is used to evaluate the severity of the user's cognitive impairment. The higher the value, the more severe the user's cognitive impairment.

[0183] The weighted summation of the facial expression assessment value and the EEG assessment value can be performed using weight values ​​of 0.3 and 0.7.

[0184] The value of the expression category is a numerical code value for each expression category. For example, the code value for a calm expression is 1, the code value for an angry expression is 2, the code value for a frowning expression is 3, the code value for a smiling expression is 4, and the code value for an excited expression is 5.

[0185] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multimodal cognitive screening device based on brain-computer interface and facial expression recognition, characterized in that, include: Brain-computer interface module, image acquisition module, signal feature extraction module, cognitive assessment module; The brain-computer interface module is used to acquire the EEG signal sequence of the user during cognitive testing; The image acquisition module is used to acquire a set of facial expression images of the user during the cognitive test; The signal feature extraction module is connected to the brain-computer interface module, the image acquisition module, and the cognitive assessment module, respectively. It is used to perform feature recognition and extraction processing on the acquired EEG signal sequences and facial expression image sets to obtain EEG feature sequences and expression category sequences, including: The signal feature extraction module preprocesses the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set. Feature extraction processing is performed on the preprocessed EEG signal sequence and the preprocessed facial expression image set to obtain EEG feature sequences and expression category sequences, including: Obtain a set of standard electroencephalogram (EEG) signals and a set of standard facial expression images; the set of standard facial expression images includes facial expression images and corresponding expression category values; The preprocessed EEG signal sequence and the standard EEG signal are subjected to differential feature extraction processing to obtain an EEG feature sequence, including: The preprocessed EEG signal sequence includes several preprocessed EEG signals; Each preprocessed EEG signal was subtracted from the standard EEG signal to obtain the corresponding difference signal. Using all the differential signals, an EEG abnormality matrix is ​​constructed; the row vectors of the EEG abnormality matrix are the differential signals. Using the difference signals as sample data, the sample covariance matrix is ​​calculated; the expression for calculating the sample covariance matrix A is: ， in, This indicates the calculation of the mean, where Z represents the EEG abnormality matrix. This represents the mean of all differential signals; The first feature vector is obtained by performing a first feature vector extraction process on the EEG abnormality matrix; The expression for the first feature vector extraction process is: ， Where m and n represent the row and column dimensions of the EEG abnormality matrix, This represents the element in the i-th row and j-th column of the EEG abnormality matrix. This represents the i-th element of the first eigenvector. This represents the first eigenvector; The second feature vector is obtained by performing a second feature vector extraction process on the EEG abnormality matrix; The expression for the second feature vector extraction process is: ， in, This represents the i-th element of the second eigenvector. This represents the second eigenvector; The first feature vector, the second feature vector, and the sample covariance matrix are processed to obtain the EEG feature sequence. Using a preset facial expression recognition model, the standard facial expression image set and the preprocessed facial expression image set are processed to obtain an expression category sequence; the expression category sequence includes the predicted expression category values ​​corresponding to all standard facial expression images in the standard facial expression image set; The cognitive assessment module is used to perform fusion assessment processing on the EEG feature sequence, expression category sequence, facial expression image set and EEG signal sequence to obtain cognitive impairment assessment results.

2. The multimodal cognitive screening device based on brain-computer interface and facial expression recognition as described in claim 1, characterized in that, The signal feature extraction module preprocesses the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set, including: The signal feature extraction module performs data cleaning processing on the acquired EEG signal sequence and facial expression image set to obtain cleaned EEG signal sequence and cleaned facial expression image set. Data normalization was performed on the cleaned EEG signal sequence and the cleaned facial expression image set to obtain the preprocessed EEG signal sequence and the preprocessed facial expression image set.

3. A multimodal cognitive screening method based on brain-computer interface and facial expression recognition, characterized in that, The multimodal cognitive screening device based on brain-computer interface and facial expression recognition, as described in any one of claims 1 to 2, includes: Using the brain-computer interface module, the EEG signal sequence of the user during cognitive testing is acquired; Using the image acquisition module, a set of facial expression images of the user during the cognitive test is acquired; Using the signal feature extraction module, feature recognition and extraction processing is performed on the acquired EEG signal sequence and facial expression image set to obtain EEG feature sequence and expression category sequence; Using the cognitive assessment module, the EEG feature sequence, expression category sequence, facial expression image set, and EEG signal sequence are fused and assessed to obtain the cognitive impairment assessment result.

4. The multimodal cognitive screening method based on brain-computer interface and facial expression recognition as described in claim 3, characterized in that, The signal feature extraction module is used to perform feature recognition and extraction processing on the acquired EEG signal sequence and facial expression image set to obtain EEG feature sequence and expression category sequence, including: Using the signal feature extraction module, the acquired EEG signal sequence and facial expression image set are preprocessed to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set. Feature extraction processing is performed on the preprocessed EEG signal sequence and the preprocessed facial expression image set to obtain EEG feature sequence and expression category sequence.

5. The multimodal cognitive screening method based on brain-computer interface and facial expression recognition as described in claim 4, characterized in that, The process of using the signal feature extraction module to preprocess the acquired EEG signal sequence and facial expression image set to obtain a preprocessed EEG signal sequence and a preprocessed facial expression image set includes: Using the signal feature extraction module, the acquired EEG signal sequence and facial expression image set are respectively cleaned to obtain cleaned EEG signal sequence and cleaned facial expression image set; Data normalization was performed on the cleaned EEG signal sequence and the cleaned facial expression image set to obtain the preprocessed EEG signal sequence and the preprocessed facial expression image set.

6. The multimodal cognitive screening method based on brain-computer interface and facial expression recognition as described in claim 5, characterized in that, The preprocessed EEG signal sequence and the preprocessed facial expression image set are subjected to feature extraction processing to obtain EEG feature sequences and expression category sequences, including: Obtain a set of standard electroencephalogram (EEG) signals and a set of standard facial expression images; the set of standard facial expression images includes facial expression images and corresponding expression category values; The preprocessed EEG signal sequence and the standard EEG signal are subjected to differential feature extraction processing to obtain the EEG feature sequence; Using a preset facial expression recognition model, the standard facial expression image set and the preprocessed facial expression image set are processed to obtain an expression category sequence; the expression category sequence includes the predicted expression category values ​​corresponding to all standard facial expression images in the standard facial expression image set.

7. The multimodal cognitive screening method based on brain-computer interface and facial expression recognition as described in claim 6, characterized in that, The step of extracting differential features from the preprocessed EEG signal sequence and the standard EEG signal to obtain an EEG feature sequence includes: The preprocessed EEG signal sequence includes several preprocessed EEG signals; Each preprocessed EEG signal was subtracted from the standard EEG signal to obtain the corresponding difference signal. Using all the differential signals, an EEG abnormality matrix is ​​constructed; the row vectors of the EEG abnormality matrix are the differential signals. Using the difference signals as sample data, the sample covariance matrix is ​​calculated; the expression for calculating the sample covariance matrix A is: ， in, This indicates the calculation of the mean, where Z represents the EEG abnormality matrix. This represents the mean of all differential signals; The first feature vector is obtained by performing a first feature vector extraction process on the EEG abnormality matrix; The expression for the first feature vector extraction process is: ， Where m and n represent the row and column dimensions of the EEG abnormality matrix, This represents the element in the i-th row and j-th column of the EEG abnormality matrix. This represents the i-th element of the first eigenvector. This represents the first eigenvector; The second feature vector is obtained by performing a second feature vector extraction process on the EEG abnormality matrix; The expression for the second feature vector extraction process is: ， in, This represents the i-th element of the second eigenvector. This represents the second eigenvector; The first feature vector, the second feature vector, and the sample covariance matrix are processed to obtain the EEG feature sequence.

8. The multimodal cognitive screening method based on brain-computer interface and facial expression recognition as described in claim 7, characterized in that, The expression for the feature calculation process is: ， in, Let be the element in the i-th row and j-th column of the sample covariance matrix. Let be the element in the i-th row and i-th column of the sample covariance matrix. is the element of the i-th row of the EEG feature sequence.

9. The multimodal cognitive screening method based on brain-computer interface and facial expression recognition as described in claim 6, characterized in that, The process involves using a pre-defined facial expression recognition model to perform calculations on the standard facial expression image set and the pre-processed facial expression image set to obtain an expression category sequence, including: Each preprocessed facial expression image in the preprocessed facial expression image set is represented as a corresponding matrix to be recognized; Each standard expression image in the set of standard expression images is represented as a standard matrix corresponding to the expression category. Subtract the matrix to be identified from the standard matrix corresponding to each expression category to obtain a set of difference matrices; the difference matrices in the set of difference matrices correspond to the expression categories. For each difference matrix, the set of eigenvalues ​​and trace values ​​are obtained by solving. The absolute value of the difference between each eigenvalue and the trace value in the eigenvalue set is calculated. The minimum value of all absolute differences is determined as the eigenvalue of the difference matrix. The expression category corresponding to the difference matrix with the minimum quality is determined, which is the predicted expression category value of the matrix to be identified; Based on the predicted expression category values ​​of the expression category of the matrix to be identified, the predicted expression category values ​​of the preprocessed facial expression image corresponding to the matrix to be identified are determined.

Images