Non-contact cognitive performance prediction method
By extracting contactless physiological and behavioral characteristic data from facial videos and combining with machine learning models, the problem of insufficient accuracy in cognitive performance prediction caused by single physiological parameter data is solved, and a more objective and accurate cognitive performance evaluation is achieved.
Patent Information
- Application Number
- CN202410914294.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-01
AI Technical Summary
In the prior art, single physiological parameter data is used to predict cognitive performance, resulting in insufficient objectivity and accuracy.
By obtaining facial visible light videos, non-contact physiological and head-face behavioral characteristic index data are extracted, dimensionality reduction is reduced based on correlation analysis, and machine learning models are used to predict cognitive performance, combining cognitive flexibility, inhibitory control and working memory dimensions, Design comprehensive cognitive performance evaluation indicators.
The objectivity and accuracy of the classification evaluation of cognitive performance by machine learning models is enhanced, and the complementarity of multimodal features is used to improve the accuracy of prediction results.
Smart Images

Figure CN118873091A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cognitive performance prediction, and in particular to a contactless cognitive performance prediction method, system, storage medium and electronic device. Background Art
[0002] The brain is considered one of the most complex biological organs in the world. As an external manifestation of the core functions of the brain, cognitive performance also has a complex influencing mechanism. Cognitive performance is related to the individual's sensory system, cerebral cortex, prefrontal cortex and nervous system, and is also affected by the individual's emotions, stress, anxiety and test environment.
[0003] In the related technologies, there are four types of physiological detection methods for predicting cognitive performance, namely functional magnetic resonance imaging, functional near-infrared spectroscopy, electroencephalogram and electrocardiogram. All methods use single data to study cognitive performance. Considering the complexity of the mechanism affecting cognitive performance, the use of single data will lead to insufficient objectivity and accuracy. The high sensitivity of physiological measurement methods enables it to accurately identify subtle changes in brain areas, but its high sensitivity also makes the collected data extremely susceptible to external factors, resulting in inaccurate physiological information. At the same time, using a single data source without any supplementary information, when errors occur in the collected data, it will lead to deviations in the prediction results of cognitive performance. Summary of the invention
[0004] 1. Technical issues to be resolved
[0005] In view of the deficiencies in the prior art, the present invention provides a non-contact cognitive performance prediction method, system, storage medium and electronic device, which solve the technical problem that using single physiological parameter data to predict cognitive performance will lead to insufficient objectivity and accuracy.
[0006] (II) Technical solution
[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0008] A non-contact cognitive performance prediction method, comprising:
[0009] Obtain the visible light video of the user's face and divide it into several video clips of equal length;
[0010] Based on the video clips, extracting non-contact physiological and head and facial behavior characteristic indicator data;
[0011] Based on the correlation analysis, the two types of extraction results are reduced in dimension respectively, and according to different cognitive performance evaluation indicators, the first number of non-contact physiological characteristic indicator data with the highest correlation ranking and the second number of head and face behavior characteristic indicator data with the highest correlation ranking are selected accordingly;
[0012] The cognitive performance evaluation indexes include the reaction time of cognitive flexibility, the accuracy of cognitive flexibility, the reaction time of inhibitory control, the accuracy of inhibitory control, the reaction time of working memory, and the accuracy of working memory;
[0013] A machine learning model corresponding to each of the cognitive performance evaluation indicators is retrieved, and the non-contact physiological and head and facial behavior characteristic indicator data selected under different cognitive performance evaluation indicators are used as input to predict the low-level or high-level cognitive performance of the user respectively.
[0014] Preferably, the extracting of non-contact physiological and head and face behavior characteristic index data based on the video clip includes:
[0015] Extracting rPPG signals based on the video clips, and extracting non-contact physiological characteristic indicator data using a HeartPy bioelectric signal tool based on the rPPG signals;
[0016] Based on the video clips, using Openface to extract head and facial behavior feature index data;
[0017] The non-contact physiological characteristic index data and head and facial behavior characteristic index data both include time domain characteristics and frequency domain characteristics under corresponding characteristic indicators; wherein the time domain characteristics are any one or a combination of any one of mean, standard deviation, maximum value, minimum value, peak value, root mean square, peak factor and waveform factor, and the frequency domain characteristics are any one or a combination of any one of center of gravity frequency, average frequency, frequency standard deviation and root mean square frequency.
[0018] Preferably, the non-contact physiological characteristic indicators include at least the following 16 types:
[0019] Heart rate, mean cardiac interval, root mean square of cardiac cycle difference, standard deviation of cardiac interval, standard deviation of adjacent cardiac cycle difference, number of adjacent cardiac cycle difference greater than 50ms, ratio of adjacent cardiac cycle difference greater than 50ms, total power less than or equal to 0.4Hz, very low frequency power between 0.003 and 0.04Hz, low frequency power between 0.04 and 0.15Hz, high frequency power between 0.15 and 0.4Hz, ratio of low frequency and high frequency power, standard deviation perpendicular to the same line in nonlinear analysis, standard deviation along the same line in nonlinear analysis, ratio of standard deviation perpendicular to the same line in nonlinear analysis to standard deviation along the same line in nonlinear analysis, area of ellipse described by standard deviation perpendicular to the same line in nonlinear analysis and standard deviation along the same line in nonlinear analysis.
[0020] Preferably, the head and face behavior characteristic indicators include at least the following:
[0021] 17 facial movement units: inner eyebrow raising, outer eyebrow raising, frowning, lifting upper eyelids, cheek raising, eye contraction, nose wrinkling, upper lip lifting, pulling mouth corners, tightening mouth corners, mouth corners down, lower lip lifting, pouting, lateral stretching of mouth corners, tightening lips, lip pressing, lips parting, chin lowering, mouth opening, eyes closed;
[0022] 8 eye gaze angles: gaze direction vector of the leftmost eye along the x-axis, gaze direction vector of the leftmost eye along the y-axis, gaze direction vector of the leftmost eye along the z-axis, gaze direction vector of the rightmost eye along the x-axis, gaze direction vector of the rightmost eye along the y-axis, gaze direction vector of the rightmost eye along the z-axis, average gaze direction of both eyes along the x-axis, average gaze direction of both eyes along the y-axis;
[0023] 6 head movement postures: away from the camera with positive z axis, the position of the head relative to the camera on the x axis, away from the camera with positive z axis, the position of the head relative to the camera on the y axis, away from the camera with positive z axis, the position of the head relative to the camera on the z axis, the deflection of the head along the x axis, the deflection of the head along the y axis, and the deflection of the head along the z axis;
[0024] Among them, the x-axis, y-axis, and z-axis refer to the three coordinate axes in the three-dimensional coordinate system constructed by spatial Cartesian coordinates.
[0025] Preferably, the first number is five.
[0026] Preferably, the second number is twenty.
[0027] Preferably, the machine learning model is any one or a combination of any of random forest, support vector machine, Bayesian classifier, logistic regression, decision tree, K-order nearest neighbor, extreme gradient boosting tree, lightweight gradient boosting machine.
[0028] Preferably, the machine learning model corresponding to the reaction time of the cognitive flexibility uses a lightweight gradient boosting machine.
[0029] Preferably, the machine learning model corresponding to the accuracy of the cognitive flexibility uses an extreme gradient boosting tree;
[0030] Preferably, a lightweight gradient boosting machine is selected as the machine learning model corresponding to the reaction time of the inhibitory control.
[0031] Preferably, a lightweight gradient boosting machine is selected as the machine learning model corresponding to the accuracy of the inhibitory control.
[0032] Preferably, the machine learning model corresponding to the reaction time of the working memory is a random forest.
[0033] Preferably, the machine learning model corresponding to the accuracy of the working memory is a random forest.
[0034] A non-contact cognitive performance prediction system, comprising:
[0035] A preprocessing module is used to obtain the visible light video of the user's face and divide it into several video clips of the same length;
[0036] An extraction module, used for extracting non-contact physiological and head and face behavior characteristic index data based on the video clip;
[0037] A selection module is used to reduce the dimensions of the two types of extraction results based on correlation analysis, and select the first number of non-contact physiological characteristic indicator data with the highest correlation ranking and the second number of head and facial behavior characteristic indicator data with the highest correlation ranking according to different cognitive performance evaluation indicators;
[0038] The cognitive performance evaluation indexes include the reaction time of cognitive flexibility, the accuracy of cognitive flexibility, the reaction time of inhibitory control, the accuracy of inhibitory control, the reaction time of working memory, and the accuracy of working memory;
[0039] The prediction module is used to retrieve the machine learning model corresponding to each of the cognitive performance evaluation indicators, and use the non-contact physiological and head and facial behavior characteristic indicator data selected under different cognitive performance evaluation indicators as input to predict the low-level or high-level cognitive performance of the user respectively.
[0040] A storage medium stores a computer program for contactless cognitive performance prediction, wherein the computer program enables a computer to execute the contactless cognitive performance prediction method as described above.
[0041] An electronic device, comprising:
[0042] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the non-contact cognitive performance prediction method as described above.
[0043] (III) Beneficial effects
[0044] The present invention provides a non-contact cognitive performance prediction method, system, storage medium and electronic device. Compared with the prior art, the present invention has the following beneficial effects:
[0045] In the present invention, the facial visible light video is divided into several video clips of the same length to extract non-contact physiological and head and facial behavior characteristic index data; based on correlation analysis, the two types of extraction results are reduced in dimension respectively, and according to different cognitive performance evaluation indicators, the non-contact physiological and head and facial behavior characteristic index data with the highest correlation ranking are selected and used as the corresponding machine learning model input to predict the user's low-level or high-level cognitive performance respectively. By utilizing the complementarity between multimodal features, and from the three dimensions of cognitive flexibility, inhibitory control and working memory, as well as the two perspectives of accuracy and reaction time, a comprehensive cognitive performance evaluation index is designed, which enhances the objectivity and accuracy of the machine learning model classification evaluation of cognitive performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 A framework diagram of a multi-execution function evaluation model provided by an embodiment of the present invention;
[0048] Figure 2 A block diagram of a non-contact cognitive performance prediction method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] The embodiments of the present application provide a non-contact cognitive performance prediction method, system, storage medium and electronic device to solve the technical problem that using single physiological parameter data to predict cognitive performance will lead to insufficient objectivity and accuracy.
[0051] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:
[0052] The applicants have recognized that heart rate variability, as an indicator of the activation level of inhibitory prefrontal cortex-subcortical structures, can be used to predict cognitive performance. In addition, an individual's cognitive performance is affected by the autonomic nervous system, and these effects are reflected in the individual's head and face behaviors.
[0053] In addition, cognitive performance involves an individual's ability to plan, control, and regulate complex behaviors. Non-contact executive function assessment is of great significance for reasonable work allocation and improving work efficiency. Based on this, the embodiment of the present invention will build a non-contact cognitive performance prediction model based on machine learning based on non-contact physiological and behavioral data to achieve classified evaluation of cognitive performance. The proposed multi-executive function assessment model framework is as follows: Figure 1 As shown in the figure, it specifically includes data preprocessing, feature selection and dimensionality reduction, training execution function evaluation models based on different machine learning methods, and comparative analysis of the results.
[0054] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0055] Embodiment 1:
[0056] like Figure 2 As shown, an embodiment of the present invention provides a non-contact cognitive performance prediction method, comprising:
[0057] S1. Obtain the visible light video of the user's face and divide it into several video clips of the same length;
[0058] S2, extracting non-contact physiological and head and facial behavior characteristic indicator data based on video clips;
[0059] S3, based on correlation analysis, dimensionality reduction is performed on the two types of extraction results respectively, and according to different cognitive performance evaluation indicators, the first number of non-contact physiological characteristic indicator data with the highest correlation ranking and the second number of head and facial behavior characteristic indicator data with the highest correlation ranking are correspondingly selected;
[0060] Among them, cognitive performance evaluation indicators include cognitive flexibility reaction time, cognitive flexibility accuracy, inhibitory control reaction time, inhibitory control accuracy, working memory reaction time, working memory accuracy;
[0061] S4. Retrieve the machine learning model corresponding to each cognitive performance evaluation indicator, and use the non-contact physiological and head and facial behavior characteristic indicator data selected under different cognitive performance evaluation indicators as input to predict the user's low-level or high-level cognitive performance respectively.
[0062] The embodiments of the present invention utilize the complementarity between multimodal features, and design comprehensive cognitive performance evaluation indicators from three dimensions of cognitive flexibility, inhibitory control and working memory, as well as two perspectives of accuracy and reaction time, thereby enhancing the objectivity and accuracy of the machine learning model classification and evaluation of cognitive performance.
[0063] Next, we will combine Figure 1 The following are the steps of the above solution:
[0064] In step S1, a visible light video of the user's face is obtained and divided into a number of video clips of the same length.
[0065] Exemplarily, in a 24-hour sleep deprivation experiment dataset collected in a laboratory, an embodiment of the present invention predicts cognitive performance based on visible light videos of the face during the sitting phase.
[0066] Specifically, this step uses the timestamp file saved during the experiment to obtain the visible light video of the face during the sitting phase, and divides the data into multiple groups of 1-minute video clips.
[0067] In step S2, non-contact physiological and head and face behavior characteristic index data are extracted based on the video clips.
[0068] The applicant recognizes that physiological characteristics and facial behavior characteristics have a high correlation with cognitive performance levels. Therefore, the embodiment of the present invention will focus on facial physiology and head and face behavior data, and continue to perform feature extraction and dimensionality reduction operations on the laboratory self-collected test data set that has completed preprocessing; including:
[0069] An rPPG signal is extracted based on the video clip, and based on the rPPG signal, a HeartPy bioelectric signal tool is used to extract non-contact physiological characteristic indicator data.
[0070] Based on the video clips, Openface is used to extract head and facial behavior feature index data.
[0071] The non-contact physiological characteristic index data and head and facial behavior characteristic index data both include time domain characteristics and frequency domain characteristics under corresponding characteristic indicators; wherein the time domain characteristics are any one or a combination of any one of mean, standard deviation, maximum value, minimum value, peak value, root mean square, peak factor and waveform factor, and the frequency domain characteristics are any one or a combination of any one of center of gravity frequency, average frequency, frequency standard deviation and root mean square frequency.
[0072] Exemplarily, the non-contact physiological characteristic indicators extracted in this step include at least 16 indicators shown in Table 1, which shows the name and meaning of each non-contact physiological characteristic indicator.
[0073] Table 1 Non-contact physiological characteristic indicators
[0074]
[0075] Exemplarily, given that the video clip data obtained after preprocessing the facial visible light video are all discrete time series, referring to the feature extraction method of time series data, the head and face behavior feature indicators extracted in this step include at least 17 facial movement units shown in Table 2, 8 eye gaze angles shown in Table 3, and 6 head movement postures.
[0076] Table 2 Facial movement units
[0077]
[0078] Table 3 Eye gaze angle and head movement posture
[0079] Indicator name meaning gaze_0_x The leftmost eye's gaze direction vector along the x-axis in the image gaze_0_y The leftmost eye's gaze direction vector along the y-axis in the image gaze_0_z The leftmost eye's gaze direction vector along the z-axis in the image gaze_1_x The rightmost eye's gaze direction vector along the x-axis in the image gaze_1_y The rightmost eye's gaze direction vector along the y-axis in the image gaze_1_z The rightmost eye's gaze direction vector along the z-axis in the image gaze_angle_x The average of the gaze directions of both eyes along the x-axis gaze_angle_y The average of the gaze directions of both eyes along the y-axis pose_Tx The position of the head relative to the camera on the x-axis (with positive z-axis away from the camera) pose_Ty The position of the head relative to the camera on the y-axis (with positive z-axis moving away from the camera) pose_Tz The position of the head relative to the camera on the z-axis (positive z is away from the camera) pose_Rx Head deflection along the x-axis pose_Ry Head deflection along the y-axis pose_Rz Head deflection along the z-axis
[0080] Table 2 shows the index name and action of each facial movement unit, and Table 3 shows the index name and meaning of each eye gaze angle and head movement posture. Among them, the x-axis, y-axis, and z-axis refer to the three coordinate axes in the three-dimensional coordinate system constructed by the spatial Cartesian coordinates.
[0081] Whether it is for facial motion units, eye gaze angles, or head motion postures, this step then extracts features from time domain, frequency domain, and other dimensions. Specifically, for all time series, common features are extracted, including time domain features such as mean, standard deviation, maximum, minimum, peak, root mean square, peak factor, and waveform factor, and frequency domain features such as centroid frequency, average frequency, frequency standard deviation, and root mean square frequency.
[0082] In step S3, the two types of extraction results are reduced in dimension based on correlation analysis, and according to different cognitive performance evaluation indicators, the first number of non-contact physiological characteristic indicator data with high correlation ranking and the second number of head and facial behavior characteristic indicator data with high correlation ranking are selected accordingly.
[0083] It is understandable that in machine learning methods, high-dimensional feature sets can better represent complex data and, to a certain extent, improve the accuracy of machine learning models. However, high-dimensional features may also lead to problems such as the curse of dimensionality, and the risk of noise and redundancy contained in the data is also higher, which increases the computational complexity of the model and makes it easier to overfit. Therefore, the embodiment of the present invention contemplates using the correlation coefficient in the feature selection filtering method to reduce the dimension and select features, and selects features by calculating the correlation between the features and the target variable.
[0084] It should also be noted that the embodiments of the present invention design comprehensive cognitive performance evaluation indicators from the three dimensions of cognitive flexibility, inhibitory control and working memory, and the two perspectives of accuracy and reaction time, specifically including the reaction time of cognitive flexibility, the accuracy of cognitive flexibility, the reaction time of inhibitory control, the accuracy of inhibitory control, the reaction time of working memory, and the accuracy of working memory.
[0085] Specifically, considering the large difference between the number of non-contact physiological indicators and the number of head and facial behavior characteristics, this step uses correlation analysis to select physiological characteristics and facial behavior characteristics respectively. The top 5 correlation indicators will be selected from the non-contact physiological characteristics, and the top 20 correlation coefficient indicators will be selected from the facial behavior characteristics. A total of 25 dimensional features will be obtained for subsequent model input. For each cognitive performance evaluation indicator, the selected features are shown in Tables 4 to 9.
[0086] Table 4 Characteristics of reaction time prediction for cognitive flexibility
[0087]
[0088] Table 5 Characteristics used for accuracy prediction of cognitive flexibility
[0089]
[0090] Table 6 Characteristics of reaction time prediction for inhibitory control
[0091]
[0092] Table 7 Features used for predicting the accuracy of inhibitory control
[0093]
[0094] Table 8. Features for predicting reaction time of working memory
[0095]
[0096] Table 9 Features used to predict working memory accuracy
[0097]
[0098] Tables 4 to 9 show the features selected for the six cognitive performance evaluation indicators. It can be seen from the table that the features of facial movement units are mainly concentrated on AU04 frowning, AU05 lifting the upper eyelid, AU07 eye contraction, AU14 tightening the mouth corners and AU23 tightening the lips. The features of eye gaze angle are mainly concentrated on the behavioral changes of the left eye, while the features of head movement posture, pitch angle and yaw angle have a more obvious effect on cognitive performance evaluation. The time domain, frequency domain and nonlinear indicators of heart rate variability are all included.
[0099] In step S4, a machine learning model corresponding to each cognitive performance evaluation indicator is retrieved, and the non-contact physiological and head and facial behavior characteristic indicator data selected under different cognitive performance evaluation indicators are used as input to predict the user's low-level or high-level cognitive performance respectively.
[0100] After determining the type of cognitive performance prediction model, it is necessary to build a suitable machine learning model for training. The selection of the model requires comprehensive consideration of the input characteristics, the nature of the problem, and the computing resources required by the model to cope with the complex and changing real-world environment.
[0101] The cognitive performance prediction targeted by the embodiments of the present invention has a wide range of application prospects, which places high demands on the adaptability and generalization ability of the model, while taking into account the deployment flexibility and interpretability of the model. For example, the following 8 machine learning models are selected to construct the cognitive performance evaluation model:
[0102] (1) Random Forest (RF): Random Forest is an ensemble learning method based on decision trees. It improves the performance of the model by constructing multiple decision trees (called forests) and integrating the prediction results of each tree. Each decision tree is trained independently, and random sampling and feature random selection techniques are used during the construction process to ensure the differences between trees and integrate their prediction results, thereby improving the accuracy and robustness of the model.
[0103] (2) Support Vector Machine (SVM): SVM performs classification by mapping data into a high-dimensional space and finding a hyperplane that maximizes the classification boundary. In a high-dimensional space, data points can be more easily divided into different categories, making problems that are linearly inseparable in the original space become linearly separable.
[0104] (3) Bayesian Classification (BS): Bayesian classifier is a type of probabilistic classification model based on Bayesian theorem. It is based on the conditional independence assumption between features, uses Bayesian theorem to calculate the posterior probability, and selects the category with the highest posterior probability as the prediction result.
[0105] (4) Logistic Regression (LR): Logistic regression uses a logistic function to model the probability of each category. It usually requires fewer resources and has good model interpretability.
[0106] (5) Decision Tree (DT): A decision tree divides data through a series of questions or tests to build a tree structure. At each decision node, the data is divided according to the value of a certain feature until the stopping condition is reached.
[0107] (6) K-Nearest Neighbor (KNN): K-Nearest Neighbor calculates the distance between the sample to be predicted and all samples in the training set, selects the k samples closest to it (neighboring samples), and then votes based on the categories of these k samples to determine the category of the sample to be predicted.
[0108] (7) Extreme Gradient Boosting (XGBoost): XGBoost is an algorithm based on gradient boosting trees. Gradient boosting trees are an ensemble learning method that trains multiple decision trees in series and combines them into a more powerful model. In gradient boosting trees, each tree is trained on the residual (or gradient) of the previous tree to gradually reduce the error and thus improve the performance of the model.
[0109] (8) Light Gradient Boosted Machine (LightGBM): LightGBM is similar to XGBoost in that it is based on a gradient boosting tree method. In addition, LightGBM uses a histogram algorithm to construct a histogram for each feature and uses these histograms to accelerate gradient calculation and split point search. This method is more efficient when processing large-scale data and uses less memory.
[0110] Therefore, it can be understood that the machine learning model called in this step can be any one or a combination of random forest, support vector machine, Bayesian classifier, logistic regression, decision tree, K-order nearest neighbor, extreme gradient boosting tree, and lightweight gradient boosting machine.
[0111] In particular, through necessary experimental verification, the embodiments of the present invention provide the following preferred combinations:
[0112] The machine learning model corresponding to the reaction time of the cognitive flexibility selects a lightweight gradient boosting machine; the machine learning model corresponding to the accuracy of the cognitive flexibility selects an extreme gradient boosting tree; the machine learning model corresponding to the reaction time of the inhibitory control selects a lightweight gradient boosting machine; the machine learning model corresponding to the accuracy of the inhibitory control selects a lightweight gradient boosting machine; the machine learning model corresponding to the reaction time of the working memory selects a random forest; and the machine learning model corresponding to the accuracy of the working memory selects a random forest.
[0113] After clarifying the selection of each machine learning model, the features selected by the different cognitive performance evaluation indicators shown in Tables 4 to 9 are used as the input of the corresponding models to predict the user's low-level or high-level cognitive performance, respectively.
[0114] It should be noted that the machine learning model adopts supervised learning training method; it should be noted that:
[0115] (1) During the training phase, the model is trained on samples that have both features and labels; the labels refer to: according to the corresponding median values, the six cognitive performance evaluation indicators are converted into two categories of "high level" and "low level" one by one.
[0116] (2) Five-fold cross validation can be used to evaluate the performance and generalization ability of the model. The original data set is divided into five data sets of equal size by random sampling. Each subset is called a fold. For each cross validation, one of the folds is selected as the validation set and the remaining four folds are used as the training set to train and test the model. Repeat the above steps and select each fold as the validation set in turn until each fold is used as the validation set. The model is trained and validated on a Windows 11 host equipped with an InterCore @ 3GHz CPU and an NVIDIA RTX 3090Ti graphics card.
[0117] (3) In order to evaluate the classification performance of the constructed executive function evaluation model, accuracy, precision, recall, and F1-score can be selected as evaluation indicators.
[0118] Embodiment 2:
[0119] An embodiment of the present invention provides a non-contact cognitive performance prediction system, comprising:
[0120] A preprocessing module is used to obtain the visible light video of the user's face and divide it into several video clips of the same length;
[0121] An extraction module, used for extracting non-contact physiological and head and face behavior characteristic index data based on the video clip;
[0122] A selection module is used to reduce the dimensions of the two types of extraction results based on correlation analysis, and select the first number of non-contact physiological characteristic indicator data with the highest correlation ranking and the second number of head and facial behavior characteristic indicator data with the highest correlation ranking according to different cognitive performance evaluation indicators;
[0123] The cognitive performance evaluation indexes include the reaction time of cognitive flexibility, the accuracy of cognitive flexibility, the reaction time of inhibitory control, the accuracy of inhibitory control, the reaction time of working memory, and the accuracy of working memory;
[0124] The prediction module is used to retrieve the machine learning model corresponding to each of the cognitive performance evaluation indicators, and use the non-contact physiological and head and facial behavior characteristic indicator data selected under different cognitive performance evaluation indicators as input to predict the low-level or high-level cognitive performance of the user respectively.
[0125] Embodiment 3:
[0126] An embodiment of the present invention provides a storage medium storing a computer program for contactless cognitive performance prediction, wherein the computer program enables a computer to execute the contactless cognitive performance prediction method as described in Embodiment 1.
[0127] Embodiment 4:
[0128] An embodiment of the present invention provides an electronic device, including:
[0129] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the non-contact cognitive performance prediction method as described in Example 1.
[0130] It is understandable that the non-contact cognitive performance prediction system, storage medium and electronic device provided in the embodiments of the present invention correspond to the non-contact cognitive performance prediction method provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the non-contact cognitive performance prediction method, which will not be repeated here.
[0131] In summary, compared with the prior art, the present invention has the following beneficial effects:
[0132] In the embodiment of the present invention, the facial visible light video is divided into several video clips of the same length to extract non-contact physiological and head and facial behavior characteristic index data; based on correlation analysis, the two types of extraction results are reduced in dimension respectively, and according to different cognitive performance evaluation indicators, the non-contact physiological and head and facial behavior characteristic index data with the highest correlation ranking are selected and used as the corresponding machine learning model input to predict the user's low-level or high-level cognitive performance respectively. By utilizing the complementarity between multimodal features, and from the three dimensions of cognitive flexibility, inhibitory control and working memory, as well as the two perspectives of accuracy and reaction time, a comprehensive cognitive performance evaluation index is designed, which enhances the objectivity and accuracy of the machine learning model classification evaluation of cognitive performance.
[0133] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0134] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A non-contact cognitive performance prediction method, characterized in that: include: Obtain the visible light video of the user's face and divide it into several video clips of equal length; Based on the video clips, extracting non-contact physiological and head and facial behavior characteristic indicator data; Based on the correlation analysis, the two types of extraction results are reduced in dimension respectively, and according to different cognitive performance evaluation indicators, the first number of non-contact physiological characteristic indicator data with the highest correlation ranking and the second number of head and face behavior characteristic indicator data with the highest correlation ranking are selected accordingly; The cognitive performance evaluation indexes include the reaction time of cognitive flexibility, the accuracy of cognitive flexibility, the reaction time of inhibitory control, the accuracy of inhibitory control, the reaction time of working memory, and the accuracy of working memory; A machine learning model corresponding to each of the cognitive performance evaluation indicators is retrieved, and the non-contact physiological and head and facial behavior characteristic indicator data selected under different cognitive performance evaluation indicators are used as input to predict the low-level or high-level cognitive performance of the user respectively.
2. The non-contact cognitive performance prediction method according to claim 1, characterized in that: The extracting of non-contact physiological and head and face behavior characteristic index data based on the video clips includes: Extracting rPPG signals based on the video clips, and extracting non-contact physiological characteristic indicator data using a HeartPy bioelectric signal tool based on the rPPG signals; Based on the video clips, using Openface to extract head and facial behavior feature index data; The non-contact physiological characteristic index data and head and facial behavior characteristic index data both include time domain characteristics and frequency domain characteristics under corresponding characteristic indicators; wherein the time domain characteristics are any one or a combination of any one of mean, standard deviation, maximum value, minimum value, peak value, root mean square, peak factor and waveform factor, and the frequency domain characteristics are any one or a combination of any one of center of gravity frequency, average frequency, frequency standard deviation and root mean square frequency.
3. The non-contact cognitive performance prediction method according to claim 2, characterized in that: The non-contact physiological characteristic indicators include at least the following 16 types: Heart rate, mean of cardiac interval, root mean square of cardiac cycle difference, standard deviation of cardiac interval, standard deviation of adjacent cardiac cycle difference, number of adjacent cardiac cycle difference greater than 50ms, ratio of adjacent cardiac cycle difference greater than 50ms, total power less than or equal to 0.4Hz, very low frequency power between 0.003 and 0.04Hz, low frequency power between 0.04 and 0.15Hz, high frequency power between 0.15 and 0.4Hz, ratio of low frequency and high frequency power, standard deviation perpendicular to the same straight line in nonlinear analysis, standard deviation along the same straight line in nonlinear analysis, ratio of standard deviation perpendicular to the same straight line in nonlinear analysis to standard deviation along the same straight line in nonlinear analysis, area of ellipse described by standard deviation perpendicular to the same straight line in nonlinear analysis and standard deviation along the same straight line in nonlinear analysis; And / or head and facial behavior characteristic indicators include at least the following: 17 facial movement units: inner eyebrow raising, outer eyebrow raising, frowning, lifting upper eyelids, cheek raising, eye contraction, nose wrinkling, upper lip lifting, pulling mouth corners, tightening mouth corners, mouth corners down, lower lip lifting, pouting, lateral stretching of mouth corners, tightening lips, lip pressing, lips parting, chin lowering, mouth opening, eyes closed; 8 eye gaze angles: gaze direction vector of the leftmost eye along the x-axis, gaze direction vector of the leftmost eye along the y-axis, gaze direction vector of the leftmost eye along the z-axis, gaze direction vector of the rightmost eye along the x-axis, gaze direction vector of the rightmost eye along the y-axis, gaze direction vector of the rightmost eye along the z-axis, average gaze direction of both eyes along the x-axis, average gaze direction of both eyes along the y-axis; 6 head movement postures: away from the camera with positive z axis, the position of the head relative to the camera on the x axis, away from the camera with positive z axis, the position of the head relative to the camera on the y axis, away from the camera with positive z axis, the position of the head relative to the camera on the z axis, the deflection of the head along the x axis, the deflection of the head along the y axis, and the deflection of the head along the z axis; Among them, the x-axis, y-axis, and z-axis refer to the three coordinate axes in the three-dimensional coordinate system constructed by spatial Cartesian coordinates.
4. The non-contact cognitive performance prediction method according to claim 1, characterized in that: The first number is five; and / or the second number is twenty.
5. The non-contact cognitive performance prediction method according to claim 1, characterized in that: The machine learning model is any one of random forest, support vector machine, Bayesian classifier, logistic regression, decision tree, K-order nearest neighbor, extreme gradient boosting tree, lightweight gradient boosting machine, or a combination of any of them.
6. The non-contact cognitive performance prediction method according to claim 1, characterized in that: A lightweight gradient boosting machine is selected as the machine learning model corresponding to the reaction time of the cognitive flexibility; and / or the machine learning model corresponding to the accuracy of the cognitive flexibility uses an extreme gradient boosting tree; and / or a lightweight gradient boosting machine is selected as the machine learning model corresponding to the reaction time of the inhibitory control; and / or a lightweight gradient boosting machine is selected as the machine learning model corresponding to the accuracy of the inhibitory control; And / or a random forest is selected as the machine learning model corresponding to the reaction time of the working memory; And / or a random forest is selected as the machine learning model corresponding to the accuracy of the working memory.
7. A non-contact cognitive performance prediction system, characterized in that: include: A preprocessing module is used to obtain the visible light video of the user's face and divide it into several video clips of the same length; An extraction module, used for extracting non-contact physiological and head and face behavior characteristic index data based on the video clip; A selection module is used to reduce the dimensions of the two types of extraction results based on correlation analysis, and select the first number of non-contact physiological characteristic indicator data with the highest correlation ranking and the second number of head and facial behavior characteristic indicator data with the highest correlation ranking according to different cognitive performance evaluation indicators; The cognitive performance evaluation indexes include the reaction time of cognitive flexibility, the accuracy of cognitive flexibility, the reaction time of inhibitory control, the accuracy of inhibitory control, the reaction time of working memory, and the accuracy of working memory; The prediction module is used to retrieve the machine learning model corresponding to each of the cognitive performance evaluation indicators, and use the non-contact physiological and head and facial behavior characteristic indicator data selected under different cognitive performance evaluation indicators as input to predict the low-level or high-level cognitive performance of the user respectively.
8. A storage medium, characterized in that: The computer program for contactless cognitive performance prediction is stored therein, wherein the computer program enables a computer to execute the contactless cognitive performance prediction method according to any one of claims 1 to 6.
9. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising instructions for executing the contactless cognitive performance prediction method according to any one of claims 1 to 6.
Citation Information
Cited By
Intelligent old people health monitoring system
CN119523442A
