A non-contact cognitive impairment identification system and method based on facial video

Abstract

The present invention relates to a non-contact cognitive impairment identification system and method based on facial video, comprising step 1: obtaining a facial video of a subject in an application scenario; step 2: determining an area of ​​interest according to the facial video obtained in step 1 and positioning facial feature points; step 3: using a high-definition facial image set as an input of an anxiety recognition network, extracting and constructing a multi-dimensional feature vector through a 3DCNN network corresponding to each dimension; step 4: an emotion regression model, and step 5: four emotion tasks, including boredom, concentration, confusion, and frustration. The advantages of the present invention are: non-contact, convenient, real-time, natural, and remotely operable; in addition, it has the advantage of low cost, usually does not require expensive contact detection equipment and consumables, and reduces the overall cost.

Description

Technical Field

[0001] The present invention relates to the technical field of cognitive impairment detection, and in particular to a non-contact cognitive impairment recognition system and method based on facial video. Background Art

[0002] Existing methods and systems for detecting cognitive dysfunction use medical instruments to conduct a series of physical examinations, wearable testing devices, or answer mental health scales. However, the scales are universal, and the results are affected by the subjective consciousness of the subject, making it impossible to accurately target and detect cognitive dysfunction. Wearable devices analyze psychological states by detecting physiological information. This method not only requires wearing electrodes or sensors during the detection process, but existing methods are mostly single-time tests and cannot determine whether the subject has cognitive dysfunction through a single test. It rarely takes into account that psychological disorders are persistent mental problems that need to be analyzed and identified in combination with historical information. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide a non-contact cognitive impairment identification system and method based on facial video, which is used to combine the underlying logic of conventional detection methods, bind physiological data and psychological data for analysis, simplify the detection process, and can repeat multiple tests anytime and anywhere to overcome the shortcomings of the above-mentioned prior art.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] A non-contact cognitive impairment identification method based on facial video, characterized by comprising the following steps:

[0006] Step 1: Obtain a facial video of the subject in an application scenario, wherein the application scenario includes: classroom, home, hospital, and public place; locate facial feature points based on the facial video to obtain a facial image set, wherein the facial feature points include: facial motion trajectory features, head posture features, eye gaze features, and eyebrow morphology features;

[0007] Step 2: Determine the region of interest based on the facial video obtained in step 1 and locate the facial feature points, and obtain a set of high-definition facial images;

[0008] Step 2.1: Detect the face and locate the region of interest from the face video, use the facial feature point positioning model to segment the face image, and perform calibration, alignment, and fill-in lighting operations on the face image to obtain a high-definition face image set;

[0009] Step 3: Use the high-definition facial image set as the input of the anxiety recognition network, and extract and construct a multi-dimensional feature vector through the 3DCNN network corresponding to each dimension;

[0010] Step 4: Emotion regression model, including: data set, preprocessing, model architecture, model training;

[0011] Step 5: Four emotion tasks, including: obtaining DAiSEE dataset, obtaining devices and scenarios, dataset size and type, data preprocessing and feature extraction, and feature classification; achieving data reasoning for four emotions: boredom, concentration, confusion, and frustration.

[0012] As the preferred embodiment of the present invention, step 3 also includes the following steps:

[0013] Step 3.1: Construct facial motion trajectory feature vector,

[0014] Obtain the position information of facial feature points in a high-definition facial image set, where the facial feature points include eyes, nose tip, mouth corners, eyebrows, and facial contour points, obtain a facial motion trajectory sequence based on the position information of the facial feature points, and use a 3DCNN network to extract facial motion trajectory feature vectors from the facial motion trajectory sequence, where the facial motion trajectory sequence is formed by tracking the motion trajectory of the facial feature points over a period of time and extracting the numerical features of the facial feature points to form a feature vector, and the facial motion trajectory feature vector includes position change, speed, and acceleration;

[0015] Step 3.2: Construct the head pose feature vector,

[0016] Obtain the position information of facial feature points in a high-definition facial image set, and perform regression prediction on the head position and its deflection direction based on the position information of the facial feature points. The regression prediction is used to quantify the rotational position angular coordinates of the head in three dimensions in space, and extract the velocity and acceleration features of the three dimensions to obtain a head pose sequence, and use a 3DCNN network to extract a head pose feature vector from the head pose sequence;

[0017] Step 3.3: Construct contactless physiological feature vector,

[0018] Based on the region of interest in the high-definition facial image set, a non-contact physiological signal sequence is obtained, and a non-contact physiological feature vector is extracted from the non-contact physiological signal sequence using a 3DCNN network; the non-contact physiological signal sequence includes respiration and heart rate;

[0019] Step 3.31: Heart rate calculation,

[0020] ‌‌According to the RPPG method, the peak of the signal corresponds to the moment when the blood vessel volume is the largest, that is, the moment when the heart pumps blood. Therefore, the interval between two peaks is a heart beat. For the collected RPPG signal, elgendi peak detection is first performed to obtain the peak position, and the heart rate corresponding to each peak is calculated in combination with the sampling frequency. In order to enhance the robustness of the result, the heart rate falling within the range of 50-130 is taken, and then the average is taken to obtain the final heart rate of the current 10-second video.

[0021] As the preferred embodiment of the present invention, step 4 also includes the following steps:

[0022] Step 4.1: Dataset,

[0023] The multimodal physical signal emotion DEAP dataset is used to obtain the signal emotions in four dimensions: arousal, emotion, liking, and dominance.

[0024] Step 4.2: Preprocessing,

[0025] Preprocessing is mainly divided into two parts: face detection and key frame selection;

[0026] First, the MTCNN face detector is used to identify the face area in the image of the high-definition face image set. The video in the multimodal physical signal emotion DEAP dataset is 60 seconds long and has a frame rate of 50Hz. The video is converted into 3000 consecutive frames of images. Then, 16 key frame images are selected from the image sequence using random sampling.

[0027] Step 4.3: Model Architecture,

[0028] Step 4.3.1: Extract facial expression features EmotionModel,

[0029] (1) Build the VGG19 feature extractor,

[0030] Using the feature extractor of the VGG19 model, we first freeze the VGG19 parameters and set the requires_grad attribute of all parameters of the VGG19 model to False to prevent all parameters from being updated during training. Then we add additional convolutional layers, where two convolutional layers are added, each followed by a ReLU activation function and a maximum pooling layer. The number of input and output channels of each convolutional layer is 512, the convolution kernel size is 2x2, the stride is 1, and the padding is 1. Finally, we perform feature embedding. After flattening the convolutional features into a one-dimensional vector, we map the features to the embedding space through a linear layer, with an input dimension of 4608 and an output dimension of 1024.

[0031] (2) rPPG signal feature extraction RPPG1DCNN performs three-layer convolutional network, ReLU activation function, maximum pooling layer and feature map calculation, where three-layer convolutional network;

[0032] Three-layer convolutional network,

[0033] The first convolutional layer: the number of input channels is 1, the number of output channels is 16, the convolution kernel size is 9, the stride is 1, and the padding is 1;

[0034] The second convolutional layer: the number of input channels is 16, the number of output channels is 32, the convolution kernel size is 9, the stride is 1, and the padding is 1;

[0035] The third convolutional layer: the number of input channels is 32, the number of output channels is 64, the convolution kernel size is 9, the stride is 1, and the padding is 1;

[0036] ReLU activation function and maximum pooling layer: used for nonlinear transformation and feature dimensionality reduction;

[0037] The input rPPG signal data is processed through three convolutional layers and a maximum pooling layer, and the convolutional layer output is flattened into a one-dimensional vector;

[0038] Feature map calculation,

[0039] Calculate the size change after three layers of convolution and pooling, and calculate the output feature map size based on the input signal length;

[0040] Step 4.3.2: Feature stitching,

[0041] Instantiate the EmotionModel and RPPG1DCNN modules, calculate the combined feature dimensions, and define the linear classifier:

[0042] The first linear layer: the input dimension is combined_feature_dim, and the output dimension is 512, where combined_feature_dim is the dimension of the combined feature;

[0043] ReLU activation function and Dropout layer: used for nonlinear transformation and to prevent overfitting;

[0044] The second linear layer: the output dimension is 2, corresponding to the two dimensions of arousal and valence of emotion recognition;

[0045] Step 4.4: Model training,

[0046] Step 4.41: Evaluation indicators,

[0047] (1) Determination of emotion categories,

[0048] Negative emotions: Valence value ≤ 2

[0049] Neutral sentiment: 2< valence ≤ 6

[0050] Positive emotions: valence value > 6

[0051] (2) Determination of emotional intensity,

[0052] Mild: wake-up value ≤ 2

[0053] Medium: 2< wake-up value ≤ 6

[0054] High: wake-up value > 6

[0055] (3) Linear mapping function,

[0056] Map it into a score based on the arousal value for further sentiment analysis;

[0057] Among them, valence indicates the positivity or pleasure of emotion. High valence means positive emotions, such as happiness, excitement, and satisfaction, while low valence means negative emotions, such as sadness, anger, and disgust.

[0058] Among them, arousal refers to the activation or intensity level of emotions; high arousal indicates strong emotions, specifically excitement, anger, and surprise; low arousal: calm emotions, specifically relaxation, boredom, and calmness.

[0059] As the preferred embodiment of the present invention, step 5 also includes the following steps:

[0060] Step 5.1: Obtain the DAiSEE dataset, which contains video recordings of subjects in an e-learning environment and annotated with labels for engagement, frustration, confusion, and boredom;

[0061] Step 5.2: Get the equipment, use a full HD webcam installed on a computer where the students are focused on watching the video; Step 5.3: Scenario, provide two different videos, one educational and one entertaining, to capture the concentration and relaxation scenes;

[0062] Step 5.4: Dataset size and variety: Obtain video clips of multiple subjects and label the four emotional states in the video clips, including engagement, frustration, confusion, and boredom; the four levels of labels are very low, low, high, and very high;

[0063] Step 5.5: Data preprocessing to achieve binary classification of four emotions;

[0064] The four labels of the four emotions are merged, very low and low are merged into low labels; high and very high are merged into high labels, and samples that cannot be recognized by face recognition are eliminated;

[0065] Step 5.6: Feature extraction,

[0066] The attention state is predicted through eye trackers, and the predicted attention is used to help students improve their learning performance;

[0067] Step 5.61: Detect students’ attention status through eye tracking and provide dialogue to re-engage students and improve learning outcomes; wherein, using attention-aware learning methods, using eye trackers to provide interventions based on students’ predictions of mind wandering, including interventions such as asking questions, reviewing content, and calling names;

[0068] According to the Daisee dataset, the camera is used to capture the students' eyes for feature extraction. The eye aspect ratio is highly correlated with the attention state. The facial video-based method is used to detect distraction behavior in online education.

[0069] Eye aspect ratio: For each facial video frame, use Mediapipe to extract facial landmarks; from the facial landmark positions, extract the eye aspect ratio, which represents the relationship between the width of the eye and the height. This ratio remains scale-invariant even if the scale of the eye changes;

[0070] The mean and standard deviation were extracted to understand the EAR including outliers; the influence of period was observed by extracting 1q, 2q, 3q and MAD as outlier robust statistics, where EAR is the eye aspect ratio and q is the quartile;

[0071] Calculate the mean and standard deviation:

[0072] (1) Mean: the average value of the EAR data points;

[0073] (2) Standard deviation: The standard deviation of the EAR data points, which is used to measure the dispersion of the data;

[0074] Compute quartiles and median absolute deviations:

[0075] (1) 1st quartile: 25th percentile of the data;

[0076] (2) The second quartile, i.e. the median: the 50th percentile of the data;

[0077] (3) 3rd quartile: 75th percentile of the data;

[0078] (4) Median absolute deviation: The median of the absolute deviations between the median and each point in the data set, used to measure the central tendency of the data and the presence of outliers;

[0079] Step 5.62: feature classification, using the initialized base classifier and the multi-output classifier;

[0080] The multi-output classifier allows the sentiment regression model to handle problems that require predicting multiple outputs simultaneously;

[0081] Parallel training: The n_jobs=-1 parameter allows the sentiment regression model to use all available CPU cores to train multiple classifiers in parallel, which significantly speeds up the training process for large datasets, where n_jobs is the number of tasks when training the base classifier in parallel;

[0082] Ensemble learning: Use the RandomForestClassifier ensemble learning method to improve generalization and robustness through the prediction of multiple decision trees of the RandomForestClassifier ensemble learning method; train the model, calculate the accuracy, precision, recall and confusion matrix of the four emotions for binary classification, and save the model in ONNX format for data reasoning.

[0083] Another object of the present invention is to provide a cognitive impairment processing system, comprising a video acquisition module, an image processing unit, a data analysis module and a result output module;

[0084] The video acquisition module is a high-definition camera, which is used to obtain the patient's facial video;

[0085] The image processing unit is used to perform preprocessing and feature extraction operations on the collected video;

[0086] The data analysis module uses a sentiment regression model to analyze and identify the extracted features;

[0087] The result output module is used to present the recognition result to the user in an intuitive manner.

[0088] The advantages and positive effects of the present invention are:

[0089] 1. The present invention is non-contact: it avoids direct contact with the patient's body, reduces the patient's discomfort and resistance, and is especially advantageous for some sensitive or uncooperative patients. It is convenient: it can be deployed and applied in various scenarios relatively easily, without complicated preparations or special equipment to connect with the patient. It is real-time: it can monitor and analyze facial videos in real time, and promptly detect potential signs of cognitive impairment, which is conducive to early intervention and treatment. It is natural: patients are evaluated in a natural state, and the results may better reflect the real situation without the interference that may be caused by contact detection.

[0090] Remote operation: It facilitates telemedicine and remote monitoring, allowing expert resources to cover patients more widely. In addition, it has the advantage of low cost and usually does not require expensive contact detection equipment and consumables, reducing overall costs.

[0091] 2. The present invention can improve screening efficiency: quickly and efficiently conduct preliminary screening of a large number of people, and improve the efficiency of cognitive impairment detection. It can improve the early diagnosis rate: identify the risk of cognitive impairment earlier, and win a valuable treatment time window for patients. It can continuously monitor feasibility: realize long-term continuous monitoring of patients, track changes in their condition and treatment effects.

[0092] It can improve patient compliance: Due to its non-contact nature, patients are more willing to cooperate with testing, which is conducive to long-term health management.

[0093] It can promote the optimization of medical resources: through more efficient identification and classification, medical resources can be rationally allocated and focused on patients who need special attention. It can promote the popularization of technology: relatively simple implementation methods can help promote its application in a wider range of regions and institutions, and improve the overall level of cognitive impairment identification. DETAILED DESCRIPTION

[0094] Example 1

[0095] This embodiment provides a non-contact cognitive impairment identification method based on facial video, comprising the following steps:

[0096] Step 1: Obtain a facial video of the subject in an application scenario, wherein the application scenario includes: classroom, home, hospital, and public place; locate facial feature points based on the facial video to obtain a facial image set, wherein the facial feature points include: facial motion trajectory features, head posture features, eye gaze features, and eyebrow morphology features.

[0097] Step 1.1: Use the high-definition facial image set as the input of the anxiety recognition network, extract and construct a multi-dimensional feature vector through the 3DCNN (3D Convolutional Neural Networks) network corresponding to each dimension, fuse the multi-dimensional feature vectors, and perform spatiotemporal attention mechanism processing on the fused feature vectors (from the paper: TemporalAttention Unit: Towards Efficient Spatiotemporal Predictive Learning). Then perform dimensionality reduction processing on the feature vectors processed by the spatiotemporal attention mechanism, specifically: map multiple features in the original data set to a lower feature space, thereby reducing the number and dimension of features. This dimensionality reduction processing can reduce noise and redundant information in the data set, thereby improving the performance of the machine learning algorithm, and finally output the recognition result of the anxiety recognition network.

[0098] Step 2: Determine the region of interest based on the facial video obtained in step 1 and locate the facial feature points, and obtain a set of high-definition facial images, where ROL (Region of Interest) represents the region of interest. It refers to the area to be processed that is outlined in the processed image in the form of a square, circle, ellipse, or irregular polygon; it is used in the fields of face recognition and image segmentation;

[0099] Step 2.1: Detect faces and locate regions of interest from facial videos, segment facial images using a facial feature point localization model (from the paper: Rapid Object Detection using a Boosted Cascade of Simple Features), and perform calibration, alignment, and fill-in lighting operations on facial images to obtain a set of high-definition facial images;

[0100] Step 3: Use the high-definition facial image set as the input of the anxiety recognition network, and extract and construct a multi-dimensional feature vector through the 3DCNN network corresponding to each dimension, where each dimension includes the motion trajectory feature of the facial image, the head position and posture feature of the facial image, and the non-contact physiological feature of the facial image;

[0101] Step 3.1: Construct facial motion trajectory feature vector,

[0102] Obtain the position information of facial feature points in a high-definition facial image set, where the facial feature points include eyes, nose tip, mouth corners, eyebrows, and face contour points. Obtain a facial motion trajectory sequence based on the position information of the facial feature points, and use a 3DCNN network to extract facial motion trajectory feature vectors from the facial motion trajectory sequence. The facial motion trajectory sequence is formed by tracing the motion trajectory of facial feature points (such as eye corners, mouth corners, etc.) over a period of time and extracting the numerical features of the facial feature points to form a feature vector. The facial motion trajectory feature vector includes position change, speed, and acceleration. The facial motion trajectory feature vector is used to reflect the dynamic change characteristics of facial expressions, which helps to distinguish, identify, and analyze different facial motion states.

[0103] Step 3.2: Construct the head pose feature vector,

[0104] Obtain the position information of facial feature points in a high-definition facial image set, and perform regression prediction on the head position and its deflection direction based on the position information of the facial feature points. The regression prediction is used to quantify the rotational position angular coordinates of the head in three dimensions in space, and extract the velocity and acceleration features of the three dimensions to obtain a head pose sequence, and use a 3DCNN network to extract a head pose feature vector from the head pose sequence;

[0105] Step 3.3: Construct contactless physiological feature vector,

[0106] Based on the region of interest in the high-definition facial image set, a non-contact physiological signal sequence is obtained, and a non-contact physiological feature vector is extracted from the non-contact physiological signal sequence using a 3DCNN network; the non-contact physiological signal sequence includes respiration and heart rate, wherein the non-contact physiological signal sequence is obtained by performing specific analysis and processing on the region of interest; for example, the region of interest can be monitored and data collected by using a suitable sensor or imaging technology, and then relevant feature information of physiological signals such as heart rate, respiration rate, blood oxygen, etc. can be extracted from the collected data, and these feature information are arranged in chronological order to form a non-contact physiological signal sequence. Common operations include analyzing the image or data of the region of interest, extracting feature points, and calculating the change trend, so as to infer and parse the relevant information of the physiological signal;

[0107] Step 3.31: Heart rate calculation,

[0108] ‌‌According to the RPPG (Remote Photoplethysmography) method, the peak of the signal corresponds to the moment when the blood vessel volume is the largest, that is, the moment when the heart pumps blood. Therefore, the interval between two peaks is a heart beat. The collected RPPG signal is firstly subjected to elgendi peak detection (reference: Systolic peak detection in accelerationphotoplethysmograms measured from emergency responders in tropicalconditions.) to obtain the peak position, and the heart rate corresponding to each peak is calculated in combination with the sampling frequency. In order to enhance the robustness of the result, the heart rate falling within the range of 50-130 is taken, and then the final heart rate of the current 10-second video is obtained by averaging. RPPG (Remote Photoplethysmography) is a non-contact measurement technology used to restore physiological signals under ambient light by analyzing the skin color changes caused by periodic heartbeats.

[0109] Step 4: Sentiment regression model,

[0110] Step 4.1: Dataset,

[0111] Using the currently public multimodal physical signal emotion DEAP dataset (available from https: / / www.eecs.qmul.ac.uk / mmv / datasets / deap / ), 32 subjects (16 females and 16 males) were asked to watch 40 music videos with a duration of 60 seconds each. The data was recorded using the BioSemi ActiveTwo system, providing 10 channels of data, namely 8 EEG channels and a single channel of electrogalvanic skin response and photoplethysmography PPG. The video frame rate was 50Hz and the duration was 60 seconds. Before watching the music video, the subjects had a 3s pre-tracking baseline time. In the experiment, the labels and data needed to be preprocessed first. The dataset also has subjective ratings, requiring participants to use a continuous digital scale from 1 to 9 after watching the video to characterize their emotional state brought about by watching the movie, including four dimensions: arousal, emotion, liking, and dominance. The four dimensions represent different aspects of emotional experience. The labels refer to the labeling and naturalization processing of the dataset by the emotion regression model.

[0112] Step 4.2: Preprocessing,

[0113] Preprocessing is mainly divided into two parts: face detection and key frame selection;

[0114] First, the MTCNN face detector is used to identify the face area in the image of the high-definition face image set. The video in the multimodal physical signal emotion DEAP dataset is 60 seconds long and has a frame rate of 50Hz. The video is converted into 3000 consecutive frames of images. Then, 16 key frame images are selected from the image sequence using random sampling.

[0115] Step 4.3: Model Architecture,

[0116] Step 4.3.1: Extract facial expression features EmotionModel,

[0117] (1) Build the VGG19 feature extractor,

[0118] The feature extractor of the VGG19 model (i.e. the convolutional layer part in front) is used. VGG19 is a deep convolutional neural network composed of multiple convolutional layers, pooling layers, and fully connected layers. VGG19 represents a total of 19 weight layers in the network, including 16 convolutional layers and 5 maximum pooling layers; each convolutional layer uses a filter with a small-sized convolution kernel (3×3) for convolution operations, and uses the Relu activation function to perform nonlinear transformations on the output. By stacking multiple convolutional layers, the network can learn more abstract and higher-level feature information. After each convolutional layer passes a certain number of convolutional layers, VGG19 reduces the feature dimension through steps such as pooling. First, freeze the VGG19 parameters and set the requires_grad attribute of all parameters of the VGG19 model to False to prevent all parameters from being updated during training. Then add additional convolutional layers, where two convolutional layers are added, each followed by a ReLU activation function and a maximum pooling layer. The number of input and output channels of each convolutional layer is 512, the convolution kernel size is 2x2, the step size is 1, and the padding is 1. Finally, perform feature embedding, flatten the convolutional features into a one-dimensional vector, and then map the features to the embedding space through a linear layer, with an input dimension of 4608 and an output dimension of 1024. All parameters refer to the preset parameters in the feature extractor, that is, similar to the initialization operation in the current scene or step.

[0119] (2) rPPG signal feature extraction RPPG1DCNN performs three-layer convolutional network, ReLU activation function, maximum pooling layer and feature map calculation, where three-layer convolutional network;

[0120] Three-layer convolutional network,

[0121] The first convolutional layer: the number of input channels is 1, the number of output channels is 16, the convolution kernel size is 9, the stride is 1, and the padding is 1;

[0122] The second convolutional layer: the number of input channels is 16, the number of output channels is 32, the convolution kernel size is 9, the stride is 1, and the padding is 1;

[0123] The third convolutional layer: the number of input channels is 32, the number of output channels is 64, the convolution kernel size is 9, the stride is 1, and the padding is 1;

[0124] ReLU activation function and maximum pooling layer: used for nonlinear transformation and feature dimensionality reduction;

[0125] The input rPPG signal data is processed through three convolutional layers and a maximum pooling layer, and the convolutional layer output is flattened into a one-dimensional vector;

[0126] Feature map calculation,

[0127] Calculate the size change after three layers of convolution and pooling, and calculate the output feature map size based on the input signal length (the image signal of the three convolution layers input);

[0128] Step 4.3.2: Feature stitching,

[0129] Instantiate the EmotionModel and RPPG1DCNN modules, calculate the combined feature dimensions, and define the linear classifier:

[0130] The first linear layer: the input dimension is combined_feature_dim, and the output dimension is 512, where combined_feature_dim is the dimension of the combined feature;

[0131] ReLU activation function and Dropout layer: used for nonlinear transformation and to prevent overfitting;

[0132] The second linear layer: the output dimension is 2, corresponding to the two dimensions of arousal and valence of emotion recognition;

[0133] Step 4.4: Model training,

[0134] Step 4.41: Evaluation indicators,

[0135] In the emotion recognition task, mapping continuous valence and arousal values ​​to discrete emotion categories is an important step. This process involves classifying the model's predictions and providing the intensity of the emotion and a valence-based emotion score.

[0136] (1) Determination of emotion categories,

[0137] Negative emotion: value ≤ 2

[0138] Neutral: 2< valence ≤ 6

[0139] Positive emotions: valence value > 6

[0140] (2) Determination of emotional intensity,

[0141] Slight: wake-up value ≤ 2

[0142] Medium: 2< wake-up value ≤ 6

[0143] High: wake-up value > 6

[0144] (3) Linear mapping function,

[0145] Map it into a score based on the arousal value for further sentiment analysis;

[0146] Among them, valence indicates the positivity or pleasure of emotion. High valence means positive emotion, such as happiness, excitement, and satisfaction, while low valence means negative emotion, such as sadness, anger, and disgust.

[0147] Arousal refers to the activation or intensity level of emotion; high arousal indicates strong emotions, such as excitement, anger, and surprise. Low arousal: calm emotions, such as relaxation, boredom, and calmness. Valence and arousal are often used together to form a two-dimensional emotional space. This two-dimensional space can effectively represent various emotional states.

[0148] Step 5: Four emotional tasks, including boredom, concentration, confusion, and frustration;

[0149] Step 5.1: Obtain the DAiSEE dataset, which contains video recordings of subjects in e-learning environments, annotated with labels for engagement, frustration, confusion, and boredom; this dataset captures the “in the wild” settings commonly seen in the real world and will be publicly available, along with individual annotations from the crowd, to facilitate open research. The database can be downloaded from: https: / / people.iith.ac.in / vineethnb / resources / daisee / index.html.

[0150] Step 5.2: Get the equipment, use a full HD webcam installed on a computer focused on the student watching the video; (1920x1080, 30 fps, focal length 3.6mm, 78◦ field of view);

[0151] Step 5.3: Scenario, provide two different videos, one educational and one entertaining, to capture the concentration and relaxation scenarios;

[0152] Step 5.4: Dataset size and variety: 112 subjects, aged 18-30, all currently enrolled students, Asian ethnicity, 32 female subjects and 80 male subjects; 9068 video clips, each 10 seconds; video clips distributed in different locations, including dormitories, crowded laboratory spaces, libraries, and three different lighting settings, bright, dark, and neutral; male to female ratio is 2.13:1. Each video clip has a unique identification number to help distinguish between video clips; four emotional state labels, engagement, frustration, confusion, and boredom; among which, the four levels of labels are very low, low, high, and very high;

[0153] Step 5.5: Data preprocessing to achieve binary classification of four emotions;

[0154] The four labels of the four emotions are merged, very low and low are merged into the low label; high and very high are merged into the high label. Samples that cannot be recognized by face are eliminated, leaving 8566 samples, 6853 samples in the training set, and 1713 samples in the test set;

[0155] Step 5.6: Feature extraction,

[0156] The attention state is predicted through eye trackers, and the predicted attention is used to help students improve their learning performance;

[0157] Step 5.61: Detect students’ attention status through eye tracking and provide dialogue to re-engage students and improve learning outcomes; among them, use the Attention Aware Learning Method (AALT) to provide intervention measures based on students’ prediction of distraction using an eye tracker, including intervention measures such as asking questions, reviewing content, and calling names;

[0158] Predicting attention states is crucial for education. Most existing mind-wandering prediction schemes utilize specific biomarkers and specialized and expensive hardware: eye trackers (e.g., gaze, fixation, saccades, and blinks), EEG, EDA, or fMRI. However, when replaced with affordable devices such as eye tracking, the prediction of attention states is not feasible.

[0159] According to the Daisee dataset, the camera is used to capture the students' eyes for feature extraction. The eye aspect ratio is highly correlated with the attention state. The facial video-based method is used to detect distraction behavior in online education.

[0160] Eye aspect ratio: For each face video frame, facial landmarks are extracted using Mediapipe; from the facial landmark locations, the eye aspect ratio (EAR) is extracted, which represents the relationship between the width of the eye and its height. This ratio remains scale-invariant even when the scale of the eye changes; in addition, the face detection and landmark detection systems are translation-invariant. Therefore, unless the eyes are detectable, the EAR is robust to any possible differences.

[0161] The mean and standard deviation (SD) were extracted to understand the EAR including outliers; the influence of the period was observed by extracting 1q, 2q, 3q and MAD as outlier robust statistics, where outliers included closed eyes, where EAR was the eye aspect ratio and q was the quartile;

[0162] Calculate the mean and standard deviation (SD):

[0163] (1) Mean: the average value of the EAR data points;

[0164] (2) Standard deviation: The standard deviation of the EAR data points, which is used to measure the dispersion of the data;

[0165] Calculate quartiles and median absolute deviation (MAD):

[0166] (1) 1st quartile (1q or Q1): 25th percentile of the data;

[0167] (2) The second quartile (2q or Q2), also known as the median: the 50th percentile of the data;

[0168] (3) 3rd quartile (3q or Q3): 75th percentile of the data;

[0169] (4) Median Absolute Deviation (MAD): The median of the absolute deviations between the median and each point in the data set, used to measure the central tendency of the data and the presence of outliers;

[0170] Step 5.62: Feature classification, using the initialized base classifier (RandomForestClassifier) ​​and the multi-output classifier (MultiOutputClassifier);

[0171] The multi-output classifier allows the sentiment regression model to handle problems that require predicting multiple outputs simultaneously;

[0172] Parallel training: The n_jobs=-1 parameter allows the sentiment regression model to use all available CPU cores to train multiple classifiers in parallel, which significantly speeds up the training process for large datasets, where n_jobs is the number of tasks when training the base classifier in parallel;

[0173] Ensemble learning: Use the RandomForestClassifier ensemble learning method to improve generalization and robustness by combining the predictions of multiple decision trees with the RandomForestClassifier ensemble learning method; this helps achieve higher accuracy and better handle overfitting.

[0174] Train the model, calculate the accuracy, precision, recall and confusion matrix of the four emotions for binary classification, and save the model in ONNX format for data reasoning.

[0175] Example 2

[0176] This embodiment provides a cognitive impairment processing system, including a video acquisition module, an image processing unit, a data analysis module and a result output module;

[0177] The video acquisition module is a high-definition camera, which is used to obtain the patient's facial video; its specific content applies step 1 in embodiment 1.

[0178] The image processing unit is used to perform preprocessing and feature extraction operations on the collected video; its specific content applies steps 2 and 3 in embodiment 1.

[0179] The data analysis module: uses the emotion regression model to analyze and identify the extracted features; its specific content applies steps 4 and 5 in embodiment 1.

[0180] The result output module is used to present the recognition result to the user in an intuitive manner.

Claims

1. A non-contact cognitive impairment recognition method based on facial video, characterized in that: The following steps are involved: Step 1: Obtain a facial video of the subject in an application scenario, wherein the application scenario includes: classroom, home, hospital, and public place; locate facial feature points based on the facial video to obtain a facial image set, wherein the facial feature points include: facial motion trajectory features, head posture features, eye gaze features, and eyebrow morphology features; Step 2: Determine the region of interest based on the facial video obtained in step 1 and locate the facial feature points, and obtain a set of high-definition facial images; Step 2.1: Detect the face and locate the region of interest from the face video, use the facial feature point positioning model to segment the face image, and perform calibration, alignment, and fill-in lighting operations on the face image to obtain a high-definition face image set; Step 3: Use the high-definition facial image set as the input of the anxiety recognition network, and extract and construct a multi-dimensional feature vector through the 3DCNN network corresponding to each dimension; Step 4: Emotion regression model, including: data set, preprocessing, model architecture, model training; Step 4.1: Dataset, The multimodal physical signal emotion DEAP dataset is used to obtain the signal emotions in four dimensions: arousal, emotion, liking, and dominance. Step 4.2: Preprocessing, Preprocessing is mainly divided into two parts: face detection and key frame selection; First, the MTCNN face detector is used to identify the face area in the image of the high-definition face image set. The video in the multimodal physical signal emotion DEAP dataset is 60 seconds long and has a frame rate of 50Hz. The video is converted into 3000 consecutive frames of images. Then, 16 key frame images are selected from the image sequence using random sampling. Step 4.3: Model Architecture, Step 4.3.1: Extract facial expression features EmotionModel, (1) Build the VGG19 feature extractor, Using the feature extractor of the VGG19 model, we first freeze the VGG19 parameters and set the requires_grad attribute of all parameters of the VGG19 model to False to prevent all parameters from being updated during training. Then we add additional convolutional layers, where two convolutional layers are added, each followed by a ReLU activation function and a maximum pooling layer. The number of input and output channels of each convolutional layer is 512, the convolution kernel size is 2x2, the stride is 1, and the padding is 1. Finally, we perform feature embedding. After flattening the convolutional features into a one-dimensional vector, we map the features to the embedding space through a linear layer, with an input dimension of 4608 and an output dimension of 1024. (2) rPPG signal feature extraction RPPG1DCNN performs three-layer convolutional network, ReLU activation function, maximum pooling layer and feature map calculation, where three-layer convolutional network; Three-layer convolutional network, The first convolutional layer: the number of input channels is 1, the number of output channels is 16, the convolution kernel size is 9, the stride is 1, and the padding is 1; The second convolutional layer: the number of input channels is 16, the number of output channels is 32, the convolution kernel size is 9, the stride is 1, and the padding is 1; The third convolutional layer: the number of input channels is 32, the number of output channels is 64, the convolution kernel size is 9, the stride is 1, and the padding is 1; ReLU activation function and maximum pooling layer: used for nonlinear transformation and feature dimensionality reduction; The input rPPG signal data is processed through three convolutional layers and a maximum pooling layer, and the convolutional layer output is flattened into a one-dimensional vector; Feature map calculation, Calculate the size change after three layers of convolution and pooling, and calculate the output feature map size based on the input signal length; Step 4.3.2: Feature stitching, Instantiate the EmotionModel and RPPG1DCNN modules, calculate the combined feature dimensions, and define the linear classifier: The first linear layer: the input dimension is combined_feature_dim, and the output dimension is 512, where combined_feature_dim is the dimension of the combined feature; ReLU activation function and Dropout layer: used for nonlinear transformation and to prevent overfitting; The second linear layer: the output dimension is 2, corresponding to the two dimensions of arousal and valence of emotion recognition; Step 4.4: Model training, Step 4.41: Evaluation indicators, (1) Determination of emotion categories, Negative emotions: Valence value ≤ 2 Neutral sentiment: 2 < valence ≤ 6 Positive emotions: valence value > 6 (2) Determination of emotional intensity, Mild: wake-up value ≤ 2 Medium: 2 < wake-up value ≤ 6 High: wake-up value > 6 (3) Linear mapping function, Map it into a score based on the arousal value for further sentiment analysis; Among them, valence indicates the positivity or pleasure of emotion. High valence means positive emotions, such as happiness, excitement, and satisfaction, while low valence means negative emotions, such as sadness, anger, and disgust. Arousal refers to the activation or intensity level of emotion; high arousal indicates strong emotions, such as excitement, anger, and surprise; low arousal indicates calm emotions, such as relaxation, boredom, and calmness; Step 5: Four emotion tasks, including: obtaining DAiSEE dataset, obtaining devices and scenarios, dataset size and type, data preprocessing and feature extraction, and feature classification; achieving data reasoning for four emotions: boredom, concentration, confusion, and frustration.

2. The non-contact cognitive impairment identification method based on facial video according to claim 1, characterized in that: Step 3 also includes the following steps: Step 3.1: Construct facial motion trajectory feature vector, Obtain the position information of facial feature points in a high-definition facial image set, where the facial feature points include eyes, nose tip, mouth corners, eyebrows, and facial contour points, obtain a facial motion trajectory sequence based on the position information of the facial feature points, and use a 3DCNN network to extract facial motion trajectory feature vectors from the facial motion trajectory sequence, where the facial motion trajectory sequence is formed by tracking the motion trajectory of the facial feature points over a period of time and extracting the numerical features of the facial feature points to form a feature vector, and the facial motion trajectory feature vector includes position change, speed, and acceleration; Step 3.2: Construct the head pose feature vector, Obtain the position information of facial feature points in a high-definition facial image set, and perform regression prediction on the head position and its deflection direction based on the position information of the facial feature points. The regression prediction is used to quantify the rotational position angular coordinates of the head in three dimensions in space, and extract the velocity and acceleration features of the three dimensions to obtain a head pose sequence, and use a 3DCNN network to extract a head pose feature vector from the head pose sequence; Step 3.3: Construct contactless physiological feature vector, Based on the region of interest in the high-definition facial image set, a non-contact physiological signal sequence is obtained, and a non-contact physiological feature vector is extracted from the non-contact physiological signal sequence using a 3DCNN network; the non-contact physiological signal sequence includes respiration and heart rate; Step 3.31: Heart rate calculation, ‌‌According to the RPPG method, the peak of the signal corresponds to the moment when the blood vessel volume is the largest, that is, the moment when the heart pumps blood. Therefore, the interval between two peaks is a heart beat. For the collected RPPG signal, elgendi peak detection is first performed to obtain the peak position, and the heart rate corresponding to each peak is calculated in combination with the sampling frequency. In order to enhance the robustness of the result, the heart rate falling within the range of 50-130 is taken, and then the average is taken to obtain the final heart rate of the current 10-second video.

3. The non-contact cognitive impairment identification method based on facial video according to claim 1, characterized in that: Step 5 also includes the following steps: Step 5.1: Obtain the DAiSEE dataset, which contains video recordings of subjects in an e-learning environment and annotated with labels for engagement, frustration, confusion, and boredom; Step 5.2: Get the equipment, use a full HD webcam installed on a computer where the students are focused on watching the video; Step 5.3: Scenario, provide two different videos, one educational and one entertaining, to capture the concentration and relaxation scenes; Step 5.4: Dataset size and variety: Obtain video clips of multiple subjects and label the four emotional states in the video clips, including engagement, frustration, confusion, and boredom; the four levels of labels are very low, low, high, and very high; Step 5.5: Data preprocessing to achieve binary classification of four emotions; The four labels of the four emotions are merged, very low and low are merged into low labels; high and very high are merged into high labels, and samples that cannot be recognized by face recognition are eliminated; Step 5.6: Feature extraction, The attention state is predicted through eye trackers, and the predicted attention is used to help students improve their learning performance; Step 5.61: Detect students’ attention status through eye tracking and provide dialogue to re-engage students and improve learning outcomes; wherein, using attention-aware learning methods, using eye trackers to provide interventions based on students’ predictions of mind wandering, including interventions such as asking questions, reviewing content, and calling names; According to the Daisee dataset, the camera is used to capture the students' eyes for feature extraction. The eye aspect ratio is highly correlated with the attention state. The facial video-based method is used to detect distraction behavior in online education. Eye aspect ratio: For each facial video frame, use Mediapipe to extract facial landmarks; from the facial landmark positions, extract the eye aspect ratio, which represents the relationship between the width of the eye and the height. This ratio remains scale-invariant even if the scale of the eye changes; The mean and standard deviation were extracted to understand the EAR including outliers; the influence of period was observed by extracting 1q, 2q, 3q and MAD as outlier robust statistics, where EAR is the eye aspect ratio and q is the quartile; Calculate the mean and standard deviation: (1) Mean: the average value of the EAR data points; (2) Standard deviation: The standard deviation of the EAR data points, which is used to measure the dispersion of the data; Compute quartiles and median absolute deviations: (1) 1st quartile: 25th percentile of the data; (2) The second quartile, i.e. the median: the 50th percentile of the data; (3) 3rd quartile: 75th percentile of the data; (4) Median absolute deviation: The median of the absolute deviations between the median and each point in the data set, used to measure the central tendency of the data and the presence of outliers; Step 5.62: feature classification, using the initialized base classifier and the multi-output classifier; The multi-output classifier allows the sentiment regression model to handle problems that require predicting multiple outputs simultaneously; Parallel training: The n_jobs=-1 parameter allows the sentiment regression model to use all available CPU cores to train multiple classifiers in parallel, which significantly speeds up the training process for large datasets, where n_jobs is the number of tasks when training the base classifier in parallel; Ensemble learning: Use the RandomForestClassifier ensemble learning method to improve generalization and robustness through the prediction of multiple decision trees of the RandomForestClassifier ensemble learning method; train the model, calculate the accuracy, precision, recall and confusion matrix of the four emotions for binary classification, and save the model in ONNX format for data reasoning.

4. A cognitive impairment processing system, applicable to the non-contact cognitive impairment identification method based on facial video as claimed in claim 1, characterized in that: It includes a video acquisition module, an image processing unit, a data analysis module and a result output module; The video acquisition module is a high-definition camera, which is used to obtain the patient's facial video; The image processing unit is used to perform preprocessing and feature extraction operations on the collected video; The data analysis module uses a sentiment regression model to analyze and identify the extracted features; The result output module is used to present the recognition result to the user in an intuitive manner.