A cognitive impairment recognition method based on facial information fusion

Abstract

The present invention discloses a method for identifying cognitive impairment based on facial information fusion, and belongs to the technical field of cognitive impairment identification. The present invention collects facial video data of cognitive impairment patients, performs preprocessing, performs face detection, and marks key points of the face; calculates EAR, rPPG signal, head motion translation and rotation vector, and optimizes by minimizing the reprojection error; obtains RR interval, performs HRV analysis based on RR interval, and extracts features from EAR signal; extracts features based on head motion translation and rotation vector, trains random forest model, obtains feature fusion ratio, and performs feature fusion; trains cognitive impairment detection model based on fused features, uses EAR, rPPG signal and head motion vector as input data of feature classification model, outputs classification result, and corresponds to the binary classification task of cognitive impairment state; applies activation function, evaluates test result, and obtains cognitive impairment identification result.

Description

Technical Field

[0001] The present invention relates to the technical field of cognitive impairment recognition, and in particular to a cognitive impairment recognition method based on facial information fusion. Background Art

[0002] Cognitive impairment refers to the decline in memory, thinking, language and judgment due to damage to brain function. This disorder is common in the elderly population, and early detection and intervention of cognitive impairment are crucial to improving the quality of life of people with cognitive impairment.

[0003] Existing technologies lack the fusion analysis of multimodal information. Multimodal information such as facial expressions, head movements, heart rate variability, etc. cannot fully utilize various physiological and behavioral data. The lack of multimodal information leads to one-sided detection results and makes it difficult to fully assess the state of cognitive impairment. Traditional methods have limitations in feature extraction and fusion, and cannot effectively capture the complex relationships in multimodal data. Feature fusion methods lack dynamic adjustment capabilities and cannot adaptively optimize feature weights according to actual conditions. Summary of the invention

[0004] The purpose of the present invention is to provide a method for identifying cognitive impairment based on facial information fusion to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A method for identifying cognitive impairment based on facial information fusion, the method comprising the following steps:

[0007] S100, collect facial video data of cognitively impaired people, perform preprocessing, perform face detection through Mediapipe, and mark facial key points; calculate EAR and rPPG signals, use the PnP algorithm to calculate the head movement translation and rotation vectors, and optimize by minimizing the reprojection error;

[0008] S200, performing peak point detection on the rPPG signal to obtain the RR interval, performing HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis based on the RR interval, and extracting features of the EAR signal;

[0009] S300, based on the translation and rotation vectors of the head movement, obtain the yaw, pitch, roll angle vectors and the x, y, z translation vectors, and extract features; preliminarily define weights, form a preliminary fused feature vector with the preliminary features of the EAR, rPPG, and head movement, train a random forest model, the goal of the model is to classify different states of cognitive impairment, obtain the feature fusion ratio of the EAR, rPPG, and head movement, and perform feature fusion;

[0010] S400, based on the fused features, the cognitive impairment detection model is trained. The optimizer selects the stochastic gradient descent Adam optimization algorithm, uses the EAR, rPPG signals and head motion vectors as the input data of the feature classification model, and outputs the classification results, corresponding to the binary classification task of the cognitive impairment state; it is mapped to the probability space through a Sigmoid activation function, and the test results are analyzed from the accuracy, precision and recall rate to obtain the cognitive impairment recognition results.

[0011] According to step S100, facial video data of a cognitively impaired person is collected by using a camera, and the collected video data is preprocessed, wherein the preprocessing includes removing redundant frames and standardizing the video frame rate and resolution; importing the Mediapipe framework, initializing the Face Mesh module of Mediapipe, detecting faces and marking three-dimensional key points; processing the preprocessed video data frame by frame, applying the Mediapipe face detection model, identifying faces in each frame and extracting their key points;

[0012] Get the key point coordinates. For each detected face in each frame, the Mediapipe model will return a set of key point coordinates, including facial contour, forehead, eyebrows, eyes, nose and mouth. The ROI area position is anchored in the video according to the key point coordinates. Mediapipe is an open source cross-platform machine learning framework that provides a pre-trained model to annotate 468 key points of the face. According to these key point coordinates, the ROI area position can be anchored in the video.

[0013] According to step S100, the eye aspect ratio EAR is selected as a feature for cognitive impairment detection and is calculated using the key point coordinates. The calculation formula is as follows:

[0014] ;

[0015] The euclidean function is the Euclidean distance between two points. The coordinates of the four upper and lower key points of the eye correspond to the 158, 159, 386, and 387 points of Mediapipe respectively. The left and right eye corners of the left eye correspond to the 33 and 133 points of Mediapipe respectively. are the left and right eye corner coordinates of the right eye, corresponding to the 263 and 362 points of Mediapipe respectively; the original signal is filtered using a Savitzky-Golay filter, the filtered signal is subtracted from the original signal, and the waveform of the high-frequency component is retained;

[0016] The heart rate information is extracted by using the mean value of the green channel pixels in the cheek area as the rPPG signal, and the rPPG signal is band-pass filtered using a Butterworth filter, and the upper and lower cutoff frequencies of the band-pass filter are set according to the frequency range of the heartbeat;

[0017] According to the facial positions of the nose, eye corners, and mouth corners, the PnP algorithm is used to calculate the head movement translation and rotation vectors. The calculation formula of the PnP algorithm is as follows:

[0018] ;

[0019] where s is the scale factor, is the homogeneous coordinate of the i-th 3D model point, is the camera intrinsic parameter matrix, is the x-axis focal length of the camera, is the y-axis focal length of the camera, is the coordinate of the camera's principal point on the image x-axis, is the coordinate of the camera's principal point on the image's y axis, R is the rotation matrix, is the translation vector, represents the homogeneous coordinates of the i-th 3D model point, T represents the transpose of the matrix, , and Represent the x-axis coordinate, y-axis coordinate and z-axis coordinate of the i-th 3D model point respectively; optimize the rotation matrix 𝑅 and translation vector by minimizing the reprojection error error , the nonlinear optimization formula is as follows:

[0020] ;

[0021] Where N is the total number of 3D model points.

[0022] According to step S200, the peak point detection of the rPPG signal is performed to obtain the RR interval, and HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis are performed based on the RR interval to obtain a feature vector; the HRV time domain analysis uses the following method:

[0023] ;

[0024] Where HR is the heart rate in bmp; M is the total number of RR intervals; It is RR interval;

[0025] ;

[0026] Among them, SDNN represents the standard deviation of all normal sinus beat intervals, in milliseconds, reflecting the degree to which heart rate variability deviates from the mean; yes RR interval; It is the average value of the RR intervals of several heart beats;

[0027] ;

[0028] Among them, RMSSD represents the root mean square of the difference between all adjacent RR intervals, in ms, reflecting the regulatory effect of the parasympathetic nerve on heart rate; It is RR interval; It is RR interval;

[0029] HRV frequency domain analysis reflects the activities of the sympathetic and parasympathetic nerves. The following method is used for HRV frequency domain analysis:

[0030] ;

[0031] Wherein, LF represents the low-frequency energy of the pulse wave of the pain subject in the frequency range [μ, ν], μ and ν are both positive decimals; PSD(f) is the power spectral density of the signal, which represents the power of the signal at frequency f;

[0032] ;

[0033] Among them, HF represents the high-frequency energy of the pulse wave of the pain subject in the frequency range [ν,ξ], and both ν and ξ are positive decimals;

[0034] By calculating LF / HF, the relative ratio and balance of sympathetic and parasympathetic nerve activity can be obtained;

[0035] To perform HRV nonlinear analysis, use the Poincare map. For a continuous heartbeat interval, use the rth heartbeat interval as the horizontal coordinate and the r+1th heartbeat interval as the vertical coordinate to draw a scatter plot on a two-dimensional plane. The formula used is as follows:

[0036] ;

[0037] Where SD1 is the semi-major axis of the ellipse, E is the length of the heartbeat interval sequence, is the rth data point of the time series;

[0038] ;

[0039] Where SD2 is the semi-minor axis of the ellipse;

[0040] ;

[0041] Here, S represents the total amount of heart rate variability, which combines short-term and long-term variability and is used to assess the overall function of the autonomic nervous system.

[0042] According to step S200, the EAR signal is subjected to feature extraction, and the feature calculation formulas are as follows:

[0043] ;

[0044] in, is the length of the signal, is the i-th EAR signal value, MEAN represents the average level of the EAR signal, reflecting the central trend of the entire signal, i is used to traverse from 1 to ;

[0045] ;

[0046] Among them, SD represents the discrete degree of EAR signal;

[0047] ;

[0048] Where Q1 represents the value of 25% of the data points in the EAR signal, and percentile(ear_signal,25) means obtaining the value of 25% of the data points in the EAR signal;

[0049] ;

[0050] Among them, Q2 represents the median value of the EAR signal, and median(ear_signal) represents obtaining the median value of the EAR signal;

[0051] ;

[0052] Among them, Q3 represents the 75% data point of the EAR signal, and percentile(ear_signal,75) means obtaining the value of the 75% data point in the EAR signal;

[0053] ;

[0054] Wherein, MAD represents the median of the deviations of each data point in the EAR signal from the median, and median(|ear_signal-median(ear_signal)|) represents the median of the deviations of each data point in the EAR signal from the median;

[0055] ;

[0056] in, is the Fourier transform result of the EAR signal, Represents the center of energy in the spectrum;

[0057] ;

[0058] Among them, BW represents the dispersion degree of the frequency components of the EAR signal, reflecting the complexity of the signal change;

[0059] ;

[0060] in, is the fundamental frequency, THD indicates the degree of harmonic distortion in the signal, the ratio of the energy of all harmonics in the signal to the energy of the fundamental frequency. For traversing from 2 to .

[0061] According to step S300, the rotation vector is converted into three Euler angles of yaw, pitch and roll to form yaw, pitch and roll angle vectors; the head movement translation vector is decomposed into x, y, z translation vectors; and vector feature extraction is performed, and the formula is as follows:

[0062] ;

[0063] ;

[0064] ;

[0065] in, represents the mean value of the yaw angle, represents the mean value of the pitch angle, represents the mean value of the roll angle, represents the standard deviation of the yaw angle, represents the standard deviation of the pitch angle, represents the standard deviation of the roll angle, represents the yaw angle vector of the i-th data point, represents the pitch angle vector of the ith data point, Represents the roll angle vector of the i-th data point;

[0066] ;

[0067] ;

[0068] ;

[0069] in, is the spectrum energy at the yaw angle, is the spectral energy at the pitch angle, is the spectrum energy at the roll angle, k is used to traverse from 1 to N / 2, and j represents the imaginary unit;

[0070] ;

[0071] ;

[0072] in, is the RMS value of the ith translation vector, which represents the overall amplitude of the head movement in space, is the RMS value of the i-th rotation vector, indicating the overall magnitude of the head rotation.

[0073] According to step S300, weights are preliminarily defined, and initial weights are set for EAR, rPPG and head motion features, each accounting for 1 / 3; EAR features, rPPG features and head motion features are sequentially spliced ​​to form a preliminarily fused feature vector;

[0074] The eye aspect ratio in the EAR feature is mapped to the concentration level of the cognitively impaired person, and the rotation angle and frequency in the head movement feature are mapped to the interaction frequency of the cognitively impaired person, and a weighted sum is performed to judge the sense of participation; the heart rate variability in the rPPG feature is mapped to the emotional fluctuation of the cognitively impaired person, and the rotation frequency in the head movement feature is mapped to the failure reaction of the cognitively impaired person, and a weighted sum is performed to judge the sense of frustration; the abnormal closure of the eyes in the EAR feature is mapped to the behavioral pause of the cognitively impaired person, and the frequency of shaking the head left and right in the head movement feature is mapped to the repeated attempts of the cognitively impaired person to judge the sense of confusion; the abnormal opening and stillness of the eyes in the EAR feature is mapped to the distraction of the cognitively impaired person, and the heart rate in the rPPG feature and the movement amplitude in the head movement feature are mapped to the slow behavior of the cognitively impaired person to judge the sense of boredom;

[0075] The initially fused feature vectors and the corresponding state labels are divided into training sets and test sets. The training set data is used to train the random forest model. The goal of the model is to classify the different states of people with cognitive impairment. The random forest is integrated with several decision trees. After the random forest model is trained, the importance score of each feature in the model is extracted. According to the feature importance score, the optimal feature weight combination under different states is generated. The weight combination under different states is dynamically adjusted according to the feature importance score. According to the feature importance evaluation results of the random forest model, the weights of EAR, rPPG and head movement features are dynamically adjusted. Under different states, high-importance features are given higher weights, and low-importance features are given lower weights. According to the adjusted weights, the fused feature vector is recalculated.

[0076] The test set data was used to verify the model performance after adjusting the weights. The evaluation indicators included accuracy, precision, and recall. According to the verification results, the random forest model and feature weight combination were further optimized to obtain the feature fusion ratio of EAR, rPPG, and head motion, and feature fusion was performed.

[0077] According to step S400, the training data of the cognitive impairment detection model comes from the Daisee database; the Daisee database contains facial video records of multiple participants when watching emotion-inducing videos. These videos are annotated and contain emotion tags, such as anger, fear, happiness, sadness, surprise, etc. The frame rate and resolution of the video data are suitable for face detection and expression analysis. Each video is annotated by experts and assigned corresponding emotion tags. These tags are based on the emotional reactions of the participants when watching the videos, helping researchers to perform emotion recognition and analysis. Emotion tags include but are not limited to participation, frustration, confusion and boredom, which can be directly used for the training of cognitive impairment detection models.

[0078] Divide the data set into training set and validation set in proportion for preprocessing; set the learning rate, training rounds and BatchSize of the model; select the stochastic gradient descent Adam optimization algorithm for the optimizer; set the exponential decay rate of the gradient first and second moment estimates; the Adam optimization algorithm formula is as follows:

[0079] ;

[0080] in and are the parameter weights at time t+1 and t respectively, lr is the learning rate, which controls the step size of each parameter update, and They are the first-order moment estimates of the gradient and the second moment estimate The exponential decay rate of is the first-order moment estimate at time t-1, is the second-order moment estimate of the previous step, is the parameter gradient at time t, is the error term;

[0081] The model selects binary cross entropy as the loss function, performs sigmoid transformation on the original output of the model, maps the output to the range of [0,1], and uses binary cross entropy to calculate the loss. , the calculation formula is as follows:

[0082] ;

[0083] in is the Logits of the original output of the model, is the true label, represents the Sigmoid function, is the BatchSize size.

[0084] According to step S400, the EAR, rPPG signal and head motion vector are used as input data of the feature classification model, and the model integrates the 1DCNN convolutional neural network of the LSTM module;

[0085] The input time series is used to extract low-level local features through a one-dimensional convolutional layer. The BatchNorm layer standardizes the data of each batch to keep the output distribution of the convolutional layer stable. The maximum pooling layer is used to reduce the size of the features, retain important features, and prevent overfitting. The second one-dimensional convolutional layer is used to further extract features. The batch normalization layer is used to standardize the output of the second convolutional layer. The maximum pooling layer is used to further reduce the size of the feature map. The bidirectional LSTM layer is used to capture sequence features. The second bidirectional LSTM layer is used to further capture sequence features. The three fully connected layers output 4-dimensional classification results, corresponding to the binary classification task of 4 cognitive impairment states.

[0086] According to step S400, linear transformation is performed on the output Logits through two fully connected layers, and mapped to the probability space through a Sigmoid activation function; the Sigmoid function formula is as follows:

[0087] ;

[0088] Among them, 𝑒 is the base of natural logarithm; the output value of Sigmoid function approaches 0 when the input is negative infinity, and approaches 1 when the input is positive infinity, mapping the Logits output by the model to a probability value between 0 and 1; the test results are analyzed from the three indicators of accuracy Acc, precision Pre and recall Rec, and the corresponding formulas are as follows:

[0089] ;

[0090] ;

[0091] ;

[0092] TP means that the samples of engagement, frustration, confusion and boredom are correctly predicted as engagement, frustration, confusion and boredom; FP means that the samples of non-engagement, non-frustration, non-confusion and non-boredom are incorrectly predicted as engagement, frustration, confusion and boredom; TN means that the samples of non-engagement, non-frustration, non-confusion and non-boredom are correctly predicted as non-engagement, non-frustration, non-confusion and non-boredom; FN means that the samples of engagement, frustration, confusion and boredom are incorrectly predicted as non-engagement, non-frustration, non-confusion and non-boredom;

[0093] When users are predicted to feel disengaged, frustrated, confused, and bored, they are judged to be cognitively impaired.

[0094] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0095] The present invention comprehensively reflects the physiological and behavioral states of people with cognitive impairment by collecting multimodal data such as EAR, rPPG signals and head movements. Multimodal information fusion improves the accuracy and robustness of the detection results.

[0096] The present invention adopts Mediapipe to perform face detection and key point annotation to ensure accurate extraction of EAR signals; uses PnP algorithm to calculate head motion vector and optimize the feature extraction process; performs peak point detection and HRV analysis on rPPG signals to extract heart rate variability features in multiple dimensions.

[0097] The present invention trains the random forest model to learn the optimal feature weight combination under different states, dynamically adjust the feature weights, and improve the self-adaptation ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0099] Figure 1 It is a schematic diagram of the steps of a method for identifying cognitive impairment based on facial information fusion of the present invention;

[0100] Figure 2 It is a schematic diagram of the implementation structure of a cognitive impairment identification method based on facial information fusion of the present invention. DETAILED DESCRIPTION

[0101] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0102] See also Figure 1 and Figure 2 , the present invention provides a technical solution:

[0103] According to one embodiment of the present invention, Figure 1 A method for identifying cognitive impairment based on facial information fusion is shown in a schematic diagram of steps. A method for identifying cognitive impairment based on facial information fusion comprises the following steps:

[0104] S100, collect facial video data of cognitively impaired people, perform preprocessing, perform face detection through Mediapipe, and mark facial key points; calculate EAR and rPPG signals, use the PnP algorithm to calculate the head movement translation and rotation vectors, and optimize by minimizing the reprojection error;

[0105] S200, performing peak point detection on the rPPG signal to obtain the RR interval, performing HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis based on the RR interval, and extracting features of the EAR signal;

[0106] S300, based on the translation and rotation vectors of the head movement, obtain the yaw, pitch, roll angle vectors and the x, y, z translation vectors, and extract features; preliminarily define weights, form a preliminary fused feature vector with the preliminary features of the EAR, rPPG, and head movement, train a random forest model, the goal of the model is to classify different states of cognitive impairment, obtain the feature fusion ratio of the EAR, rPPG, and head movement, and perform feature fusion;

[0107] S400, based on the fused features, the cognitive impairment detection model is trained. The optimizer selects the stochastic gradient descent Adam optimization algorithm, uses the EAR, rPPG signals and head motion vectors as the input data of the feature classification model, and outputs the classification results, corresponding to the binary classification task of the cognitive impairment state; it is mapped to the probability space through a Sigmoid activation function, and the test results are analyzed from the accuracy, precision and recall rate to obtain the cognitive impairment recognition results.

[0108] According to step S100, a 10-second video of the face is captured by a camera with a sampling rate of 30 Hz, and the collected video data is preprocessed, wherein the preprocessing includes removing redundant frames and standardizing the video frame rate and resolution; the Mediapipe framework is imported, the Face Mesh module of Mediapipe is initialized, the face is detected and the three-dimensional key points are marked; the preprocessed video data is processed frame by frame, and the Mediapipe face detection model is applied to identify the face in each frame and extract its key points;

[0109] Get the key point coordinates. For each detected face in each frame, the Mediapipe model will return a set of key point coordinates, including facial contour, forehead, eyebrows, eyes, nose and mouth. The ROI area position is anchored in the video according to the key point coordinates. Mediapipe is an open source cross-platform machine learning framework that provides a pre-trained model to annotate 468 key points of the face. According to these key point coordinates, the ROI area position can be anchored in the video.

[0110] According to step S100, the eye aspect ratio EAR is selected as a feature for cognitive impairment detection and is calculated using the key point coordinates. The calculation formula is as follows:

[0111] ;

[0112] The euclidean function is the Euclidean distance between two points. The coordinates of the four upper and lower key points of the eye correspond to the 158, 159, 386, and 387 points of Mediapipe respectively. The left and right eye corners of the left eye correspond to the 33 and 133 points of Mediapipe respectively. are the left and right eye corner coordinates of the right eye, corresponding to the 263 and 362 points of Mediapipe respectively; the original signal is filtered using a Savitzky-Golay filter, the filtered signal is subtracted from the original signal, and the waveform of the high-frequency component is retained;

[0113] The heart rate information is extracted by taking the mean value of the green channel pixels in the cheek area as the rPPG signal, and the rPPG signal is band-pass filtered using a Butterworth filter. The upper and lower cutoff frequencies of the band-pass filter are set to 0.5 Hz and 2 Hz based on the frequency range of the heartbeat.

[0114] According to the facial positions of the nose, eye corners, and mouth corners, the PnP algorithm is used to calculate the head movement translation and rotation vectors. The calculation formula of the PnP algorithm is as follows:

[0115] ;

[0116] where s is the scale factor, is the homogeneous coordinate of the i-th 3D model point, is the camera intrinsic parameter matrix, is the x-axis focal length of the camera, is the y-axis focal length of the camera, is the coordinate of the camera's principal point on the image x-axis, is the coordinate of the camera's principal point on the image's y axis, R is the rotation matrix, is the translation vector, represents the homogeneous coordinates of the i-th 3D model point, T represents the transpose of the matrix, , and Represent the x-axis coordinate, y-axis coordinate and z-axis coordinate of the i-th 3D model point respectively; optimize the rotation matrix 𝑅 and translation vector by minimizing the reprojection error error , the nonlinear optimization formula is as follows:

[0117] ;

[0118] Where N is the total number of 3D model points.

[0119] According to step S200, the peak point detection of the rPPG signal is performed to obtain the RR interval, and HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis are performed based on the RR interval to obtain a feature vector; the HRV time domain analysis uses the following method:

[0120] ;

[0121] Where HR is the heart rate in bmp; M is the total number of RR intervals; It is RR interval;

[0122] ;

[0123] Among them, SDNN represents the standard deviation of all normal sinus beat intervals, in milliseconds, reflecting the degree to which heart rate variability deviates from the mean; yes RR interval; It is the average value of the RR intervals of several heart beats;

[0124] ;

[0125] Among them, RMSSD represents the root mean square of the difference between all adjacent RR intervals, in ms, reflecting the regulatory effect of the parasympathetic nerve on heart rate; It is RR interval; It is RR interval;

[0126] HRV frequency domain analysis reflects the activities of the sympathetic and parasympathetic nerves. The following method is used for HRV frequency domain analysis:

[0127] ;

[0128] Wherein, LF represents the low-frequency energy of the pulse wave of the pain subject in the frequency range [μ, ν], and both μ and ν are positive decimals; PSD(f) is the power spectral density of the signal, which represents the power of the signal at frequency f; in this embodiment, μ is 0.04 and ν is 0.15.

[0129] ;

[0130] Wherein, HF represents the high-frequency energy of the pulse wave of the pain subject within the frequency range [ν,ξ], and both ν and ξ are positive decimals; in this embodiment, ξ is 0.4.

[0131] By calculating LF / HF, the relative ratio and balance of sympathetic and parasympathetic nerve activity can be obtained;

[0132] To perform HRV nonlinear analysis, use the Poincare map. For a continuous heartbeat interval, use the rth heartbeat interval as the horizontal coordinate and the r+1th heartbeat interval as the vertical coordinate to draw a scatter plot on a two-dimensional plane. The formula used is as follows:

[0133] ;

[0134] Where SD1 is the semi-major axis of the ellipse, E is the length of the heartbeat interval sequence, is the rth data point of the time series;

[0135] ;

[0136] Where SD2 is the semi-minor axis of the ellipse;

[0137] ;

[0138] Here, S represents the total amount of heart rate variability, which combines short-term and long-term variability and is used to assess the overall function of the autonomic nervous system.

[0139] According to step S200, the EAR signal is subjected to feature extraction, and the feature calculation formulas are as follows:

[0140] ;

[0141] in, is the length of the signal, is the i-th EAR signal value, MEAN represents the average level of the EAR signal, reflecting the central trend of the entire signal, i is used to traverse from 1 to ;

[0142] ;

[0143] Among them, SD represents the discrete degree of EAR signal;

[0144] ;

[0145] Where Q1 represents the value of 25% of the data points in the EAR signal, and percentile(ear_signal,25) means obtaining the value of 25% of the data points in the EAR signal;

[0146] ;

[0147] Among them, Q2 represents the median value of the EAR signal, and median(ear_signal) represents obtaining the median value of the EAR signal;

[0148] ;

[0149] Among them, Q3 represents the 75% data point of the EAR signal, and percentile(ear_signal,75) means obtaining the value of the 75% data point in the EAR signal;

[0150] ;

[0151] Wherein, MAD represents the median of the deviations of each data point in the EAR signal from the median, and median(|ear_signal-median(ear_signal)|) represents the median of the deviations of each data point in the EAR signal from the median;

[0152] ;

[0153] in, is the Fourier transform result of the EAR signal, Represents the center of energy in the spectrum;

[0154] ;

[0155] Among them, BW represents the dispersion degree of the frequency components of the EAR signal, reflecting the complexity of the signal change;

[0156] ;

[0157] in, is the fundamental frequency, THD indicates the degree of harmonic distortion in the signal, the ratio of the energy of all harmonics in the signal to the energy of the fundamental frequency. For traversing from 2 to .

[0158] According to step S300, the rotation vector is converted into three Euler angles of yaw, pitch and roll to form yaw, pitch and roll angle vectors; the head movement translation vector is decomposed into x, y, z translation vectors; and vector feature extraction is performed, and the formula is as follows:

[0159] ;

[0160] ;

[0161] ;

[0162] in, represents the mean value of the yaw angle, represents the mean value of the pitch angle, represents the mean value of the roll angle, represents the standard deviation of the yaw angle, represents the standard deviation of the pitch angle, represents the standard deviation of the roll angle, represents the yaw angle vector of the i-th data point, represents the pitch angle vector of the ith data point, Represents the roll angle vector of the i-th data point;

[0163] ;

[0164] ;

[0165] ;

[0166] in, is the spectrum energy at the yaw angle, is the spectral energy at the pitch angle, is the spectrum energy at the roll angle, k is used to traverse from 1 to N / 2, and j represents the imaginary unit;

[0167] ;

[0168] ;

[0169] in, is the RMS value of the ith translation vector, which represents the overall amplitude of the head movement in space, is the RMS value of the i-th rotation vector, indicating the overall magnitude of the head rotation.

[0170] According to step S300, weights are preliminarily defined, and initial weights are set for EAR, rPPG and head motion features, each accounting for 1 / 3; EAR features, rPPG features and head motion features are sequentially spliced ​​to form a preliminarily fused feature vector;

[0171] The eye aspect ratio in the EAR feature is mapped to the concentration level of the cognitively impaired person, and the rotation angle and frequency in the head movement feature are mapped to the interaction frequency of the cognitively impaired person, and a weighted sum is performed to judge the sense of participation; the heart rate variability in the rPPG feature is mapped to the emotional fluctuation of the cognitively impaired person, and the rotation frequency in the head movement feature is mapped to the failure reaction of the cognitively impaired person, and a weighted sum is performed to judge the sense of frustration; the abnormal closure of the eyes in the EAR feature is mapped to the behavioral pause of the cognitively impaired person, and the frequency of shaking the head left and right in the head movement feature is mapped to the repeated attempts of the cognitively impaired person to judge the sense of confusion; the abnormal opening and stillness of the eyes in the EAR feature is mapped to the distraction of the cognitively impaired person, and the heart rate in the rPPG feature and the movement amplitude in the head movement feature are mapped to the slow behavior of the cognitively impaired person to judge the sense of boredom;

[0172] The initially fused feature vectors and the corresponding state labels are divided into training sets and test sets. The training set data is used to train the random forest model. The goal of the model is to classify the different states of people with cognitive impairment. The random forest is integrated with several decision trees. After the random forest model is trained, the importance score of each feature in the model is extracted. According to the feature importance score, the optimal feature weight combination under different states is generated. The weight combination under different states is dynamically adjusted according to the feature importance score. According to the feature importance evaluation results of the random forest model, the weights of EAR, rPPG and head movement features are dynamically adjusted. Under different states, high-importance features are given higher weights, and low-importance features are given lower weights. According to the adjusted weights, the fused feature vector is recalculated.

[0173] The test set data was used to verify the model performance after adjusting the weights. The evaluation indicators included accuracy, precision, and recall. According to the verification results, the random forest model and feature weight combination were further optimized to obtain the feature fusion ratio of EAR, rPPG, and head motion, and feature fusion was performed.

[0174] According to step S400, the training data of the cognitive impairment detection model comes from the Daisee database; the Daisee database contains facial video records of multiple participants when watching emotion-inducing videos. The Daisee database contains 112 experimental subjects, 9068 video clips, each clip is 10 seconds long, a total of 25.2 hours of experimental data, and labels for four states: engagement, frustration, confusion, and boredom. The data set is divided into a training set and a validation set at a ratio of 80% and 20%, that is, 6853 experimental subjects are selected as training data, 1713 experimental subjects are selected as test data, and 10 seconds is used as the time window.

[0175] In this embodiment, the model learning rate is set to 0.001, the training rounds are 50 rounds, the BatchSize is 32, the optimizer selects the stochastic gradient descent Adam optimization algorithm, and the exponential decay rates of the gradient first and second moment estimates are set to 0.9 and 0.999 respectively. The Adam optimization algorithm formula is as follows:

[0176] ;

[0177] in and are the parameter weights at time t+1 and t respectively, lr is the learning rate, which controls the step size of each parameter update, and They are the first-order moment estimates of the gradient and the second moment estimate The exponential decay rate of is the first-order moment estimate at time t-1, is the second-order moment estimate of the previous step, is the parameter gradient at time t, is the error term;

[0178] The model selects binary cross entropy as the loss function, performs sigmoid transformation on the original output of the model, maps the output to the range of [0,1], and uses binary cross entropy to calculate the loss. , the calculation formula is as follows:

[0179] ;

[0180] in is the Logits of the original output of the model, is the true label, represents the Sigmoid function, is the BatchSize size.

[0181] According to step S400, the EAR, rPPG signal and head motion vector are used as input data of the feature classification model, and the model integrates the 1DCNN convolutional neural network of the LSTM module;

[0182] The input time series is used to extract low-level local features through a one-dimensional convolutional layer. The BatchNorm layer standardizes the data of each batch to keep the output distribution of the convolutional layer stable. The maximum pooling layer is used to reduce the size of the features, retain important features, and prevent overfitting. The second one-dimensional convolutional layer is used to further extract features. The batch normalization layer is used to standardize the output of the second convolutional layer. The maximum pooling layer is used to further reduce the size of the feature map. The bidirectional LSTM layer is used to capture sequence features. The second bidirectional LSTM layer is used to further capture sequence features. The three fully connected layers output 4-dimensional classification results, corresponding to the binary classification task of 4 cognitive impairment states.

[0183] According to step S400, according to another embodiment of the present invention, Figure 2 The implementation structure diagram of a cognitive impairment recognition method based on facial information fusion is shown in the figure. The output shapes of each layer of the feature classification model are shown in Table 1:

[0184] Table 1 Output shape of each layer of feature classification model

[0185]

[0186] After two layers of fully connected layers, linear transformation is performed to output Logits, which are mapped to the probability space through a Sigmoid activation function; the Sigmoid function formula is as follows:

[0187] ;

[0188] Among them, 𝑒 is the base of natural logarithm; the output value of Sigmoid function approaches 0 when the input is negative infinity, and approaches 1 when the input is positive infinity, mapping the Logits output by the model to a probability value between 0 and 1; the test results are analyzed from the three indicators of accuracy Acc, precision Pre and recall Rec, and the corresponding formulas are as follows:

[0189] ;

[0190] ;

[0191] ;

[0192] TP means that the samples of engagement, frustration, confusion and boredom are correctly predicted as engagement, frustration, confusion and boredom; FP means that the samples of non-engagement, non-frustration, non-confusion and non-boredom are incorrectly predicted as engagement, frustration, confusion and boredom; TN means that the samples of non-engagement, non-frustration, non-confusion and non-boredom are correctly predicted as non-engagement, non-frustration, non-confusion and non-boredom; FN means that the samples of engagement, frustration, confusion and boredom are incorrectly predicted as non-engagement, non-frustration, non-confusion and non-boredom;

[0193] When users are predicted to feel disengaged, frustrated, confused, and bored, they are judged to be cognitively impaired.

[0194] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0195] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying cognitive impairment based on facial information fusion, characterized in that: The method comprises the following steps: S100, collect facial video data of cognitively impaired people, perform preprocessing, perform face detection through Mediapipe, and mark facial key points; calculate EAR and rPPG signals, use the PnP algorithm to calculate the head movement translation and rotation vectors, and optimize by minimizing the reprojection error; S200, performing peak point detection on the rPPG signal to obtain the RR interval, performing HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis based on the RR interval, and extracting features of the EAR signal; S300, based on the translation and rotation vectors of the head movement, obtain the yaw, pitch, roll angle vectors and the x, y, z translation vectors, and extract features; preliminarily define weights, form a preliminary fused feature vector with the preliminary features of the EAR, rPPG, and head movement, train a random forest model, the goal of the model is to classify different states of cognitive impairment, obtain the feature fusion ratio of the EAR, rPPG, and head movement, and perform feature fusion; S400, based on the fused features, the cognitive impairment detection model is trained. The optimizer selects the stochastic gradient descent Adam optimization algorithm, uses the EAR, rPPG signals and head motion vectors as the input data of the feature classification model, and outputs the classification results, corresponding to the binary classification task of the cognitive impairment state; it is mapped to the probability space through a Sigmoid activation function, and the test results are analyzed from the accuracy, precision and recall rate to obtain the cognitive impairment recognition results.

2. The method for identifying cognitive impairment based on facial information fusion according to claim 1, characterized in that: According to step S100, facial video data of the cognitively impaired person is collected by using a camera, and the collected video data is preprocessed, wherein the preprocessing includes removing redundant frames and standardizing the video frame rate and resolution; Import the Mediapipe framework, initialize the Face Mesh module of Mediapipe, detect faces and mark 3D key points; Process the preprocessed video data frame by frame, apply the Mediapipe face detection model, identify faces in each frame and extract their key points; Get the key point coordinates. For each detected face in each frame, the Mediapipe model will return a set of key point coordinates, including facial contour, forehead, eyebrows, eyes, nose and mouth. The ROI area position is anchored in the video according to the key point coordinates.

3. The method for identifying cognitive impairment based on facial information fusion according to claim 2, characterized in that: According to step S100, the eye aspect ratio EAR is selected as a feature for cognitive impairment detection and is calculated using the key point coordinates. The calculation formula is as follows: ； The euclidean function is the Euclidean distance between two points. The coordinates of the four upper and lower key points of the eye correspond to the 158, 159, 386, and 387 points of Mediapipe respectively. The left and right eye corners of the left eye correspond to the 33 and 133 points of Mediapipe respectively. are the left and right eye corner coordinates of the right eye, corresponding to the 263 and 362 points of Mediapipe respectively; the original signal is filtered using a Savitzky-Golay filter, the filtered signal is subtracted from the original signal, and the waveform of the high-frequency component is retained; The heart rate information is extracted by using the mean value of the green channel pixels in the cheek area as the rPPG signal, and the rPPG signal is band-pass filtered using a Butterworth filter, and the upper and lower cutoff frequencies of the band-pass filter are set according to the frequency range of the heartbeat; According to the facial positions of the nose, eye corners, and mouth corners, the PnP algorithm is used to calculate the head movement translation and rotation vectors. The calculation formula of the PnP algorithm is as follows: ； where s is the scale factor, is the homogeneous coordinate of the i-th 3D model point, is the camera intrinsic parameter matrix, is the x-axis focal length of the camera, is the y-axis focal length of the camera, is the coordinate of the camera's principal point on the image x-axis, is the coordinate of the camera's principal point on the image's y axis, R is the rotation matrix, is the translation vector, represents the homogeneous coordinates of the i-th 3D model point, T represents the transpose of the matrix, , and Represent the x-axis coordinate, y-axis coordinate and z-axis coordinate of the i-th 3D model point respectively; optimize the rotation matrix 𝑅 and translation vector by minimizing the reprojection error error , the nonlinear optimization formula is as follows: ； Where N is the total number of 3D model points.

4. The method for identifying cognitive impairment based on facial information fusion according to claim 1, characterized in that: According to step S200, the peak point detection of the rPPG signal is performed to obtain the RR interval, and HRV time domain analysis, HRV frequency domain analysis and HRV nonlinear analysis are performed based on the RR interval to obtain a feature vector; the HRV time domain analysis uses the following method: ； Where HR is the heart rate in bmp; M is the total number of RR intervals; It is RR interval; ； Among them, SDNN represents the standard deviation of all normal sinus beat intervals, in milliseconds, reflecting the degree to which heart rate variability deviates from the mean; yes RR interval; It is the average value of the RR intervals of several heart beats; ； Among them, RMSSD represents the root mean square of the difference between all adjacent RR intervals, in ms, reflecting the regulatory effect of the parasympathetic nerve on heart rate; It is RR interval; It is RR interval; HRV frequency domain analysis reflects the activities of the sympathetic and parasympathetic nerves. The following method is used for HRV frequency domain analysis: ； Wherein, LF represents the low-frequency energy of the pulse wave of the pain subject in the frequency range [μ, ν], μ and ν are both positive decimals; PSD(f) is the power spectral density of the signal, which represents the power of the signal at frequency f; ； Among them, HF represents the high-frequency energy of the pulse wave of the pain subject in the frequency range [ν,ξ], and both ν and ξ are positive decimals; By calculating LF / HF, the relative ratio and balance of sympathetic and parasympathetic nerve activity can be obtained; To perform HRV nonlinear analysis, use the Poincare map. For a continuous heartbeat interval, use the rth heartbeat interval as the horizontal coordinate and the r+1th heartbeat interval as the vertical coordinate to draw a scatter plot on a two-dimensional plane. The formula used is as follows: ； Where SD1 is the semi-major axis of the ellipse, E is the length of the heartbeat interval sequence, is the rth data point of the time series; ； Where SD2 is the semi-minor axis of the ellipse; ； Here, S represents the total amount of heart rate variability, which combines short-term and long-term variability and is used to assess the overall function of the autonomic nervous system.

5. The method for identifying cognitive impairment based on facial information fusion according to claim 4, characterized in that: According to step S200, the EAR signal is subjected to feature extraction, and the feature calculation formulas are as follows: ； in, is the length of the signal, is the i-th EAR signal value, MEAN represents the average level of the EAR signal, reflecting the central trend of the entire signal, i is used to traverse from 1 to ; ； Among them, SD represents the discrete degree of EAR signal; ； Among them, Q1 represents the value of 25% data points in the EAR signal, and percentile(ear_signal,25) means obtaining the value of 25% data points in the EAR signal; ； Wherein, Q2 represents the median value of the EAR signal, and median(ear_signal) represents obtaining the median value of the EAR signal; ； Among them, Q3 represents the 75% data point of the EAR signal, and percentile(ear_signal,75) means obtaining the value of the 75% data point in the EAR signal; ； Wherein, MAD represents the median of the deviations of each data point in the EAR signal from the median, and median(|ear_signal-median(ear_signal)|) represents the median of the deviations of each data point in the EAR signal from the median; ； in, is the Fourier transform result of the EAR signal, Represents the center of energy in the spectrum; ； Among them, BW represents the dispersion degree of the frequency components of the EAR signal, reflecting the complexity of the signal change; ； in, is the fundamental frequency, THD indicates the degree of harmonic distortion in the signal, the ratio of the energy of all harmonics in the signal to the energy of the fundamental frequency. For traversing from 2 to .

6. The method for identifying cognitive impairment based on facial information fusion according to claim 1, characterized in that: According to step S300, the rotation vector is converted into three Euler angles of yaw, pitch and roll to form yaw, pitch and roll angle vectors; the head movement translation vector is decomposed into x, y, z translation vectors; and vector feature extraction is performed, and the formula is as follows: ； ； ； in, represents the mean value of the yaw angle, represents the mean value of the pitch angle, represents the mean value of the roll angle, represents the standard deviation of the yaw angle, represents the standard deviation of the pitch angle, represents the standard deviation of the roll angle, represents the yaw angle vector of the i-th data point, represents the pitch angle vector of the ith data point, Represents the roll angle vector of the i-th data point; ； ； ； in, is the spectrum energy at the yaw angle, is the spectral energy at the pitch angle, is the spectrum energy at the roll angle, k is used to traverse from 1 to N / 2, and j represents the imaginary unit; ； ； in, is the RMS value of the ith translation vector, which represents the overall amplitude of the head movement in space, is the RMS value of the i-th rotation vector, indicating the overall magnitude of the head rotation.

7. The method for identifying cognitive impairment based on facial information fusion according to claim 6, characterized in that: According to step S300, weights are preliminarily defined, and initial weights are set for EAR, rPPG and head motion features, each accounting for 1 / 3; EAR features, rPPG features and head motion features are sequentially spliced ​​to form a preliminarily fused feature vector; The eye aspect ratio in the EAR feature is mapped to the concentration level of the cognitively impaired person, and the rotation angle and frequency in the head movement feature are mapped to the interaction frequency of the cognitively impaired person, and a weighted sum is performed to judge the sense of participation; the heart rate variability in the rPPG feature is mapped to the emotional fluctuation of the cognitively impaired person, and the rotation frequency in the head movement feature is mapped to the failure reaction of the cognitively impaired person, and a weighted sum is performed to judge the sense of frustration; the abnormal closure of the eyes in the EAR feature is mapped to the behavioral pause of the cognitively impaired person, and the frequency of shaking the head left and right in the head movement feature is mapped to the repeated attempts of the cognitively impaired person to judge the sense of confusion; the abnormal opening and stillness of the eyes in the EAR feature is mapped to the distraction of the cognitively impaired person, and the heart rate in the rPPG feature and the movement amplitude in the head movement feature are mapped to the slow behavior of the cognitively impaired person to judge the sense of boredom; The initially fused feature vectors and corresponding state labels are divided into training sets and test sets. The training set data is used to train the random forest model. The goal of the model is to classify the different states of people with cognitive impairment. The random forest is integrated with several decision trees. After the random forest model is trained, the importance score of each feature in the model is extracted. According to the feature importance score, the optimal feature weight combination under different states is generated. The weight combination under different states is dynamically adjusted according to the feature importance score. According to the feature importance evaluation results of the random forest model, the weights of EAR, rPPG and head movement features are dynamically adjusted. In different states, high-importance features are given higher weights, and low-importance features are given lower weights; Recalculate the fused feature vector according to the adjusted weights; The test set data was used to verify the model performance after adjusting the weights. The evaluation indicators included accuracy, precision, and recall. According to the verification results, the random forest model and feature weight combination were further optimized to obtain the feature fusion ratio of EAR, rPPG, and head motion, and feature fusion was performed.

8. The method for identifying cognitive impairment based on facial information fusion according to claim 1, characterized in that: According to step S400, the training data of the cognitive impairment detection model comes from the Daisee database; the data set is divided into a training set and a validation set in proportion for preprocessing; the learning rate setting, training rounds and BatchSize of the model are set, the optimizer selects the stochastic gradient descent Adam optimization algorithm, and the exponential decay rate of the gradient first and second moment estimation is set. The formula of the Adam optimization algorithm is as follows: ； in and are the parameter weights at time t+1 and t respectively, lr is the learning rate, which controls the step size of each parameter update, and They are the first-order moment estimates of the gradient and the second moment estimate The exponential decay rate of is the first-order moment estimate at time t-1, is the second-order moment estimate of the previous step, is the parameter gradient at time t, is the error term; The model selects binary cross entropy as the loss function, performs sigmoid transformation on the original output of the model, maps the output to the range of [0,1], and uses binary cross entropy to calculate the loss. , the calculation formula is as follows: ； in is the Logits of the original output of the model, is the true label, represents the Sigmoid function, is the BatchSize size.

9. The method for identifying cognitive impairment based on facial information fusion according to claim 8, characterized in that: According to step S400, the EAR, rPPG signal and head motion vector are used as input data of the feature classification model, and the model integrates the 1DCNN convolutional neural network of the LSTM module; The input time series is passed through a one-dimensional convolutional layer to extract low-level local features. The BatchNorm layer normalizes the data of each batch to keep the output distribution of the convolutional layer stable. The maximum pooling layer is used to reduce the size of features, retain important features, and prevent overfitting. The second one-dimensional convolutional layer is used to further extract features. The batch normalization layer standardizes the output of the second convolutional layer; the maximum pooling layer further reduces the size of the feature map; the bidirectional LSTM layer captures the sequence features; the second bidirectional LSTM layer further captures the sequence features; and the three fully connected layers output 4-dimensional classification results, corresponding to the binary classification task of 4 cognitive impairment states.

10. The method for identifying cognitive impairment based on facial information fusion according to claim 9, characterized in that: According to step S400, linear transformation is performed on the output Logits through two fully connected layers, and mapped to the probability space through a Sigmoid activation function; the Sigmoid function formula is as follows: ； Among them, 𝑒 is the base of natural logarithm; the output value of Sigmoid function approaches 0 when the input is negative infinity, and approaches 1 when the input is positive infinity, mapping the Logits output by the model to a probability value between 0 and 1; the test results are analyzed from the three indicators of accuracy Acc, precision Pre and recall Rec, and the corresponding formulas are as follows: ； ； ； TP means that the samples of engagement, frustration, confusion and boredom are correctly predicted as engagement, frustration, confusion and boredom; FP means that the samples of non-engagement, non-frustration, non-confusion and non-boredom are incorrectly predicted as engagement, frustration, confusion and boredom; TN means that the samples of non-engagement, non-frustration, non-confusion and non-boredom are correctly predicted as non-engagement, non-frustration, non-confusion and non-boredom; FN means that the samples of engagement, frustration, confusion and boredom are incorrectly predicted as non-engagement, non-frustration, non-confusion and non-boredom; When users are predicted to feel disengaged, frustrated, confused, and bored, they are judged to be cognitively impaired.

Images