A method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos

Abstract

The present invention discloses a method for quantifying brain cognitive load based on the extraction of physiological indicators from facial videos. The facial video of the subject is recorded in a piano-playing and memorizing scene, and pulse variability and blood oxygen saturation are extracted through the facial video. The time of reproducing notes is recorded at the same time, and the above indicators are used to quantify and score cognitive load, thereby overcoming the subjectivity of the self-assessment scale commonly used internationally and the inconvenience of contact device measurement. The steps are as follows: the facial video of the subject memorizing the note sequence is captured by a camera, and the time of reproducing notes is recorded at the same time; the pulse wave is restored from the facial video and denoised, and a variety of physiological indicators are extracted; based on statistical analysis, the physiological indicators and the time of reproducing notes are used together to quantify and score cognitive load. The present invention can provide a non-invasive and non-invasive means for quantifying cognitive load.

Description

Technical Field

[0001] The present invention relates to the fields of information technology and smart health technology, and in particular to a method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos. Background Art

[0002] At present, the most commonly used method for evaluating brain cognitive load internationally is the subjective scale method, including the PAAS scale, WP scale, and TLX scale. These scales basically require the subjects to answer questions in the scale, and the experimenters score the answers to judge the subjects' brain cognitive load. This method of measuring brain cognitive load is highly subjective and time-consuming; another way to measure brain cognitive load is to measure it through performance, such as the time, speed or correctness of the subject's task completion, critical errors, and false starts, but this method can obtain limited information. There are also methods to measure brain cognitive load through behavior, such as observing the characteristic patterns of the subject's interactive behavior, such as language or dialogue patterns. This method is also subjective and not very universal.

[0003] With the development of modern sensor technology, various physiological measurement methods have been developed for the assessment of brain cognitive load. Commonly used methods for measuring brain cognitive load use signals such as brain waves, eye activity, breathing, heart rate, skin conductance, and language. However, the traditional contact measurement method stimulates the subject and affects the measurement of brain cognitive load itself.

[0004] Therefore, the method of using a non-contact camera to capture the subject's facial video, extract various physiological indicators, and then measure cognitive load provides a non-destructive, non-invasive, objective and efficient measurement solution, which is also the main development direction of future smart healthcare. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of the subjectivity of the common self-assessment scale of brain cognitive load in the prior art and the inconvenience of contact equipment measurement, and provide a method for quantifying brain cognitive load based on the extraction of physiological indicators from facial videos, design a scene of playing the piano to induce different levels of cognitive load, and gradually increase the difficulty of memory by having the subjects memorize different sequences of notes and reproduce them on the electronic piano, so as to control different levels of cognitive load, and at the same time combine the hands and the brain to measure cognitive load more comprehensively. In this scenario, the cognitive load of the subject is quantified in a multi-dimensional manner through the physiological indicators extracted from the face video and the recorded time of the reproduced notes. This non-contact method provides a non-invasive and non-invasive means for measuring cognitive load.

[0006] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0007] A method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos, the method comprising the following steps:

[0008] S1. Collect facial videos of subjects while memorizing musical note sequences under different cognitive loads, and record the time of reproducing musical notes; wherein the cognitive load is the magnitude of cognitive pressure on the subject's brain, and different cognitive load levels are divided by memorizing musical note sequences of different lengths;

[0009] The facial video is collected for subsequent extraction of physiological indicators and to measure the cognitive load level from an objective perspective; the note repetition time is the total time for the subject to recall the note and play it completely after the memory process is completed, and the note repetition time measures the cognitive load level by reflecting the tension of the brain; the note sequence refers to any combination of seven single notes within an octave from the middle note "do" to the middle note "xi";

[0010] S2, locate the facial region of interest and extract the pulse wave: perform face recognition on the collected face video, locate the region of interest, obtain the grayscale value of the skin pixels in the region of interest, and extract the original pulse wave signal; wherein the region of interest is the face region used to extract the pulse wave signal; based on the consistency between the grayscale value change of the skin pixel and the pulse change, the original pulse wave signal is extracted;

[0011] S3. De-noising the original pulse wave signal: The restored original pulse wave signal is subjected to noise removal using a cascaded uniform phase empirical mode decomposition algorithm and a Butterworth bandpass filter to obtain a denoised pulse wave signal; this composite filtering method can well solve the problem of low signal-to-noise ratio of signals collected by non-contact devices;

[0012] S4. Extract blood oxygen saturation and pulse variability from the denoised pulse wave signal, wherein the blood oxygen saturation is obtained by fitting the pulse waves of two different color channels, and the pulse variability is obtained by extracting the peak-to-peak time interval of the pulse wave; since pulse variability includes the modulation of the pulse by the human sympathetic and parasympathetic nerves, and is directly related to the brain cognitive load, blood oxygen saturation reflects the brain's call for resources under different cognitive loads, and can indirectly reflect the level of cognitive load, so these two physiological indicators are selected for extraction;

[0013] S5. Statistically analyze the above-mentioned blood oxygen saturation, pulse variability and recurring note time respectively, construct the distribution of each indicator, thereby establishing the distribution curve equation corresponding to each indicator, and screen the indicators that obey the normal distribution for subsequent quantitative scoring;

[0014] S6. Collect the facial video of the subject, repeat steps S1-S4, and obtain blood oxygen saturation, pulse variability, and time to reproduce musical notes; repeat step S5 to screen indicators, select indicators that obey the normal distribution, input blood oxygen saturation, pulse variability, and time to reproduce musical notes into the corresponding curve equations, and perform quantitative scoring of cognitive load.

[0015] Furthermore, the process of step S1 is as follows:

[0016] S101. The subject sits in front of a display screen with an external camera. The display screen is controlled by an external intelligent terminal and displays the sequence of notes to be memorized. The camera captures an unobstructed facial video of the subject, and the subject plays the memorized notes on the electronic piano in front of him. The display screen in this step is used to display the sequence of notes to be memorized, and generates cognitive load by stimulating the scheduling of memory resources in the human brain; this step selects simple musical notation symbols to represent the sequence of notes to prevent subjects with zero music theory knowledge from not being able to understand the questions; the facial video captured by the camera in this step is used for subsequent extraction of physiological indicators to measure the level of cognitive load; the electronic piano in this step expands the examination of cognitive load from a single memory resource scheduling to the auditory dimension, which can examine a more comprehensive cognitive load;

[0017] S102, a sequence of notes to be memorized is played on the display screen and a countdown begins. The subject faces the display screen and memorizes the sequence of notes. An external camera records the subject's facial video. This step requires the subject to memorize the sequence of notes within a limited time. If the memorization time is too short, it will cause too much cognitive load, causing the subject to give up sexual behavior. If the memorization time is too long, it will cause too little cognitive load, making it difficult to detect. This step requires the subject to stay still as much as possible to avoid excessive motion noise, which will cause large errors in the extraction results of physiological indicators.

[0018] S103, after the memory time countdown returns to zero, the display automatically switches to the next page, and the subject plays the corresponding note sequence on the electronic piano in front of him, and the subject's note reproduction time is recorded. The note reproduction time recorded in this step reflects the level of cognitive load from the perspective of task performance, avoiding the incompleteness of measuring cognitive load with a single physiological indicator; in this step, the subject needs to reproduce the note sequence within a limited time, and the purpose of setting the limited time is the same as setting the memory time limit in step S102;

[0019] S104, repeating the process of steps S101-S103, and after memorizing, the subject plays the note sequence of the previous round or the previous two rounds. This step requires the subject to repeat multiple rounds of testing, with the difficulty of the note sequence to be memorized in each round of testing being different, so as to generate different levels of cognitive load, increase the types of data that can be evaluated, and verify the universality of the cognitive load measurement method.

[0020] Furthermore, the process of step S2 is as follows:

[0021] S201, dividing the collected facial video into frames and enhancing the brightness: firstly divide the collected facial video into frames to obtain a continuous color picture sequence. The picture format is RGB, where R represents the red channel, G represents the green channel, and B represents the blue channel;

[0022] Then convert the image from RGB color space to YCrCb color space. Based on the theory of positive correlation between brightness and signal-to-noise ratio, multiply the value of Y space containing only brightness information by 1.5 to enhance brightness and increase signal-to-noise ratio. Y represents brightness, Cr represents the difference between the red part of RGB input signal and the brightness value of RGB signal, and Cb represents the difference between the blue part of RGB input signal and the brightness value of RGB signal.

[0023] This step first divides the facial video into frames and cuts the video into continuous images to facilitate subsequent face recognition; this step enhances the image brightness and increases the signal-to-noise ratio based on the theory that the brightness of the facial image is positively correlated with the signal-to-noise ratio of the extracted signal; this step converts the original image from the RGB color space to the YCrCb color space, and only enhances the component Y related to the brightness, avoiding direct enhancement in the RGB color space and destroying the color information related to the extracted pulse wave height;

[0024] S202, face recognition and location of regions of interest: for each frame of the image after brightness enhancement, a regression tree method based on gradient boosting learning is used to detect key points of the face. This method can quickly detect key points of the face and locate the forehead part as the region of interest based on the theoretical basis that the forehead has a high signal-to-noise ratio. This step is based on the theory that different parts of the face have different signal-to-noise ratios. By detecting key points of the face, the forehead with a high signal-to-noise ratio is selected as the region of interest for subsequent pulse wave extraction;

[0025] S203, extracting the original pulse wave from the region of interest: performing channel separation on the image of the region of interest to obtain images of three channels: R, G, and B. Images of different channels contain different information, the G channel is used to extract pulse variability, and the R and B channels are used to extract blood oxygen saturation;

[0026] For each channel image, the idea of ​​reducing noise by using mean filtering is used, and the grayscale mean of all pixels is used as the amplitude of the pulse wave signal. Steps S202-S203 are processed for multiple images of a video, and they are sorted by time to obtain the original pulse wave signals of the three channels;

[0027] This step separates the channels of the image. For pulse variability, which can be extracted with a single channel signal, the G channel with the highest signal-to-noise ratio is selected as the channel for subsequent index extraction. For blood oxygen saturation, which requires signals from two different channels to extract, the R and B channels are selected for blood oxygen saturation extraction based on the theory that hemoglobin has a higher absorption rate for red and blue light;

[0028] S204, compare the grayscale mean with a pre-specified threshold. If it is less than the pre-specified threshold, it is determined that the forehead cannot be used as the region of interest for extracting the pulse wave, and the cheek of the suboptimal solution is selected as the region of interest, and the operation of step S203 is performed to obtain the original pulse wave signals of the three channels. The basis for the threshold judgment in this step is that the grayscale of black hair is low, which will lower the overall grayscale mean, and the hair does not contain pulse information. Therefore, when the grayscale mean is too low, the cheek is selected as the region of interest.

[0029] Furthermore, the process of step S3 is as follows:

[0030] S301, remove the baseline drift of the original pulse wave: Due to the baseline modulation effect of breathing on the pulse wave, the original pulse wave restored in step S2 is subjected to the least square method to remove the linear trend of the pulse wave, complete the baseline drift removal, and obtain the original pulse wave signal without baseline drift. In order to solve the modulation effect of breathing on the pulse wave and avoid the error caused by baseline drift in the subsequent extraction of physiological indicators, this step uses the least square method based on mathematical principles to remove the baseline drift. The least square method obtains the equation of the baseline trend through linear fitting and subtracts it from the original signal;

[0031] S302, using a multi-level uniform phase empirical mode decomposition algorithm to decompose, filter, and reconstruct the original pulse wave signal without baseline drift obtained in step S301 to obtain a low-noise signal;

[0032] The signal-to-noise ratio of non-contact video signals is low, and uniform phase empirical mode decomposition can solve the problem of modal aliasing in traditional decomposition; multi-level decomposition can further subdivide the signal frequency band and extract low-noise pulse waves;

[0033] S303: Input the low-noise signal obtained in step S302 into a Butterworth fifth-order filter to obtain a filtered pulse wave signal. The Butterworth fifth-order filter can filter out high-frequency and low-frequency noise irrelevant to the pulse, and finally output a pure pulse wave signal.

[0034] Furthermore, the process of step S4 is as follows:

[0035] S401, perform peak detection: For the pulse wave signal after denoising in step S3, select the signal of channel G for peak detection, extract the time of all peaks, and use it to extract pulse variability; this is because the time change of the peak can reflect the change of the pulse. This step detects the peak point by judging the relative size of the adjacent data points of the pulse wave signal. The characteristic of the peak point is that the amplitude of the data points on its left and right are smaller than it. The peak point that meets this condition is detected; the time when the peak point appears can be used to extract the pulse variability signal;

[0036] S402, restore the pulse variability signal: differentiate the time of the wave peak to obtain the pulse beat interval sequence, use the cubic spline interpolation method to interpolate the beat interval sequence to obtain the pulse variability signal; the original signal obtained by differentiation in this step is a non-uniform sampling signal, which contains too little information, so the signal is interpolated using the cubic spline interpolation method to obtain a pulse variability signal containing a large amount of information;

[0037] S403, extracting multiple pulse variability indices: extracting features from the pulse variability signal obtained in step S402, preferably selecting seven pulse variability indices, wherein features are extracted in the time domain to obtain a mean value MEAN and a standard deviation SDPP, solving the power spectrum of the pulse variability signal, and extracting frequency domain features to obtain a total signal power TP, a very low frequency power VLF, a low frequency power LF, a high frequency power HF, and a power ratio LF / HF between low and high frequencies; the indices selected in this step are all related to the activities of the sympathetic and parasympathetic nerves of the human body, and have a high correlation with the generation of cognitive load;

[0038] S404, calculate blood oxygen saturation: use the pulse wave signals of the R channel and the B channel, use the blood oxygen saturation empirical formula containing unknown parameters and the measured data of the standard finger-clip oximeter to perform linear fitting to solve the unknown parameters of the blood oxygen saturation empirical formula; use the fitted blood oxygen saturation empirical formula to calculate the current blood oxygen saturation.

[0039] Since the empirical formula of blood oxygen saturation contains two unknown parameters, two different sets of signals must be used for fitting. Therefore, the pulse wave signals of the R channel and the B channel extracted in steps S2 and S3 are used, and the empirical formula of blood oxygen saturation is linearly fitted with the measured data of the standard finger-clip oximeter to solve the unknown parameters of the empirical formula of blood oxygen saturation. The current blood oxygen saturation is calculated using the fitted empirical formula of blood oxygen saturation and the time domain parameters of the pulse wave signals of the R channel and the B channel, including the mean and standard deviation. Blood oxygen saturation can well reflect the human body's use of oxygen resources under different cognitive loads, and can therefore also be used to measure the size of cognitive load.

[0040] Furthermore, the process of step S5 is as follows:

[0041] S501. Draw frequency histograms for the three types of indicators: Draw corresponding frequency histograms for the time of reproducing notes, seven pulse variability indicators, and blood oxygen saturation indicators; among the three types of indicators statistically analyzed in this step, blood oxygen saturation and pulse variability indicators measure the level of brain cognitive load from the perspective of physiological indicators, and the time of reproducing notes measures the level of brain cognitive load from the perspective of task performance. The multi-dimensional evaluation method reduces the misjudgment rate of single indicator evaluation; Special note: The statistical analysis of the three types of indicators is based on indicators obtained by multiple subjects;

[0042] S502, setting the bin distance: according to the data span of different indicators, different histogram bin distances are set to merge the data to avoid distribution errors caused by too large or too small bin distances. Reasonable bin distance setting can avoid errors in solving the equation of the subsequent distribution curve.

[0043] S503, constructing a distribution curve: taking the midpoints of all histograms, connecting them smoothly into a curve as the distribution curve of the current indicator, and obtaining the equation of the distribution curve; since the frequency can be used to replace the probability when the amount of data is large enough, the frequency histogram is used to fit the distribution curve, and the distribution curve is essentially the distribution of the probability density;

[0044] S504, perform normality test: Based on the law of large numbers, most scores obey normal distribution, so the distribution curves of various indicators are tested for normality, and indicators that obey normal distribution are selected as subsequent quantitative scoring indicators. This step uses the idea of ​​normal distribution of scores to test the normality of the distribution curves of various indicators obtained. Indicators that do not satisfy normal distribution cannot map the corresponding brain cognitive load scores.

[0045] Furthermore, the process of step S6 is as follows:

[0046] S601, executing steps S1-S4 for a subject, respectively obtaining the following three types of indicators, namely pulse variability, blood oxygen saturation, and note repetition time; this step is based on the distribution of various indicators constructed in S5, and is the beginning of the quantitative test of brain cognitive load. Two types of physiological indicators are extracted from the facial video of the subject participating in the test: pulse variability and blood oxygen saturation, and task performance indicators such as note repetition time are recorded at the same time. These three types of indicators are used to quantify and score the brain cognitive load;

[0047] S602, according to the result of the normality test in step S5, several indicators that obey the normal distribution are screened out, and for each indicator that obeys the normal distribution, the respective distribution means are mapped into an intermediate score; the indicator measured in step S601 is input into the corresponding distribution curve equation, and a score with the intermediate score as a reference is mapped out according to the distribution curve equation; this step first eliminates the non-normal distribution indicators that cannot be quantified, selects the indicators that obey the normal distribution, and maps the corresponding quantitative scores according to the corresponding distribution curve equation;

[0048] S603, averaging the scores obtained in step S602 to obtain the brain cognitive load score of the current subject; in order to comprehensively judge the brain cognitive load score of the subject, this step performs a weighted average of the quantitative scores mapped by the various indicators in step S602 to obtain the final brain cognitive load quantitative score.

[0049] Compared with the prior art, the present invention has the following advantages and effects:

[0050] 1) The present invention uses physiological indicators to measure the size of the brain's cognitive load, which solves the subjectivity of the cognitive load scale measurement. It uses a non-contact camera to capture facial videos, which solves the discomfort of contact devices to subjects and avoids the interference of objective conditions on the measurement results. At the same time, a scene of playing the piano and memorizing music is designed for experimentation. The display screen is used to display the sequence of notes to be memorized, which stimulates the scheduling of human brain memory resources to generate cognitive load; simple musical notation symbols are selected to represent the sequence of notes to avoid subjects with zero music theory foundation from not being able to understand the questions; the method of combining the completion time of the reproduced notes with physiological indicators to jointly measure cognitive load creates an experimental environment with controllable cognitive load levels and the use of both hands and brains, expanding the examination of cognitive load from a single memory resource scheduling to the auditory dimension, making the examination more comprehensive.

[0051] 2) The present invention proposes a quantitative scoring method for brain cognitive load based on statistics. The blood oxygen saturation, pulse variability series of indicators and the recurrence time of notes in the recording scene calculated by facial video inversion are subjected to histogram statistical analysis and normality test. The idea of ​​fractional normal distribution is used to construct a distribution model, fit the distribution curve, and quantify the score. The results are intuitive and can objectively reflect the level of brain cognitive load. The blood oxygen saturation and pulse variability series of indicators reflect the changes in cognitive load from the perspective of changes in physiological indicators, and the recurrence time of notes indicators reflects the changes in cognitive load from the perspective of task performance, avoiding the incompleteness of single physiological indicators.

[0052] 3) The present invention proposes an improved empirical mode decomposition method, which uses a cascaded uniform phase empirical mode decomposition algorithm to decompose, screen and reconstruct the original pulse wave, which effectively solves the problem of excessive harmonics in non-contact signals, low signal-to-noise ratio, and difficulty in restoring a clean signal. At the same time, the use of a uniform phase empirical mode decomposition algorithm can also effectively solve the problem of modal aliasing in traditional empirical mode decomposition algorithms.

[0053] 4) The present invention proposes an adaptive region of interest selection strategy, which determines whether the forehead of the preferred region of interest is excessively blocked by hair by setting a fixed regional pixel grayscale threshold, and then selects a suitable region of interest to extract high-quality pulse wave signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0055] Figure 1 is a flow chart of a method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos disclosed in Example 1 of the present invention;

[0056] Figure 2 is a flow chart of a strategy for selecting a region of interest in Embodiment 1 of the present invention;

[0057] Figure 3 is a flow chart of an improved empirical mode decomposition algorithm in Example 1 of the present invention;

[0058] Figure 4 is a flow chart of extracting physiological indicators in Example 1 of the present invention;

[0059] Figure 5 is a system block diagram for quantitatively scoring brain cognitive load in Example 1 of the present invention;

[0060] Figure 6 is the original pulse wave waveform diagram in Example 2 of the present invention;

[0061] Figure 7 is a pulse wave waveform diagram after denoising in Example 2 of the present invention;

[0062] Figure 8 This is a blood oxygen saturation distribution curve diagram in Example 2 of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Example 1

[0065] This embodiment discloses a method for measuring brain cognitive load based on extraction of physiological indicators from facial videos. Figure 1 As shown, the specific steps are as follows:

[0066] S1. Collect facial videos of subjects during the process of memorizing notes under different cognitive loads and the time to reproduce notes. Facial videos are used to extract relevant physiological indicators to measure the level of cognitive load from an objective perspective; the time to reproduce notes is the total time it takes for subjects to recall notes and play them completely after the memorization process is completed, which measures the level of cognitive load by reflecting the tension of the brain.

[0067] The subjects watched the sequence of notes on the display screen and memorized them within a limited time. The camera was used to record the unobstructed video of the subject's face while the subject memorized the notes. Different cognitive loads were controlled by increasing the difficulty of memorizing the note sequence. After memorization, the subject was required to play the memorized note sequence on the electronic keyboard and record the time it took to reproduce the notes.

[0068] The note sequence refers to the result of any combination of seven single notes within an octave from the middle note "do" to the middle note "xi";

[0069] The difficulty of memory is determined by measuring the number of notes in the note sequence. The more notes there are, the more difficult it is to remember. The difficulty of memory is linked to the level of cognitive load the brain is currently bearing. By changing the number of notes in the note sequence, different levels of cognitive load are induced. Three notes are defined as low cognitive load, four or five notes are defined as medium cognitive load, and six or seven notes are defined as high cognitive load.

[0070] Table 1. Correspondence between the number of single tones and different levels of brain cognitive load

[0071] Cognitive load level Low middle high Number of notes 3 4、5 6、7

[0072] S2. Locate the region of interest and extract the pulse wave: For the collected facial video, first divide the video into frames to obtain continuous pictures. The pictures are preprocessed by channel transformation and brightness enhancement, and then face recognition is performed using the method of facial feature point detection to locate the region of interest, obtain the grayscale value of the skin pixels in the region of interest, and extract the original pulse wave signal. An adaptive region of interest selection method is designed, using the forehead as the optimal solution and the cheek as the suboptimal solution to deal with the situation where the forehead is covered by hair. Finally, the original pulse waves of the three channels of R, G, and B are obtained for the extraction of different indicators.

[0073] S3. Remove noise from the pulse wave: For the restored original pulse wave signal, first remove the baseline drift, then perform two uniform phase modal decompositions to extract the signal most relevant to the target signal frequency band, and finally input a Butterworth bandpass filter to obtain the denoised pulse wave signal, thereby extracting different physiological indicators.

[0074] S4. Extract physiological indicators from the denoised pulse wave: For the denoised pulse wave signal, extract blood oxygen saturation and pulse variability. The blood oxygen saturation is obtained by fitting the pulse wave signals of the R and B channels, and the pulse variability is obtained by extracting the peak-to-peak time interval of the G channel pulse wave.

[0075] S5. Construct a distribution model and perform a normality test: Conduct statistical analysis on the above-mentioned blood oxygen saturation, pulse variability, and note recurrence time, construct the distribution of each indicator, and thus establish the distribution curve equation corresponding to each indicator, and select indicators that obey the normal distribution for quantitative scoring.

[0076] S6. Quantitative scoring: Collect the face video of the subject, repeat steps S1-S4 to obtain three types of indicators: blood oxygen saturation, pulse variability, and note recurrence time. Repeat step S5 to screen indicators, select indicators that obey the normal distribution, input blood oxygen saturation, pulse variability, and note recurrence time into the corresponding curve equation, and perform quantitative scoring of cognitive load. Average the scoring results of all indicators to obtain the subject's brain cognitive load score.

[0077] The specific process of the above-mentioned image preprocessing and adaptive region of interest selection is as follows Figure 2 Shown:

[0078] S201, frame the collected facial video and enhance the brightness: first frame the collected facial video to obtain a continuous color image sequence, the image format is RGB. R represents the red channel, G represents the green channel, and B represents the blue channel. Then convert the image from the RGB color space to the YCrCb color space, multiply the value of the Y space by 1.5, enhance the brightness, increase the signal-to-noise ratio, and then convert the enhanced image back to the RGB color space. Y represents brightness, Cr represents the difference between the red part of the RGB input signal and the brightness value of the RGB signal, and Cb represents the difference between the blue part of the RGB input signal and the brightness value of the RGB signal.

[0079] The conversion formula between RGB color space and YCrCb color space is as follows:

[0080] Y=0.257*R+0.504*G+0.098*B+16 Formula (1)

[0081] Cb=-0.148*R-0.291*G+0.439*B+128 Formula (2)

[0082] Cr=0.439*R-0.368*G-0.071*B+128 Formula (3)

[0083] S202, recognizing faces and locating regions of interest: for each frame of the image after brightness enhancement, a regression tree method based on gradient boosting learning is used to detect key points of the face, and a traditional 68-feature point face template is used to detect key points of the face, and the forehead is located as the region of interest.

[0084] S203, extracting the original pulse wave from the region of interest: performing channel separation on the image of the region of interest to obtain images of three channels: R, G, and B. Images of different channels contain different information, the G channel is used to extract pulse variability, and the R and B channels are used to extract blood oxygen saturation;

[0085] For each channel image, the grayscale mean of all pixels is used as the amplitude of the pulse wave signal. S202-S203 are processed for multiple images of a video and sorted by time to obtain the original pulse wave signals of three channels.

[0086] S204. Compare the grayscale mean with a pre-specified threshold value, which is set to 30. If it is less than 30, it is determined that the region cannot be used as the region of interest for extracting the pulse wave. The cheek of the suboptimal solution is selected as the region of interest, and the operation of step S203 is performed to obtain the original pulse wave signals of the three channels.

[0087] The denoising process of the original signal is as follows: Figure 3 As shown:

[0088] S301, removing baseline drift: for the original pulse wave restored in step S2, the linear trend of the pulse wave is removed using the least square method to complete the baseline drift removal and obtain the original pulse wave signal without baseline drift.

[0089] S302, denoising using cascaded uniform phase empirical mode decomposition algorithm: using uniform phase empirical mode decomposition to perform two-level decomposition on the pulse wave signal without baseline drift, calculating the correlation coefficient between each layer of the first-level decomposition signal and the original signal, and selecting the top three signals for superposition according to the size of the correlation coefficient. Perform a second uniform phase empirical mode decomposition on the superimposed signal, and select the signal with a frequency band of 1-4Hz as the final output pulse wave signal according to the result of Hilbert transform;

[0090] The uniform phase empirical mode decomposition method is an improved version of the classical empirical mode decomposition algorithm, that is, an improved version of the EMD algorithm. The execution steps of the EMD algorithm are as follows:

[0091] Find all the maximum and minimum points of the original signal x(t);

[0092] Use the cubic spline interpolation method to fit all extreme points to obtain the maximum point signal and the minimum point signal, and average these two signals to obtain m(t);

[0093] Calculate the difference signal between the original signal x(t) and m(t), as shown in the following formula:

[0094] h(t)=x(t)-m(t) Formula (4)

[0095] Determine whether h(t) has reached the convergence condition. There are generally three convergence conditions: m(t) is close to 0, the difference between the number of local extreme points and zero-crossing points of h(t) does not exceed 1, and the set iteration threshold is reached. As long as one of the three conditions is met, it is considered that the convergence condition has been reached;

[0096] If the convergence condition is not reached, the previous operation is repeated, otherwise the intrinsic component IMF1 = m(t) and the residual res1 = h(t) are obtained;

[0097] Continue to use the residual res1 as the new input signal and repeat the previous steps until the entire decomposition process is completed;

[0098] The decomposed signal is in the form of:

[0099]

[0100] Where i represents the i-th group of decomposed signals, and N represents the total number of decomposed signals into N groups;

[0101] Since EMD decomposition of the signal will lead to the problem of modal aliasing, the uniform phase EMD algorithm is used here to decompose the original pulse wave. Its basic idea is the same as EMD, except that a series of uniformly distributed sine functions are added to the original signal as perturbations, and the mean of the perturbed EMD results is calculated as the result of each layer of decomposition. The perturbation function is designed as follows:

[0102] w=ε m ·cos(2π·f ω ·t+θ k ) Formula (6)

[0103] f ω =1 / 2 m Formula (7)

[0104] θ k =2π(k-1)n imf Formula (8)

[0105] n imf =log2n formula (9)

[0106] Among them, w is the perturbation function, ε m represents the standard deviation of the residual signal, f ω represents the frequency of the current signal, θ k Indicates the signal phase, m = 1:n imf , k=1,2,3,…,n imf , n represents the length of the current signal.

[0107] S303, the low-noise signal obtained in step S302 is input into a Butterworth fifth-order filter, whose upper and lower cut-off frequencies are 3 Hz and 1 Hz respectively, corresponding to the normal heart rate range of 60-180 beats / min, to obtain a filtered pulse wave signal.

[0108] The flow chart of pulse variability and blood oxygen extraction process is as follows Figure 4 The specific implementation process is as follows:

[0109] S401. After the pulse wave signal has been denoised in step S3, the signal of the G channel is selected for peak detection to extract continuous peaks, requiring that the peak amplitude is greater than 0 and the peak-to-peak interval is greater than 0.5 seconds.

[0110] S402, performing differentiation on continuous peak-to-peak values ​​to obtain a pulse inter-beat interval sequence, and using a cubic spline interpolation method to interpolate the inter-beat interval sequence to obtain a pulse variability signal.

[0111] S403, extracting multiple pulse variability indices: performing time domain feature extraction on the pulse variability signal obtained in step S402, preferably selecting seven pulse variability indices, wherein features are extracted in the time domain to obtain a mean MEAN and a standard deviation SDPP, performing power spectrum solution on the pulse variability signal, and extracting frequency domain features to obtain a total signal power TP, a very low frequency power VLF, a low frequency power LF, a high frequency power HF, and a power ratio LF / HF between the low frequency and high frequency bands.

[0112] S404, using the pulse wave signals of the R channel and the B channel extracted in steps S2-S3, and performing linear fitting using the blood oxygen saturation empirical formula and the measured data of the standard finger-clip oximeter, to obtain the unknown parameters in the blood oxygen saturation empirical formula. The current blood oxygen saturation is calculated using the fitted blood oxygen saturation empirical formula and the pulse wave parameters of the R channel and the B channel;

[0113] The empirical formula for blood oxygen saturation is:

[0114] SPO2=A+K*X Formula (10)

[0115] SPO2 represents blood oxygen saturation, A and K are constants obtained by linear fitting, and X is the independent variable, which is calculated by the R channel pulse wave and the B channel pulse wave;

[0116] The calculation formula for X is:

[0117]

[0118] where std r Indicates the standard deviation of the R channel, std b Indicates the standard deviation of the B channel, mean b Indicates the mean of the B channel, mean r Represents the mean of the R channel.

[0119] The flowchart for the construction of the indicator distribution model and the quantitative scoring of cognitive load is as follows: Figure 5 The specific implementation process is as follows:

[0120] S501, drawing a frequency histogram: drawing corresponding frequency histograms for the recurring note time, seven pulse variability indices, and blood oxygen saturation sub-indices.

[0121] S502, setting group distance: setting different histogram group distances according to the data spans of different indicators to merge the data;

[0122] The formula for determining the group distance is as follows:

[0123] Group distance = range / number of groups formula (12)

[0124] The range represents the difference between the maximum and minimum values ​​within a group, and the number of groups is set to 8.

[0125] S503, constructing a distribution curve: taking the midpoints of all histograms, connecting them smoothly into a curve as the distribution curve of the current indicator, and obtaining the equation of the distribution curve.

[0126] S504. Perform normality test: Based on the law of large numbers, most scores obey normal distribution. Therefore, the distribution curve of each indicator is tested for normality, and indicators that obey normal distribution are selected as subsequent quantitative scoring indicators.

[0127] The normality test uses the skewness-kurtosis test method, where skewness refers to the measurement of the direction and degree of skewness of the statistical data distribution. Skewness equals 0, indicating that it is consistent with the normal distribution; kurtosis reflects the sharpness of the peak, and kurtosis equals 3, indicating that it is consistent with the normal distribution. When the skewness is -0.5 to 0.5 and the kurtosis is 2.5-3.5, it is considered to obey the normal distribution.

[0128] S601, calculate the measured indicators: perform steps S1-S4 on a subject to obtain the following three indicators, namely pulse variability, blood oxygen saturation, and note recurrence time.

[0129] S602, quantitatively scoring a single indicator: According to the results of the normality test in step S5, multiple indicators that obey the normal distribution are screened out. For each indicator that obeys the normal distribution, the respective distribution mean is mapped to 80 points. The indicator measured in step S601 is input into the corresponding distribution curve equation, and a score with 80 points as a reference is mapped according to the distribution curve equation; the indicator outside the left boundary of the Laida criterion is set to 0 points, and the indicator outside the right boundary of the Laida criterion is set to 100 points.

[0130] S603. Averaging the scores obtained in step S602 to obtain the brain cognitive load score of the current subject.

[0131] Example 2

[0132] This embodiment discloses a method for measuring brain cognitive load based on extraction of physiological indicators from facial videos. Taking blood oxygen saturation as a reference indicator, the process of using blood oxygen saturation as a physiological indicator to score brain cognitive load is described. The specific steps are as follows:

[0133] S1. The subject sits in front of a display screen with an external camera. The display screen is controlled by an external intelligent terminal and displays the sequence of notes to be memorized. The camera captures the subject's unobstructed facial video. The subject plays the memorized notes on the electronic piano set in front of him.

[0134] A sequence of notes to be memorized is played on the display screen and a countdown begins. The subject faces the display screen and memorizes the sequence of notes. The subject is asked to keep still, try not to shake his head, and not to cover his face with his hands or other objects. An external camera will record the subject's facial video. After the memorization is over, the subject has 60 seconds to play the memorized sequence of notes.

[0135] After one round, the next round began and the process of memorizing the note sequence and playing it was repeated. Each subject underwent five rounds of testing, and facial videos of the memorized note sequence were recorded in the first, third, and fifth rounds.

[0136] S2. Frame the collected facial video of the subject to obtain a continuous facial image, convert the image from the RGB color space to the YCrCb color space, and perform brightness enhancement. The enhancement operation is to multiply the Y channel value by 1.5 times. For the enhanced image, select the forehead as the region of interest, separate this region into three channels of R, G, and B, and for each channel, average the grayscale values ​​of all pixels to obtain the amplitude of the three channel signals.

[0137] All the images obtained by frame division are sorted by time and the above operation is performed to obtain the original pulse wave signals of the three channels of R, G, and B, as shown in Figure 6 shown.

[0138] S3. Use the improved uniform mode decomposition algorithm to decompose each signal, select the components of the fourth, fifth and sixth layers to reconstruct the signal, and then input a bandpass filter with a passband frequency of 1Hz-3Hz to obtain the filtered pulse wave signal, such as Figure 7 shown.

[0139] S4. Since only the extraction of blood oxygen saturation is discussed, the signals of the R channel and the B channel are used to fit and calculate the blood oxygen saturation. The equation to be fitted is shown in formula (10) of Example 1. According to the fitting results of the measured data and the blood oxygen saturation data collected by the oximeter, the fitted equation is:

[0140] SPO2=98.9397-1.6853*X formula (13)

[0141] For the denoised pulse wave signal restored from a facial video of a subject, X is calculated according to formula (11) of Example 1 to be 0.9492, and the blood oxygen saturation is 97.34%.

[0142] S5. Draw a frequency histogram based on the blood oxygen saturation of all subjects, and use the normal distribution to fit the distribution curve, such as Figure 8 shown.

[0143] S6. The curve follows a normal distribution with a mean of 97.2494 and a standard deviation of 0.1592. Therefore, the score corresponding to the mean of 97.2494 is set to 80 points. According to the curve equation, the score mapping table is as follows:

[0144] Table 2. Mapping table of blood oxygen saturation and cognitive load score

[0145]

[0146] According to the score mapping table, the subject with a blood oxygen saturation of 97.34% had a brain cognitive load score of 90 points.

[0147] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos, characterized in that: The method for quantifying brain cognitive load comprises the following steps: S1. Collect facial videos of subjects while memorizing musical note sequences under different cognitive loads, and record the time of reproducing musical notes; wherein the cognitive load is the magnitude of cognitive pressure on the subject's brain, and different cognitive load levels are divided by memorizing musical note sequences of different lengths; The note repetition time is the total time for the subject to recall the notes and play them completely after the memory process is completed. The note repetition time measures the cognitive load level by reflecting the tension of the brain; the note sequence refers to any combination of seven single notes within an octave from the middle note "do" to the middle note "xi"; S2. Perform face recognition on the collected face video, locate the region of interest, obtain the grayscale value of the skin pixel in the region of interest, and extract the original pulse wave signal; wherein the region of interest is the face region used to extract the pulse wave signal; the process is as follows: S201, firstly divide the collected facial video into frames to obtain a continuous color picture sequence, the format of the picture is RGB, where R represents the red channel, G represents the green channel, and B represents the blue channel, and then convert the picture from the RGB color space to the YCrCb color space to enhance the brightness, where Y represents brightness, Cr represents the difference between the red part of the RGB input signal and the brightness value of the RGB signal, and Cb represents the difference between the blue part of the RGB input signal and the brightness value of the RGB signal; S202, for each frame of the image after brightness enhancement, using a regression tree method based on gradient boosting learning to detect key points of the face, and locating the forehead part based on the key points of the face, taking the forehead part as the region of interest; S203, performing channel separation on the image of the region of interest to obtain images of three channels, namely, R, G, and B; for the image of each channel, using the grayscale mean of all pixels as the amplitude of the pulse wave signal; performing the same processing on multiple images of a video frame, and sorting them according to time to obtain original pulse wave signals of three channels; S204, comparing the grayscale mean with a pre-specified threshold. If the grayscale mean is less than the pre-specified threshold, the forehead is determined to be unsuitable for extracting the pulse wave. The cheek of the suboptimal solution is selected as the region of interest. The operation of step S203 is performed to obtain the original pulse wave signals of the three channels. S3, using a cascaded uniform phase empirical mode decomposition algorithm and a Butterworth bandpass filter to remove noise from the restored original pulse wave signal to obtain a denoised pulse wave signal; S4. Extracting blood oxygen saturation and pulse variability from the denoised pulse wave signal, wherein the blood oxygen saturation is obtained by fitting the pulse waves of two different color channels, and the pulse variability is obtained by extracting the peak-to-peak time interval of the pulse wave, and the process is as follows: S401, for the pulse wave signal after denoising in step S3, select the signal of the G channel to perform peak detection, extract the time of all peaks, and use it to extract the pulse variability; S402, differentiating the time of the wave peaks to obtain a pulse inter-beat interval sequence, and interpolating the inter-beat interval sequence using a cubic spline interpolation method to obtain a pulse variability signal; S403, extracting features from the pulse variability signal obtained in step S402, preferably selecting seven pulse variability indices, wherein features are extracted in the time domain to obtain a mean value MEAN and a standard deviation SDPP, solving the power spectrum of the pulse variability signal, and extracting frequency domain features to obtain a total signal power TP, a very low frequency power VLF, a low frequency power LF, a high frequency power HF, and a power ratio LF / HF between a low frequency and a high frequency; S404, using the pulse wave signals of the R channel and the B channel, using the blood oxygen saturation empirical formula containing unknown parameters and the measured data of the standard finger-clip oximeter to perform linear fitting to solve the unknown parameters of the blood oxygen saturation empirical formula; using the fitted blood oxygen saturation empirical formula to calculate the current blood oxygen saturation; S5. Statistically analyze the above-mentioned blood oxygen saturation, pulse variability and recurring note time respectively, construct the distribution of each indicator, thereby establishing the distribution curve equation corresponding to each indicator, and screen the indicators that obey the normal distribution for subsequent quantitative scoring; S6. Collect the facial video of the subject, repeat steps S1-S4, and obtain blood oxygen saturation, pulse variability, and time to reproduce musical notes; repeat step S5 to screen indicators, select indicators that obey the normal distribution, input blood oxygen saturation, pulse variability, and time to reproduce musical notes into the corresponding curve equations, and perform quantitative scoring of cognitive load.

2. The method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos according to claim 1, characterized in that: The process of step S1 is as follows: S101, the subject sits in front of a display screen connected to an external camera, the display screen is controlled by an external intelligent terminal, and displays a sequence of musical notes to be memorized. The camera collects an unobstructed facial video of the subject, and the subject plays the memorized musical notes on an electronic keyboard set in front of him; S102, a sequence of notes to be memorized is played on the display screen and a countdown begins, the subject faces the display screen and memorizes the sequence of notes, and an external camera records a video of the subject's face; S103, after the memory time countdown returns to zero, the display automatically switches to the next page, and the subject plays the corresponding note sequence on the electronic piano in front of him, and the time for the subject to reproduce the notes is recorded; S104, repeating the process of steps S101-S103, and after memorizing, the subject plays the note sequence of the previous round or the previous two rounds.

3. The method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos according to claim 1, characterized in that: The process of step S3 is as follows: S301, for the original pulse wave restored in step S2, the linear trend of the pulse wave is removed by using the least square method, the baseline drift is removed, and the original pulse wave signal without baseline drift is obtained; S302, using a multi-level uniform phase empirical mode decomposition algorithm to decompose, filter, and reconstruct the original pulse wave signal without baseline drift obtained in step S301 to obtain a low-noise signal; S303: Input the low-noise signal obtained in step S302 into a Butterworth fifth-order filter to obtain a filtered pulse wave signal.

4. The method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos according to claim 1, characterized in that: The process of step S5 is as follows: S501, plotting corresponding frequency histograms for the recurring note time, seven pulse variability indices, and blood oxygen saturation sub-indices; S502, according to the data span of different indicators, set different histogram group intervals to merge the data; S503, taking the midpoints of all histograms, connecting them into a smooth curve as the distribution curve of the current indicator, and obtaining the equation of the distribution curve; S504. Perform a normality test on the distribution curve of each indicator, and select indicators that obey the normal distribution as subsequent quantitative scoring indicators.

5. The method for quantifying brain cognitive load based on extraction of physiological indicators from facial videos according to claim 1, characterized in that: The process of step S6 is as follows: S601, performing steps S1-S4 on a subject to obtain the following three indicators, namely pulse variability, blood oxygen saturation, and note recurrence time; S602, according to the result of the normality test in step S5, several indicators that obey the normal distribution are screened out, and for each indicator that obeys the normal distribution, the respective distribution means are mapped into a middle score; the indicator measured in step S601 is input into the corresponding distribution curve equation, and a score with the middle score as a reference is mapped out according to the distribution curve equation; S603. Averaging the scores obtained in step S602 to obtain the brain cognitive load score of the current subject.

Images