Working memory load level identification method based on facial video

Abstract

The present invention discloses a method for identifying working memory load level based on facial video. The method collects facial videos under different working memory loads through a camera, calculates multiple indicators such as heart rate, performs feature selection and load level classification on the indicators, and identifies the working memory load level. The steps are as follows: display the questions on the screen, and continuously increase the difficulty of memory; starting from the second round of testing, the testee memorizes the questions of the current round and the previous round, and writes down the questions of the previous round; at the beginning, middle and end of the test, collects facial videos through a camera, extracts pulse wave signals, performs signal reconstruction and denoising, calculates multiple indicators such as heart rate, performs feature selection and load level classification, and identifies the working memory load level. The present invention can provide a non-destructive and non-invasive convenient means for objectively identifying the working memory load level.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a working memory load level recognition method based on facial video. Background Art

[0002] Memory, as a key topic in cognitive science research, is mainly divided into two categories: long-term memory and short-term memory. Compared with long-term memory, short-term memory has a shorter storage time for information and a more limited information storage capacity. Working memory comes from the concept of short-term memory. It is a memory system with limited capacity that temporarily processes and stores information. It is used to temporarily maintain and store information. It is an energy-based temporary memory with a short storage time and limited capacity, like the memory of a computer, which disappears when the power is off.

[0003] The current methods for measuring working memory load are mainly divided into two categories: subjective methods and objective methods. Questionnaires, inquiries and other subjective methods for evaluating working memory load have shortcomings such as subjective tendency and time lag. Objective detection methods such as electrocardiogram, electroencephalogram, skin galvanic response, nuclear magnetic resonance and brain magnetism have high accuracy, but at the same time, due to the characteristics of expensive equipment, large equipment size, and long data collection time, they cannot be detected in real time, and require users to wear cumbersome equipment, which is highly invasive and difficult to promote and use in real production and life fields outside of clinical practice. For example, in the paper "Measuring Load on Working Memory: The Use of Heart Rate as a Means of Measuring Chemistry Students' Cognitive Load", the experiment designed chemistry questions representing various load levels, requiring the subjects to wear heart rate monitors while doing chemistry questions of various difficulty levels, record the time and accuracy of the questions, and measure the correlation between heart rate changes and load levels. However, compared with Arabic numerals, the chemistry questions used in the test are less universal and are not suitable for measurements of non-chemistry majors; the experimental results only proved the significant correlation between heart rate changes and load levels, but did not accurately identify the load levels.

[0004] Therefore, a measurement method that uses non-contact devices to measure physiological signals and extract corresponding physiological parameters from physiological signals to measure the load level has emerged. For non-contact devices, low cost and easy deployment are more practical requirements. In the non-contact measurement system using a camera, extracting physiological parameters from facial videos to measure different load levels has become the focus of research. Therefore, it is of great practical significance to explore a non-contact identification scheme for working memory load level that is non-destructive, non-invasive, scientific and simple, and breaks away from existing contact devices. Summary of the invention

[0005] The purpose of the present invention is to improve the subjectivity of traditional working memory load identification methods and the inconvenience of contact device measurement, and to provide a working memory load level measurement method based on facial video.

[0006] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0007] A working memory load level recognition method based on facial video is intended to improve the subjectivity of traditional working memory load level recognition methods and the inconvenience of contact devices. The recognition method comprises the following steps:

[0008] S1. Setting the working memory load level and setting questions; the working memory load level is set to three different levels, namely: high load, medium load and low load level; setting questions means designing different digital memory questions according to different load levels;

[0009] S2. Design digital memory rules, and the subjects will take working memory load test according to the digital memory rules, complete memory questions, and calculate the memory accuracy of the subjects;

[0010] S3, using a camera to collect facial video of the subject when he / she is memorizing numbers during a specified memory period; wherein the specified memory period refers to the beginning, middle and near the end of the working memory load test;

[0011] S4, performing face detection and face segmentation according to the captured facial video, taking the segmented face as the region of interest and performing channel separation, performing signal decomposition and reconstruction on the separated G channel, and extracting the pulse wave using a wavelet denoising algorithm and a bandpass filter;

[0012] S5. Perform time domain and frequency domain analysis on the extracted pulse wave to calculate the heart rate and heart rate variability indexes;

[0013] S6. Give several subjects questions with different working memory load levels, repeat steps S1 to S5, obtain the subjects' memory accuracy, heart rate, and heart rate variability indicators, form a data set required for model construction, and then perform feature selection and load level classification based on the data set required for model construction to construct a working memory load physiological model; the data set required for model construction is composed of memory accuracy, heart rate, and heart rate variability indicators collected from multiple subjects; based on the data set required for model construction, perform feature selection and load level classification to construct a working memory load physiological model;

[0014] S7. In the actual measurement of the working memory load of the subject, the memory accuracy, as well as the heart rate and heart rate variability indicators extracted from the collected facial video are input into the working memory load physiological model to identify the current working memory load level of the subject.

[0015] Furthermore, the rules for designing different digital memory questions according to different load levels are as follows:

[0016] For low-load questions, there are 5 questions with 6 or 7 digits, and the digit memory time is set to 10 seconds; for medium-load questions, there are 5 questions with 8 digits, and the digit memory time is set to 15 seconds; for high-load questions, there are 6 questions with 9 or 10 digits, and the digit memory time is set to 20 seconds. This step is to induce different levels of working memory load. The difficulty of memory is increased by increasing the number of memorized digits. The higher the difficulty, the more tense the brain nerves, thus inducing different levels of working memory load; in order to avoid erroneous test results caused by misunderstanding the meaning of the questions or too high professional requirements, questions based on numbers are chosen. Arabic numerals are highly universal, have low requirements on the professional level of the testees, and can reduce the interference caused by irrelevant factors during the test.

[0017] Furthermore, the process of step S2 is as follows:

[0018] S201. Design the rules for the working memory load test and make the test subjects answer the questions according to the rules; the test rules for generating different working memory load levels according to the number memory questions are as follows: display the number questions on the screen, and continuously increase the difficulty of remembering the number questions; starting from the second round of testing, remember the numbers of the current round and the previous round, and write down the numbers of the previous round on the computer. In order to ensure the continuous variability of the working memory load, this step designs the rules for answering the questions to apply a continuous and variable parameter load on the working memory, which can continuously stimulate the brain, thereby ensuring the scientific nature of the test;

[0019] S202, record the numbers written by the subject on the computer, and calculate the subject's memory accuracy by comparing with the questions. This step takes into account the limited capacity of working memory. Usually, the memory of one thing will cover another thing, which is prone to memory loss. Therefore, the memory accuracy is regarded as an important indicator for identifying the working memory load level.

[0020] Furthermore, the process of step S3 is as follows:

[0021] The designated memory period refers to the period at the beginning, middle and near the end of the working memory load test. The designed working memory test requires the testee to write down 5 rounds of numbers. In the digital memory period of the first test in the first round, the digital memory period of the third test in the third round, and the digital memory period of the third test in the fifth round, facial videos were taken under sufficient light conditions at a distance of 40 to 45 cm, and the shooting time was 10 seconds, 15 seconds and 20 seconds respectively. This step is based on the continuous change of working memory load, and the designated memory period corresponds to low load, medium load and high load working memory scenarios respectively.

[0022] Furthermore, the process of extracting the pulse wave from the captured facial video in step S4 is as follows:

[0023] S401, dividing the video into frames: calculating the frame rate of the captured video, and dividing the video into frames according to the frame rate, so as to divide the video into image sequences;

[0024] S402, perform face segmentation and select region of interest: for the image sequence, detect the face area and segment it according to the face edge, remove the area irrelevant to the face, and use the segmented face as the region of interest; this step takes into account that the selection of the region of interest is the key to extracting the pure pulse wave signal, selects the segmented face as the region of interest, improves the signal-to-noise ratio, and avoids excessive errors in the calculation of heart rate and heart rate variability indicators caused by the poor quality of the selected region of interest.

[0025] S403, perform channel separation and select the G channel signal as the initial pulse wave signal: perform channel separation on each image in the image sequence, divide it into three color channels of R, G, and B, and select the G channel signal as the initial pulse wave signal; in order to extract the initial pulse wave signal, this step takes into account that the G channel is most sensitive to changes in heart rate and the blood has the strongest absorption capacity for green light, so the G channel signal is selected as the initial pulse wave signal.

[0026] S404, decomposing and reconstructing the G channel signal: decomposing the G channel signal into multiple components, and calculating the Pearson correlation coefficient between each component and the original G channel signal; selecting several components with the highest correlation coefficient and reconstructing the signal; this step takes into account that the G channel signal contains too much high-frequency noise, and the decomposition and reconstruction of the signal can produce a good effect on removing the high-frequency noise.

[0027] S405, performing wavelet denoising on the reconstructed signal; this step takes into account that the reconstructed signal still contains low-frequency noise such as environmental and power frequency, and wavelet denoising has an obvious effect on removing low-frequency noise.

[0028] S406, after filtering the signal after wavelet denoising, the pulse wave of the captured facial video is obtained. The signal after wavelet denoising is filtered using a Butterworth 3rd order bandpass filter, wherein the bandpass frequency band is set to 0.8-4 Hz; this step takes into account that under normal circumstances, the human heart rate range is 50-240 times per minute, so the bandpass frequency band is set to 0.8-4 Hz, and filtering is performed to finally obtain the pulse wave signal extracted according to the captured facial video.

[0029] Furthermore, the process of calculating the heart rate and heart rate variability according to the obtained pulse wave signal in step S5 is as follows:

[0030] S501, calculating the heart rate: according to the obtained pulse wave signal, convert the signal into the frequency domain, find the frequency corresponding to the highest peak of the amplitude spectrum, which is the number of heartbeats per second of the subject; multiply the obtained number of heartbeats per second by 60, which is the heart rate;

[0031] S502. Calculate the heart rate variability index based on the obtained pulse wave time domain signal, wherein the heart rate variability index includes but is not limited to the standard deviation of heart rate interval SDNN, the standard deviation of the difference between adjacent heart beat intervals SDSD, the root mean square of the difference between adjacent heart beat intervals RMSSD, the mean heart rate Mean_HR, the maximum heart rate Max_HR, the minimum heart rate Min_HR, the standard deviation of heart rate STD_HR, the total signal power TP, the low frequency band power LF, the high frequency band power HF, the ratio of low frequency band to high frequency band power LF / HF, and the very low frequency band power VLF.

[0032] Furthermore, the process of step S6 is as follows:

[0033] S601, labeling the collected data set, wherein the low load label is 0, the medium load label is 1, and the high load label is 2, to obtain the data set required for model construction; this step takes into account the subsequent steps of working memory load level classification and recognition accuracy calculation, and labels the working memory load level, thereby obtaining label data representing different working memory load levels;

[0034] S602, splitting the data set into a training set and a test set; wherein the training set is used to train the model, and the test set is used to test the model;

[0035] S603, normalize the indicators in the training set. Assume that the amount of data in the training set is an m×n feature matrix F, and the elements in F are all finite real numbers X i , and the data in each row are not equal. The data in the test set is a k×n feature matrix t s , then the normalized element Y j for:

[0036]

[0037] In the formula, the normalized range is set to [-1,1], then Y max The value is 1, Y min The value is -1, max() and min() represent the maximum and minimum values ​​respectively; this step takes into account that the extreme maximum and minimum values ​​in the training set are likely to affect the training of subsequent models, and normalizes the indicators in the training set to reduce the impact of the extreme maximum and minimum values ​​on the model.

[0038] S604: Normalize the indicators in the test set. Use the indicators in the training set as a mapping and normalize the indicators in the test set. s Perform the same normalization operation as above, and the normalized test set Ts obtained is:

[0039]

[0040] Where Y max The value is 1, Y min The value is -1, max() and min() represent the maximum and minimum values ​​respectively; this step is to prevent the indicators in the test set from carrying the information in the training set, which may lead to overfitting of the model training results. The indicators in the training set are used as mapping and normalized, thereby reducing the risk of overfitting in model training.

[0041] S605, integrating the normalized training set and test set together to obtain a normalized data set;

[0042] S606, feature selection is performed on the normalized data set to obtain a feature subset; feature selection is to treat each indicator as a feature, eliminate irrelevant features from several indicators such as memory accuracy, heart rate, and heart rate variability, select appropriate features, and form a feature subset. This step takes into account the problem that too many features are prone to dimensionality disasters, and feature selection can remove irrelevant features, reduce the difficulty of learning tasks in model training, and thus improve the efficiency of the model;

[0043] S607, construct a physiological model of working memory load. First, the normalized data set is used for training. The key step of training is to use the classifier for classification and perform K-fold cross-validation on the classification results. The purpose of training is to determine the hyperparameters of the classifier. After the hyperparameters are determined, the physiological model of working memory load is constructed. In order to evaluate the performance of the model, this step uses K-fold cross-validation to verify the classification results, which can reduce the risk of overfitting to a certain extent. In addition, this step takes into account that the setting of hyperparameters will affect the efficiency of the model, and trains the hyperparameters for the classifier to obtain the most ideal working memory load recognition accuracy for the classifier.

[0044] Furthermore, the step S6 further includes:

[0045] The hyperparameters obtained through training and the feature subsets obtained through feature selection are input into the classifier and tested on the test set to judge the recognition accuracy of the physiological model of working memory load. This step is to solve the problem of identifying the level of working memory load. Through accurate classification by the classifier, a mapping relationship between multiple indicators in the feature subset and the level of working memory load is established, thereby identifying the level of working memory load.

[0046] Furthermore, the process in step S7 is as follows:

[0047] S701, the subject memorizes the digital memory questions representing different loads according to the rules, writes down the results of the questions on the computer, and calculates the memory accuracy;

[0048] S702, collecting face videos for the memory period and extracting pulse waves;

[0049] S703, extracting heart rate and heart rate variability indicators through the pulse wave;

[0050] S704, normalizing and feature selecting the memory accuracy, heart rate and heart rate variability indicators, and then inputting them into the working memory load physiological model to identify the current working memory load level of the subject.

[0051] Compared with the prior art, the present invention has the following advantages and effects:

[0052] 1) The present invention extracts physiological parameters from pulse waves based on facial videos to measure the working memory load level. It is an objective, non-contact method, which improves the subjective tendency, time lag and other deficiencies existing in subjective evaluation methods such as questionnaires and inquiries, and the inconvenience caused by contact measurement in objective evaluation methods.

[0053] 2) In order to induce different levels of working memory load, the present invention adopts a digital memory test, and the questions are highly universal and can easily cause the subjects to produce a continuous working memory load with variable parameters. At the same time, the present invention takes into account the limited capacity of working memory and calculates the memory accuracy index closely related to working memory, thereby realizing a scientific and reliable working memory load level identification scheme.

[0054] 3) The present invention segments the face, removes areas irrelevant to the face, and uses the segmented entire face as the region of interest, thereby avoiding excessive errors in the calculation of physiological indicators caused by the poor quality of the selected region of interest; the present invention combines signal reconstruction and wavelet denoising algorithms to denoise the signal, removes the noise existing in the pulse wave signal, extracts high-quality pulse waves, and improves the accuracy of the extracted physiological indicators.

[0055] 4) The present invention regards each indicator as a feature and performs feature selection on the extracted indicators, thereby reducing the impact of feature redundancy, ensuring the correlation between the feature subset and the working memory load, and improving the accuracy of the classifier for multi-classification; the present invention uses a classifier to classify different working memory loads, establishes a mapping relationship between multiple features and working memory loads, and realizes objective identification of the working memory load level. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0057] Figure 1 It is a flow chart of a method for identifying working memory load level based on facial video disclosed by the present invention;

[0058] Figure 2 Schematic diagram of the LinkNet network architecture used in face segmentation in the present invention;

[0059] Figure 3 It is a schematic diagram of the encoder and decoder structure of the LinkNet network in the present invention;

[0060] Figure 4 It is a flow chart of performing signal processing on the G channel signal to obtain the pulse wave signal in the present invention;

[0061] Figure 5 is a flow chart of the CEEMDAN algorithm of the present invention;

[0062] Figure 6 is a flow chart of the random forest algorithm in the present invention;

[0063] Figure 7 is a schematic diagram of an initial pulse wave signal of a subject in Example 2 of the present invention, that is, a G channel signal after channel separation;

[0064] Figure 8 is a schematic diagram of a pulse wave signal of a subject after improved wavelet threshold denoising in Example 2 of the present invention;

[0065] Fig. 9 3 is a graph showing the relationship between the cross-validation mean and the number of decision trees obtained through training in Example 2 of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example 1

[0068] Figure 1 : is a flow chart of a method for identifying working memory load level based on facial video provided by an embodiment of the present invention. This embodiment is implemented in a piano playing scenario and can realize non-contact identification of working memory load level. The specific steps are as follows:

[0069] S1. Set the working memory load level and set questions. The working memory load level is set to three different levels: high load, medium load and low load level; design different music score questions according to different load levels, where the music score is represented by Arabic numerals. The specific steps are as follows:

[0070] S101. Design different music score topics according to different working memory load levels;

[0071] Among them, for low-load questions, the score contains 3 or 4 notes, there are 5 questions in total, and the score memory time is set to 10 seconds; for medium-load questions, the score contains 5 notes, there are 5 questions in total, and the score memory time is set to 15 seconds; for high-load questions, the score contains 6 or 7 notes, there are 6 questions in total, and the score memory time is set to 20 seconds;

[0072] S102. Randomly generate a corresponding number of music scores according to the level, thus completing the question setting.

[0073] S2. Design a music score playing rule. The subject performs a working memory load test according to the music score playing rule, completes the music score playing, and calculates the subject's playing accuracy. The specific steps are as follows:

[0074] S201. The test rules for generating different working memory load levels according to playing the music score are as follows:

[0075] The music scores are displayed on the screen, and the difficulty of the scores is continuously increased; starting from the second round of testing, the scores of the current and previous rounds are memorized, and the scores of the previous round are played;

[0076] S202, the 88-key electronic keyboard in this embodiment has been modified, and a circuit board with multiple sensors is installed under the keys. When the keys are pressed, the sensors can record the keys pressed by the subject through the serial port. After the subject finishes playing the current score, he presses the 88th key at the end of the electronic keyboard to indicate that the current score has ended. After all the scores have been played, the data recorded by the serial port is output as a txt file;

[0077] S203, opening the corresponding piano key data record in Pycharm, taking key 88 as the dividing point, obtaining the playing data of each question of the testee, and saving it in an array, and calculating the correct rate of each question;

[0078] S3. Use a camera to capture the subject's facial video while memorizing the music score during the specified memory period. The designed working memory test requires the subject to play 5 rounds of music scores. During the music score memorization period of the first test in the first round, the music score memorization period of the third test in the third round, and the music score memorization period of the third test in the fifth round, facial videos are taken under sufficient lighting conditions at a distance of 40 to 45 cm for 10 seconds, 15 seconds, and 20 seconds respectively.

[0079] S4. Perform face detection based on the captured facial video, use the LinkNet network to perform face segmentation, and use the segmented face as the region of interest and perform channel separation. Perform signal decomposition and reconstruction on the separated G channel, and use the improved wavelet denoising algorithm and Butterworth bandpass filter to extract the pulse wave. The specific steps are as follows:

[0080] S401, calculating the frame rate of the captured video, and performing frame processing on the video according to the frame rate, dividing the video into a sufficient number of frame images;

[0081] S402, the collected videos and the Caltech Faces face video dataset are combined into a new dataset, and the dataset is split into a training set ratio of 0.8 and a test set ratio of 0.2, and the LinkNet network is trained. The LinkNet network architecture is as follows: Figure 2 As shown, the structure of the encoding block and the decoding block is as follows Figure 3As shown in the figure; the LinkNet network contains 4 encoders and 4 decoders, and the input of the encoder is connected to the output of the corresponding decoder; the encoder is designed based on the ResNet18 network. Starting from the first encoding block, the module performs convolution operations on the input image of size 7*7 and stride 2; batch normalization is performed between each convolution layer, and a ReLu layer is added for optimization.

[0082] S403, for each frame of the image, use the pre-trained LinkNet model to detect the face area and segment it according to the face edge, remove the area irrelevant to the face, and use the segmented face as the region of interest;

[0083] S404, separate the channels into three color channels: R, G, and B, and calculate the grayscale mean of the G channel. The G channel signal is selected as the initial pulse wave signal, wherein the flow chart of processing the G channel signal to obtain the pulse wave signal is as follows: Figure 4 As shown;

[0084] S405, decomposing and reconstructing the initial pulse wave signal. The baseline drift of the initial pulse wave signal is removed, and the CEEMDAN algorithm is used to decompose it to obtain 10 intrinsic mode components (IMFs), and the Pearson correlation coefficient between each IMF component and the initial pulse wave signal is calculated; after testing, it is found that the correlation coefficient difference between the first 5 IMF components and the last 5 IMF components is large, so the 5 IMF components with the highest correlation coefficient are selected to reconstruct the signal and remove the high-frequency part;

[0085] S406, perform wavelet threshold denoising on the reconstructed signal. Perform baseline drift removal on the reconstructed signal, and use the wavelet threshold denoising algorithm with improved threshold to denoise the reconstructed signal, so as to reduce the influence of the discontinuity of the hard threshold function at the threshold point and the constant error problem of the soft threshold function on the denoising effect. The improved wavelet threshold function formula is as follows:

[0086]

[0087] Among them, ω j,k is the wavelet decomposition coefficient, W j,k is the wavelet coefficient after threshold processing, λ is the threshold, α, β are the parameters of the threshold function. In this embodiment, the wavelet threshold parameters are α=1, β=1, db3 wavelet is selected for decomposition, and the number of decomposition layers is 2; wherein, the signal after wavelet denoising is as follows Figure 7 As shown, compared with the G channel signal graph, the signal has removed many burrs and has become smooth;

[0088] S407, use a Butterworth 3rd order bandpass filter to filter the signal after wavelet denoising. Under normal circumstances, the heart rate range of a person is 50 to 240 times per minute, so the passband is set to 0.8 to 4 Hz and bandpass filtering is performed to finally obtain a pulse wave signal with higher quality;

[0089] S5. Perform time domain and frequency domain analysis on the extracted pulse wave to calculate indicators such as heart rate and heart rate variability. The specific steps are as follows:

[0090] S501, according to the obtained pulse wave signal, convert the signal into the frequency domain, find the frequency corresponding to the highest peak of the amplitude spectrum, which is the number of heartbeats per second of the subject; multiply the obtained number of heartbeats per second by 60 to obtain the heart rate;

[0091] S502, the pulse rate variability signal and the heart rate variability signal are highly consistent, so the heart rate variability index can be calculated through the pulse wave signal. In this embodiment, multiple indexes related to heart rate variability are calculated, among which the indexes related to heart rate variability include 26 indexes such as heart rate interval standard deviation SDNN, adjacent heart rate interval difference standard deviation SDSD, adjacent heart rate interval difference root mean square RMSSD, heart rate mean Mean_HR, maximum heart rate Max_HR, minimum heart rate Min_HR, heart rate standard deviation STD_HR, signal total power TP, low frequency band power LF, high frequency band power HF, low frequency band to high frequency band power ratio LF / HF, very low frequency band power VLF;

[0092] S6. Give several subjects music problems with different working memory load levels, repeat steps S1 to S5, obtain the subjects' memory accuracy, heart rate, and heart rate variability indicators, and form the data set required for model construction. Based on the data set required for model construction, use the recursive feature elimination algorithm for feature selection, and use the random forest algorithm for load level classification to construct a working memory load physiological model. The specific steps are as follows:

[0093] S601, labeling the data set required for model construction, where the low load label is 0, the medium load label is 1, and the high load label is 2;

[0094] S602, split the above data set, wherein the ratio of the training set is 0.7 and the ratio of the test set is 0.3;

[0095] S603, normalize the indicators in the training set so that the extreme maximum and minimum values ​​in some indicators have less impact on the overall model. Suppose the amount of data in the training set is an m×n feature matrix F, and the elements in F are all finite real numbers X i , and the data in each row are not equal. The data in the test set is a k×n feature matrix t s, then the normalized element Y j for:

[0096]

[0097] In the formula, the normalized range is set to [-1,1], then Y max The value is 1, Y min The value is -1, max() and min() represent the maximum and minimum values ​​respectively;

[0098] S604: Normalize the indicators in the test set. To prevent the data in the test set from carrying information in the training set, which may lead to overfitting of the model training results, the data in the training set is selected as the mapping, and the data in the test set is normalized. s Perform the same normalization operation as above, and the normalized test set Ts obtained is:

[0099]

[0100] Where Y max The value is 1, Y min The value is -1, max() and min() represent the maximum and minimum values ​​respectively.

[0101] S605. Perform feature selection on the normalized data set to obtain a feature subset. This embodiment uses a recursive feature elimination algorithm to perform feature selection, and regards each indicator as a feature. During feature selection, several indicators such as the accuracy of playing the music score, heart rate, and heart rate variability are initialized as a feature set containing 28 indicators, and the importance measure of each feature is calculated, and irrelevant features are eliminated. The recursive process is continuously performed until no further features are retained, and finally appropriate features are selected to form a feature subset.

[0102] S606. Construct a physiological model of working memory load. The working memory load model is a model that uses a recursive feature elimination algorithm to select features, uses a random forest algorithm to classify working memory load levels, and uses 10-fold cross validation to evaluate the model. First, use the normalized data set for training, use the random forest algorithm for classification, and perform 10-fold cross validation on the classification results; determine the number of decision trees n_estimators of the random forest algorithm; the range of n_estimators is [0,200], and within this range, find the number of decision trees n_estimators with the highest accuracy for the working memory load physiological model to obtain the working memory load physiological model;

[0103] S607, input the number of decision trees obtained by training and the feature subset obtained by feature selection into the random forest, and test on the test set to obtain the recognition accuracy of the working memory load physiological model. The calculation formula of the recognition accuracy is as follows:

[0104]

[0105] In the formula, TP represents the number of minority class samples correctly predicted as minority class, FN represents the number of minority class samples incorrectly predicted as majority class, FP represents the number of majority class samples incorrectly predicted as minority class, and TN represents the number of majority class samples correctly predicted as majority class. The structure diagram of random forest algorithm is shown in Figure 6 As shown; In addition, considering the size of the data set, K is set to 10 for cross-validation to judge the model;

[0106] S7. In the actual measurement of the working memory load of the subject, the memory accuracy rate, as well as the heart rate and heart rate variability indexes extracted from the collected facial video, are input into the working memory load physiological model to identify the current working memory load level of the subject. The specific steps are as follows:

[0107] S701, the subject memorizes the music scores representing different loads according to the rules, and plays the corresponding notes on the electronic keyboard, and the memory accuracy is calculated;

[0108] S702, collecting face video during the memory period targeted by step S3, and extracting pulse waves;

[0109] S703, extracting heart rate and heart rate variability indicators through the pulse wave;

[0110] S704, normalizing and feature selecting the memory accuracy, heart rate and heart rate variability indicators, and then inputting them into the working memory load physiological model to identify the current working memory load level of the subject.

[0111] Example 2

[0112] Figure 1 : is a flowchart of a method for identifying working memory load level based on facial video provided by an embodiment of the present invention, which can realize non-contact identification of working memory load level, and the specific steps are as follows:

[0113] S1. Set the working memory load level and set questions. The working memory load level is set to three different levels: high load, medium load and low load, and carries a label, the low load label is 0, the medium load label is 1, and the high load label is 2; setting questions means generating different numbers of numbers using a random number generator according to different load levels. The specific steps are as follows:

[0114] S101. Design different digital memory questions according to different working memory load levels;

[0115] Among them, for the low load level, the number memory questions contain 5 numbers, there are 5 questions in total, and the number memory time is set to 10 seconds; for the medium load level, the number memory questions contain 8 numbers, there are 4 questions in total, and the number memory time is set to 15 seconds; for the high load level, the number memory questions contain 10 numbers, there are 3 questions in total, and the number memory time is set to 20 seconds;

[0116] S102, using a random number generation program to generate different numbers of digital memory questions;

[0117] S2. Design digital memory rules. The subjects will perform working memory load test according to the digital memory rules, complete memory questions, and calculate the memory accuracy of the subjects. The specific steps are as follows:

[0118] S201. The test rules for generating different working memory load levels based on digital memory questions are as follows:

[0119] The randomly generated number memory questions are displayed on the screen, and the program records the currently generated random numbers; after the current memory time is over, the subject writes down the numbers remembered in the previous round on the computer and presses the Enter key to end the current round of answering questions;

[0120] S202, record the generated random numbers and the numbers written by the testee, and generate a corresponding txt file; open the corresponding txt file in Pycharm, use the Enter key as the dividing point, obtain the memory data of each question of the testee, and save it in an array, and calculate the accuracy of the testee's memory of the numbers;

[0121] S3. Collect the subject's facial video during the specified memory period. The designed working memory test requires the subject to memorize three rounds of digital questions. In the digital memory period of the first test in the first round, the digital memory period of the third test in the second round, and the digital memory period of the third test in the third round, facial videos are taken under good lighting conditions at a shooting distance of about 40 cm, with shooting durations of 10 seconds, 15 seconds, and 20 seconds respectively;

[0122] S4, refer to the corresponding steps in Example 1, and will not be elaborated here. The G channel signal obtained is as follows Figure 7 As shown, the signal after improved wavelet threshold denoising is as follows Figure 8 As shown;

[0123] S5. Refer to the corresponding steps in Example 1 and implement them, which will not be elaborated here.

[0124] S6, refer to the corresponding steps in Example 1, and will not be elaborated here. Among them, using 10-fold cross validation, the best decision tree parameter n_estimators for the model is 79, and the corresponding cross validation mean is 0.8008. At this time, the accuracy of the model for each load classification is shown in Table 1, and the relationship between the cross validation mean and the number of decision trees is shown in Fig. 9 As shown;

[0125] Table 1. Recognition accuracy of each load level in Example 2

[0126]

[0127] S7. In the actual measurement of the working memory load of the subject, the memory accuracy rate, as well as the heart rate and heart rate variability indexes extracted from the collected facial video, are input into the working memory load physiological model to identify the current working memory load level of the subject. The specific steps are as follows:

[0128] S701, the subject memorizes the digital memory questions representing different loads according to the rules, writes down the results of the questions on the computer, and calculates the memory accuracy;

[0129] S702, collecting face video during the memory period targeted by step S3, and extracting pulse waves;

[0130] S703, extracting heart rate and heart rate variability indicators through the pulse wave;

[0131] S704, normalize and feature-select the memory accuracy, heart rate, and heart rate variability indicators, and then input them into the working memory load physiological model to identify the current working memory load level of the subject, as shown in Table 2.

[0132] Table 2. Identification results of working memory load levels of subjects at different stages in Example 2

[0133]

[0134] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A working memory load level identification method based on facial video, characterized in that: The identification method comprises the following steps: S1. Setting the working memory load level and setting questions; the working memory load level is set to three different levels, namely: high load, medium load and low load level; setting questions means designing different digital memory questions according to different load levels; S2. Design digital memory rules, and the subjects will take working memory load test according to the digital memory rules, complete memory questions, and calculate the memory accuracy of the subjects; S3, using a camera to collect facial video of the subject when he / she is memorizing numbers during a specified memory period; wherein the specified memory period refers to the beginning, middle and near the end of the working memory load test; S4, performing face detection and face segmentation according to the captured facial video, taking the segmented face as the region of interest and performing channel separation, performing signal decomposition and reconstruction on the separated G channel, and extracting the pulse wave using a wavelet denoising algorithm and a bandpass filter; S5. Perform time domain and frequency domain analysis on the extracted pulse wave to calculate the heart rate and heart rate variability indexes; S6, giving several subjects questions with different working memory load levels, repeating steps S1 to S5, obtaining the subjects' memory accuracy, heart rate, and heart rate variability indicators, forming a data set required for model construction, and then performing feature selection and load level classification based on the data set required for model construction to construct a working memory load physiological model; S7. In the actual measurement of the working memory load of the subject, the memory accuracy, as well as the heart rate and heart rate variability indicators extracted from the collected facial video are input into the working memory load physiological model to identify the current working memory load level of the subject.

2. The working memory load level identification method based on facial video according to claim 1 is characterized in that: The rules for designing different digital memory questions according to different load levels are as follows: For low-load questions, the questions contain 6 or 7 digits, there are 5 questions in total, and the digit memory time is set to 10 seconds; for medium-load questions, the questions contain 8 digits, there are 5 questions in total, and the digit memory time is set to 15 seconds; for high-load questions, the questions contain 9 or 10 digits, there are 6 questions in total, and the digit memory time is set to 20 seconds.

3. The working memory load level identification method based on facial video according to claim 1, characterized in that: The process of step S2 is as follows: S201. Designing the rules for the working memory load test and making the subjects answer the questions according to the rules; wherein the test rules for generating different working memory load levels according to the number memory questions are as follows: displaying the number questions on the screen, and continuously increasing the difficulty of remembering the number questions; starting from the second round of testing, memorizing the numbers of the current round and the previous round, and writing down the numbers of the previous round on the computer; S202. Record the numbers written by the subject on the computer, and calculate the subject's memory accuracy rate by comparing with the questions.

4. The working memory load level identification method based on facial video according to claim 1, characterized in that: The process of step S3 is as follows: The designated memory period refers to the period at the beginning, middle and near the end of the working memory load test. The designed working memory test requires the subjects to write down 5 rounds of numbers. During the digital memory period of the first test in the first round, the digital memory period of the third test in the third round, and the digital memory period of the third test in the fifth round, facial videos were taken under external conditions with sufficient lighting. The shooting distance was 40 to 45 cm, and the shooting time was 10 seconds, 15 seconds and 20 seconds respectively.

5. The working memory load level identification method based on facial video according to claim 1, characterized in that: The process of extracting the pulse wave according to the captured facial video in step S4 is as follows: S401, calculating the frame rate of the captured video, and performing frame processing on the video according to the frame rate, so as to divide the video into an image sequence; S402, for the image sequence, detecting the face area and segmenting it according to the face edge, removing the area irrelevant to the face, and taking the segmented face as the region of interest; S403, performing channel separation on each image in the image sequence into three color channels of R, G, and B, and selecting the G channel signal as the initial pulse wave signal; S404, decomposing the G channel signal to obtain multiple components, and calculating the Pearson correlation coefficient between each component and the original G channel signal; Select several components with the highest correlation coefficients and reconstruct the signal; S405, performing wavelet denoising processing on the reconstructed signal; S406 , after filtering the signal after wavelet denoising, the pulse wave of the captured facial video is obtained.

6. The working memory load level identification method based on facial video according to claim 1, characterized in that: In step S5, the process of calculating the heart rate and heart rate variability according to the obtained pulse wave signal is as follows: S501, calculating the heart rate: according to the obtained pulse wave signal, convert the signal into the frequency domain, find the frequency corresponding to the highest peak of the amplitude spectrum, which is the number of heartbeats per second of the subject; multiply the obtained number of heartbeats per second by 60, which is the heart rate; S502. Calculate the heart rate variability index according to the obtained pulse wave time domain signal, wherein the heart rate variability index includes the standard deviation of heart rate interval SDNN, the standard deviation of the difference between adjacent heart beat intervals SDSD, the root mean square of the difference between adjacent heart beat intervals RMSSD, the mean heart rate Mean_HR, the maximum heart rate Max_HR, the minimum heart rate Min_HR, the standard deviation of heart rate STD_HR, the total signal power TP, the low frequency band power LF, the high frequency band power HF, the ratio of low frequency band to high frequency band power LF / HF, and the very low frequency band power VLF.

7. The working memory load level identification method based on facial video according to claim 1, characterized in that: The process of step S6 is as follows: S601, labeling the collected data set, wherein the low load label is 0, the medium load label is 1, and the high load label is 2, to obtain the data set required for model construction; S602, splitting the data set into a training set and a test set, wherein the training set is used to train the model, and the test set is used to verify the model; S603, normalizing the indicators in the training set; S604, normalizing the indicators in the test set; S605, integrating the normalized training set and test set together to obtain a normalized data set; S606, performing feature selection on the normalized data set, treating each indicator as a feature, eliminating irrelevant features from several indicators such as memory accuracy, heart rate, and heart rate variability, selecting appropriate features, and forming a feature subset; S607, constructing a physiological model of working memory load. First, using the normalized data set for training, using a classifier for classification, and performing K-fold cross validation on the classification results to determine the hyperparameters of the classifier, thereby obtaining a physiological model of working memory load.

8. The working memory load level identification method based on facial video according to claim 7 is characterized in that: The step S6 further comprises: The hyperparameters obtained from training and the feature subsets obtained from feature selection were input into the classifier and tested on the test set to evaluate the recognition accuracy of the working memory load physiological model.

9. The working memory load level identification method based on facial video according to claim 1, characterized in that: The process of step S7 is as follows: S701, the subject memorizes the digital memory questions representing different loads according to the rules, writes down the results of the questions on the computer, and calculates the memory accuracy; S702, collecting face videos for the memory period and extracting pulse waves; S703, extracting heart rate and heart rate variability indicators through the pulse wave; S704, normalizing and feature selecting the memory accuracy, heart rate and heart rate variability indicators, and then inputting them into the working memory load physiological model to identify the current working memory load level of the subject.

Images