Student learning state monitoring method based on facial recognition mental stress analysis

Abstract

The invention relates to a student learning state monitoring method based on facial recognition mental stress analysis. Comprising the following steps of A, environment acquisition and equipment installation, B, time and frequency acquisition, C, face monitoring and tracking, D, region-of-interest selection, E, rPPG signal extraction, F, feature extraction, G, pressure analysis and H, student pressure level monitoring, and the pressure analysis in the step G is applied to actual student learning state monitoring in the step H. Acquiring facial image data of the student in real time, performing mental stress analysis, regularly evaluating the performance of the optimized stress evaluation model, and updating and improving the optimized stress evaluation model according to new data and practical application feedback; the method has the advantages that the mental stress level of a student is evaluated by analyzing the facial expression of the student; facial expression data of students can be obtained in real time, and instant state changes of the students in the learning process can be reflected in time.

Description

Technical Field

[0001] The present invention relates to the technical field of mental stress analysis based on face recognition; in particular, to a method and system for monitoring mental stress in a student's learning state based on RPPG technology. Background Art

[0002] With the intensification of educational competition, students' learning pressure is increasing day by day. Timely and accurate monitoring of students' learning status and mental stress is of great significance to protecting students' physical and mental health and improving learning outcomes. Traditional stress assessment methods usually rely on subjective questionnaires or direct measurement of physiological indicators, such as blood tests, electrocardiograms, etc. However, these methods have limitations such as strong invasiveness, complex operations, and difficulty in real-time monitoring. RPPG technology, as a non-contact physiological signal measurement method, obtains physiological information such as heart rate by analyzing skin color changes in facial videos, providing new possibilities for monitoring students' learning stress. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide a student learning status monitoring method based on facial recognition and mental stress analysis, which is used for effective student learning status monitoring to overcome the shortcomings of the above-mentioned prior art.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] A method for monitoring student learning status based on facial recognition mental stress analysis includes the following steps:

[0006] Step A: environment collection and equipment installation, including step A1 environment collection, step A2 equipment installation, step A3 camera installation at multiple angles, step A4 light and environment optimization, step A5: unified processing of the background of the learning place;

[0007] Step B: Collection time and frequency, including step B1 learning stage coverage, step B2 covering different course types, step B3 time length and continuity, step B4 obtaining more representative data, and step B5 collection frequency setting;

[0008] Step C: facial monitoring and tracking, including step C1 facial detection algorithm and step C2 facial tracking algorithm;

[0009] Step D: selection of regions of interest, including step D1 region determination principle, step D2 automatic selection method, and step D3 manual adjustment;

[0010] Step E: rPPG signal extraction, including E1 data preprocessing, step E2 rPPG signal extraction, step E3 physiological feature calculation;

[0011] Step F: feature extraction, including step F1 feature screening, step F2 time domain analysis, step F3 frequency domain analysis, and step F4 nonlinear analysis;

[0012] Step G: stress analysis, including step G1: stress identification, step G2: feature association, and step G3: stress classification and evaluation;

[0013] Step H: Monitoring of students’ stress levels, including step H1 real-time monitoring, step H2 long-term monitoring, and step H3 personalized intervention.

[0014] As a preferred embodiment of the present invention, in step A;

[0015] Step A1: Environmental collection, select representative learning places for data collection, including classrooms, library study rooms, and laboratories;

[0016] Step A2: Equipment installation: Install a high-definition, wide-angle camera with a resolution of 1080P or above to ensure that clear images of students' faces can be obtained;

[0017] Step A3: The camera is installed at multiple angles, with cameras installed at the front, back and sides of the learning place to capture students' facial expressions and movements in all directions. At the same time, the camera has automatic focus and light adjustment functions to adapt to different light conditions and changes in students' positions;

[0018] Step A4: Optimize the light and environment. Install lighting equipment to avoid shadows and strong light areas. Install blackout curtains on windows to adjust when the light is too strong to prevent misjudgment of facial expressions due to light problems.

[0019] Step A5: Perform uniform processing on the background of the learning place to reduce the interference of complex background patterns or colors on data collection.

[0020] As the preferred embodiment of the present invention, in step B:

[0021] Step B1: Coverage of learning stages. The data collection time covers all stages of students' daily learning, including classroom learning, homework, and exam review. By obtaining different learning tasks at each stage and changes in facial expressions under pressure, their mental stress level is assessed.

[0022] Step B2: Cover different types of courses, including liberal arts courses, science courses, and art courses, and collect data on the different levels of mental stress and emotional reactions caused by different courses in students through targeted data collection;

[0023] Step B3: Time length and continuity. The length of each data collection is greater than 30 minutes to capture the changing trend of students’ learning status during this time.

[0024] Step B4: To obtain more representative data, the data collection time should be distributed on different days and time periods, including morning, afternoon and evening, to take into account the physiological and psychological differences of students at different time periods;

[0025] Step B5: Set the acquisition frequency. Set the data acquisition frequency to capture subtle changes in students' faces. The acquisition frequency should be greater than 15 frames per second to record dynamic changes in students' facial expressions.

[0026] As a preferred embodiment of the present invention, in step C;

[0027] Step C1: Face detection algorithm: Use a deep learning-based face detection algorithm (MTCNN (Multi-task Cascaded Convolutional Networks) or RetinaFace) to quickly and accurately locate the position of the face in the video frame;

[0028] Step C2: Face tracking algorithm: After the face is detected in step C1, a tracking algorithm based on optical flow (KLT (Kanade-Lucas-Tomasi) algorithm) or feature point matching (SIFT (Scale-Invariant Feature Transform) algorithm) is used to track the movement of the face in real time to reduce errors caused by factors such as head shaking.

[0029] As preferred embodiment of the present invention, in step D:

[0030] Step D1: Principles of region determination Based on the study of facial vascular distribution and RPPG signal intensity, the forehead, cheeks, and nose bridge are selected as regions of interest, i.e., ROIs. These regions usually have more obvious changes in blood perfusion.

[0031] Step D2: The automatic selection method automatically segments the selected RO1 based on the results of face detection and tracking, combined with skin color and texture features;

[0032] Step D3: Manual adjustment, providing the function of manually adjusting the ROI to cope with special situations or improve signal quality.

[0033] As a preferred embodiment of the present invention, in step E;

[0034] Step E1: Data preprocessing,

[0035] Frame data: The front end sends Base64-encoded image data through http. The image data needs to be losslessly compressed. The frame_front.py in the project encodes the frame read by the camera into binary data in the memory buffer through the cv2.Imencode function. The image format is png and the compression rate is 0. It is then Base64-encoded and transmitted to the back end via JSON.

[0036] Video data: The front end uses the read function to read the MP4 video in binary read mode and uploads the data through application / octet-stream;

[0037] The backend first needs to perform a check on the integrity of the received data, and then parse the binary data into images or videos for preprocessing;

[0038] First, the FaceMesh model is loaded through mediapipe for each frame of the image, and the key points of the face are calibrated; then the EAR (Eye Aspect Ratio) curve is calculated based on the key points, and the points (362, 263) and (386, 374) are selected as the length and height of the eyes respectively, and the ratio of the two is calculated to obtain the EAR; the points (12, 15) and (78, 308) are selected as the length and width of the mouth, and the MAR (Mouth Aspect Ratio) is calculated; in order to calculate the rPPG signal, the area (200, 431, 411, 340, 349, 120, 111, 187, 211) is selected as the ROI according to the distribution density of capillaries, and the three color channels in the ROI area are extracted; the RGB channels, EAR and MAR are saved as the results of data preprocessing in the corresponding container;

[0039] Step E2: rPPG signal extraction,

[0040] For the frame interface, the rPPG signal is calculated when the continuous frame length reaches 300. For the video interface, the number of video frames is checked to be between 260 and 340 (this range needs to be modified according to the business scenario). After passing the verification, the rPPG signal is calculated.

[0041] The rPPG calculation uses the POS algorithm (derived from the paper: Algorithmic principles of remote PPG); that is, the noise is eliminated by the projection plane; specifically, the video captured by the camera contains the DC part, the AC part caused by the change of light intensity, the AC part in the mirror reflection and the rPPG signal; first, the original signal matrix is ​​normalized, and then the inner product projection is performed through a projection matrix orthogonal to the light intensity (experimentally proposed) to eliminate the influence of the light intensity change; the orthogonal matrix is:

[0042]

[0043] Among them, p is the pulse, T is the time, is the standard deviation operator. The signal projection on the orthogonal plane will produce two directional components S1 and S2. It is necessary to separate the specular reflection and rPPG from these two components. Since rPPG and specular noise occupy different dominant positions at different times, it is necessary to combine S1 and S2 to dynamically adjust the algorithm parameters.

[0044]

[0045] The projected rPPG signal also needs to be band-pass filtered to remove the non-pulse part; according to the human heart rate range (60-120) and signal frequency (30Hz), the selected band-pass filter range is 0.5-2Hz;

[0046] Step E3: Physiological characteristics calculation,

[0047] (1) Heart rate calculation,

[0048] According to the rPPG principle, the peak of the signal corresponds to the moment when the blood vessel volume is the largest, that is, the moment when the heart pumps blood; therefore, the interval between two peaks is one heart beat; for the collected rPPG signal, first perform elgendi peak detection (reference: Systolic peak detection in acceleration photoplethysmograms measuredfrom emergency responders in tropical conditions.) to obtain the peak position, and calculate the heart rate corresponding to each peak in combination with the sampling frequency. In order to enhance the robustness of the result, take the heart rate that falls within a reasonable range (50-130), and then take the average to obtain the final heart rate of the current 10-second video;

[0049] (2) Blood oxygen calculation,

[0050] Blood oxygen saturation (SpO2) refers to the percentage of oxygenated hemoglobin (i.e. hemoglobin combined with oxygen) in the blood to the total hemoglobin; oxygenated hemoglobin (HbO2) and deoxyhemoglobin (Hb) have different absorption characteristics for light of different wavelengths; the absorption coefficient of Hb is higher between 600-800nm, and the absorption coefficient of HbO2 is higher between 800-1000nm; since the wavelengths of green light and red light are relatively closer, it is not easy to calculate; therefore, red light and blue light are selected as reference signals in rPPG; and the mean is used as the DC component, and the standard deviation is used as the AC component; according to the formula:

[0051]

[0052] The blood oxygen saturation is calculated, where A and B are laboratory fitting values, A=100, B=2.39; the value of B is adjusted according to the actual situation.

[0053] As a preferred embodiment of the present invention, in step F;

[0054] Step F1: Feature screening, heart rate variability (HRV) refers to the temporal fluctuation of heartbeat intervals, which reflects the ability of the autonomic nervous system to regulate the heart; when sympathetic nerve activity increases, the heart rate increases and heart rate variability decreases; when parasympathetic nerve activity increases, the heart rate slows down and heart rate variability increases; therefore, heart rate variability is the result of the interaction between sympathetic and parasympathetic nerves; heart rate variability analysis can be used to further measure psychological state and cognitive ability; this is also the basis for using rPPG as model training data;

[0055] Step F2: Time domain analysis. The time domain analysis method needs to draw the HRV change curve according to the RR interval (the time interval between two adjacent R waves), and use mathematical statistics methods to calculate the characteristic value. SDNN (standard deviation of RR interval), PNN50 (the percentage of RR intervals greater than 50ms in the total number), RMSSD (the square root of the mean of the sum of squares of the differences between adjacent RR intervals), and SDSD (the standard deviation of the differences between adjacent RRs) have become representative time domain indicators in the field of mental stress research;

[0056] Step F3: Frequency domain analysis. HRV frequency domain analysis is to convert the time domain signal into the frequency domain, and use the Welch power spectrum to perform frequency domain analysis, which is based on the Fast Fourier Transform (FFT) at the average window end. Frequency domain analysis reflects the activities of the sympathetic and parasympathetic nerves, as well as the balance of the autonomic nervous system. The energy of the normal HRV power spectrum of the human body is between 0 and 0.4, which can be divided into high frequency information (HF), low frequency information (LF), and very low frequency information (VLF). The energy of the spectrum is calculated to obtain the frequency domain characteristics.

[0057] Step F4: nonlinear analysis. The nonlinear analysis adopts the Poincare scatter plot analysis method. The current heartbeat cycle is affected by its previous one. In order to express the correlation between the two, the RR interval sequence is adopted. The two adjacent intervals are used as the coordinates of a factor of the Poincare point, that is, the current interval is the horizontal coordinate and the next interval is the vertical coordinate. Each point is drawn in the figure, thereby obtaining a scatter plot. The scatter plot draws an ellipse, the center of which is the average value of the heartbeat interval. Then, the major axis and minor axis of the ellipse can be obtained as SD2 (standard deviation along the marked line) and SD1 (standard deviation of the Poincare plot on the straight line perpendicular to the marked line). In order to obtain characteristic parameters that can accurately reflect stress changes and observe the changing trends of characteristic parameters from relaxation to stress state, the 17 groups of features extracted were screened according to the experimental results; SPSS (Statistical Package for the Social Sciences) was used to analyze the data, and all HRV characteristics (30 groups of experimental data) before and after stress induction were tested for significance (confidence interval 95%) through paired t-test, and the features with significant differences were screened out as relevant indicators for mental stress analysis. Only features with significant differences can be used for subsequent classification; According to the significant difference results, the differences of the three characteristic parameters of std_hr (heart rate standard deviation), SD1 / SD2 (ratio of SD2 and SD1) before and after stress induction were not significant (p>0.05), so it can be considered that these three characteristic indicators are difficult to accurately reflect the state between relaxation and stress, so these three characteristic parameters were discarded in subsequent studies;

[0058] As a preferred embodiment of the present invention, in step G,

[0059] Step G1: Stress identification, mental stress assessment Based on the stress induction experiment, characteristic parameters (14 dimensions) related to the stress state are obtained. Based on the actual life stress collection experiment, the model is built to realize mental stress analysis. The training set and the test set are divided into 7:3, 70% of the data are selected as the training set and 30% of the data are selected as the test set; the input of the model is 14-dimensional HRV features;

[0060] Based on the random forest model, the 14-dimensional features obtained were used to identify mental stress; a cross-validated grid search was used for parameter optimization; the default value of n_estimators (the number of random forest subtrees) was 100, and the parameter range was set to 1-101 with a step length of 10; the max_features (the maximum number of features in the decision tree establishment process) parameter range was set to 1-11 with a step length of 1; the max_depth (the maximum depth of the decision tree) range was set to 5-15 with a step length of 1; a grid search was performed on n_estimators, max_features and max_depth within the set parameter range, and the average score of each group of parameter random forest models was calculated and compared. According to the cross-validation grid optimization, the parameter combination with the highest final verification score of the ten-fold crossover was selected as the final model parameter result, and the optimal parameter combination of n_estimators, max_features and max_depth was determined; and finally the optimal parameters were used to train a new random forest model on the entire training set, thereby obtaining the optimal model; according to the above cross-validation grid search method, the best model parameters were obtained;

[0061] Step G2: Analyze the correlation between the extracted RPPG features and known stress-related indicators; through correlation analysis and feature importance evaluation, screen out features that have significant contributions to stress assessment;

[0062] Step G3: Stress classification and evaluation Input the RPPG features of the students to be evaluated into the trained stress model to obtain stress classification results (such as low stress, medium stress, high stress) and stress evaluation values.

[0063] As a preferred embodiment of the present invention, in step H;

[0064] Step H1: Real-time monitoring: During the students’ learning process, real-time collection and analysis of RPPG data are performed to provide real-time feedback on changes in students’ stress levels;

[0065] Step H2: Long-term monitoring: monitor students’ stress over a long period of time, establish stress profiles, and track stress trends;

[0066] Step H3: Personalized intervention: Provide educators with personalized intervention suggestions based on students’ stress levels and changing trends, including adjusting learning tasks and providing psychological counseling.

[0067] The advantages and positive effects of the present invention are:

[0068] It provides an effective method for monitoring students' learning status and assesses their mental stress level by analyzing their facial expressions. It can obtain students' facial expression data in real time and promptly reflect the changes in students' status during the learning process. It can monitor multiple students at the same time and is suitable for large-scale classroom environments. It helps to detect excessive stress or poor learning status of students at an early stage so that timely intervention measures can be taken. DETAILED DESCRIPTION

[0069] Example 1

[0070] This embodiment provides a method for monitoring student learning status based on facial recognition mental stress analysis, including the following steps:

[0071] Step A: environment collection and equipment installation, including step A1 environment collection, step A2 equipment installation, step A3 camera installation at multiple angles, step A4 light and environment optimization, step A5: unified processing of the background of the learning place;

[0072] Step B: Collection time and frequency, including step B1 learning stage coverage, step B2 covering different course types, step B3 time length and continuity, step B4 obtaining more representative data, and step B5 collection frequency setting;

[0073] Step C: facial monitoring and tracking, including step C1 facial detection algorithm and step C2 facial tracking algorithm;

[0074] Step D: selection of regions of interest, including step D1 region determination principle, step D2 automatic selection method, and step D3 manual adjustment;

[0075] Step E: rPPG signal extraction, including E1 data preprocessing, step E2 rPPG signal extraction, step E3 physiological feature calculation;

[0076] Step F: feature extraction, including step F1 feature screening, step F2 time domain analysis, step F3 frequency domain analysis, and step F4 nonlinear analysis;

[0077] Step G: stress analysis, including step G1: stress identification, step G2: feature association, and step G3: stress classification and evaluation;

[0078] Step H: Monitoring of students’ stress levels, including step H1 real-time monitoring, step H2 long-term monitoring, and step H3 personalized intervention.

[0079] Further, in step A of this embodiment;

[0080] Step A1: Environmental collection, select representative learning places for data collection, including classrooms, library study rooms, and laboratories;

[0081] Step A2: Equipment installation: Install a high-definition, wide-angle camera with a resolution of 1080P or above to ensure that clear images of students' faces can be obtained;

[0082] Step A3: The camera is installed at multiple angles, with cameras installed at the front, back and sides of the learning place to capture students' facial expressions and movements in all directions. At the same time, the camera has automatic focus and light adjustment functions to adapt to different light conditions and changes in students' positions;

[0083] Step A4: Optimize the light and environment. Install lighting equipment to avoid shadows and strong light areas. Install blackout curtains on windows to adjust when the light is too strong to prevent misjudgment of facial expressions due to light problems.

[0084] Step A5: Perform uniform processing on the background of the learning place to reduce the interference of complex background patterns or colors on data collection.

[0085] Further, in step B of this embodiment:

[0086] Step B1: Coverage of learning stages. The data collection time covers all stages of students' daily learning, including classroom learning, homework, and exam review. By obtaining different learning tasks at each stage and changes in facial expressions under pressure, their mental stress level is assessed.

[0087] Step B2: Cover different types of courses, including liberal arts courses, science courses, and art courses, and collect data on the different levels of mental stress and emotional reactions caused by different courses in students through targeted data collection;

[0088] Step B3: Time length and continuity. The length of each data collection is greater than 30 minutes to capture the changing trend of students’ learning status during this time.

[0089] Step B4: To obtain more representative data, the data collection time should be distributed on different days and time periods, including morning, afternoon and evening, to take into account the physiological and psychological differences of students at different time periods;

[0090] Step B5: Set the acquisition frequency. Set the data acquisition frequency to capture subtle changes in students' faces. The acquisition frequency should be greater than 15 frames per second to record dynamic changes in students' facial expressions.

[0091] Further, in step C of this embodiment;

[0092] Step C1: Face detection algorithm: Use a deep learning-based face detection algorithm (MTCNN (Multi-task Cascaded Convolutional Networks) or RetinaFace) to quickly and accurately locate the position of the face in the video frame;

[0093] Step C2: Face tracking algorithm: After the face is detected in step C1, a tracking algorithm based on optical flow (KLT (Kanade-Lucas-Tomasi) algorithm) or feature point matching (SIFT (Scale-Invariant Feature Transform) algorithm) is used to track the movement of the face in real time to reduce errors caused by factors such as head shaking.

[0094] 5. The method for monitoring student learning status based on facial recognition and mental stress analysis according to claim 1, characterized in that in step D:

[0095] Step D1: Principles of region determination Based on the study of facial vascular distribution and RPPG signal intensity, the forehead, cheeks, and nose bridge are selected as regions of interest (ROI); these regions usually have more obvious blood perfusion changes;

[0096] Step D2: The automatic selection method automatically segments the selected RO1 based on the results of face detection and tracking, combined with skin color and texture features;

[0097] Step D3: Manual adjustment, providing the function of manually adjusting the ROI to cope with special situations or improve signal quality.

[0098] Further, in step E of this embodiment;

[0099] Step E1: Data preprocessing,

[0100] Frame data: The front end sends Base64-encoded image data through http. The image data needs to be losslessly compressed. The frame_front.py in the project encodes the frame read by the camera into binary data in the memory buffer through the cv2.Imencode function. The image format is png and the compression rate is 0. It is then Base64-encoded and transmitted to the back end via JSON.

[0101] Video data: The front end uses the read function to read the MP4 video in binary read mode and uploads the data through application / octet-stream;

[0102] The backend first needs to perform a check on the integrity of the received data, and then parse the binary data into images or videos for preprocessing;

[0103] First, the FaceMesh model is loaded through mediapipe for each frame of the image, and the key points of the face are calibrated; then the EAR (Eye Aspect Ratio) curve is calculated based on the key points, and the points (362, 263) and (386, 374) are selected as the length and height of the eyes respectively, and the ratio of the two is calculated to obtain the EAR; the points (12, 15) and (78, 308) are selected as the length and width of the mouth, and the MAR (Mouth Aspect Ratio) is calculated; in order to calculate the rPPG signal, the area (200, 431, 411, 340, 349, 120, 111, 187, 211) is selected as the ROI according to the distribution density of capillaries, and the three color channels in the ROI area are extracted; the RGB channels, EAR and MAR are saved as the results of data preprocessing in the corresponding container;

[0104] Step E2: rPPG signal extraction,

[0105] For the frame interface, the rPPG signal is calculated when the continuous frame length reaches 300. For the video interface, the number of video frames is checked to be between 260 and 340 (this range needs to be modified according to the business scenario). After passing the verification, the rPPG signal is calculated.

[0106] The rPPG calculation uses the POS algorithm (derived from the paper: Algorithmic principles of remote PPG); that is, the noise is eliminated by the projection plane; specifically, the video captured by the camera contains the DC part, the AC part caused by the change of light intensity, the AC part in the mirror reflection and the rPPG signal; first, the original signal matrix is ​​normalized, and then the inner product projection is performed through a projection matrix orthogonal to the light intensity (experimentally proposed) to eliminate the influence of the light intensity change; the orthogonal matrix is:

[0107]

[0108] Among them, p is the pulse, T is the time, is the standard deviation operator. The signal projection on the orthogonal plane will produce two directional components S1 and S2. It is necessary to separate the specular reflection and rPPG from these two components. Since rPPG and specular noise occupy different dominant positions at different times, it is necessary to combine S1 and S2 to dynamically adjust the algorithm parameters.

[0109]

[0110] The projected rPPG signal also needs to be band-pass filtered to remove the non-pulse part; according to the human heart rate range (60-120) and signal frequency (30Hz), the selected band-pass filter range is 0.5-2Hz;

[0111] Step E3: Physiological characteristics calculation,

[0112] (1) Heart rate calculation,

[0113] According to the rPPG principle, the peak of the signal corresponds to the moment when the blood vessel volume is the largest, that is, the moment when the heart pumps blood; therefore, the interval between two peaks is one heart beat; for the collected rPPG signal, first perform elgendi peak detection (reference: Systolic peak detection in acceleration photoplethysmograms measuredfrom emergency responders in tropical conditions.) to obtain the peak position, and calculate the heart rate corresponding to each peak in combination with the sampling frequency. In order to enhance the robustness of the result, take the heart rate that falls within a reasonable range (50-130), and then take the average to obtain the final heart rate of the current 10-second video;

[0114] (2) Blood oxygen calculation,

[0115] Blood oxygen saturation (SpO2) refers to the percentage of oxygenated hemoglobin (i.e. hemoglobin combined with oxygen) in the blood to the total hemoglobin; oxygenated hemoglobin (HbO2) and deoxyhemoglobin (Hb) have different absorption characteristics for light of different wavelengths; the absorption coefficient of Hb is higher between 600-800nm, and the absorption coefficient of HbO2 is higher between 800-1000nm; since the wavelengths of green light and red light are relatively closer, it is not easy to calculate; therefore, red light and blue light are selected as reference signals in rPPG; and the mean is used as the DC component, and the standard deviation is used as the AC component; according to the formula:

[0116]

[0117] The blood oxygen saturation is calculated, where A and B are laboratory fitting values, A=100, B=2.39; the value of B is adjusted according to the actual situation.

[0118] Further, in step F of this embodiment;

[0119] Step F1: Feature screening, heart rate variability (HRV) refers to the temporal fluctuation of heartbeat intervals, which reflects the ability of the autonomic nervous system to regulate the heart; when sympathetic nerve activity increases, the heart rate increases and heart rate variability decreases; when parasympathetic nerve activity increases, the heart rate slows down and heart rate variability increases; therefore, heart rate variability is the result of the interaction between sympathetic and parasympathetic nerves; heart rate variability analysis can be used to further measure psychological state and cognitive ability; this is also the basis for using rPPG as model training data;

[0120] Step F2: Time domain analysis. The time domain analysis method needs to draw the HRV change curve according to the RR interval (the time interval between two adjacent R waves), and use mathematical statistics methods to calculate the characteristic value. SDNN (standard deviation of RR interval), PNN50 (the percentage of RR intervals greater than 50ms in the total number), RMSSD (the square root of the mean of the sum of squares of the differences between adjacent RR intervals), and SDSD (the standard deviation of the differences between adjacent RRs) have become representative time domain indicators in the field of mental stress research;

[0121] Step F3: Frequency domain analysis. HRV frequency domain analysis is to convert the time domain signal into the frequency domain, and use the Welch power spectrum to perform frequency domain analysis, which is based on the Fast Fourier Transform (FFT) at the average window end. Frequency domain analysis reflects the activities of the sympathetic and parasympathetic nerves, as well as the balance of the autonomic nervous system. The energy of the normal HRV power spectrum of the human body is between 0 and 0.4, which can be divided into high frequency information (HF), low frequency information (LF), and very low frequency information (VLF). The energy of the spectrum is calculated to obtain the frequency domain characteristics.

[0122] Step F4: nonlinear analysis. The nonlinear analysis adopts the Poincare scatter plot analysis method. The current heartbeat cycle is affected by its previous one. In order to express the correlation between the two, the RR interval sequence is adopted. The two adjacent intervals are used as the coordinates of a factor of the Poincare point, that is, the current interval is the horizontal coordinate and the next interval is the vertical coordinate. Each point is drawn in the figure, thereby obtaining a scatter plot. The scatter plot draws an ellipse, the center of which is the average value of the heartbeat interval. Then, the major axis and minor axis of the ellipse can be obtained as SD2 (standard deviation along the marked line) and SD1 (standard deviation of the Poincare plot on the straight line perpendicular to the marked line). In order to obtain characteristic parameters that can accurately reflect stress changes and observe the changing trends of characteristic parameters from relaxation to stress state, the 17 groups of features extracted were screened according to the experimental results; SPSS (Statistical Package for the Social Sciences) was used to analyze the data, and all HRV characteristics (30 groups of experimental data) before and after stress induction were tested for significance (confidence interval 95%) through paired t-test, and the features with significant differences were screened out as relevant indicators for mental stress analysis. Only features with significant differences can be used for subsequent classification; According to the significant difference results, the differences of the three characteristic parameters of std_hr (heart rate standard deviation), SD1 / SD2 (ratio of SD2 and SD1) before and after stress induction were not significant (p>0.05), so it can be considered that these three characteristic indicators are difficult to accurately reflect the state between relaxation and stress, so these three characteristic parameters were discarded in subsequent studies;

[0123] Further, in step G of this embodiment,

[0124] Step G1: Stress identification, mental stress assessment Based on the stress induction experiment, characteristic parameters (14 dimensions) related to the stress state are obtained. Based on the actual life stress collection experiment, the model is built to realize mental stress analysis. The training set and the test set are divided into 7:3, 70% of the data are selected as the training set and 30% of the data are selected as the test set; the input of the model is 14-dimensional HRV features;

[0125] Based on the random forest model, the 14-dimensional features obtained were used to identify mental stress; a cross-validated grid search was used for parameter optimization; the default value of n_estimators (the number of random forest subtrees) was 100, and the parameter range was set to 1-101 with a step length of 10; the max_features (the maximum number of features in the decision tree establishment process) parameter range was set to 1-11 with a step length of 1; the max_depth (the maximum depth of the decision tree) range was set to 5-15 with a step length of 1; a grid search was performed on n_estimators, max_features and max_depth within the set parameter range, and the average score of each group of parameter random forest models was calculated and compared. According to the cross-validation grid optimization, the parameter combination with the highest final verification score of the ten-fold crossover was selected as the final model parameter result, and the optimal parameter combination of n_estimators, max_features and max_depth was determined; and finally the optimal parameters were used to train a new random forest model on the entire training set, thereby obtaining the optimal model; according to the above cross-validation grid search method, the best model parameters were obtained;

[0126] Step G2: Analyze the correlation between the extracted RPPG features and known stress-related indicators; through correlation analysis and feature importance evaluation, screen out features that have significant contributions to stress assessment;

[0127] Step G3: Stress classification and evaluation Input the RPPG features of the students to be evaluated into the trained stress model to obtain stress classification results (such as low stress, medium stress, high stress) and stress evaluation values.

[0128] Further, in step H of this embodiment;

[0129] Step H1: Real-time monitoring: During the students’ learning process, real-time collection and analysis of RPPG data are performed to provide real-time feedback on changes in students’ stress levels;

[0130] Step H2: Long-term monitoring: monitor students’ stress over a long period of time, establish stress profiles, and track stress trends;

[0131] Step H3: Personalized intervention: Provide educators with personalized intervention suggestions based on students’ stress levels and changing trends, including adjusting learning tasks and providing psychological counseling.

[0132] Further, the establishment of the stress profile in step H includes the following steps: M1: receiving data, step M2: display interface, step M3: report generation, step M4: early warning mechanism, step M5: personalized suggestions, step M6: application scenarios;

[0133] Step M1: receiving data, transmitting the data collected in step H1 to the cloud server terminal in real time through the network for processing; the pressure model trained in step G3 is stored in the cloud server terminal;

[0134] Step M2: Display interface, including teacher end and student end;

[0135] Step M2.1: On the teacher's computer screen, the current stress level of each student is displayed in real time in the form of a list or chart. Each student's name corresponds to a number or color label, which is convenient for the teacher to quickly browse;

[0136] Step M2.2: On the student side, the student's current stress status is displayed on the student's personal learning device, where the stress value and prompt information are displayed in a pop-up window;

[0137] Step M3: Report generation, including: data storage, report content;

[0138] Step M31: data storage, storing the stress assessment data of each student within a week or a month in a database;

[0139] Step M32: Report content, including pressure change trend graph, average pressure level statistics, pressure peak analysis, and comparison with the class average pressure;

[0140] Among them, the stress change trend chart: with time as the horizontal axis and stress level as the vertical axis, it shows the fluctuations of students' stress during the assessment period;

[0141] Among them, average stress level statistics: calculate the average stress value of students during the evaluation period to evaluate the overall learning stress status;

[0142] Among them, stress peak analysis: points out the specific time when the stress peak occurs and the corresponding course, helping teachers understand which teaching links cause greater pressure on students;

[0143] Among them, comparison with the average stress of the class: comparing the stress data of students with the average level of the class, so that teachers and students can understand their relative position in the class;

[0144] Step M4: early warning mechanism, including: threshold setting, early warning method;

[0145] Step M4.1: Threshold setting: teachers set personalized stress thresholds for each student based on the overall situation of the class and individual differences of students;

[0146] Step M4.2: Warning method. When the student's stress exceeds the stress threshold of step M231, the warning information is automatically sent to the teacher via an email or mobile phone text message via the cloud server terminal. The warning information includes the student's name, current stress value, the magnitude of the stress exceeding the threshold, and intervention measures.

[0147] Step M5: Personalized recommendations, including: analysis basis and recommendations,

[0148] Step M51: Analyze the basis, based on the student's stress level, change trend, study course and time factors;

[0149] Step M52: Suggestions, including learning, relaxation and psychological adjustment;

[0150] Step M521: In terms of study, adjust the study plan, optimize the study method, and allocate time reasonably;

[0151] Step M522: In terms of relaxation, relax activities are recommended and specific time suggestions are given;

[0152] Step M523: Psychological adjustment, providing psychological adjustment techniques, including positive self-suggestion and deep breathing exercises;

[0153] Step M6: Application scenarios, including: classroom teaching, examination period and independent learning.

[0154] Step M61: During classroom teaching, including: overall stress assessment A, stress source analysis A, and suggestions and interventions A,

[0155] Step M611: Overall pressure assessment A, including: low pressure A, medium pressure A and high pressure A;

[0156] Low stress A: Students show a relaxed and focused look in class, with natural facial muscles, bright and stable eyes, and upturned corners of the mouth, indicating that they can easily keep up with the teaching progress and enjoy the learning process;

[0157] Moderate stress A: The student's facial expression is slightly serious, his brows are slightly furrowed, and he occasionally blinks faster, indicating that he is trying hard to understand and absorb knowledge and feels a certain learning challenge;

[0158] High Stress A: Students have tense facial expressions, sweat on their foreheads, clenched teeth, and anxious or wandering eyes, indicating that they encounter great difficulties in class and have difficulty keeping up with the teaching pace;

[0159] Step M612: Analyze the sources of stress A, including: from teachers, from classmates, and from oneself.

[0160] From teachers: Students have stress reactions when teachers ask questions, explain specific content, or have direct interactions with teachers, indicating that they feel stressed about the teacher's teaching style, the difficulty of questions, or the attention;

[0161] From classmates: When students show signs of stress during group activities, class presentations, or competition with classmates, they are affected by their classmates' performance or interaction;

[0162] From within: Observe that students show signs of stress when thinking, taking notes, or practicing on their own, indicating that they feel pressured about their learning ability and performance expectations;

[0163] Step M613: suggestions and interventions A, including: for low-stress student A, for medium-stress student A, and for high-stress student A;

[0164] For low-stress students A: Encourage students to stay positive and try more challenging learning tasks;

[0165] For moderately stressed students A: provide additional tutoring resources or learning strategy guidance to help students better cope with learning challenges;

[0166] For high-stress student A: communicate with each other individually, understand the specific problem, and provide psychological support and specific help in study;

[0167] Step M62: During the exam, including: overall stress assessment B, stress source analysis B, and suggestions and interventions B,

[0168] Step M621: Overall pressure assessment, including: low pressure B, medium pressure B and high pressure B,

[0169] Low stress B: The student's expression is relatively calm, breathing is steady, and the movements are smooth when answering questions, with no obvious signs of tension;

[0170] Moderate stress B: Students may experience slight hand tremors, lick their lips frequently, and frown slightly, but they can still answer questions attentively.

[0171] High Stress B: The student looks pale, sweats profusely, breathes rapidly, rubs eyes or scratches hair frequently, and hesitates or pauses when answering questions;

[0172] Step M622: Analysis of stress sources, including test content, test environment, and suggestions and interventions C,

[0173] Exam content: Students experience increased stress when encountering specific question types, knowledge points, or difficult questions, indicating that they have not mastered the relevant content;

[0174] Examination environment: the atmosphere of the examination room, the behavior of the invigilator or the behavior of the students around them trigger the students' stress response;

[0175] Self-expectations: Excessive self-expectations and excessive concerns about test results cause students to be under high pressure during the test

[0176] Step M622: Suggestions and interventions B, including for low-stress student B, for medium-stress student B, and for high-stress student B.

[0177] For low-stress students B: remind them to stay careful, check their answers, and allocate their time wisely;

[0178] For students B with moderate stress: Give them appropriate time reminders and encourage them to take a deep breath, relax, and focus on the questions;

[0179] For high-stress student B: After the exam, communicate with the student to understand the cause of stress and provide psychological counseling and study suggestions;

[0180] Step M63: Self-study includes: overall stress assessment C, stress source analysis C, and suggestions and interventions C.

[0181] Step M631: Overall pressure assessment C, including: low pressure C, medium pressure C and high pressure C,

[0182] Low stress C: Students sit in a relaxed posture, have a relaxed and happy facial expression, can switch between learning content and activities freely, and have no obvious signs of anxiety;

[0183] Moderate stress C: The student is focused, occasionally biting his lips or resting his chin on his hand while thinking, but still able to maintain a certain learning pace;

[0184] High Stress C: Students lean forward, look serious, sigh or rub their temples frequently, and are easily distracted or interrupted during learning;

[0185] Step M632: Analyze the sources of stress, including learning tasks, learning environment, and self-management.

[0186] Learning tasks: Too many, too difficult or lack of clear goals of learning tasks lead to stress;

[0187] Learning environment: Uncomfortable seats, noisy background noise or distractions affect students’ learning mentality;

[0188] Self-management: Lack of effective time management, self-discipline or improper learning methods can cause stress;

[0189] Step M633: Suggestions and interventions C, including: for low-stress students C, for medium-stress students C, and for high-stress students C,

[0190] For low-stress student C: continue to maintain a good state and encourage students to set higher learning goals;

[0191] For medium-stress students C: provide study tips and time management advice to help students optimize their learning process;

[0192] For high-stress students C: assist students in developing reasonable study plans, adjusting the learning environment, providing psychological counseling, and reducing the pressure brought by independent learning.

[0193] The above description is only a preferred example of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modification, equivalent substitution, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for monitoring student learning status based on facial recognition mental stress analysis, characterized in that: The following steps are involved: Step A: environment collection and equipment installation, including step A1 environment collection, step A2 equipment installation, step A3 camera installation at multiple angles, step A4 light and environment optimization, step A5: unified processing of the background of the learning place; Step B: Collection time and frequency, including step B1 learning stage coverage, step B2 covering different course types, step B3 time length and continuity, step B4 obtaining more representative data, and step B5 collection frequency setting; Step C: facial monitoring and tracking, including step C1 facial detection algorithm and step C2 facial tracking algorithm; Step D: selection of regions of interest, including step D1 region determination principle, step D2 automatic selection method, and step D3 manual adjustment; Step E: rPPG signal extraction, including E1 data preprocessing, step E2 rPPG signal extraction, step E3 physiological feature calculation; Step F: feature extraction, including step F1 feature screening, step F2 time domain analysis, step F3 frequency domain analysis, and step F4 nonlinear analysis; Step G: stress analysis, including step G1: stress identification, step G2: feature association, and step G3: stress classification and evaluation; Step H: Monitoring of students’ stress levels, including step H1 real-time monitoring, step H2 long-term monitoring, and step H3 personalized intervention.

2. According to claim 1, a method for monitoring student learning status based on facial recognition mental stress analysis is characterized in that: In step A; Step A1: Environmental collection, select representative learning places for data collection, including classrooms, library study rooms, and laboratories; Step A2: Equipment installation: Install a high-definition, wide-angle camera with a resolution of 1080P or above to ensure that clear images of students' faces can be obtained; Step A3: The camera is installed at multiple angles, with cameras installed at the front, back and sides of the learning place to capture students' facial expressions and movements in all directions. At the same time, the camera has automatic focus and light adjustment functions to adapt to different light conditions and changes in students' positions; Step A4: Optimize the light and environment. Install lighting equipment to avoid shadows and strong light areas. Install blackout curtains on windows to adjust when the light is too strong to prevent misjudgment of facial expressions due to light problems. Step A5: Perform uniform processing on the background of the learning place to reduce the interference of complex background patterns or colors on data collection.

3. The method for monitoring student learning status based on facial recognition and mental stress analysis according to claim 1 is characterized in that: In step B: Step B1: Coverage of learning stages. The data collection time covers all stages of students' daily learning, including classroom learning, homework, and exam review. By obtaining different learning tasks at each stage and changes in facial expressions under pressure, their mental stress level is assessed. Step B2: Cover different types of courses, including liberal arts courses, science courses, and art courses, and collect data on the different levels of mental stress and emotional reactions caused by different courses in students through targeted data collection; Step B3: Time length and continuity. The length of each data collection is greater than 30 minutes to capture the changing trend of students’ learning status during this time. Step B4: To obtain more representative data, the data collection time should be distributed on different days and time periods, including morning, afternoon and evening, to take into account the physiological and psychological differences of students at different time periods; Step B5: Set the acquisition frequency. Set the data acquisition frequency to capture subtle changes in students' faces. The acquisition frequency should be greater than 15 frames per second to record dynamic changes in students' facial expressions.

4. The method for monitoring student learning status based on facial recognition and mental stress analysis according to claim 1, characterized in that: In step C; Step C1: Face detection algorithm: A deep learning-based face detection algorithm is used to quickly and accurately locate the position of the face in the video frame; Step C2: Face tracking algorithm: After the face is detected in step C1, a tracking algorithm based on optical flow or feature point matching is used to track the movement of the face in real time to reduce errors caused by factors such as head shaking.

5. The method for monitoring student learning status based on facial recognition and mental stress analysis according to claim 1, characterized in that: In step D: Step D1: Region determination principle Based on the study of facial vascular distribution and RPPG signal intensity, the forehead, cheeks, and nose bridge are selected as regions of interest, i.e., ROIs; Step D2: The automatic selection method automatically segments the selected RO1 based on the results of face detection and tracking, combined with skin color and texture features; Step D3: Manual adjustment, providing the function of manually adjusting the ROI to cope with special situations or improve signal quality.

6. The method for monitoring student learning status based on facial recognition and mental stress analysis according to claim 1, characterized in that: In step E; Step E1: Data preprocessing, Frame data: The front end sends Base64-encoded image data through http. The image data needs to be losslessly compressed. The frame_front.py in the project encodes the frame read by the camera into binary data in the memory buffer through the cv2.Imencode function. The image format is png and the compression rate is 0. It is then Base64-encoded and transmitted to the back end via JSON. Video data: The front end uses the read function to read the MP4 video in binary read mode and uploads the data through application / octet-stream; The backend first needs to perform a logic check on the integrity of the received data, and then parse the binary data into images or videos for preprocessing; First, the FaceMesh model is loaded through mediapipe for each frame of the image, and the key points of the face are calibrated. Then, the EAR curve is calculated based on the key points, and the points (362, 263) and (386, 374) are selected as the length and height of the eyes respectively, and the ratio of the two is calculated to obtain the EAR. The points (12, 15) and (78, 308) are selected as the length and width of the mouth, and the MAR is calculated. In order to calculate the rPPG signal, the area (200, 431, 411, 340, 349, 120, 111, 187, 211) is selected as the ROI according to the distribution density of capillaries, and the three color channels in the ROI area are extracted. The RGB channels, EAR and MAR are saved as the results of data preprocessing in the corresponding container. Step E2: rPPG signal extraction, For the frame interface, the rPPG signal is calculated when the continuous frame length reaches 300; for the video interface, the number of video frames is checked to be between 260 and 340, and the rPPG signal is calculated after passing the check; The rPPG calculation uses the POS algorithm; That is, noise is eliminated through the projection plane; specifically, the video captured by the camera contains the DC part, the AC part caused by the change of light intensity, the AC part in the mirror reflection and the rPPG signal; first, the original signal matrix is ​​normalized, and then the inner product projection is performed through a projection matrix orthogonal to the light intensity to eliminate the influence of the light intensity change; the orthogonal matrix is: Among them, p is the pulse, T is the time, is the standard deviation operator. The signal projection on the orthogonal plane will produce two directional components S1 and S2. It is necessary to separate the specular reflection and rPPG from these two components. Since rPPG and specular noise occupy different dominant positions at different times, it is necessary to combine S1 and S2 to dynamically adjust the algorithm parameters. The projected rPPG signal also needs to be band-pass filtered to remove the non-pulse part; according to the human heart rate range (60-120) and signal frequency (30Hz), the selected band-pass filter range is 0.5-2Hz; Step E3: Physiological characteristics calculation, (1) Heart rate calculation, According to the rPPG principle, the peak of the signal corresponds to the moment when the blood vessel volume is the largest, that is, the moment when the heart pumps blood; therefore, the interval between two peaks is one heart beat; for the collected rPPG signal, elgendi peak detection is first performed to obtain the peak position, and the heart rate corresponding to each peak is calculated in combination with the sampling frequency. In order to enhance the robustness of the result, the heart rate that falls within a reasonable range (50-130) is taken, and then the average is taken to obtain the final heart rate of the current 10-second video; (2) Blood oxygen calculation, Blood oxygen saturation refers to the percentage of oxygenated hemoglobin in the blood to the total hemoglobin. Oxygenated hemoglobin and deoxygenated hemoglobin have different absorption characteristics for light of different wavelengths. The absorption coefficient of Hb is higher between 600 and 800 nm, and the absorption coefficient of HbO2 is higher between 800 and 1000 nm. Since the wavelengths of green light and red light are relatively close, it is not easy to calculate. Therefore, red light and blue light are selected as reference signals in rPPG. The mean is used as the DC component, and the standard deviation is used as the AC component. According to the formula: The blood oxygen saturation is calculated, where A and B are laboratory fitting values, A=100, B=2.39; the value of B is adjusted according to the actual situation.

7. The method for monitoring student learning status based on facial recognition and mental stress analysis according to claim 1 is characterized in that: In step F; Step F1 : Feature screening, heart rate variability refers to the time fluctuation of heartbeat intervals, which reflects the ability of the autonomic nervous system to regulate the heart; when sympathetic nerve activity increases, the heart rate increases and the heart rate variability decreases; When parasympathetic nerve activity increases, heart rate slows down and heart rate variability increases; therefore, heart rate variability is the result of the interaction between sympathetic and parasympathetic nerves; heart rate variability analysis can be used to further measure psychological state and cognitive ability; this is also the basis for rPPG as model training data; Step F2: Time domain analysis. The time domain analysis method requires drawing the HRV change curve according to the RR interval and using mathematical statistical methods to calculate the characteristic value. SDNN, PNN50, RMSSD, and SDSD have become representative time domain indicators in the field of mental stress research. Step F3: Frequency domain analysis. HRV frequency domain analysis is to convert the time domain signal into the frequency domain, and use the Welch power spectrum to perform frequency domain analysis, which is based on the fast Fourier transform at the average window end. Frequency domain analysis reflects the activities of the sympathetic and parasympathetic nerves, as well as the balance state of the autonomic nervous system. The energy of the normal HRV power spectrum of the human body is between 0 and 0.4, which can be divided into high-frequency information, low-frequency information, and very low-frequency information. The energy of the spectrum graph is calculated to obtain the frequency domain characteristics; Step F4: nonlinear analysis. Nonlinear analysis uses the Poincare scatter plot analysis method. The current heartbeat cycle is affected by its previous one. In order to express the correlation between the two, the RR interval sequence is used. The two adjacent intervals are used as the coordinates of a factor of the Poincare point, that is, the current interval is the horizontal coordinate and the next interval is the vertical coordinate. Each point is drawn in the figure, thereby obtaining a scatter plot. The scatter plot draws an ellipse, the center of which is the average value of the heartbeat interval. Then, the major axis and minor axis of the ellipse can be obtained as SD2 and SD1 respectively.

8. The method for monitoring student learning status based on facial recognition and mental stress analysis according to claim 1, characterized in that: In step G, Step G1: Stress identification, mental stress assessment is based on stress induction experiments to obtain characteristic parameters related to stress state, and based on real life stress collection experiments, a model is built to realize mental stress analysis. The training set and the test set are split in a ratio of 7:3, 70% of the data is selected as the training set and 30% of the data is selected as the test set; the input of the model is 14-dimensional HRV features; Based on the random forest model, the 14-dimensional features obtained were used to identify mental stress; cross-validation grid search was used for parameter optimization; the default value of n_estimators was 100, and the parameter range was set to 1-101 with a step size of 10; the max_features parameter range was set to 1-11 with a step size of 1; Set max_depth to 5-15 with a step size of 1; perform grid search on n_estimators, max_features and max_depth within the set parameter range, calculate and compare the average scores of the random forest model for each group of parameters, and select the parameter combination with the highest final validation score of the ten-fold crossover as the final model parameter result according to the cross-validation grid optimization, and determine the optimal parameter combination of n_estimators, max_features and max_depth; and finally use the optimal parameters to train a new random forest model on the entire training set to obtain the optimal model; obtain the optimal model parameters according to the above cross-validation grid search method; Step G2: Analyze the correlation between the extracted RPPG features and known stress-related indicators; through correlation analysis and feature importance evaluation, screen out features that have significant contributions to stress assessment; Step G3: Stress classification and evaluation Input the RPPG features of the students to be evaluated into the trained stress model to obtain the stress classification results and stress evaluation values.

9. The method for monitoring student learning status based on facial recognition and mental stress analysis according to claim 1, characterized in that: In step H; Step H1: Real-time monitoring: During the students’ learning process, real-time collection and analysis of RPPG data are performed to provide real-time feedback on changes in students’ stress levels; Step H2: Long-term monitoring: monitor students’ stress over a long period of time, establish stress profiles, and track stress trends; Step H3: Personalized intervention: Provide educators with personalized intervention suggestions based on students’ stress levels and changing trends, including adjusting learning tasks and providing psychological counseling.

Images