Heart and physiological feature prediction method and device based on face information

Abstract

The invention discloses a heart physiological feature prediction method and device based on face information. The problems that in the prior art, abnormal heart physiological feature recognition is weak, generalization is poor, and interpretability is low are solved. The method comprises the following steps: deploying a camera in a target scene to collect an object face video, and extracting AU features, micro expressions and key point motion features based on an FACS standard after preprocessing; outputting basic emotion, abnormal emotion and behavior abnormity prediction results through standardization and multi-scale time sequence modeling; and continuously monitoring and generating an interpretable early warning report containing the thermodynamic diagram. The device comprises a data acquisition module, a feature extraction module, a time sequence modeling module and the like. The method realizes non-inductive and accurate monitoring, improves the abnormal recognition accuracy, and is suitable for education, workplaces and other scenes.

Description

Technical Field

[0001] This invention belongs to the field of information data processing technology, specifically relating to a method and device for predicting psychophysiological features based on facial information. Background Technology

[0002] Currently, whether it's tracking the psychological state of individuals in educational settings, assessing employee stress in the workplace, or monitoring patient emotions during medical follow-ups, timely and accurate acquisition of individual psychophysiological characteristics is crucial to support subsequent interventions. This need is particularly urgent in educational settings. Current monitoring of individuals' psychological states largely relies on periodic questionnaires or teachers' subjective observations. These methods are not only inherently lagging—for example, end-of-semester questionnaires often fail to capture sudden emotional fluctuations mid-semester—but are also prone to bias due to subjective judgment, failing to consistently reflect the true psychological changes of individuals.

[0003] Even in computer vision-based psychological state recognition technologies, existing solutions still have many technical limitations. Some solutions use convolutional neural networks to extract features from single-frame facial images and directly classify emotions, but can only recognize six basic emotions such as happiness, sadness, and anger, ignoring subtle changes in facial action units and the instantaneous nature of micro-expressions. Other solutions use simple temporal models such as recurrent neural networks and long short-term memory networks to process facial features, but lack modeling of long-term temporal contextual relationships, making it difficult to uncover the temporal evolution patterns of facial action unit combinations. At the same time, existing technologies have not systematically integrated with facial action coding system standards, resulting in insufficient utilization of facial action unit combination patterns related to complex psychological states, leading to insufficient ability to recognize abnormal psychological states such as anxiety, depression, and stress.

[0004] In addition, existing technologies face three major problems: First, they are weak in capturing and analyzing micro-expressions that are short in duration, and some key information indicating psychophysiological state is missed; second, the model decision-making process is black box-like and has poor interpretability, making it impossible for psychological counselors, medical staff and others to understand the basis for the generation of prediction results; third, in real-world scenarios, factors such as changes in lighting, head posture adjustment and partial occlusion can lead to a decrease in the model's generalization performance, which cannot meet the needs of practical applications. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides a method and apparatus for predicting cardiac physiological characteristics based on facial information. Through high frame rate facial video acquisition, refined facial action unit and micro-expression feature extraction, multi-scale temporal context modeling, and an interpretable early warning mechanism, it achieves imperceptible, continuous, and accurate prediction of individual cardiac physiological characteristics and early warning of abnormal risks.

[0006] The technical solution adopted in this invention is as follows: In a first aspect, the present invention provides a method for predicting psychophysiological features based on facial information, which extracts and analyzes facial expression information of a group of objects in a target scene, including the following steps: Step S1. First, deploy cameras in the target scene to collect facial video data of the target group, and perform face detection, alignment and cropping operations on the facial video data; Step S2. Then, based on the FACS standard, extract facial action unit (AU) features, micro-expression features, and facial key point motion features from the preprocessed face video data; Step S3. Align the obtained AU features, micro-expression features, and facial key point motion features along the time axis, and process the aligned feature sequences using sliding window processing and standardization algorithms to construct analysis samples; Step S4. Then, use a time series model to perform multi-scale time series modeling on the standardized feature sequence to fuse long-term and short-term contextual information. The classifier outputs the prediction results of the object's psychophysiological characteristics, which include basic emotions, abnormal emotions, and behavioral abnormalities. Step S5. Finally, repeat steps S1-S4 to continuously process the real-time face video stream, combine the historical psychophysiological feature prediction results to determine the risk pattern, and trigger an early warning and generate a visual report when a high-risk pattern is detected.

[0007] In conjunction with the first aspect, the present invention provides a first embodiment of the first aspect, which further includes an occlusion region detection step before performing feature extraction in step S2: The U-Net semantic segmentation model is used to identify the occluded regions in each frame of face image, generate pixel-level occlusion masks, and divide the face region into unoccluded and occluded regions.

[0008] In conjunction with the first aspect, the present invention provides a second implementation of the first aspect, wherein in step S2, the intensity value and occurrence probability of the core AU are extracted first from the unoccluded region; when an occluded region is detected, the association model is invoked to complete the AU features of the occluded region: If the object is an existing object with existing unoccluded data records, then an individual-specific association model is used to complete the data based on the historical statistical relationship between the unoccluded AU intensity value and the occluded AU. If it is a new object, the group prior association model is adopted and dynamically updated to an individual model through transfer learning; The extraction of facial key point motion features includes: adopting a 106-point facial key point expansion scheme, adding points on the forehead, temples, jawline, and periorbital area to the 68-point base; using the Kanade-Lucas-Tomasi optical flow tracing algorithm to calculate motion parameters for unoccluded key points; and predicting and continuing the motion trajectory based on the motion mapping relationship between adjacent unoccluded key points when a key point is occluded.

[0009] Among them, AU includes at least three of AU1, AU2, AU4, AU5, AU6, AU7, AU9, AU10, AU12, AU14, AU15, AU17, AU23, and AU24.

[0010] In conjunction with the first aspect, the present invention provides a third embodiment of the first aspect, wherein in step S3, the overlap rate of the sliding window processing is 50%, and the length of the sliding window is adjustable and the adjustment range is 0.5 seconds to 10 seconds; The standardization algorithm is Z-score standardization, and the calculation formula is z=(x-μ) / σ, where x is the original feature value, μ is the mean of the feature sequence, and σ is the standard deviation of the feature sequence. The labels of the analyzed samples are determined by experts in conjunction with the context of the facial video, and the label types are consistent with the types of psychophysiological features described in step S4.

[0011] In conjunction with the first aspect, the present invention provides a fourth implementation of the first aspect, wherein in step S4, the time series model is dynamically selected according to the number of monitored objects: when the number of objects is ≤10, a bidirectional long short-term memory network with added attention mechanism is used; when the number of objects is 10-30, a gated recurrent unit with added residual connection is used; and when the number of objects is >30, a Transformer encoder is used. Execution occurs when dynamically switching timing models: Feature caching: Real-time caching of the standardized feature sequence of the most recent 3 seconds as input to the new model; Delayed switching: A switch is triggered only after a 30-second period of change in the number of objects. For a smooth transition, the first 15 frames during the switch use a weighted fusion of the old and new models, with the weights gradually changing from 8:2 to 2:8.

[0012] Multi-scale temporal modeling is achieved through a hierarchical Transformer structure or a parallel multi-window feature extraction path. The scales of the parallel multi-window include 0.5 seconds, 3 seconds, and 10 seconds. The classifier includes a fully connected layer and a Softmax layer. The fully connected layer maps the fused temporal features into feature vectors, and the Softmax layer converts the feature vectors into predicted probabilities of cardiac and physiological features.

[0013] In conjunction with the first aspect, the present invention provides a fifth implementation of the first aspect, which further includes a model training and optimization step, using a weighted cross-entropy loss function to calculate the prediction loss, and assigning a weight of 1.5-3 times that of the base emotion samples to samples of abnormal emotion and abnormal behavior categories. The Adam optimizer is used for model optimization. During the optimization process, gradient clipping with a threshold of 1.0-2.0 is used to prevent gradient explosion, and early stopping is used to prevent overfitting. Attention regularization with weights of 0.01-0.1 is used to constrain the model to focus on the AU region consistent with psychological priors.

[0014] In conjunction with the first aspect, the present invention provides a sixth implementation of the first aspect, wherein attention regularization is adopted in the model training stage, and key regions such as forehead, eyelids, and corners of mouth are masked based on the FACS standard and psychological prior annotation, and a regularization term λ·L_reg is added to the loss function, with λ taking the value of 0.01-0.1, to constrain the model to reduce the attention weight of irrelevant regions.

[0015] In conjunction with the sixth embodiment of the first aspect, the present invention provides a seventh embodiment of the first aspect, wherein the visualization report in step S5 includes: Grad-CAM decision heatmaps are generated by weighted stacking of feature maps from the model's convolutional layers and then superimposed onto the original face image. The AU time sequence logic table records the occurrence time, duration, intensity value, and psychological interpretation of the combination rules of key AUs within the detection period; as well as risk level labeling and intervention suggestions.

[0016] It is worth noting that regularization is a training constraint, while Grad-CAM is a post-hoc interpretation. During the training phase, it constrains the model's prior attention, and during the inference phase, it uses Grad-CAM to visualize whether the actual areas of attention match psychological patterns.

[0017] In conjunction with the first aspect, the present invention provides an eighth embodiment of the first aspect, wherein in step S5, the continuous processing period is 30 seconds, and a cardiac physiological characteristic index is output once per period, wherein the index is a weighted sum of the predicted probabilities of each cardiac physiological characteristic.

[0018] Secondly, the present invention also provides a cardiac physiological feature prediction device based on facial information, used to perform the above-mentioned cardiac physiological feature prediction method based on facial information, comprising: Data acquisition and preprocessing module: used to deploy cameras in the target scene to collect facial video data of the target object, perform face detection, alignment and cropping operations on the facial video data, and cooperate with experts to complete the labeling of psychophysiological features; Multi-dimensional feature extraction module: used to extract AU features, micro-expression features and facial landmark motion features from preprocessed face video data based on the FACS standard; Feature standardization module: used to align the AU features, micro-expression features, and facial key point motion features along the time axis, and construct analysis samples through sliding window processing and Z-score standardization algorithm; The temporal modeling and classification module dynamically selects a temporal model based on the number of objects, performs multi-scale temporal modeling on standardized feature sequences, and outputs cardiac physiological feature prediction results through a classifier; and Continuous monitoring and early warning module: It is used to continuously process real-time face video streams, calculate the decline of the cardiophysiological characteristic index by combining historical prediction results, identify high-risk patterns and trigger early warnings, and generate a visual report containing Grad-CAM heatmap.

[0019] The camera in the data acquisition and preprocessing module can reuse existing equipment in educational scenarios, and the frame rate can be increased from 25-30fps to 60fps through a frame rate enhancement algorithm; the time series modeling and classification module is deployed on edge computing devices or servers such as smart gateways; the warning output methods of the continuous monitoring and early warning module include local pop-ups, remote terminal push, or SMS notifications.

[0020] The multi-dimensional feature extraction module includes: AU Feature Extraction Unit: Configured with OpenFace toolkit or CNN model to output AU intensity value and occurrence probability; Micro-expression extraction unit: Configured with detection algorithms based on optical flow features or spatiotemporal features to output micro-expression category, duration, peak intensity, and frame position parameters; and Key point motion extraction unit: used to extract 68 or 106 facial key points, calculate displacement vectors, velocities and accelerations and generate motion trajectory sequences.

[0021] The beneficial effects of this invention are as follows: (1) This invention can significantly improve the early recognition accuracy of abnormal psychophysiological states. Existing technologies are insufficient in recognizing abnormal states such as anxiety and depression because they ignore the combination patterns of micro-expressions and facial action units. This invention, based on the facial action coding system standard, accurately extracts the appearance, intensity, duration and micro-expression characteristics of facial action units from high frame rate videos. At the same time, it combines the motion trajectory of facial key points to fully capture subtle signals related to abnormal psychophysiological states. Furthermore, by integrating long and short-term contextual information through multi-scale temporal modeling, it deeply explores the evolutionary rules of facial action unit combinations. Compared with traditional visual methods, it improves the recognition accuracy and recall rate of abnormal states such as anxiety and depression, and can capture the precursor signals of abnormal psychophysiological states earlier. (2) In response to interference factors such as changes in lighting, head posture adjustment, and partial occlusion in real scenes, this invention adopts Z-score standardization in the feature processing stage to eliminate individual differences and lighting effects. In the data acquisition stage, it supports collaborative processing of multi-view videos to make up for the information loss caused by occlusion. At the same time, it balances feature information of different time dimensions through multi-scale temporal modeling to reduce the impact of single scene factors on model prediction results. (3) This invention improves the interpretability of model decision-making and enhances the credibility of results. Attention regularization is introduced during the model training stage to guide the model to focus on facial action unit regions that are consistent with psychological priors. At the same time, Grad-CAM technology is used to generate decision heatmaps. During the early warning stage, the generated visualization report includes specific evidence of facial action unit combinations, micro-expression parameters and heatmaps, so that psychological counselors, medical staff and others can clearly understand the basis for the generation of prediction results, enhance their trust in the system results, and provide support for the formulation of subsequent intervention measures. Detailed Implementation

[0022] The present invention will be further explained below with reference to specific embodiments.

[0023] Example 1: This embodiment discloses a method for predicting psychophysiological features based on facial information. Targeting a specific scene and a group of objects within that scene, this embodiment outlines the basic implementation framework for building a seamless and accurate psychophysiological state monitoring system. It addresses core issues of traditional methods, such as limited feature dimensions and lack of temporal modeling. The specific process is as follows: Step 1: Data Acquisition and Preprocessing First, cameras are deployed in the target scene to collect facial video data of the subjects. During the model training phase, a labeling group composed of three or more psychological experts observes the AU combinations, micro-expression characteristics, and key point motion trajectories in consecutive frames of the collected facial video data, and jointly evaluates the basic emotion / abnormal emotion / abnormal behavior labels corresponding to each video segment. Samples with labeling consistency of over 90% are included in the training set. The collected facial video data is preprocessed, and the MTCNN algorithm is used for face detection to locate the facial regions. Based on the detected facial key points, an affine transformation algorithm is used to achieve face alignment and eliminate the influence of head pose shift. Finally, the aligned facial regions are cropped, background interference is removed, and a standardized facial image sequence is obtained.

[0024] Step 2: Multi-dimensional facial feature extraction Based on the FacialActionCodingSystem standard, three types of features are extracted from the preprocessed face image sequence: facial action unit features are processed by the OpenFace toolkit to process continuous frames, outputting the intensity value and occurrence probability of each facial action unit, with a focus on extracting facial action units related to psychophysiological state; Micro-expression features: Micro-expression segments are detected using optical flow. By calculating pixel motion vectors between adjacent frames, short-duration micro-expressions are identified, and the micro-expression category, duration, intensity peak, and start and end frame positions are extracted. Facial key point motion features: 68 facial key points were extracted using the Dlib library. The displacement vectors, motion velocities, and motion accelerations of the key points were calculated between consecutive frames to form a sequence of key point motion trajectories.

[0025] Step 3: Feature sequence standardization The extracted facial motion unit feature sequences, micro-expression feature sequences, and facial key point motion feature sequences are aligned along the time axis to ensure synchronization in the time dimension. A sliding window process is used to segment the aligned feature sequences, with a sliding window overlap rate set to 50%. The features within each sliding window are processed using the Z-score normalization algorithm to eliminate interference from individual facial differences and lighting variations. The normalized features within each sliding window constitute an analysis sample. Psychological experts then label the samples based on the normalized features within the sliding windows, combined with facial video clips from the corresponding time periods.

[0026] Step 4: Multi-scale temporal modeling and classification A Transformer encoder is used as the temporal model to perform multi-scale temporal modeling on the standardized feature sequence. A self-attention mechanism is employed to capture the synergistic or inhibitory effects between different facial action units, as well as long-distance temporal dependencies. Simultaneously, different time windows are set to adapt to feature analysis of micro-expressions, combinations of facial action units, and state trends. The multi-scale fused features output from the Transformer encoder are input into a fully connected layer. The fully connected layer maps the fused features into fixed-dimensional feature vectors, which are then converted into predicted probabilities for each cardiac and physiological feature category via a Softmax layer, outputting the predicted cardiac and physiological feature results.

[0027] Step 5: Continuous monitoring and explainable early warning The trained model is deployed to an edge computing device (such as a smart gateway) in the classroom to continuously process real-time facial video streams, repeating steps 1-4 for a fixed processing cycle. Based on historical psychophysiological feature predictions, the model determines whether the current psychophysiological state conforms to a high-risk pattern. If a high-risk pattern is detected, an early warning mechanism is triggered, and a visual report is generated. The report includes key facial movement unit combinations, micro-expression parameters, and decision-making basis for reference by psychological counselors.

[0028] The above-mentioned scheme can predict psychophysiological characteristics under conditions of no occlusion and sufficient computing power. However, in actual deployment in educational scenarios, it was found that problems such as facial occlusion, limited computing power of edge devices, and black box of model decision-making led to insufficient system robustness. To address this, the following Examples 2-4 progressively optimize occlusion robustness, computational efficiency, and interpretability, respectively.

[0029] Example 2: This embodiment addresses the issues of interrupted feature extraction and a sharp drop in prediction accuracy in the basic embodiment under facial occlusion scenarios. It provides an occlusion-adaptive feature completion and trajectory continuation scheme. Facial occlusion is a common problem in fixed multi-object environments such as educational scenarios, requiring optimized design.

[0030] In daily teaching, students may have their face, mouth, nose, jaw, and other areas obscured due to behaviors such as long bangs covering their face, looking down and covering their mouth and nose with books, or touching their face with their hands.

[0031] In the above embodiments, the facial motion units corresponding to the occluded area cannot be effectively extracted, and the tracking of facial key points is interrupted, which leads to deviations in the subsequent judgment of abnormal psychophysiological states such as anxiety and depression.

[0032] To address this issue, we focus on two aspects: facial motion unit feature completion and facial key point trajectory continuation, to ensure the integrity and continuity of feature extraction in occluded scenarios.

[0033] Construction of occlusion area determination module An occlusion region detection model is built using the U-Net semantic segmentation algorithm. The specific steps are as follows: Model training: Collect a dataset of faces occluded in educational scenarios, and label the occluded and unoccluded areas of each image.

[0034] The dataset was divided into training, validation, and test sets in a 7:2:1 ratio to train the U-Net semantic segmentation model. The model was trained to identify occluded regions in the input face image in real time and output a pixel-level mask of the occluded region. The mask value of 1 indicates occlusion and 0 indicates no occlusion. The accuracy of the model in identifying occluded regions should reach more than 95%.

[0035] Real-time judgment process In the feature extraction stage, the preprocessed single-frame face image is first input into the trained U-Net model to obtain the occlusion region mask. Based on the mask, the unoccluded region and the occluded region are divided, and different feature extraction priorities are assigned to the two types of regions. Facial action units in the unoccluded region are processed first, and then facial action units in the occluded region are completed through the association model.

[0036] Facial motion units in unobstructed areas are extracted first. Based on the FacialActionCodingSystem standard, the core facial motion units in the unobstructed area were identified and extracted using the OpenFace toolkit.

[0037] The selection of core unobstructed AUs focuses on the forehead, eyes, and eyebrows. The key extractions are AU1 (inner eyebrow elevation), AU2 (outer eyebrow elevation), AU4 (brow furrowing), AU5 (upper eyelid elevation), and AU7 (eyelid tension). All of these AUs are located in the unobstructed area and are not affected by occlusion.

[0038] Then, the image patch of the unoccluded area is input into the OpenFace toolkit. The toolkit extracts the texture and grayscale variation features of the image patch through convolutional layers, outputs the intensity value and occurrence probability of each core AU through fully connected layers, and records the occurrence time and duration of each AU in consecutive frames.

[0039] Facial motion unit correlation completion in occluded areas We construct an individual-specific unoccluded-occluded AU association model, and supplement the features of occluded AUs by analyzing the intensity and occurrence patterns of unoccluded AUs, as detailed below: Association Model Establishment: During the period when the object is unobstructed, facial motion unit data is continuously collected, and the strength correlation between the unobstructed area AU and the occluded area AU is statistically analyzed.

[0040] For example, by calculating the Pearson correlation coefficient between the two, it was found that: When the AU4 intensity value is ≥3, the probability of the AU15 intensity value being ≥2 is 88%, and the probability of the AU23 intensity value being ≥1 is 82%. Based on this statistical law, a linear regression model is established, with the AU4 intensity value as the independent variable and the AU15 and AU23 intensity values ​​as the dependent variables.

[0041] When an occluded region is detected, the object's dedicated association model is invoked. The real-time extracted AU intensity values ​​of the unoccluded region are input into the model, and the model outputs the predicted intensity value of the AU of the occluded region. At the same time, the predicted value is fine-tuned by combining the historical intensity trend of the AU of the occluded region in the 10 frames prior to occlusion. For example, the predicted intensity value of AU15 is adjusted to level 2.1 to ensure that the completed AU features conform to the object's action pattern and avoid abrupt jumps.

[0042] If the object is a new object and there is a lack of historical unoccluded data to build an individual-specific unoccluded-occluded AU association model, AU feature completion is achieved through a group prior model and dynamic accumulation of individual data. The specific steps are as follows: Population Prior AU Association Model Construction We collected an unoccluded face dataset of subjects from the same scene and age group, and labeled the intensity values ​​and occurrence patterns of core Active Faces (AUs) based on the FACS standard. Then, we used the Pearson correlation coefficient to calculate the intensity correlation between unoccluded and occluded AUs. For example, statistics showed that when AU4 intensity ≥ 3, the group probability of AU15 intensity ≥ 2 was 82%, and the group probability of AU23 intensity ≥ 1 was 79%. Based on this pattern, we constructed a linear regression model: AU occlude =β 0 +β 1 ×AU4+β 2 ×AU7+ϵ in β 0 For the intercept term, β 1 , β 2 The regression coefficients are obtained through training with population data and are controlled to be within 0.3.

[0043] The group prior model is pre-set into the multi-dimensional feature extraction module as the initial completion model when new users are occluded.

[0044] Dynamic accumulation and model transfer of individual new user data When a new user enters the target scene for the first time, the system automatically starts the lightweight unobstructed data acquisition mode. Under the natural state of the object looking up and not being obstructed, it prioritizes the acquisition of 30 minutes of 60fps video data and extracts the AU sequence under the unobstructed state without manual intervention.

[0045] Then, transfer learning is used to apply the parameters of the group prior model as initial values. Every 5000 frames of valid, unobstructed data are accumulated, and the coefficients of the linear regression model are adjusted accordingly. β 1 , β 2 Perform iterative updates, with the update formula as follows: β t =0.95×β t−1 +0.05×β individual in β t−1 The coefficient for the previous round, β individual The coefficient is calculated based on individual data of new users, and 0.95 is the forgetting factor.

[0046] Completion model switching threshold: The completion accuracy is calculated in real time. When the completion accuracy is ≥85% for 3 consecutive times, the completion model is automatically switched from the group prior model to the new user individual model to ensure long-term completion accuracy.

[0047] Next, the extraction and optimization of facial key point motion features, specifically the continuation of occluded region trajectories, were performed. Specifically, for the 106-point facial key point extension design, 38 new key points were added to the basic implementation of 68 facial key points, focusing on supplementing the subdivided points of the facial contour and unoccluded areas, as follows: The newly added point distribution includes 8 points in the forehead area, 6 points in the temporal region, 12 points in the jawline, and 12 points around the eyes.

[0048] Keypoint annotation and model training adopted the keypoint detection framework of the Dlib library. Based on the expanded 106-point annotation dataset, that is, 38 additional point annotations were added to the original 68-point dataset, resulting in a total of 50,000 annotated images. The keypoint detection model was retrained so that the model can output the pixel coordinates of 106 keypoints in real time, and the keypoint detection error of the model is controlled within 2 pixels.

[0049] Then, the Kanade-Lucas-Tomasi optical flow tracing algorithm is used, combined with the motion law of key points in the unoccluded area, to realize the continuation of the trajectory of key points in the occluded area. When there is no occlusion, the pixel coordinate changes of 106 key points in consecutive frames are tracked through the KLT algorithm, and the displacement vector, motion velocity and acceleration of each key point are calculated to form the motion trajectory sequence of each key point.

[0050] When a keypoint is occluded, the adjacent unoccluded keypoints are first identified. The motion vectors and velocities of the adjacent unoccluded keypoints are calculated using the KLT algorithm. Based on the correlation of facial muscle movements, a motion mapping relationship between the adjacent keypoints and the occluded keypoints is established. According to this mapping relationship, the pixel coordinates of the occluded keypoints in consecutive frames are predicted, and their motion trajectory is continued. When the occlusion disappears, the model automatically switches the predicted coordinates to the actual detected coordinates, achieving a smooth transition of the trajectory.

[0051] For example, in a high school classroom with 50 students, 30 wore masks, and 10 occasionally covered their mouths and noses with books. A one-month verification test was conducted to compare the feature extraction effects of the basic implementation and this optimized implementation. Verification shows that in the basic embodiment, the effective extraction rate of AU in the occluded area is only 32%; in this optimized embodiment, after correlation completion, the effective extraction rate of the above AU is increased to 91%, of which the extraction accuracy of AU15 reaches 89%, which can effectively support the subsequent judgment of abnormal cardiophysiological state.

[0052] In the basic embodiment, the occlusion caused a 45% interruption rate in the trajectory of key points; in this optimized embodiment, the trajectory interruption rate was reduced to 8%. Even in a scenario where the mask covers the face for 10 seconds, the trajectory continuity error of key points around the mouth and nose is still controlled within 3 pixels, ensuring the continuity of features such as motion speed and acceleration.

[0053] For the two abnormal states of anxiety and depression, the prediction accuracy of the basic implementation is 68% and 65% respectively in the occluded scenario; the prediction accuracy of this optimized implementation is improved to 86% and 83% respectively, which is close to the prediction level in the unoccluded scenario and fully meets the monitoring needs in the education scenario.

[0054] However, the problem of resource waste caused by the dynamic change of the number of objects in different scale scenarios and the high computational complexity of the Transformer model has not yet been solved. Therefore, Example 3 will further elaborate on the computational efficiency optimization scheme.

[0055] Example 3: Furthermore, the Transformer encoder used in the above embodiments is based on a self-attention mechanism, and its computational complexity increases with the square of the feature sequence length. In educational scenarios of different scales, the amount of data varies significantly. For example, in lower elementary grades with fewer than 10 students per class, the amount of facial data to be processed per frame is small, but the Transformer's self-attention layer still needs to complete the full sequence association calculation, resulting in wasted computing power on edge devices and excessively long processing time per frame. In contrast, in middle school with fewer than 30 students per class, the amount of data is moderate, and while the computational time of the Transformer is acceptable, there is still room for optimization. Directly using it would increase device power consumption.

[0056] To address the issue of mismatched computing resources caused by differences in data volume, this embodiment provides a data volume-model complexity matching and seamless switching mechanism, specifically offering two types of time series model alternatives to achieve data volume-model complexity matching.

[0057] Option 1: Add a bidirectional LSTM with an attention mechanism to adapt to small datasets in a single classroom. (1) Construction of bidirectional LSTM infrastructure Using the standardized feature sequence from the basic embodiment as input, where the sequence length is the number of frames within the sliding window (e.g., a 3-second sliding window corresponds to 180 frames), and the feature dimension is the total dimension of AU intensity, micro-expression parameters, and key point motion features (e.g., 64 dimensions), a bidirectional LSTM network is constructed.

[0058] The forward LSTM layer and the backward LSTM layer are set to process the feature sequence in parallel. The forward LSTM propagates forward from frame 1 to frame 180 to capture the temporal dependency from the past to the present. The backward LSTM propagates backward from frame 180 to frame 1 to capture the temporal dependency from the present to the future.

[0059] The number of hidden layer units in both the forward and backward LSTMs is set to 256, and the activation function is tanh to avoid gradient saturation. The output of each LSTM layer is the hidden state sequence in that direction (H_forward for forward and H_backward for backward). The two are concatenated in the time dimension to obtain the initial output H_bi=[H_forward,H_backward] of the bidirectional LSTM.

[0060] (2) Integration of attention mechanisms A time-step attention layer is added after the bidirectional LSTM output to enhance the focus on key time nodes, as follows: Attention weights are calculated by performing a linear transformation on H_bi to obtain a feature matrix with a dimension of 180×64. The attention weight α_t for each time step is calculated using the softmax function, where α_t∈[0,1] and the sum of all α_t is 1. The higher the weight value, the more critical the feature of that time step is to the judgment of the psychophysiological state.

[0061] The attention-weighted output multiplies H_bi at each time step with the corresponding α_t, and then sums them over all time steps to obtain a fixed-dimensional global feature vector F_att. This vector has been fused with key temporal information and redundant frame features have been removed.

[0062] The classification mapping involves inputting F_att into a fully connected layer, activating it via ReLU, and then inputting it into a Softmax layer to output the predicted probability of cardiac physiological features. The process is consistent with the basic implementation.

[0063] Option 2: Add a GRU with residual connections to adapt to medium data volumes in a single classroom. (1) Simplified design of GRU infrastructure GRU reduces the number of parameters by 30% and lowers computational complexity by merging the input gate and forget gate of LSTM into the update gate, retaining only the update gate and reset gate. The specific structure is as follows: Gated computation: For the input feature sequence X, calculate the update gate z_t and the reset gate r_t: z_t=σ(W_z・[h_{t-1},X_t]), r_t=σ(W_r・[h_{t-1},X_t]), where σ is the sigmoid function, W_z and W_r are weight matrices, and h_{t-1} is the hidden state at the previous time step; Candidate hidden state calculation: h_t'=tanh(W_h・[r_t⊙h_{t-1},X_t]), where ⊙ is element-wise multiplication and W_h is the candidate hidden state weight matrix; The final hidden state is calculated as: h_t=(1-z_t)⊙h_{t-1}+z_t⊙h_t'. The fusion ratio between the previous state and the current candidate state is controlled by the update gate, thereby realizing the transmission of temporal information.

[0064] (2) Residual connection integration To address the vanishing gradient problem caused by increasing GRU depth, residual connections are added after the hidden layer output of each GRU layer: The residual path design directly connects the output h_i of the i-th layer GRU to the input X_i of that layer (i.e., the residual path is X_i→h_i+X_i). If the dimensions of h_i and X_i are inconsistent, the dimensions of X_i are adjusted to match h_i through a 1×1 convolutional layer.

[0065] Gradient propagation optimization and residual connections enable gradients to propagate backward through direct paths, avoiding gradual decay of gradients in multiple GRU layers. Even with four stacked GRU layers, the gradient magnitude can still maintain more than 60% of the initial value, ensuring effective training even after the model depth is increased.

[0066] For feature output and classification, the hidden states of the last GRU layer (after residual connection) are globally averaged to obtain the feature vector F_gru (dimension 256), which is input into the fully connected layer and the Softmax layer, and the output is the predicted probability.

[0067] (3) Optimization effect verification The validation was conducted in two typical educational scenarios, with the following hardware environment: The classroom edge smart gateway, with a quad-core ARM Cortex-A55 CPU and 2GB of memory, compares the performance of the basic embodiment with this optimized embodiment: Scene 1: Elementary school classroom (lower grades) Computation time: The basic implementation takes 120ms to process a single frame, while this optimized solution 1 takes 72ms to process a single frame, a reduction of 40% in processing time; Prediction accuracy: The accuracy of anxiety state recognition decreased from 92% in the basic implementation to 90%, and the accuracy of depression tendency recognition decreased from 90% to 88%, both of which are within acceptable ranges and meet the requirements for real-time monitoring. Scene 2: Middle School Classroom The basic implementation takes 200ms to process a single frame, while this optimized solution 2 takes 150ms, a 25% reduction in processing time. The accuracy of anxiety state recognition is reduced from 93% to 92.2%, and the accuracy of depression tendency recognition is reduced from 91% to 90.5%, a decrease of less than 1%. At the same time, the device's CPU utilization is reduced from 75% to 50%, preventing the device from lag due to high load.

[0068] Furthermore, when objects in a scene change dynamically due to special reasons, such as the number of objects in a school scene changing dynamically due to lateness, leave, or temporary extra seating, a combination of feature caching, delayed switching, and resource reservation is used to avoid prediction interruptions caused by model switching. The specific steps are as follows: Multi-model input feature unification and caching mechanism By unifying the bidirectional LSTM, adding residual connections to the GRU, and fixing the input feature dimension of the Transformer to 64 dimensions after fusing AU features, micro-expression features, and key point motion features, different models can directly call the same feature sequence.

[0069] Then, a feature cache pool is built in the temporal modeling and classification module to cache the standardized feature sequences of the most recent 3 seconds in real time and mark the integrity label of each sequence. When the model switching is triggered, the effective feature sequences in the cache pool are read first as the initial input of the new model, without having to process the real-time video stream from scratch, thus avoiding 2-3 seconds of prediction interruption.

[0070] Hysteresis threshold and smooth transition strategy Set a dynamic switching threshold for the number of objects. When the number of objects increases from ≤10 to >10, the state must be maintained for 30 seconds before the bidirectional LSTM to GRU switch is triggered. When the number of objects decreases from >10 to ≤10, the state must be maintained for 20 seconds before the GRU to bidirectional LSTM switch is triggered, thus avoiding frequent switching in a short period of time.

[0071] During model switching, a weighted fusion output strategy is adopted. The prediction result of the first 5 frames = old model output × 0.8 + new model output × 0.2, the result of the 6th to 10th frames = old model output × 0.5 + new model output × 0.5, the result of the 11th to 15th frames = old model output × 0.2 + new model output × 0.8, and from the 16th frame onwards, the new model output is used completely, which reduces the accuracy fluctuation at the moment of switching from 10%-15% to less than 3%.

[0072] Edge device resource reservation When the edge computing device starts up, a bidirectional LSTM, GRU, and Transformer backbone network are preloaded, keeping memory usage below 200MB. CPU utilization is monitored in real time; when utilization is ≥70%, non-core layer parameters of inactive models are automatically released. When a model switch is triggered, CPU resources are prioritized for the new model to avoid lag due to insufficient resources during the switch.

[0073] Example 4: Furthermore, the output of the above embodiment only includes the predicted probability of psychophysiological characteristics and a simple list of AUs, such as an anxiety probability of 85%, involving AU4 and AU7. In practice, there are problems such as opaque model decision-making process and lack of temporal logic for AU combination. This embodiment is optimized and limited accordingly.

[0074] 1. During the training phase, attention regularization constraints Based on FACS standards and psychological research findings, facial regions strongly correlated with abnormal psychophysiological states are identified as key areas for the model to focus on. Anxiety-related areas include the forehead and upper eyelid area. Areas associated with depressive tendencies include the corners of the mouth and the chin area. In the face images of the training dataset, the pixel ranges of the above regions are manually labeled. The forehead region is labeled with y coordinates 0-50 and x coordinates 20-80; the upper eyelid region is labeled with y coordinates 50-80 and x coordinates 30-70. A priori region mask is then generated.

[0075] 2. Incorporating the attention regularization term into the loss function Based on the weighted cross-entropy loss function of the basic embodiment, an attention regularization term is added to constrain the model's attention to irrelevant regions: Basic loss function: Weighted cross-entropy loss L_ce=-Σ(y_i・log(p_i)・w_i), where y_i is the true label, p_i is the predicted probability, and w_i is the class weight (w_i=2 for abnormal states and w_i=1 for basic sentiment).

[0076] Attention regularization term calculation: Obtain the attention weight map A (dimension consistent with the input face image) output by the model convolutional layer, calculate the sum of attention weights of irrelevant regions L_reg=Σ(A・(1-M)), where M is the prior region mask; the regularization term is finally λ・L_reg, where λ is the regularization coefficient, set to 0.05, to balance the loss and constraint strength.

[0077] The total loss function is: L_total = L_ce + λ・L_reg. During model training, L_total is minimized so that the model automatically reduces the attention weight on irrelevant regions and strengthens the attention to prior regions while learning the classification task.

[0078] 3. Optimization of model training process The Adam optimizer was used, with a training cycle of 30 epochs. The model's attention rate to the prior region was validated every 5 epochs. In the initial training phase: the attention rate of the prior area increased from 45% to 70%, while the attention weight of irrelevant areas gradually decreased; In the later stages of training: the prior region attention rate stabilizes at over 85%, ensuring that the model's decisions mainly rely on psychologically relevant facial region features.

[0079] 4. Output Stage: Visualizing Decision Basis and Logical Labeling Using Grad-CAM technology, the key areas of interest in the model are visualized. Specific steps include: Feature Map and Gradient Extraction: Input the face image to be analyzed, obtain the feature map F of the last convolutional layer of the model, and calculate the gradient G of the target class with respect to F; Channel weight calculation: Calculate the spatial average of the gradient G for each channel to obtain the channel weight α_c=(1 / (H×W))・Σ(G_c,h,w). The larger α_c is, the more important the channel feature is to the target category. Heatmap generation: The feature map F is weighted and summed with the channel weights α_c to obtain a coarse heatmap F_cam=Σ(α_c・F_c); ReLU activation is applied to F_cam, and then the size is enlarged to match the input face image through bilinear interpolation to obtain the final heatmap; Heatmap overlay: The heatmap is overlaid on the original face image with a red gradient to visually show the area of ​​interest of the model.

[0080] This report analyzes the relationship between AU combinations and psychophysiological states from a temporal perspective, and includes a time-series table of AU combinations and logical explanations. AU combination timing table: records the occurrence time, duration, and intensity value of key AUs within the detection period, for example: AU type Time of occurrence (seconds) Duration (seconds) Strength value AU4 May 25 20 3 AU7 August 22 14 2 AU12 0-30 2 1 Logical explanation: Based on the interpretation of the AU combination using psychological rules, AU4 (frowning) appears continuously from 5 to 25 seconds (intensity level 3), AU7 (eyelid tension) appears concurrently from 8 to 22 seconds (intensity level 2), and the frequency of AU12 (smile) decreases from 5 times / minute to 0.07 times / minute (only once in 30 seconds). This is consistent with the typical characteristics of frowning, eyelid tension, and reduced pleasant expressions in an anxious state, so it is judged to be an anxious emotion.

[0081] The report comprises three main modules to ensure that mental health counselors can quickly understand it: Heatmap module: Displays the overlaid face heatmap, highlighting areas of high interest; AU Timing Module: Displays the timing table and intensity change curve of key AUs, where the intensity of AU4 increases from level 2 at 5 seconds to level 3 at 10 seconds, and then stabilizes at 25 seconds; Logical conclusion module: Clearly indicate the basis for judgment, namely the AU combination and frequency changes and risk level, and provide preliminary intervention suggestions. It is recommended to communicate with the subject within 1 hour and pay attention to recent academic pressure.

[0082] 5. Effect Verification A two-month verification was conducted in the psychological counseling room of a high school: Increased trust: Through a questionnaire survey, the psychological counselors' trust in the prediction results increased from 6.0 points in the basic implementation example to 8.8 points. Eight counselors indicated that the heat map and AU time series table could be directly used as the basis for communication. Report interpretation efficiency: Twenty warning reports were randomly selected, and the interpretation time for instructors was statistically analyzed. The basic implementation plan averaged 5 minutes per report, while this optimized implementation plan averaged 2.5 minutes per report, a reduction of 50%. Improved intervention efficiency: The time from receiving the warning report to completing the first communication and intervention with the subject was shortened from 4 hours in the basic implementation to 2 hours. The number of abnormal status cases successfully intervened within 2 months increased from 32 to 46, and the intervention success rate increased by 43.75%.

[0083] Example 5: The above implementation only covers images with conventional resolution. To further address the issue of low-resolution images, the following optimizations are proposed.

[0084] When the resolution of face images in the target scene is low, the visualization accuracy of Grad-CAM heatmaps is improved through multi-scale convolutional feature fusion and key point correction. The specific steps are as follows: Multi-scale convolutional feature fusion The feature extraction logic of the original Grad-CAM was modified. In addition to retaining the last convolutional layer of the model, a third convolutional layer of the model was added as a feature source.

[0085] Calculate the gradient weights of the target category with respect to the high-level and mid-level convolutional feature maps, respectively. The channel weights of the high-level feature maps are as follows: The channel weights of the mid-layer feature map are as follows: Where G is the gradient value, and H and W are the feature map sizes; Heatmap generation: The final heatmap is generated using a weighted fusion formula. in , (These are high-level and mid-level feature maps, with weights of 0.6 and 0.4 respectively). The fused heatmap is enlarged to the same size as the original face image through bilinear interpolation, reducing the localization error from 5 pixels to less than 2 pixels.

[0086] Facial landmark coordinate correction Based on the detection results of 106 facial landmarks, unoccluded stable landmarks were selected as anchor points, such as the eye region, eyebrow region, and facial contour region, to ensure accurate detection of anchor points even in low-resolution images. The affine transformation matrix M between the low-resolution face image and the standard face template (128×128 pixels) was calculated using the following formula: Where a, b, d, and e are scaling and rotation coefficients, and c and f are translation coefficients; the coordinates of the fused heatmap are corrected through matrix M to ensure that the visualization positions of the forehead region corresponding to AU4 and the eyelid region corresponding to AU7 are consistent with the standard face, avoiding regional offset caused by low resolution.

[0087] Dynamic resolution adaptation mode A resolution judgment unit is added to the continuous monitoring and early warning module to calculate the resolution of the input face image in real time.

[0088] When the resolution is less than 64×64 pixels, the detail enhancement mode is automatically enabled, increasing the weight of mid-level features to 0.5, and the motion trajectory of key points is superimposed on the heatmap; when the resolution is greater than or equal to 64×64 pixels, the original high-level feature-dominated mode is maintained, balancing accuracy and computational efficiency.

[0089] This invention is not limited to the optional embodiments described above, and anyone can derive other various forms of products based on the inspiration of this invention. The specific embodiments described above should not be construed as limiting the scope of protection of this invention; the scope of protection of this invention should be determined by the claims, and the specification can be used to interpret the claims.

Claims

1. A method for predicting psychophysiological features based on facial information, which extracts and analyzes facial expression information of a group of objects in a target scene, characterized in that... Includes the following steps: Step S1. First, deploy cameras in the target scene to collect facial video data of the target group, and perform face detection, alignment and cropping operations on the facial video data; Step S2. Then, based on the FACS standard, extract facial action unit (AU) features, micro-expression features, and facial key point motion features from the preprocessed face video data; Step S3. Align the obtained AU features, micro-expression features, and facial key point motion features along the time axis, and process the aligned feature sequences using sliding window processing and standardization algorithms to construct analysis samples; Step S4. Then, use a time series model to perform multi-scale time series modeling on the standardized feature sequence to fuse long-term and short-term contextual information. The classifier outputs the prediction results of the object's psychophysiological characteristics, which include basic emotions, abnormal emotions, and behavioral abnormalities. Step S5. Finally, repeat steps S1-S4 to continuously process the real-time face video stream, combine the historical psychophysiological feature prediction results to determine the risk pattern, and trigger an early warning and generate a visual report when a high-risk pattern is detected.

2. The method for predicting cardiac physiological features based on facial information according to claim 1, characterized in that, Before performing feature extraction in step S2, an occlusion region detection step is also included: The U-Net semantic segmentation model is used to identify the occluded regions in each frame of face image, generate pixel-level occlusion masks, and divide the face region into unoccluded and occluded regions.

3. The method for predicting psychophysiological features based on facial information according to claim 1, characterized in that, In step S2, the intensity value and occurrence probability of the core AU are extracted first from the unoccluded region; when an occluded region is detected, the association model is called to complete the occluded region AU. feature: If the object is an existing object with existing unoccluded data records, then an individual-specific association model is used to complete the data based on the historical statistical relationship between the unoccluded AU intensity value and the occluded AU. If it is a new object, the group prior association model is adopted and dynamically updated to an individual model through transfer learning; The extraction of facial key point motion features includes: adopting a 106-point facial key point expansion scheme, adding points on the forehead, temples, jawline, and periorbital area to the existing 68 points; For unoccluded keypoints, the Kanade-Lucas-Tomasi optical flow tracing algorithm is used to calculate motion parameters; when a keypoint is occluded, its motion trajectory is predicted and continued based on the motion mapping relationship of adjacent unoccluded keypoints.

4. The method for predicting psychophysiological features based on facial information according to claim 1, characterized in that, In step S3, the overlap rate of the sliding window processing is 50%, and the length of the sliding window is adjustable within a range of 0.5 seconds to 10 seconds.

5. The method for predicting psychophysiological features based on facial information according to claim 1, characterized in that, In step S4, the time series model is dynamically selected according to the number of monitored objects: when the number of objects is ≤10, a bidirectional long short-term memory network with added attention mechanism is used; when the number of objects is 10-30, a gated recurrent unit with added residual connection is used; and when the number of objects is >30, a Transformer encoder is used. Execution occurs when dynamically switching timing models: Feature caching: Real-time caching of the standardized feature sequence of the most recent 3 seconds as input to the new model; Delayed switching: A switch is triggered only after a 30-second period of change in the number of objects. For a smooth transition, the first 15 frames during the switch use a weighted fusion of the old and new models, with the weights gradually changing from 8:2 to 2:

8.

6. The method for predicting cardiac physiological features based on facial information according to claim 1, characterized in that, It also includes model training and optimization steps, using a weighted cross-entropy loss function to calculate the prediction loss, and assigning a weight of 1.5-3 times that of the basic emotion samples to samples of abnormal emotions and abnormal behaviors, all of which are samples based on visual feature annotation.

7. The method for predicting cardiac physiological features based on facial information according to claim 6, characterized in that, During the model training phase, attention regularization is employed. Based on the FACS standard and psychological prior annotations, key regions such as the forehead, eyelids, and corners of the mouth are masked. A regularization term λ·L_reg is added to the loss function, with λ ranging from 0.01 to 0.1, to constrain the model to reduce the attention weights on irrelevant regions.

8. The method for predicting cardiac physiological features based on facial information according to claim 7, characterized in that, The visualization report in step S5 includes: Grad-CAM decision heatmap is generated by weighted stacking of feature maps from the model's convolutional layers and then superimposed onto the original face image. The AU time sequence logic table records the occurrence time, duration, intensity value, and psychological interpretation of the combination rules of key AUs within the detection period; as well as risk level labeling and intervention suggestions.

9. The method for predicting cardiac physiological features based on facial information according to claim 1, characterized in that, In step S5, the continuous processing period is 30 seconds, and the cardiac physiological characteristic index is output once per cycle. The index is the weighted sum of the predicted probabilities of each cardiac physiological characteristic.

10. An apparatus for performing the psychophysiological feature prediction method based on facial information according to any one of claims 1-9, characterized in that, include: The system includes a data acquisition and preprocessing module, a multi-dimensional feature extraction module, a feature standardization module, a time series modeling and classification module, and a continuous monitoring and early warning module. The data acquisition and preprocessing module reuses existing equipment in educational scenarios for its camera; the multi-dimensional feature extraction module includes: an AU feature extraction unit, a micro-expression extraction unit, and a key point motion extraction unit.

Images