Non-contact assessment method of motion fatigue based on facial visible light video

Abstract

The application provides a kind of non-contact evaluation method of motion fatigue based on facial visible light video, it is related to non-contact motion fatigue detection field.Through dual-mode feature fusion and enhancement, a multi-modal index system covering heart rate, heart rate variability and various facial behavior characteristics is constructed, breaking through the limitations of traditional single physiological signal dependence, achieving more comprehensive fatigue state perception.At the same time, the signal quality driven sparse gating expert model is introduced, the outputs of multiple basic classifiers are dynamically fused, the weights are adaptively adjusted according to the signal quality characteristics, and the prediction accuracy and generalization ability in complex environment are significantly improved.A benchmark-impact two-dimensional evaluation framework is proposed: combining the fatigue probability distribution before and after exercise, calculating the basic fatigue index through multi-modal weighted calculation, and introducing physiological change rate and facial behavior change rate correction, finally generating fatigue impact index, effectively distinguishing cumulative fatigue and exercise-induced loss, accurately quantifying individual actual body loss.

Description

Technical Field

[0001] This invention relates to the field of non-contact motion fatigue detection, specifically to a non-contact motion fatigue assessment method based on facial visible light video. Background Technology

[0002] Non-contact assessment technology for human exercise fatigue is an important research direction in the fields of sports science and health monitoring.

[0003] Among the related technologies, the following two types of solutions can achieve non-contact fatigue assessment: (1) Video-based behavioral feature detection, such as patent CN121033812A which assesses instantaneous fatigue through driver's facial movements (blinking, yawning). (2) Multimodal non-contact solutions, such as patent CN120616480B which combines video rPPG signals with motion posture features.

[0004] However, the above scheme predicts the "absolute value of instantaneous fatigue" and belongs to "in-process or post-process monitoring", which cannot accurately quantify the actual physical damage caused by a single exercise to a specific individual, thus limiting the guiding significance of the assessment results. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a non-contact assessment method for motion fatigue based on facial visible light video, solving the technical problem of accurately quantifying the actual physical wear and tear caused by a single exercise to a specific individual.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: A non-contact assessment method for motion fatigue based on facial visible light video includes: Collect visible light videos of the user's face before and after exercise; Based on visible light facial video, the first physiological feature is extracted through time-frequency analysis and facial behavioral features are extracted through OpenFace. After statistical verification and expert knowledge, a mixed feature set is obtained. The second physiological feature is extracted through a deep neural network. Finally, the mixed feature set and the second physiological feature are fused to obtain a comprehensive feature vector. The first and second physiological features both include heart rate and heart rate variability features, and the facial behavioral features include head posture, eye gaze direction, and facial motion unit feature set. Based on the comprehensive feature vector and the preset motion fatigue state label, multiple basic expert models are trained. Combined with the signal quality features of the facial visible light video, a hybrid expert model integration framework with signal quality features as conditional gating and Top-k sparse activation is constructed to obtain the probability distribution of fatigue state before and after motion. Among them, the signal quality features include at least the difference in comprehensive feature vectors before and after motion. Based on the probability distribution of fatigue state before and after exercise, a multimodal probability weighted calculation is used to calculate the basic fatigue index, and physiological and facial behavior change rates are introduced to jointly correct the basic fatigue index in a product form; and the difference between the corrected fatigue index before and after exercise is calculated to obtain the fatigue impact index to incrementally assess the actual fatigue loss intensity.

[0007] Preferably, the step of extracting the first physiological feature through time-frequency analysis and facial behavioral features through OpenFace, and then combining statistical tests with expert knowledge to obtain a mixed feature group, includes: rPPG signals were extracted from visible light videos of the face and filtered. Heart rate characteristics were calculated through spectral analysis, and heart rate variability characteristics were obtained based on time domain, frequency domain and nonlinear analysis. Facial key points were detected using OpenFace, and temporal features of head pose, eye gaze direction, and facial motion units were extracted. The mean, standard deviation, maximum, minimum, median, and quartile statistics of each temporal feature were calculated to obtain feature sets of head pose, eye gaze direction, and facial motion units. Univariate ANOVA and mutual information screening were performed on heart rate and heart rate variability features extracted through time-frequency analysis, as well as head posture, eye gaze direction and facial motion unit features extracted through OpenFace. These features were combined with features that were significantly related to exercise fatigue state based on expert knowledge retention to form a mixed feature set.

[0008] Preferably, the extraction of the second physiological feature through a deep neural network includes: Based on a deep rPPG neural network containing 3D convolutional layers and a temporal coding module, physiological change features are extracted, including: Extracting spatiotemporal features from video sequences using 3D-CNN; A temporal network is used to encode features to generate physiological change features; The rPPG waveform was reconstructed by time-frequency analysis to extract heart rate and heart rate variability features to form a second physiological characteristic.

[0009] Preferably, the signal quality features further include video brightness stability, face detection confidence, image sharpness, and rPPG signal quality index; the signal quality feature vector construction process includes: Calculate the average luminance standard deviation of the video sequence and convert it into a luminance stability index; The average confidence index of the entire video segment is calculated based on the face detection confidence score; Image sharpness metrics are calculated using the variance of the Laplacian operator. rPPG signals are extracted from visible light videos of the face, and the energy ratio inside and outside the heart rate frequency band is extracted from the rPPG signal spectrum to calculate the signal-to-noise ratio. The Euclidean distance between the combined feature vectors before and after the motion is calculated as the difference in the combined feature vectors.

[0010] Preferably, based on the comprehensive feature vector and preset motion fatigue state labels, multiple basic expert models are trained. Combined with the signal quality features of the facial visible light video, a hybrid expert model ensemble framework is constructed, using signal quality features as conditional gating and Top-k sparse activation, to obtain the probability distribution of fatigue states before and after motion, including: Based on comprehensive feature vectors and exercise fatigue state labels, a basic expert model is independently trained using support vector machine, multilayer perceptron, and logistic regression, respectively, to output the probability distribution of exercise fatigue state. , , ;in Represents the comprehensive feature vector; The probability outputs of each basic model are concatenated into a composite feature vector. In addition, the signal quality characteristics are input into the gating network to generate gating weights; The gating weights are sparsified by using the Top-k sparse activation mechanism, and the outputs of the k experts with the largest weights are retained. The sparsified weights are normalized, and the probability outputs of each basic model are weighted and fused to obtain the final fatigue state probability distribution.

[0011] Preferably, the gating network and sparse activation mechanism are implemented in the following ways: a) The gated network is a two-layer fully connected structure, and the input is a composite feature vector. With signal quality characteristics splicing The output is gating weights generated using the Softmax function. ;in These are the gating weights for support vector machines, multilayer perceptrons, and logistic regression, respectively. b) Employing a Top-k sparse activation mechanism, only the k experts with the largest weights are retained, while the rest are reset to zero and renormalized to form sparse weights. ; c) The output is a weighted sum of the probabilities of each expert: ; d) Weighted expert probabilities The final classification result is obtained through linear transformation. .

[0012] Preferably, the baseline fatigue index is calculated using a multimodal probability weighting method based on the probability distribution of fatigue state before and after exercise, and physiological and facial behavior change rates are introduced to jointly correct the baseline fatigue index in a product form, including: Based on the probability distribution of fatigue state before and after exercise According to the preset weight vector Perform a weighted summation to map to continuous values. ;in These represent the predicted probabilities for low, moderate, and severe fatigue states, respectively. These are the preset weights for low, medium, and severe fatigue states, respectively. This indicates the baseline fatigue index, including the baseline fatigue index before exercise. Basal fatigue index after exercise ; The relative rates of change of heart rate and heart rate variability before and after exercise were extracted respectively. After weighted fusion, a comprehensive physiological change quantity is generated. ; Based on the facial motion unit feature set after movement, the change in facial behavior is calculated using weighted methods. ; The multiplicative adjustment is applied to the base index using the following formula: in, , This is an adjustable coefficient; This indicates the adjusted fatigue index, including the pre-exercise adjusted fatigue index. Post-exercise corrected fatigue index .

[0013] A non-contact motion fatigue assessment system based on facial visible light video includes: The video capture module is used to capture visible light videos of the user's face before and after movement. The feature extraction module is used to extract the first physiological feature and the facial behavioral feature based on the visible light video of the face through time-frequency analysis and OpenFace. After statistical testing and expert knowledge, a mixed feature group is obtained. The second physiological feature is extracted through a deep neural network. The mixed feature group and the second physiological feature are fused to obtain a comprehensive feature vector. The first and second physiological features include heart rate and heart rate variability features. The facial behavioral features include head posture, eye gaze direction and facial motion unit feature group. The probability prediction module is used to train multiple basic expert models based on comprehensive feature vectors and preset motion fatigue state labels. It combines the signal quality features of facial visible light video to construct a hybrid expert model integration framework with signal quality features as conditional gating and Top-k sparse activation, so as to obtain the probability distribution of fatigue state before and after motion. The signal quality features include at least the difference in comprehensive feature vectors before and after motion. The fatigue assessment module is used to calculate the basic fatigue index based on the probability distribution of fatigue state before and after exercise, using multimodal probability weighting, and to incorporate physiological and facial behavior change rates to jointly correct the basic fatigue index in a product form; and to calculate the difference between the corrected fatigue index before and after exercise to obtain the fatigue impact index to incrementally assess the actual fatigue loss intensity.

[0014] A storage medium storing a computer program, wherein the computer program causes a computer to perform the non-contact motion fatigue assessment method based on facial visible light video as described above.

[0015] An electronic device, comprising: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the non-contact motion fatigue assessment method based on visible light facial video as described above.

[0016] (III) Beneficial Effects This invention provides a non-contact assessment method for motion fatigue based on facial visible light video. Compared with existing technologies, it has the following advantages: This invention provides a non-contact motion fatigue assessment method based on facial visible light video. Through dual-modal feature fusion and enhancement, a multimodal index system covering heart rate, heart rate variability, and various facial behavioral features is constructed, overcoming the limitations of traditional single physiological signal dependence and achieving more comprehensive fatigue state perception. Simultaneously, a signal quality-driven sparse gating expert model is introduced, dynamically fusing the outputs of multiple basic classifiers and adaptively adjusting weights based on signal quality features, significantly improving prediction accuracy and generalization ability in complex environments. Furthermore, a baseline-impact dual-dimensional assessment framework is proposed: combining the fatigue probability distribution before and after exercise, a baseline fatigue index is calculated through multimodal weighting, and physiological and behavioral change rates are introduced for correction, ultimately generating a fatigue impact index. This effectively distinguishes between cumulative fatigue and exercise-induced fatigue, accurately quantifying the actual physical damage to an individual. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a technical concept diagram of a non-contact motion fatigue assessment method based on facial visible light video provided by an embodiment of the present invention; Figure 2 A block diagram illustrating a non-contact motion fatigue assessment method based on facial visible light video, provided in an embodiment of the present invention. Figure 3 A flowchart of a non-contact motion fatigue assessment method based on facial visible light video provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This application provides a non-contact assessment method for motion fatigue based on facial visible light video. The overall technical concept is as follows: Figure 1 As shown, this solves the technical problem of accurately quantifying the actual physical damage caused to a specific individual by a single movement. The key points for solving the aforementioned technical problem are as follows: 1. Non-contact motion fatigue state detection: This invention provides a non-contact method for assessing motion fatigue state using single-spectrum technology based on facial visible light video. Compared with traditional motion fatigue state detection systems, it avoids the impact of wearable devices on comfort, achieving an interference-free and low-cost assessment method.

[0021] 2. Joint perception of physiological and behavioral characteristics: The embodiments of the present invention construct a multimodal index system that integrates physiological and behavioral characteristics. This system breaks through the limitations of traditional load assessment that relies on a single physiological signal (such as electrocardiogram or heart rate), and realizes a holistic and three-dimensional expression of the body's response after exercise, making the assessment results more comprehensive, reliable and more sensitive to different exercise intensities.

[0022] 3. Sparse Gated Expert Fusion Model Driven by Signal Quality: This invention innovatively introduces a sparse gated hybrid expert model into the motion fatigue state prediction framework. The gating network dynamically selects and weights multiple basic expert models (such as Support Vector Machine (SVM), Multilayer Perceptron (MLP), Logistic Regression (LR), etc.) based on information such as the signal quality, video stability, and multimodal feature consistency of the input samples, retaining only the optimal expert combination for inference. Compared with traditional fixed-weight or single-model structures, this mechanism can adaptively adjust model weights according to different motion intensities, individual differences, and video quality, significantly improving the prediction accuracy and generalization ability of the system in complex environments.

[0023] 4. A dual-dimensional assessment framework of "benchmark-impact" was established: This embodiment of the invention not only focuses on the state during exercise but also emphasizes monitoring throughout the entire process. By comparing the fatigue index before (resting state) and after exercise, a "Fatigue Impact Index (FII)" was constructed. This key point solves the problem that the original technology could not distinguish between "individual fatigue" and "fatigue caused by this exercise," and achieves decoupling between cumulative fatigue and instantaneous fatigue.

[0024] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0025] Example 1: like Figure 2 As shown, this embodiment of the invention provides a non-contact assessment method for motion fatigue based on facial visible light video, including: S1. Collect visible light videos of the user's face before and after movement; S2. Based on visible light video of the face, the first physiological feature is extracted through time-frequency analysis and the facial behavior feature is extracted through OpenFace. After statistical verification and expert knowledge, a mixed feature group is obtained. The second physiological feature is extracted through a deep neural network. The mixed feature group and the second physiological feature are fused to obtain a comprehensive feature vector. The first and second physiological features both include heart rate and heart rate variability features. The facial behavior features include head posture, eye gaze direction and facial motion unit feature group. S3. Based on the comprehensive feature vector and the preset motion fatigue state label, train multiple basic expert models, and combine the signal quality features of the facial visible light video to construct a hybrid expert model integration framework with signal quality features as conditional gating and Top-k sparse activation, so as to obtain the probability distribution of fatigue state before and after motion; wherein, the signal quality feature is at least the difference in comprehensive feature vector before and after motion. S4. Based on the probability distribution of fatigue state before and after exercise, the basic fatigue index is calculated using multimodal probability weighting, and physiological and facial behavior change rates are introduced to jointly correct the basic fatigue index in a product form; and the difference between the corrected fatigue index before and after exercise is calculated to obtain the fatigue impact index to incrementally assess the actual fatigue loss intensity.

[0026] This invention, based on visible light facial video, constructs a non-contact motion fatigue assessment system by fusing physiological and behavioral features. It innovatively employs a signal quality-driven sparse gating expert model, dynamically fusing multi-classifier outputs to improve assessment accuracy. Furthermore, by calculating the difference in the corrected fatigue index (FII) before and after exercise, it achieves precise quantification of motion-induced fatigue loss, solving the problem of traditional methods' inability to distinguish between inherent fatigue and motion-induced fatigue, and offering advantages such as interference-free operation and high sensitivity.

[0027] like Figure 3 As shown, Figure 3 A flowchart of a non-contact motion fatigue assessment method based on facial visible light video, provided for embodiments of the present invention, is disclosed. The following will be combined with... Figure 3 The steps of the above scheme are described in detail: In step S1, visible light videos of the user's face are captured before and after the user moves.

[0028] This step involves video capture and preprocessing, as detailed below: Video capture: A visible light camera was used to capture 90 seconds of facial video of the user before exercise (resting state, usually referring to a state where the body is at rest and not engaging in significant physical activity) and after exercise. The resting state video was used to establish an individual's physiological baseline, while the post-exercise video reflected changes in exercise-induced fatigue.

[0029] Region of Interest (ROI) extraction: Facial regions are located using face detection algorithms, and stable areas such as the forehead and cheeks are selected as ROIs to reduce interference from head movements or changes in lighting.

[0030] It should be noted that the metabolic level at rest is generally considered to be basal metabolism, with physiological indicators such as heart rate, blood pressure, and respiratory rate at a low level. Its characteristics include: 1) energy consumption is close to the basal metabolic rate; 2) heart rate and blood pressure are at baseline levels; and 3) no additional oxygen intake is required to cope with fatigue. The state after exercise stimulation is determined based on biochemical and physiological indicators to determine whether it is a moderate or high-intensity state.

[0031] Although the pre-exercise state is generally considered to be a relatively well-rested basic physiological state, meaning the body has not experienced recent high-intensity exercise loads or has completed sufficient recovery, and its overall manifestation is a typical state of no or low fatigue, it is important to note that under continuous training or high-frequency exercise conditions, individuals may also experience varying degrees of residual or cumulative fatigue before exercise begins. While this state may appear as a resting posture, the internal physiological baseline may have shifted, for example, with increased resting heart rate, decreased heart rate variability, or increased facial muscle tension.

[0032] Therefore, in this embodiment of the invention, the pre-exercise resting state is regarded as the observation window of the individual's fatigue baseline state. By comparing and analyzing the state after exercise, the immediate fatigue induced by exercise and the potential cumulative fatigue level can be further identified.

[0033] In step S2, based on the visible light video of the face, the first physiological feature is extracted through time-frequency analysis and the facial behavior feature is extracted through OpenFace. After statistical verification and expert knowledge, a mixed feature group is obtained. The second physiological feature is extracted through a deep neural network. The mixed feature group and the second physiological feature are then fused to obtain a comprehensive feature vector. The first and second physiological features both include heart rate and heart rate variability features, and the facial behavior features include head posture, eye gaze direction, and facial motion unit feature group.

[0034] This step performs feature extraction, including traditional methods for extracting physiological features, deep learning-based feature extraction, and feature fusion, as detailed below: 1) Extraction of physiological features from facial visible light video using traditional methods In an optional implementation, this step involves extracting the first physiological feature through time-frequency analysis and facial behavioral features through OpenFace. After statistical testing and combining expert knowledge, a mixed feature set is obtained, including: S201. Extract rPPG signals from facial visible light video and filter them. Calculate heart rate characteristics through spectrum analysis and obtain heart rate variability characteristics based on time domain, frequency domain and nonlinear analysis. S202. Detect facial key points using OpenFace, extract temporal features of head pose, eye gaze direction, and facial motion units, and calculate the mean, standard deviation, maximum, minimum, median, and quartile statistics of each temporal feature to obtain feature groups of head pose, eye gaze direction, and facial motion units. S203. Univariate ANOVA and mutual information screening are performed on the heart rate and heart rate variability features extracted by time-frequency analysis, as well as the head posture, eye gaze direction and facial motion unit features extracted by OpenFace. These features are then combined with features that are significantly related to exercise fatigue state based on expert knowledge retention to form a mixed feature group.

[0035] Specifically: Corresponding to step S201 above, the heart rate estimation algorithm is used to extract the rPPG (Remote Photoplethysmography) signal from the facial video and filter it. Finally, the peak detection algorithm is used to perform spectrum analysis on the processed rPPG signal and calculate the heart rate feature (HR) by combining the sampling frequency of the signal.

[0036] Furthermore, after noise removal and preservation of the heart rate-related frequency range, the filtered signal is used to calculate key heart rate variability (HRV) features using the `process` function of the HeartPy heart rate analysis toolkit, including: Temporal characteristics: NN intervals, NN interval standard deviation (SDNN), percentage of adjacent NN intervals with a difference greater than 50ms (pNN50), mean NN interval (AVNN), root mean square of adjacent NN interval differences (RMSSD); where NN represents a normal sinus beat.

[0037] Frequency domain characteristics: Low-frequency power (LF), high-frequency power (HF), and the power ratio of low-frequency to high-frequency (LF / HF). Here, low frequency refers to 0.04-0.15Hz, and high frequency refers to 0.15-0.4Hz.

[0038] Nonlinear characteristics: instantaneous heart rate variability standard deviation (sd1), long-term heart rate variability standard deviation (sd2), and the ratio of instantaneous heart rate variability standard deviation to long-term heart rate variability standard deviation (sd12).

[0039] Corresponding to step S202 above, based on OpenFace, faces are detected and 68 facial key points are located from the acquired visible light video frames. Based on these key points, temporal features such as head pose, eye gaze direction, and facial motion units (AU) are obtained, including: Head pose features: Pitch (X), representing the vertical tilt of the head; Yaw (Y), representing the horizontal rotation of the head; Roll (Z), representing the tilt of the head.

[0040] Facial movement unit (AU) feature groups: AU01: inner eyebrow raised; AU02: outer eyebrow raised; AU04: eyebrow lowered; AU05: upper eyelid raised; AU06: cheek raised; AU07: eyelid tightened; AU09: nose wrinkled; AU10: upper lip raised; AU12: corner of mouth pulled up; AU14: corner of mouth tightened; AU15: corner of mouth lowered; AU17: chin raised; AU20: lips stretched; AU23: lips tightened; AU25: lips opened; AU26: jaw opened; AU28: lips sucking; AU45: blinking.

[0041] Eye gaze direction feature set: gaze direction vector, represented by a three-dimensional vector (x,y,z), describes the direction of the gaze in three-dimensional space.

[0042] Furthermore, based on the original Pose, Gaze, and AU features extracted from OpenFace, which are sequence data for each frame of the video, in order to refine the high-dimensional and redundant original sequence into a low-dimensional, robust, and interpretable semantic description of facial behavior, the statistical features of each feature are calculated, including mean, standard deviation, minimum, maximum, median, 25% (first quartile), and 75% (third quartile), as the final facial behavior features.

[0043] Corresponding to step S203 above, firstly, univariate analysis of variance (F-test) is used to assess the correlation between each feature and the load label, and the FDR method is used to correct for false positives caused by multiple comparisons, screening out features with statistical significance. Further, the mutual information between each feature and the label is calculated to identify features closely related to the load, and the intersection of the results obtained through the F-test and mutual information screening is used as the candidate feature set. Then, the correlation coefficients between each pair of candidate features are calculated to identify highly correlated feature pairs, and these features are grouped using the connected component method in graph theory. Within each feature group, the feature with the smallest p-value is selected as the representative feature, ensuring that the final retained features are both concise and have the strongest statistical significance.

[0044] For example, the features obtained from the final screening include: 1) Facial AU features: 'AU02_r_75%', 'AU04_r_median', 'AU05_r_median','AU05_r_std', 'AU06_c_std','AU06_r_max','AU06_r_mean','AU06_c_max', 'AU07_c_std','AU07_c_max','AU07_r_max', 'AU09_c_max','AU09_c_std', 'AU10_r_max','AU10_c_std', 'AU12_c_max','AU12_r_max','AU12_c_std', 'AU14_r_max','AU14_c_max','AU14_c_std', 'AU15_r_median','AU15_c_max','AU15_r_mean', 'AU17_r_mean','AU17_c_std','AU17_r_max','AU17_c_mean','AU17_c_max', 'AU20_c_max','AU20_c_mean', 'AU23_r_max','AU23_r_std', 'AU25_c_std','AU25_r_std','AU25_c_max','AU25_r_max', 'AU26_c_std','AU26_r_max', 'AU45_r_std','AU45_r_max'.

[0045] 2) pose features: 'pose_Tx_std','pose_Ty_max','pose_Ry_25%','pose_Rx_25%','pose_Rz_std','pose_Rx_min','pose_Tx_min','pose_Ry_std','pose_Ry_min','pose_Tz_min','pose_Ry_max'.

[0046] 3) gaze features: 'gaze_1_y_25%','gaze_0_z_min','gaze_0_x_75%', 'gaze_0_y_min','gaze_0_x_max','gaze_1_z_min','gaze_angle_x_75%','gaze_0_x_std','gaze_1_x_min'.

[0047] 4) Heart rate variability characteristics: 'pNN 50','sd12','RMSSD','sd2','LF / HF'.

[0048] Related technologies indicate that as exercise intensity increases, the human body undergoes a series of corresponding physiological and behavioral changes to adapt to the rising load level. For example, under high load conditions, the heart rate increases significantly, and cardiac activity accelerates to meet the body's oxygen demand; simultaneously, sympathetic nerve activity increases, resulting in a typical stress response. Facial expressions also change, such as frowning, clenching teeth, and tightly closing lips, reflecting the individual's subjective perception of exercise stress. Head posture often leans forward, and the gaze becomes more focused to improve sensory input efficiency and motor control. These changes collectively constitute the typical physiological and behavioral response pattern of the human body during exercise fatigue.

[0049] By summarizing existing literature on exercise fatigue, it is clear that there is a significant relationship between physiological characteristics such as heart rate (HR) and standard deviation of NN interval (SDNN) and exercise fatigue state. Therefore, this step retains 'SDNN' and 'HR' based on the above expert knowledge.

[0050] It is understandable that the above indicator system is applicable to feature extraction from video data before and after motion. That is, features are extracted from visible light video before and after motion using the above indicator system, and the number and types of features extracted before and after motion are the same, and they are used as a mixed feature group. These are then input into subsequent models.

[0051] 2) Feature extraction based on deep learning In an optional implementation, this step involves extracting the second physiological feature using a deep neural network, including: S201', Based on a deep rPPG neural network containing 3D convolutional layers and a temporal coding module, extracts physiological change features, including: Extracting spatiotemporal features from video sequences using 3D-CNN; A temporal network is used to encode features to generate physiological change features; S202' Reconstructs rPPG waveforms through time-frequency analysis to extract heart rate and heart rate variability features to constitute the second physiological characteristic.

[0052] Specifically: Corresponding to step S201' above, in order to extract potential blood volume change information from color temporal changes, a deep rPPG neural network structure containing 3D convolutional layers and temporal coding modules is designed to perform spatiotemporal feature extraction and temporal physiological coding operations, respectively.

[0053] Spatiotemporal feature extraction: Facial visible light video (ROI image sequence) is input as a spatiotemporal tensor into a 3D convolutional network to extract local spatiotemporal features. in, For ROI image sequences, This is the spatiotemporal feature map output by 3D convolution.

[0054] Understandably, 3D convolution can simultaneously capture subtle changes in pixels over time, periodic perturbations in skin-reflected light signals, and local stabilization patterns of illumination variations and motion artifacts, thereby significantly improving the robustness of rPPG signals.

[0055] Temporal physiological coding: To further capture the periodicity and long-term dependence of rPPG signals, a temporal coding module is introduced to encode the 3D-CNN output, which can be represented as: in, These are physiological changes. It can be a Temporal Convolutional Network (TCN) or a Lightweight Transformer.

[0056] Understandably, the time-series coding module can learn implicit patterns related to changes in heart rate, respiration, and hemodynamics, and the resulting index sequence constitutes a rich set of physiological changes before and after exercise.

[0057] Corresponding to step S202' above, firstly, the output of the timing coding module is... Used to reconstruct rPPG waveforms: in, Approximate PPG (pulse wave) signal generated for depth encoder.

[0058] Subsequently, several key physiological indicators were estimated through frequency and time domain analysis: A function to extract heart rate from a signal (usually through frequency domain analysis, such as FFT). Functions that extract heart rate variability metrics from signals (such as RMSSD, SDNN).

[0059] Finally, summarizing the heart rate and heart rate variability features extracted from deep rPPG signals through frequency and time domain analysis, including heart rate features (HR) and heart rate variability features (RMSSD, SDNN, RR, LF, HF), the second physiological feature is represented as: 3) Feature fusion The second physiological feature obtained from deep rPPG branching Compared with the original hybrid feature group obtained based on the indicator system The features are concatenated to obtain the enhanced comprehensive feature vector. This serves as input for subsequent multi-model expert (SVM / MLP / LR) and sparse-gated hybrid expert model (Sparse-MoE) ensemble frameworks.

[0060] In step S3, based on the comprehensive feature vector and the preset motion fatigue state label, multiple basic expert models are trained. Combined with the signal quality features of the facial visible light video, a hybrid expert model integration framework with signal quality features as conditional gating and Top-k sparse activation is constructed to obtain the fatigue state probability distribution before and after the motion. The signal quality features include at least the difference in the comprehensive feature vector before and after the motion.

[0061] In an optional implementation, this step includes: S31. Based on the comprehensive feature vector and the label of exercise fatigue state, the basic expert model is independently trained using support vector machine, multilayer perceptron and logistic regression, and outputs the probability distribution of exercise fatigue state respectively. , , ;in Represents the comprehensive feature vector; S32. Concatenate the probability outputs of each basic model into a composite feature vector. In addition, the signal quality characteristics are input into the gating network to generate gating weights; S32. The gating weights are sparsified by the Top-k sparse activation mechanism, and the k expert outputs with the largest weights are retained. S33. Normalize the sparsified weights, and then weight and fuse the probability outputs of each basic model to obtain the final fatigue state probability distribution.

[0062] For example, the following provides a feasible way to construct a label for exercise fatigue status: First, multiple biochemical indicators, including blood lactate, creatine kinase, serum testosterone, urinary protein, and cortisol, were collected before and after exercise. Simultaneously, the individual's perceived exertion score (RPE) was recorded to comprehensively reflect the physiological and psychological impact of exercise fatigue. Based on changes in blood lactate concentration, and combined with relevant technologies' blood lactate-based exercise fatigue grading standards, and further integrated with the RPE subjective score, a composite load rating system was formed that considers both objective physiological indicators and subjective fatigue perception. This system clearly classifies each exercise fatigue state into three levels: low, moderate, and severe (fatigue state), providing a scientific and accurate quantitative basis for exercise training, rehabilitation intervention, and fatigue monitoring, significantly improving the practicality and reliability of exercise fatigue state assessment.

[0063] For the analysis and evaluation of biochemical indicators, statistical methods such as correlation analysis and multiple regression were used to conduct in-depth analysis of various biochemical indicators. Finally, the core indicator that is most closely related to the state of exercise fatigue—blood lactate—was selected as the objective basis for load classification. The rule for blood lactate as a single classification standard is: 1~2 mmol / L—low fatigue state, 2~4 mmol / L—moderate fatigue state, and 4~10 mmol / L—severe fatigue state.

[0064] For the RPE scale analysis and assessment, the Borg 6-20 scale was used. This scale is a tool for assessing exercise fatigue status and is a 15-point scale, scored from 6 to 20. In this protocol, the RPE scale was used for basic assessment of exercise fatigue status, and its assessment and grading rules were: 6-11 points—low load, 11-17 points—medium load, 17-20 points—high load.

[0065] Comprehensive Analysis and Assessment: Based on the basic assessment standards, two key milestones need to be considered separately: "11" and "17". If the RPE scale results are at these two milestones, the ambiguity in distinguishing between them is significant, potentially hindering accurate grading of the current exercise fatigue state. Combining the previously conducted biochemical index analysis, the key biochemical indicator "blood lactate" is introduced for auxiliary assessment. The assessment rules are as follows: a) When an individual's RPE scale result is "11", it's impossible to accurately determine whether their current exercise state is low or high load. In this case, blood lactate levels are used for assessment. If the individual's blood lactate level is between 1 and 2 mmol / L, their current state is considered low exercise fatigue; if it's between 2 and 4 mmol / L, their current state is considered moderate exercise fatigue. b) When an individual's RPE scale result is "17", blood lactate levels are used for assessment. If the individual's blood lactate level is between 2 and 4 mmol / L, their current state is considered moderate exercise fatigue; if it's between 4 and 10 mmol / L, their current state is considered high exercise fatigue. c) During data collection, there were no significant discrepancies between the RPE scale and blood lactate levels. When the RPE scale result is "11" or "17", the blood lactate grading rules can be used to effectively label the individual.

[0066] Based on this, specifically: Corresponding to step S31 above, in the basic expert model training phase, various machine learning algorithms are used to learn and predict the processed features. Each machine learning algorithm is trained separately, including: Support Vector Machine (SVM), Decision Tree, Random Forest, K-Nearest Neighbors (KNN), Naive Bayes, XGBoost, Multilayer Perceptron (MLP), and Logistic Regression (LR).

[0067] Based on the results of each machine learning algorithm, the final selected basic classification algorithms include Support Vector Machine (SVM), Multilayer Perceptron (MLP), and Logistic Regression (LR). Each algorithm has its unique advantages: SVM excels at handling non-linear relationships, MLP can capture complex hierarchical data features, and LR is suitable for linear classification problems. By integrating these three models, the system can process data from multiple perspectives and capture the complex patterns hidden within the data.

[0068] 1) Support Vector Machine (SVM) Decision function: in, These are the high-dimensional features after SVM kernel function mapping. This is the model parameter vector (the weights that need to be learned). The bias term is a scalar used to adjust the position of the decision boundary.

[0069] Category probability: in, The normalized activation function will output... Convert to a probability distribution form.

[0070] 2) Multilayer Perceptron (MLP) Forward propagation: in, for The output vector of the first hidden layer This is the output vector of the second hidden layer. This is the weight matrix from the input layer to the first hidden layer. This is the weight matrix from the first hidden layer to the second hidden layer or the output layer. This is the bias vector of the first hidden layer. This is the bias vector for the second hidden layer or the output layer. for Nonlinear activation function.

[0071] Category probability: in, This is the weight matrix from the second hidden layer to the output layer. For the bias term of the output layer, This is the normalized activation function.

[0072] 3) Logistic Regression (LR) Category probability: in, This is the model parameter vector (the weights that need to be learned). The bias term is a scalar used to adjust the position of the decision boundary. This is the normalized activation function, and the superscript T is the transpose.

[0073] Corresponding to steps S32~S33 above, after completing the independent training of the three basic expert models (SVM, MLP, LR), this embodiment of the invention no longer adopts the traditional multi-head attention integration method, but proposes a hybrid expert model integration framework based on sparse gating.

[0074] This framework introduces a gating network to dynamically select and weight the outputs of different expert models, enabling adaptive fusion of the capabilities of multiple models and significantly improving the prediction stability and generalization ability of the system under different motion states, different individuals, and different video quality conditions.

[0075] Specifically: First, the probability outputs of the three base models (SVM, MLP, LR) are concatenated to form a composite feature vector: in, For the spatial dimension of composite features, It represents the set of real numbers.

[0076] Secondly, in order to enable the gating network to perform dynamic expert selection based on the quality of the input samples, this embodiment of the invention constructs additional signal quality features q, including but not limited to video brightness stability, face detection confidence, image sharpness, rPPG signal quality index, and feature differences before and after motion.

[0077] In an optional implementation, the signal quality feature vector construction process includes: 1) Calculate the average luminance standard deviation of the video sequence and convert it into a luminance stability index.

[0078] Video brightness stability is used to describe the overall degree of illumination variation in a video sequence and calculates the average brightness value of each frame. ,in Indicates the first The first frame grayscale value of each pixel. Indicates the total number of pixels in the image. Indicates the first The average brightness of each frame. Further, the standard deviation of the brightness across the entire video sequence is calculated. ,in Indicates the total number of frames in the video. This represents the average brightness of all frames.

[0079] Based on the above results, the video brightness stability index is defined as follows: Understandably, the smaller the change in illumination, the smaller the standard deviation of brightness, indicating that the video brightness is more stable.

[0080] 2) Calculate the average confidence index of the entire video based on the face detection confidence score.

[0081] In the face detection process, a detection confidence score can be obtained for each frame. Its value is usually located in the range The confidence score for face detection in the entire video is defined as follows: in, Indicates the first Frame-based face detection confidence. This indicates the total number of frames.

[0082] Understandably, The larger the value, the more reliable the facial region detection in the video, and the higher the reliability of subsequent ROI extraction and feature analysis.

[0083] 3) Calculate the image sharpness index using the variance of the Laplacian operator.

[0084] Image sharpness reflects the degree to which edge and texture details are preserved in a video frame, and can be characterized by the response intensity of the Laplacian operator. For the th A frame image, its sharpness is defined as ,in Represents the Laplace operator. This represents variance calculation. Indicates the first Frame image.

[0085] Furthermore, the average resolution metric for the entire video segment is defined as follows: Understandably, The larger the value, the clearer the image and the weaker the effects of motion blur and defocus.

[0086] 4) Extract rPPG signals from facial visible light video, and extract the energy ratio inside and outside the heart rate frequency band from the rPPG signal spectrum to calculate the signal-to-noise ratio index.

[0087] To evaluate the reliability of the rPPG signal extracted from facial video, this invention introduces a signal quality index based on frequency domain analysis. Let the extracted rPPG time-domain signal be... Its spectral power density is expressed as First, the effective frequency range corresponding to heart rate is defined as follows: This frequency band covers the common range of human heart rate variations. The effective signal energy within the heart rate frequency band is then defined as... The noise energy outside the heart rate frequency band is defined as .

[0088] Based on this, the signal-to-noise ratio (SNR) of the rPPG signal is defined as follows: in, This represents the logarithmic function with base 10.

[0089] Understandably, The larger the value, the more significant the effective pulse component in the rPPG signal, and the higher the signal quality.

[0090] 5) Calculate the Euclidean distance between the composite feature vectors before and after the motion as the composite feature vector difference.

[0091] To describe the overall changes in physiological and behavioral states before and after exercise, let's assume the comprehensive feature vector extracted before exercise is... The comprehensive feature vector extracted after the motion is Then, the difference in features before and after the motion is defined as the Euclidean distance between the two: in, It is a 2-norm.

[0092] Understandably, The larger the value, the more significant the change in state before and after exercise, indicating a more pronounced physiological or behavioral impact of the exercise on the body.

[0093] 6) Final signal quality characteristics In summary, the constructed signal quality feature vector can be expressed as: Finally, the signal quality characteristics This is combined with the composite feature vector obtained by concatenating the outputs of various basic expert models. A common input gating network is used to achieve dynamic selection and weighted fusion of different expert models, thereby improving the prediction stability and generalization ability of the system under different video quality, different individual states and different motion scenarios.

[0094] In an optional implementation, the gating network and sparse activation mechanism are implemented in the following way: a) The gated network is a two-layer fully connected structure, and the input is a composite feature vector. With signal quality characteristics splicing The output is gating weights generated using the Softmax function. ;in These are the gating weights for support vector machines, multilayer perceptrons, and logistic regression, respectively.

[0095] The gated network is a two-layer fully connected structure, which can be specifically represented as follows: in, These are the weight matrices for a two-layer fully connected structure. These are the bias terms for a two-layer fully connected structure. It is a non-linear activation function.

[0096] b) Employing a Top-k sparse activation mechanism, only the k experts with the largest weights are retained, while the rest are reset to zero and renormalized to form sparse weights. .

[0097] It should be noted that the predictive capabilities of different expert models may vary significantly under different sample conditions. For example, when the input features exhibit a clear linear relationship, the LR model shows better stability, while SVM or MLP models may be more advantageous in complex nonlinear scenarios. Therefore, even with a limited number of expert models (SVM, MLP, and LR), introducing a Top-k sparse activation mechanism remains significant. This mechanism can dynamically select the model with the largest weight based on the output of the gating network. This design assigns a number of expert models to each model, thereby suppressing noise interference from low-weight experts and improving the model's predictive stability and generalization ability. Furthermore, it provides a scalable structure for adding new expert models in the future.

[0098] In this method, only the k experts with the largest weights are retained, and the weights of the rest are reset to zero, as follows: in, For the first Updated gating weights for each basic expert model.

[0099] Then it is renormalized to form sparse weights. : in, Indicates the first Sparse weights of a basic expert model Let be the updated gating weights of the j-th basic expert model.

[0100] Understandably, this sparse strategy ensures that only a small number of experts are activated for each sample, improving model stability and interpretability.

[0101] c) The output is a weighted sum of the probabilities of each expert: .

[0102] Understandably, the above structure allows the gating network to adaptively select the optimal expert combination under different samples and different motion stages.

[0103] d) Weighted expert probabilities The final classification result is obtained through linear transformation. ,in This is the weight matrix of the final output layer. This is the bias term for the final output layer.

[0104] In step S4, based on the probability distribution of fatigue state before and after exercise, a multimodal probability weighted calculation is used to calculate the basic fatigue index, and physiological and facial behavior change rates are introduced to jointly correct the basic fatigue index in a product form; and the difference between the corrected fatigue index before and after exercise is calculated to obtain the fatigue impact index to incrementally assess the actual fatigue loss intensity.

[0105] like Figure 3 As shown, the embodiments of the present invention are based on the final classification results. A multi-level quantitative method based on the baseline-impact dimension is used to assess users' exercise fatigue, including calculating the baseline fatigue index and the fatigue impact index. Details are provided below. This step aims to transform the discrete classification probabilities output by the model into a continuous fatigue index (FI), thereby enabling a refined quantitative assessment of the body's fatigue state.

[0106] In an optional implementation, this step calculates the baseline fatigue index based on the probability distribution of fatigue state before and after exercise using a multimodal probability weighting method, and incorporates physiological and facial behavior change rates to jointly correct the baseline fatigue index in a product form, including: S401. Based on the probability distribution of fatigue state before and after exercise According to the preset weight vector Perform a weighted summation to map to continuous values. ;in These represent the predicted probabilities for low, moderate, and severe fatigue states, respectively. These are the preset weights for low, medium, and severe fatigue states, respectively. This indicates the baseline fatigue index, including the baseline fatigue index before exercise. Basal fatigue index after exercise .

[0107] It should be noted that the embodiments of the present invention preferentially set =0.1 (Weight of low fatigue / resting state). =0.5 (weight for moderate fatigue) =0.9 (weight for severe fatigue). Of course, an adaptive weight adjuster can also be introduced; for example, if a user's baseline heart rate is detected to be high, the model can be dynamically fine-tuned. The slope makes the system more personalized.

[0108] Understandably, the above formula for calculating the basic fatigue index is applied to the probability output obtained from the pre-movement video features (generating the pre-movement basic fatigue index). ) and the probability output obtained from the video features after exercise (generating the baseline fatigue index after exercise) The result is multiplied by a constant coefficient to normalize the evaluation result to a preset numerical range, so as to facilitate horizontal comparison between different individuals and different scenarios.

[0109] S402. Extract the relative rate of change of heart rate and heart rate variability features before and after exercise, respectively. After weighted fusion, a comprehensive physiological change quantity is generated. .

[0110] S403. Based on the facial motion unit feature set after motion, calculate the weighted change in facial behavior. .

[0111] To reduce the impact of inter-individual physiological differences on the assessment results and improve the sensitivity to the response to exercise-induced physiological changes, this embodiment of the invention further introduces a correction to the baseline fatigue index based on physiological indicators and facial action unit (AU) feature groups.

[0112] 1) Correction for physiological rate of change: It should be noted that the physiological change rate correction is calculated based on changes in multiple physiological indicators before and after exercise. Compared with using only a single heart rate indicator, multiple physiological indicators can more comprehensively reflect the changes in the body's autonomic nervous system regulation level, circulatory system response, and recovery state under exercise load. Therefore, this embodiment of the invention uses a unified model for multiple key physiological indicators to obtain a comprehensive physiological change.

[0113] set up Indicates the first For each physiological indicator, its relative rate of change is defined as: in, Indicates the first movement The values ​​of each physiological indicator, Indicates the first time after the exercise The values ​​of each physiological indicator, This indicates the relative change of the indicator before and after exercise. The physiological indicator may include, but is not limited to, heart rate (HR), heart rate variability indicators (such as SDNN, RMSSD, LF / HF), etc.

[0114] Furthermore, to comprehensively characterize the overall changes of multiple physiological indicators, a weighted fusion method is introduced to construct a comprehensive physiological change measure: in, This indicates the number of physiological indicators used in the calculation. Indicates the first The weight coefficients corresponding to each physiological indicator are determined based on historical sample data, correlation analysis results, or regression modeling results.

[0115] Understandably, through the above method, changes in multiple physiological signals can be uniformly mapped into a single comprehensive physiological correction quantity, which can be used to characterize the degree of internal physiological activation induced by exercise.

[0116] 2) Correction for rate of change in behavior: The behavioral change rate correction is constructed based on facial action unit features after exercise to characterize the overt behavioral stress response exhibited by individuals under fatigue. Existing studies have shown that when exercise load increases or fatigue intensifies, individuals may exhibit typical facial expression changes such as frowning, opening their mouths, closing their lips, and increasing facial tension. These features can reflect subjective fatigue to some extent.

[0117] in, Indicates the first Features of each action unit Indicates the number of action units included in the calculation. This represents the weight coefficient of the corresponding action unit. The action unit features may include, but are not limited to, AU04 (frowning), AU25 (opening mouth), AU06 (cheek lifting), etc. It can be represented by the mean, maximum value, standard deviation or weighted statistics of the corresponding action unit.

[0118] Understandably, the above method can integrate multiple behavioral features into a single behavioral change correction value, which can be used to characterize the explicit amplification effect of facial behavioral patterns on fatigue state after exercise.

[0119] S404. Perform a multiplicative correction on the base index. That is, based on the base fatigue index, comprehensively consider physiological and behavioral corrections, and use a multiplicative correction method to obtain the final fatigue index: in, , This is an adjustable coefficient; This indicates a revised fatigue index, including adjusting the base fatigue index. Substituting into the above formula, the corrected fatigue index before exercise is calculated. And the basic fatigue index Substituting into the above formula, the corrected fatigue index after exercise is obtained. .

[0120] Understandably, the weights of each indicator... , and adjustment coefficient , The method can be determined through regression analysis, correlation analysis, or model training of historical sample data to improve the adaptability and individualization of fatigue assessment results for different populations and sports scenarios. Furthermore, the multiplicative correction method described above can effectively reflect the amplification or inhibition of physiological feedback and behavioral stress on fatigue states: when the level of internal physiological activation is high after exercise, or facial behavioral characteristics show obvious fatigue stress responses, the final fatigue index will increase accordingly; conversely, when the physiological changes are small and the overt behavioral stress is weak, the change in the corrected fatigue index is relatively limited, thus avoiding unreasonably high assessment values ​​under low baseline load conditions.

[0121] Based on the changes in the load index before and after exercise, this embodiment of the invention further defines the Fatigue Impact Index (FII) to quantify the actual increase in internal load on an individual's body caused by a single exercise activity. The system defines the Fatigue Impact Index by comparing the quantitative differences before and after exercise. Understandably, this index reflects the actual fatigue intensity caused to an individual's body by the exercise. When the fatigue impact index exceeds a preset threshold, the system can assess that the individual is in a state of significant physiological stress and suggest extending the recovery period. The threshold can be set based on population statistical distribution, individual historical load levels, or application scenarios. An exemplary value is provided in this embodiment of the invention, for example: If the FII is high (e.g., >50), the system assessment indicates "the body is in a state of deep damage," requiring a longer rehabilitation period; if A reading significantly higher than the normal baseline indicates that the individual has accumulated fatigue at rest. The system will automatically issue a warning and suggest reducing the intensity of exercise that day to prevent sports injuries.

[0122] In an alternative implementation, after obtaining the fatigue index... and fatigue impact index Subsequently, in order to achieve intuitive expression of the evaluation results and decision support, this embodiment of the invention further maps the continuous fatigue evaluation score into discrete fatigue state level intervals to form an interpretable three-class fatigue judgment result.

[0123] Specifically, the fatigue index The probabilities of each fatigue level are determined according to a preset level weight vector. A weighted sum is performed and multiplied by a normalization coefficient to map to a preset numerical range. The weights are used to characterize the relative contribution of different fatigue states to the body's internal physiological load. Low fatigue corresponds to a body load close to the resting level and is therefore assigned a smaller weight; moderate fatigue corresponds to a moderate exercise load and is assigned a medium weight; while high fatigue typically corresponds to a phase close to the anaerobic threshold or high-intensity exercise and is therefore assigned a higher weight.

[0124] Building upon this, to further reinterpret the continuous fatigue index as a more readable discrete level result, it is necessary to determine the boundary thresholds between low and moderate fatigue, and between moderate and severe fatigue. Since the low, medium, and high states are respectively represented in this mapping space by… , , As a representative position, the most natural boundary between two adjacent classes should be located at the geometric center of their representative values, i.e., the midpoint of their adjacent weights. This means that when the fatigue index is closer to the representative value corresponding to low fatigue, it should be classified as low fatigue; when it is closer to the representative value corresponding to moderate fatigue, it should be classified as moderate fatigue; similarly, when it is closer to the representative value corresponding to severe fatigue, it should be classified as severe fatigue. Therefore, using the midpoint as the boundary threshold ensures that the classification boundary remains equidistant and unbiased for adjacent classes. Based on the above mapping method, and combined with the midpoint of adjacent fatigue level weights as the boundary threshold, it completely disregards the sample distribution and relies only on the defined level weights. Low / Medium Boundary: Medium / heavy boundary: .

[0125] Fatigue Index The fatigue state can be further divided into the following three ranges: when When this occurs, it is determined to be a state of low fatigue, corresponding to an individual being in a state of relatively full recovery or light overload; when When the condition is determined to be moderate fatigue, the individual is at a moderate level of physiological load; when When this condition is identified as severe fatigue, the individual exhibits a significant physiological stress response or a state of high-load fatigue.

[0126] The aforementioned grading thresholds are determined based on the ordered relationship of the fatigue level weights, have clear physiological significance, and do not depend on a specific sample distribution. They can serve as a general method for grading fatigue states.

[0127] In an optional implementation, to further interpretably classify the degree of impact from the exercise load, this embodiment of the invention divides the fatigue impact index into intervals based on the relative proportion of the total range of the fatigue index. Let the total range of the fatigue index be: In one embodiment, there is , ,therefore .

[0128] Furthermore, two scaling factors are defined. and ,in Then the threshold of the fatigue effect index can be expressed as: Exemplary take , Thus we get: Based on the above thresholds, the fatigue impact of a single exercise load can be divided into the following intervals: when When the impact is low, it is considered a low-impact state, indicating that the fatigue increment caused by the exercise is small; when At that time, it was determined to be a moderate impact state, indicating that the exercise caused a significant but usually recoverable physiological fatigue response; when If the condition is identified as a high-impact or deep-fatigue state, it indicates that the exercise has caused significant physiological load to the individual, requiring an extended recovery period or a reduction in the intensity of subsequent training.

[0129] It should be noted that the above threshold division is determined based on the relative proportion of the total fatigue index range, enabling the fatigue impact index to reflect the relative position of the fatigue increment caused by a single exercise within the overall fatigue scale. Therefore, it has good interpretability and scale consistency. Furthermore, this proportional coefficient can be adaptively adjusted according to different population characteristics, historical training load levels, or specific application scenarios without affecting the overall technical solution of this invention.

[0130] In practical applications, the fatigue index It can be used for baseline fatigue assessment before exercise and early warning of cumulative fatigue, and the fatigue impact index... This can be used to assess the immediate fatigue impact of a single exercise or training session on an individual. Through simultaneous analysis... and This allows for a more comprehensive depiction of individual fatigue state changes, providing a reference for training load regulation and recovery strategy formulation.

[0131] Example 2: This invention provides a non-contact motion fatigue assessment system based on facial visible light video, comprising: The video capture module is used to capture visible light videos of the user's face before and after movement. The feature extraction module is used to extract the first physiological feature and the facial behavioral feature based on the visible light video of the face through time-frequency analysis and OpenFace. After statistical testing and expert knowledge, a mixed feature group is obtained. The second physiological feature is extracted through a deep neural network. The mixed feature group and the second physiological feature are fused to obtain a comprehensive feature vector. The first and second physiological features include heart rate and heart rate variability features. The facial behavioral features include head posture, eye gaze direction and facial motion unit feature group. The probability prediction module is used to train multiple basic expert models based on comprehensive feature vectors and preset motion fatigue state labels. It combines the signal quality features of facial visible light video to construct a hybrid expert model integration framework with signal quality features as conditional gating and Top-k sparse activation, so as to obtain the probability distribution of fatigue state before and after motion. The signal quality features include at least the difference in comprehensive feature vectors before and after motion. The fatigue assessment module is used to calculate the basic fatigue index based on the probability distribution of fatigue state before and after exercise, using multimodal probability weighting, and to incorporate physiological and facial behavior change rates to jointly correct the basic fatigue index in a product form; and to calculate the difference between the corrected fatigue index before and after exercise to obtain the fatigue impact index to incrementally assess the actual fatigue loss intensity.

[0132] Example 3: This invention provides a storage medium storing a computer program that causes a computer to execute the non-contact motion fatigue assessment method based on facial visible light video as described in Embodiment 1.

[0133] Example 4: This invention provides an electronic device, comprising: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing a non-contact motion fatigue assessment method based on facial visible light video as described in Example 1.

[0134] It is understood that the non-contact motion fatigue assessment system, storage medium, and electronic device based on facial visible light video provided in the embodiments of the present invention correspond to the non-contact motion fatigue assessment method based on facial visible light video provided in the embodiments of the present invention. The explanations, examples, and beneficial effects of the relevant contents can be referred to the corresponding parts of the method, and will not be repeated here.

[0135] In summary, compared with existing technologies, it has the following beneficial effects: 1. The depth and comprehensiveness of the assessment dimensions: Compared to related patents that rely solely on external behavioral characteristics (such as facial expressions and posture) for fatigue assessment, this invention introduces physiological signal monitoring based on remote photoplethysmography (rPPG) technology. This non-contact method acquires cardiovascular activity indicators such as heart rate and heart rate variability, supplementing objective physiological information beyond behavioral characteristics. Multimodal fusion helps assess an individual's exercise load status by combining the changing trends of physiological indicators, even when behavioral representations are not significant.

[0136] 2. The deep rPPG temporal encoder constructed in this invention can stably extract key physiological signals in complex motion environments such as motion artifacts, lighting changes, and facial pose changes. Combined with a quality-aware multi-expert dynamic fusion mechanism, the model can automatically select the optimal expert combination based on video quality and feature stability.

[0137] 3. This invention introduces the Fatigue Impact Index (FII) as a core assessment parameter. By collecting and modeling an individual's baseline physiological data before exercise, real-time monitoring values ​​during exercise are dynamically compared with their baseline. This eliminates, to some extent, assessment biases introduced by differences in basal metabolic rate, physical characteristics, and exercise capacity. Compared to generalized assessment methods based on population statistical thresholds, FII uses an individual's own physiological baseline as a reference, reflecting the relative physiological impact of a specific exercise load on that individual, thus improving the applicability of the assessment results to different populations.

[0138] 4. This invention is entirely based on visible light video, requiring no contact with the body, thus solving the problems of skin constriction, allergy risks, and frequent charging associated with wearable devices (such as ECG straps and sports watches). Users only need to face the camera to complete professional-grade physiological status monitoring.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact assessment method for motion fatigue based on facial visible light video, characterized in that, include: Collect visible light videos of the user's face before and after exercise; Based on visible light facial video, the first physiological feature was extracted through time-frequency analysis and facial behavioral features were extracted through OpenFace. After statistical testing and expert knowledge, a mixed feature group was obtained. Second physiological features are extracted using deep neural networks; In addition, a comprehensive feature vector is obtained by fusing the hybrid feature group and the second physiological feature; wherein, both the first and second physiological features include heart rate and heart rate variability features, and facial behavior features include head posture, eye gaze direction and facial motion unit feature group. Based on the comprehensive feature vector and the preset motion fatigue state label, multiple basic expert models are trained. Combined with the signal quality features of the facial visible light video, a hybrid expert model integration framework with signal quality features as conditional gating and Top-k sparse activation is constructed to obtain the probability distribution of fatigue state before and after motion. Among them, the signal quality features include at least the difference in comprehensive feature vectors before and after motion. Based on the probability distribution of fatigue state before and after exercise, a multimodal probability weighted calculation is used to calculate the basic fatigue index, and physiological and facial behavior change rates are introduced to jointly correct the basic fatigue index in a product form; and the difference between the corrected fatigue index before and after exercise is calculated to obtain the fatigue impact index to incrementally assess the actual fatigue loss intensity. The baseline fatigue index is calculated using a multimodal probability weighting method based on the probability distribution of fatigue state before and after exercise. Physiological and facial behavioral change rates are also incorporated and multiplied to correct the baseline fatigue index, including: Based on the probability distribution of fatigue state before and after exercise According to the preset weight vector Perform a weighted summation to map to continuous values. ;in These represent the predicted probabilities for low, moderate, and severe fatigue states, respectively. These are the preset weights for low, medium, and severe fatigue states, respectively. This indicates the baseline fatigue index, including the baseline fatigue index before exercise. Basal fatigue index after exercise ; The relative rates of change of heart rate and heart rate variability before and after exercise were extracted respectively. After weighted fusion, a comprehensive physiological change quantity is generated. ; Based on the facial motion unit feature set after movement, the change in facial behavior is calculated using weighted methods. ; The multiplicative adjustment is applied to the base index using the following formula: in, , This is an adjustable coefficient; This indicates the adjusted fatigue index, including the pre-exercise adjusted fatigue index. Post-exercise corrected fatigue index .

2. The non-contact assessment method for exercise fatigue as described in claim 1, characterized in that, The process involves extracting the first physiological feature through time-frequency analysis and facial behavioral features through OpenFace. After statistical testing and combining expert knowledge, a mixed feature set is obtained, including: rPPG signals were extracted from visible light videos of the face and filtered. Heart rate characteristics were calculated through spectral analysis, and heart rate variability characteristics were obtained based on time domain, frequency domain and nonlinear analysis. Facial key points were detected using OpenFace, and temporal features of head pose, eye gaze direction, and facial motion units were extracted. The mean, standard deviation, maximum, minimum, median, and quartile statistics of each temporal feature were calculated to obtain feature sets of head pose, eye gaze direction, and facial motion units. Univariate ANOVA and mutual information screening were performed on heart rate and heart rate variability features extracted through time-frequency analysis, as well as head posture, eye gaze direction and facial motion unit features extracted through OpenFace. These features were combined with features that were significantly related to exercise fatigue state based on expert knowledge retention to form a mixed feature set.

3. The non-contact assessment method for exercise fatigue as described in claim 1, characterized in that, The extraction of the second physiological feature through a deep neural network includes: Based on a deep rPPG neural network containing 3D convolutional layers and a temporal coding module, physiological change features are extracted, including: Extracting spatiotemporal features from video sequences using 3D-CNN; A temporal network is used to encode features to generate physiological change features; The rPPG waveform was reconstructed by time-frequency analysis to extract heart rate and heart rate variability features to form a second physiological characteristic.

4. The non-contact assessment method for exercise fatigue as described in claim 1, characterized in that, The signal quality features also include video brightness stability, face detection confidence, image sharpness, and rPPG signal quality index; the signal quality feature construction process includes: Calculate the average luminance standard deviation of the video sequence and convert it into a luminance stability index; The average confidence index of the entire video segment is calculated based on the face detection confidence score; Image sharpness metrics are calculated using the variance of the Laplacian operator. rPPG signals are extracted from visible light videos of the face, and the energy ratio inside and outside the heart rate frequency band is extracted from the rPPG signal spectrum to calculate the signal-to-noise ratio. The Euclidean distance between the combined feature vectors before and after the motion is calculated as the difference in the combined feature vectors.

5. The non-contact assessment method for exercise fatigue as described in claim 1, characterized in that, The method trains multiple basic expert models based on comprehensive feature vectors and preset motion fatigue state labels. Combining this with signal quality features from facial visible light video, a hybrid expert model ensemble framework is constructed, using signal quality features as conditional gating and Top-k sparse activation, to obtain the probability distribution of fatigue states before and after motion, including: Based on comprehensive feature vectors and exercise fatigue state labels, a basic expert model is independently trained using support vector machine, multilayer perceptron, and logistic regression, respectively, to output the probability distribution of exercise fatigue state. , , ;in Represents the comprehensive feature vector; The probability outputs of each basic model are concatenated into a composite feature vector. In addition, the signal quality characteristics are input into the gating network to generate gating weights; The gating weights are sparsified by using the Top-k sparse activation mechanism, and the outputs of the k experts with the largest weights are retained. The sparsified weights are normalized, and the probability outputs of each basic model are weighted and fused to obtain the final fatigue state probability distribution.

6. The non-contact assessment method for exercise fatigue as described in claim 5, characterized in that, The gated network and sparse activation mechanism are implemented in the following ways: a) The gated network is a two-layer fully connected structure, and the input is a composite feature vector. With signal quality characteristics splicing The output is gating weights generated using the Softmax function. ;in These are the gating weights for support vector machines, multilayer perceptrons, and logistic regression, respectively. b) Employing a Top-k sparse activation mechanism, only the k experts with the largest weights are retained, while the rest are reset to zero and renormalized to form sparse weights. ; c) The output is a weighted sum of the probabilities of each expert: ; d) Weighted expert probabilities The final classification result is obtained through linear transformation. .

7. A non-contact motion fatigue assessment system based on facial visible light video, characterized in that, include: The video capture module is used to capture visible light videos of the user's face before and after movement. The feature extraction module is used to extract the first physiological features and facial behavioral features based on visible light facial videos through time-frequency analysis and OpenFace, and obtain a mixed feature group after statistical testing and expert knowledge. Second physiological features are extracted using deep neural networks; In addition, a comprehensive feature vector is obtained by fusing the hybrid feature group and the second physiological feature; wherein, both the first and second physiological features include heart rate and heart rate variability features, and facial behavior features include head posture, eye gaze direction and facial motion unit feature group. The probability prediction module is used to train multiple basic expert models based on comprehensive feature vectors and preset motion fatigue state labels. It combines the signal quality features of facial visible light video to construct a hybrid expert model integration framework with signal quality features as conditional gating and Top-k sparse activation, so as to obtain the probability distribution of fatigue state before and after motion. The signal quality features include at least the difference in comprehensive feature vectors before and after motion. The fatigue assessment module is used to calculate the basic fatigue index based on the probability distribution of fatigue state before and after exercise, using multimodal probability weighting, and to incorporate physiological and facial behavior change rates to jointly correct the basic fatigue index in a product form; and to calculate the difference between the corrected fatigue index before and after exercise to obtain the fatigue impact index to incrementally assess the actual fatigue loss intensity. The baseline fatigue index is calculated using a multimodal probability weighting method based on the probability distribution of fatigue state before and after exercise. Physiological and facial behavioral change rates are also incorporated and multiplied to correct the baseline fatigue index, including: Based on the probability distribution of fatigue state before and after exercise According to the preset weight vector Perform a weighted summation to map to continuous values. ;in These represent the predicted probabilities for low, moderate, and severe fatigue states, respectively. These are the preset weights for low, medium, and severe fatigue states, respectively. This indicates the baseline fatigue index, including the baseline fatigue index before exercise. Basal fatigue index after exercise ; The relative rates of change of heart rate and heart rate variability before and after exercise were extracted respectively. After weighted fusion, a comprehensive physiological change quantity is generated. ; Based on the facial motion unit feature set after movement, the change in facial behavior is calculated using weighted methods. ; The multiplicative adjustment is applied to the base index using the following formula: in, , This is an adjustable coefficient; This indicates the adjusted fatigue index, including the pre-exercise adjusted fatigue index. Post-exercise corrected fatigue index .

8. A storage medium, characterized in that, It stores a computer program, wherein the computer program causes the computer to perform the non-contact assessment method for motion fatigue based on facial visible light video as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the non-contact motion fatigue assessment method based on facial visible light video as described in any one of claims 1 to 6.

Images