A method and system for rppg-oriented video heart rate extractability evaluation

Abstract

The application discloses a video heart rate extractability evaluation method for rPPG, comprising the following steps: acquiring video data containing a facial skin area, dividing the video data into a plurality of time windows according to a preset window length, and establishing a window index for each time window; performing face detection and facial key point positioning on video frames in each time window to obtain a window-level effective skin frame set; extracting window-level video features based on the window-level effective skin frame set; performing time sequence derivation, feature screening and standardization processing on the window-level video features to obtain an extended feature vector for classification; inputting the extended feature vector into a trained extractability evaluation model to obtain a heart rate extractability probability or category corresponding to the time window. The probability or category output by the application is used to represent the extractability degree of the rPPG heart rate signal in the video window, and is suitable for remote medical treatment, video conference health monitoring, driver monitoring and rPPG data screening.

Description

Technical Field

[0001] This invention belongs to the fields of non-contact vital sign monitoring, video quality assessment, and machine learning technology, specifically relating to a method and system for assessing the extractability of video heart rate using remote photoplethysmography (rPPG). Background Technology

[0002] Heart rate is an important physiological indicator reflecting the state of the human circulatory system, exercise load, and emotional changes. Traditional heart rate monitoring usually relies on electrocardiograms, electrode pads, finger clip pulse oximeters, or wristband photoplethysmography devices. Although these devices are relatively mature, they require direct contact between the sensor and the human skin, which presents problems such as inconvenience in long-term monitoring, remote consultation, driving safety, infant monitoring, burn patient monitoring, and non-invasive health management, as well as issues related to wearing them, behavioral constraints, or skin irritation.

[0003] rPPG technology uses a regular RGB camera to record subtle periodic changes in reflected light from the face or other skin areas, estimating blood volume, pulse signals, and heart rate from the video. It offers advantages such as being non-contact, low-cost, and easy to deploy. With the widespread adoption of mobile devices, web conferencing, telemedicine, and intelligent monitoring equipment, video-based heart rate measurement has broad application value.

[0004] However, rPPG signals are essentially small-amplitude color periodic components within video pixels, making them susceptible to changes in lighting, head movement, facial expressions, skin color differences, automatic exposure, camera noise, resolution, frame rate, and video compression distortion. In real-world applications, videos are typically transmitted, stored, or encoded and compressed over a network. Even if a face appears visually clear, it may no longer contain the faint physiological information necessary for reliable heart rate estimation.

[0005] Most existing technologies focus on the backend heart rate estimation algorithm itself, such as using methods like Chromaticometry (CHROM), Orthogonal Projection of the Skin (POS), Independent Component Analysis (ICA), Orthogonal Matrix Image Transformation (OMIT), and deep learning networks to recover BVP (blood volume pulse wave) or BPM (heart rate per minute) from a given video. These methods assume that the input video has a certain quality, but lack a front-end judgment mechanism to determine whether the input video window is suitable for rPPG analysis before heart rate estimation.

[0006] In addition, traditional video quality evaluation metrics are mainly based on human subjective perception and may not necessarily reflect factors such as skin color temporal continuity, heart rate band energy, ROI stability, and coding strength required for rPPG tasks. Summary of the Invention

[0007] The purpose of this invention is to provide a video heart rate extractability assessment method for rPPG. This method does not aim to directly output heart rate values, but rather judges whether the video window has reliable heart rate extraction conditions before backend rPPG heart rate estimation, thereby realizing front-end quality screening, re-acquisition prompts, and heart rate confidence correction.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a method for evaluating the extractability of video heart rate for rPPG, the method comprising the following steps:

[0009] S1, acquire video data containing facial skin areas, divide the video data into several time windows according to a preset window length, and create a window index for each time window;

[0010] S2, perform face detection and facial key point localization on video frames within each time window, generate skin regions of interest based on facial key points, and remove eye, mouth and invalid skin regions to obtain a window-level set of valid skin frames;

[0011] S3, extracts window-level video features based on the window-level effective skin frame set;

[0012] S4, perform temporal derivation, feature filtering and standardization on the window-level video features to obtain an extended feature vector for classification;

[0013] S5, input the extended feature vector into the trained extractability evaluation model to obtain the heart rate extractability probability or category for the corresponding time window.

[0014] This invention also proposes a video heart rate extractability assessment system for rPPG, including a video window segmentation module, a skin region of interest extraction module, a window-level feature extraction module, a feature derivation and filtering module, and an extractability classification output module;

[0015] The video window segmentation module is used to acquire video data containing facial skin areas, divide the video data into several time windows according to a preset window length, and establish a window index for each time window.

[0016] The skin region of interest extraction module is used to perform face detection and facial key point localization on video frames within each time window, generate skin regions of interest based on facial key points, and remove eye, mouth and invalid skin regions to obtain a window-level set of valid skin frames.

[0017] The window-level feature extraction module is used to extract window-level video features based on a set of valid skin frames at the window level.

[0018] The feature derivation and filtering module is used to perform temporal derivation, feature filtering and standardization on the window-level video features to obtain an extended feature vector for classification.

[0019] The extractability classification output module is used to input the extended feature vector into the trained extractability evaluation model to obtain the heart rate extractability probability or category for the corresponding time window.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects:

[0021] (1) The present invention adds front-end extractability assessment before back-end heart rate estimation, which can identify low-quality windows in advance and reduce the output of unconfidence rate caused by unusable videos.

[0022] (2) This invention makes comprehensive use of multiple features such as frequency domain signal, color distribution, brightness and chromaticity, motion disturbance and coding strength, rather than relying solely on traditional video clarity or bit rate indicators, which better meets the special needs of rPPG tasks for subtle skin color changes.

[0023] (3) The present invention constructs training labels by voting using the multi-rPPG method, taking into account both BPM error and BVP waveform structure similarity, making the labels more stable and more interpretable for the task.

[0024] (4) The present invention performs training and verification by video or subject group, which can avoid data leakage caused by highly overlapping sliding windows and make the model evaluation results closer to real deployment.

[0025] (5) This invention can be directly deployed in scenarios such as telemedicine, video conferencing health monitoring, driver monitoring, mobile selfie video analysis and large-scale rPPG data screening, with low hardware cost and strong engineering interpretability. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the overall process of the video heart rate extractability assessment method for rPPG according to the present invention.

[0027] Figure 2 This is a flowchart illustrating the multi-method voting label generation process during the training phase of this invention.

[0028] Figure 3 This is a flowchart illustrating the window-level feature construction, filtering, and model training process of this invention.

[0029] Figure 4 This is a schematic diagram of the modules of the online deployment system of the present invention. Detailed Implementation

[0030] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the following embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Without departing from the core idea of ​​the present invention, those skilled in the art can make equivalent substitutions or combinations of window length, number of features, classification model, and threshold strategy.

[0031] A method for assessing the extractability of heart rate in rPPG videos, comprising:

[0032] S1, acquire video data containing facial skin areas, divide the video data into multiple time windows according to a preset window length, and establish a window index for each time window;

[0033] like Figure 1 As shown, this invention first receives facial video or a real-time video stream. The video can come from a regular RGB camera, a mobile phone camera, a remote conferencing camera, a vehicle-mounted camera, or a surveillance camera. It is then buffered and segmented using a sliding window of fixed length.

[0034] In this embodiment, the window length is preferably 12 seconds, and the step size is preferably 1 second. A 12-second window can provide sufficient spectral resolution for the heart rate band, while avoiding the increased latency and in-window non-stationarity caused by a 20-second equal-length window.

[0035] S2, perform face detection and facial key point localization on video frames within each time window, generate skin regions of interest based on facial key points, and remove eye, mouth and invalid skin regions to obtain a window-level set of valid skin frames;

[0036] In each window, face detection and keypoint localization are performed on the video frames. Preferably, a face mesh keypoint localization algorithm (e.g., MediaPipe Face Mesh) or an equivalent keypoint localization algorithm is used to obtain 468 facial keypoints. A facial convex hull is generated based on the keypoints, and areas such as the eyes and mouth are excluded to obtain a skin ROI mask for each frame. Frames with too few effective skin pixels, failed keypoint localization, or incomplete faces are marked as invalid; the remaining effective frames within the window and their corresponding skin ROI pixel sets are recorded as the window-level effective skin frame set. If the number of effective frames within a window is lower than a preset threshold, the window is directly determined as unextractable.

[0037] S3, extract window-level video features based on the effective skin frame set. The window-level video features include signal-level features, pixel-level features, color distribution features, luminance and chrominance features, geometric motion features, and coding parameter features.

[0038] After obtaining the effective set of skin frames, signal-level features are extracted. Let mu_c(t) be the average color of channel c within the region of interest (ROI) of the skin in frame t, where c belongs to the R, G, and B channels. Then, the time standard deviation, peak-to-peak value, and heart rate band power percentage of this channel within the window can be calculated. The heart rate band can be set from 0.65Hz to 4Hz, corresponding to approximately 39bpm to 240bpm. A higher heart rate band power percentage generally indicates a more pronounced periodic component related to heart rate within that window.

[0039] Furthermore, pixel-level features are extracted. The spatial variance of RGB pixels within each skin ROI is calculated, and the mean value is taken within a window to describe the spatial dispersion within the skin region.

[0040] Furthermore, color distribution features were extracted. Sixteen normalized histograms were constructed for each of the three RGB channels to characterize dark areas, saturated areas, and color quantization clustering. These features can reflect the impact of low bitrate compression, uneven lighting, shadows, and local occlusion on rPPG extractability.

[0041] Furthermore, luminance and chromaticity features are extracted, specifically YUV color space histograms. To distinguish between luminance variations and chromaticity shifts, the RGB image is converted to the YUV color space, and eight-segment histograms for the Y, U, and V channels are calculated within the skin ROI. The Y channel describes exposure and brightness distribution, while the U / V channels describe skin tone shifts, white balance changes, and chromaticity perturbations caused by compression. In this embodiment, the YUV histogram and RGB histogram are used together as color distribution inputs.

[0042] Geometric motion features include the skin region area as a percentage of the entire frame (face_ratio) and inter-frame motion perturbations. Let the video resolution be W×H, and the number of effective skin pixels in frame t be N_t, then face_ratio(t) = N_t / (W×H). The window-level skin region area as a percentage of the entire frame (face_ratio) is the average of the effective frames' skin region area as a percentage of the entire frame (face_ratio(t)). Motion perturbations between adjacent effective frames are preferably calculated by taking the mean of the RGB absolute differences within the intersection of two skin ROI masks to reduce spurious motion caused by mask boundary jitter. The mean, standard deviation, maximum value, and high perturbation ratio are further calculated for each inter-frame difference within the window.

[0043] Encoding parameter features are used to describe the video sampling and compression intensity. The video frame rate (fps), average bitrate (B), file size, and window-equivalent file size are read or estimated, and the number of bits per pixel per frame (bpp) is calculated, where bpp = B / (fps × W × H). Compared to bitrate alone, bpp more fairly describes the number of available coded bits per pixel across different resolutions and frame rates, making it more comparable for assessing rPPG extractability.

[0044] S4, perform temporal derivation, feature filtering and standardization on the window-level video features to obtain an extended feature vector for classification;

[0045] In a preferred embodiment, the aforementioned basic features comprise 93 dimensions. The feature table and window index table are merged, and samples with missing labels, empty columns, constant columns, and index columns not used in training are removed. Missing values ​​are then processed consistently. For the training set, the features are further sorted by video identifier and window index. For each basic feature, the moving average, moving standard deviation, and first difference are calculated for window widths of 5 and 15, thus forming an extended feature set containing temporal context. This design can reflect the local statistics and trends of adjacent windows. Finally, a predetermined number of features are selected using gradient boosting tree feature importance.

[0046] S5, input the extended feature vector into the trained extractability evaluation model, and output the heart rate extractability probability or category for the corresponding time window; wherein, the extractability evaluation model includes an input feature receiving unit, a feature filtering unit, a feature standardization unit, and a binary classification discriminant unit.

[0047] like Figure 2 As shown, before model training, this embodiment first generates weakly supervised labels for supervised training. This label generation process is independent of the internal discriminative structure of the extractability evaluation model, and its purpose is to provide training targets for each time window. First, the original video is constructed into multiple bitrate versions using an H.264 / AVC encoder, keeping the resolution and frame rate constant, making the bitrate the primary control variable. Then, the video of each bitrate version is divided into 12-second sliding windows to form window-level samples. For each window, rPPG methods such as Chromaticity Mapping (CHROM), Orthogonal Projection of the Skin (POS), and Orthogonal Matrix Image Transformation (OMIT) are run to obtain the blood volume pulse wave (BVP). The heart rate per minute (BPM) estimate is then calculated based on the BVP. For each rPPG method, the absolute error of BPM (MAE) for that method within the corresponding window is calculated.

[0048] The absolute error (MAE) of BPM is compared with a threshold of 5. If MAE < 5, the single-method labels generated by the rPPG method for that time window are extractable; if MAE ≥ 5, the single-method labels generated by the rPPG method for that time window are not extractable. A majority vote is performed on the single-method binary classification labels obtained by multiple rPPG methods within the same time window to obtain window-level binary classification weakly supervised labels; if a tie occurs, a preset conservative strategy can be applied. Therefore, Figure 2The final output is a binary classification weakly supervised label corresponding to each time window. This label is used to characterize whether the corresponding window meets the conditions for extracting rPPG heart rate signals, and serves as... Figure 3 The supervision objective in the model training process shown.

[0049] like Figure 3 As shown, in the training process of the extractability evaluation model, the extended feature vector obtained in S4 is combined with... Figure 2 The generated binary weakly supervised labels form the training samples, and are processed sequentially starting from the input feature receiving unit. The input feature receiving unit receives the extended feature vector and its corresponding binary weakly supervised label for each time window; the feature selection unit preferably uses the LightGBM gradient boosting tree model to evaluate the importance of the input extended features, and selects a preset number of features according to the importance ranking, such as selecting the top 150 features, to obtain a selected feature set for subsequent training; the feature standardization unit standardizes the selected feature set according to the training set statistics, and reuses the same standardization parameters in the validation and inference phases.

[0050] The preferred binary classification discriminant unit is the XGBoost binary classification model, which uses standardized screening features as input. Figure 2 The generated binary weakly supervised labels serve as the supervision target, learning the mapping relationship between feature vectors and extractable and non-extractable classes, and obtaining the binary classification result for the corresponding time window. Specifically, the non-extractable probability is output through the binary logistic regression objective function, and then converted into an extractable or non-extractable class based on a preset threshold. During model training, the overall training objective is to minimize the binary logistic loss with a regularization term. When the number of non-extractable and extractable windows is unbalanced, the weight of minority class samples in the loss function is increased through class imbalance weight parameters or an equivalent class weighting mechanism. The class imbalance weight parameters are preferably set according to the ratio of the number of extractable to the number of non-extractable samples. The classification threshold can be fixed at 0.5, or it can be scanned within the training set using nested grouped cross-validation, and the optimal threshold can be selected based on the macro-average F1, the non-extractable class recall, or the applied cost function.

[0051] To avoid data leakage caused by highly overlapping sliding windows, model training and validation preferably employ a cross-validation strategy grouped by video or subject. All windows for the same video or subject can only appear in the same training or validation fold. This setting prevents adjacent windows with almost identical content from appearing in the training and validation sets, making performance evaluation more closely resemble real-world deployment scenarios.

[0052] In one experimental embodiment, validation was performed on the second subset of the Burgundy-Franche-Comté University Remote Photoplethysmography (UBFC2) dataset and the publicly available rPPG dataset of the Pulse Rate Detection (PURE) dataset, along with their multi-bitrate compressed versions, resulting in a total of 42,925 cleaned window samples. In the binary classification task, using both extractable and non-extractable classes as targets, the combined training yielded a macro-mean precision-recall harmonic mean (Macro-F1) of approximately 0.715 and an area under the receiver operating characteristic (AUC) of approximately 0.824, significantly higher than virtual baselines such as majority class prediction, uniform random prediction, and prior random prediction. This demonstrates that the model effectively utilizes video visual, motion, and coded features to achieve discrimination.

[0053] Experiments also show that features such as heart rate band energy ratio, bit rate, bpp, motion perturbation, YUV chromaticity distribution, RGB peak-to-peak value and luminance histogram contribute significantly to the model, indicating that the feature system of this invention can characterize rPPG extractability from multiple perspectives, including physiological signal visibility, compression intensity, motion stability and color state.

[0054] The heart rate extractability probability or category is used to characterize the extractability of rPPG heart rate signals within the corresponding time window. When the model outputs an unextractable probability, a higher probability value indicates that the time window is less suitable for rPPG heart rate signal extraction; when the model outputs a category, an extractable category indicates that the window meets the preset extractability conditions, and an unextractable category indicates that the window does not meet the preset extractability conditions.

[0055] like Figure 4 As shown, the present invention can also be implemented as a video heart rate extractability assessment system for rPPG, including a video window segmentation module, a skin region of interest extraction module, a window-level feature extraction module, a feature derivation and filtering module, and an extractability classification output module;

[0056] The video window segmentation module is used to acquire video data containing facial skin areas, divide the video data into several time windows according to a preset window length, and establish a window index for each time window.

[0057] The skin region of interest extraction module is used to perform face detection and facial key point localization on video frames within each time window, generate skin regions of interest based on facial key points, and remove eye, mouth and invalid skin regions to obtain a window-level set of valid skin frames.

[0058] The window-level feature extraction module is used to extract window-level video features based on a set of valid skin frames at the window level.

[0059] The feature derivation and filtering module is used to perform temporal derivation, feature filtering and standardization on the window-level video features to obtain an extended feature vector for classification.

[0060] The extractability classification output module is used to input the extended feature vector into the trained extractability evaluation model to obtain the heart rate extractability probability or category for the corresponding time window.

[0061] In scenarios such as telemedicine, video conferencing health monitoring, driver monitoring, and dataset construction, the heart rate extractability probability or category can serve as a quantitative annotation result of the video window quality, indicating whether the window meets the conditions for rPPG heart rate signal extraction. This evaluation result itself constitutes the output of this invention. Subsequent heart rate estimation, service prompts, or acquisition control can be configured separately by external application systems based on this evaluation result, and are not considered necessary steps in the method of this invention.

[0062] This invention is not limited to the three rPPG methods of Chromaticity Mapping (CHROM), Orthogonal Plane Projection (POS), and Orthogonal Matrix Image Transformation (OMIT), nor is it limited to the XGBoost model. Any equivalent technical solution that utilizes the visual, motion, and coded features of a video window to output the probability or category of heart rate extractability should fall within the protection scope of this invention.

Claims

1. A method for evaluating the extractability of heart rate in rPPG videos, characterized in that, The method includes the following steps: S1, acquire video data containing facial skin areas, divide the video data into several time windows according to a preset window length, and create a window index for each time window; S2, perform face detection and facial key point localization on video frames within each time window, generate skin regions of interest based on facial key points, and remove eye, mouth and invalid skin regions to obtain a window-level set of valid skin frames; S3, extracts window-level video features based on the window-level effective skin frame set; S4, perform temporal derivation, feature filtering and standardization on the window-level video features to obtain an extended feature vector for classification; S5, input the extended feature vector into the trained extractability evaluation model to obtain the heart rate extractability probability or category for the corresponding time window.

2. The video heart rate extractability assessment method for rPPG according to claim 1, characterized in that, Face detection and facial landmark localization are performed on video frames within each time window using a face mesh-based landmark localization method. The skin region of interest is composed of the facial bulge after removing the eye and mouth bulges.

3. The video heart rate extractability assessment method for rPPG according to claim 1, characterized in that, The window-level video features include signal-level features, pixel-level features, color distribution features, luminance and chrominance features, geometric motion features, and coding parameter features.

4. The video heart rate extractability assessment method for rPPG according to claim 3, characterized in that, The signal-level characteristics include the time standard deviation of the RGB channels, peak-to-peak value, and the power ratio of the heart rate frequency band. The pixel-level features include the skin pixel spatial variance of the RGB channels; The color distribution features include RGB channel normalized histograms; The brightness and chromaticity features include a YUV color space histogram; The geometric motion features include the skin region area ratio, the mean, standard deviation, maximum value, and high perturbation ratio of the inter-frame difference within the intersection of skin regions of adjacent valid frames; The encoding parameter features include frame rate, bit rate, window equivalent file size, and bits per pixel per frame.

5. The video heart rate extractability assessment method for rPPG according to claim 4, characterized in that, The formula for calculating bits per pixel per frame (bpp) is: bpp = B / (fps × W × H); Where B is the average video bitrate, fps is the video frame rate, and W and H are the video frame width and height, respectively.

6. The video heart rate extractability assessment method for rPPG according to claim 1, characterized in that, The training labels for the extractability evaluation model are generated through a multi-method voting process, specifically: The original video is constructed into several bitrate versions while keeping the resolution and frame rate constant, making the bitrate a control variable. For each time window under each bitrate version, at least two of the following methods are used to calculate the blood volume pulse wave signal: chromaticity method, skin orthogonal plane projection method, and orthogonal matrix image transformation method; The corresponding heart rate per minute is calculated using the blood volume pulse wave signal, and the absolute error of the heart rate per minute is calculated. The absolute error of heart rate per minute is compared with a set threshold to generate binary labels for single-method extractable and non-extractable classes; Window-level quality labels are obtained by voting on several binary classification labels of single-method extractable and non-extractable classes.

7. The video heart rate extractability assessment method for rPPG according to claim 6, characterized in that, The extractability evaluation model includes an input feature receiving unit, a feature filtering unit, a feature standardization unit, and a binary classification discriminant unit. The extended feature vector and the corresponding window-level quality label are used to form training samples, and the input feature receiving unit receives the training samples. The feature selection unit evaluates the importance of the extended feature vectors in the training samples and selects a preset number of features according to the importance ranking to obtain a selected feature set; the feature standardization unit standardizes the selected feature set according to the training set statistics; the binary classification discrimination unit takes the standardized selected features as input, uses the window-level quality label as the supervision target, learns the mapping relationship between the feature vector and the extractable class and the non-extractable class, and obtains the binary classification discrimination result for the corresponding time window.

8. A video heart rate extractability assessment system for rPPG based on any one of claims 1 to 7, characterized in that, It includes a video window segmentation module, a skin region of interest extraction module, a window-level feature extraction module, a feature derivation and filtering module, and an extractability classification output module; The video window segmentation module is used to acquire video data containing facial skin areas, divide the video data into several time windows according to a preset window length, and establish a window index for each time window. The skin region of interest extraction module is used to perform face detection and facial key point localization on video frames within each time window, generate skin regions of interest based on facial key points, and remove eye, mouth and invalid skin areas to obtain a window-level set of valid skin frames. The window-level feature extraction module is used to extract window-level video features based on a set of valid skin frames at the window level. The feature derivation and filtering module is used to perform temporal derivation, feature filtering and standardization on the window-level video features to obtain an extended feature vector for classification. The extractability classification output module is used to input the extended feature vector into the trained extractability evaluation model to obtain the heart rate extractability probability or category for the corresponding time window.

Images