Method for extracting rPPG signal in compressed domain

Abstract

The invention provides a method for extracting rPPG signals in a compressed domain, and the method comprises the steps: firstly carrying out the image group (GOP) division of an input video, and judging the frame type of the input video to be an I frame or a P frame frame by frame; and if the frame is the I frame, completely decoding the I frame to obtain a face image, and extracting a DC coefficient reconstructed in the decoding process at the same time. The method comprises the following steps: acquiring skin as a region of interest (ROI) through face detection, mapping the ROI to transform blocks divided in coding, calculating the weight of skin pixels in each block, marking the ROI block (transform block corresponding to the skin region) according to a set threshold value, and recording the position and weight information of the ROI block. And if the frame is the P frame, partially decoding the P frame to a DCT domain, extracting a DC coefficient of a residual error, performing accumulation by combining a reconstructed DC value of the reference frame, and updating a DC value of the current frame. And screening out the ROI block of each frame in the GOP according to the position and weight information, and carrying out weighted averaging on the pixel mean value of the ROI block. And repeating the process until all GOPs are processed, and finally converting the YUV mean value sequence into an RGB mean value sequence so as to realize efficient rPPG signal extraction without complete decoding.

Description

Technical Field

[0001] This invention belongs to the technical field of non-contact physiological signal monitoring, specifically relating to a method for extracting rPPG signals in the video compression domain. Background Technology

[0002] Heart rate, as a crucial parameter for assessing human health, plays a vital role in chronic disease prevention and daily vital sign monitoring. Currently common heart rate detection methods (such as electrocardiograms and PPG-based pulse oximeters) are mostly contact-based measurements. While these offer high accuracy, prolonged use can cause discomfort, making daily monitoring inconvenient and posing a risk of cross-infection in certain scenarios. Therefore, heart rate detection technology based on remote photoplethysmography (rPPG) has gradually become a research hotspot. This method extracts the blood volume pulse signal (BVP) by capturing subtle color changes in facial skin caused by heartbeats and then calculates the heart rate. It offers advantages such as low cost, ease of operation, and suitability for remote monitoring.

[0003] In this non-contact heart rate detection, video serves as the signal carrier. Traditional rPPG methods primarily process data in the pixel domain, requiring complete decoding of the compressed video, resulting in high computational complexity and processing latency, making them unsuitable for mobile terminals, medical devices, and large-scale monitoring systems. In contrast, compression-domain methods do not require generating complete pixel images; instead, they directly utilize information from the compressed video bitstream, such as transform coefficients, inverse quantization data, motion vectors, and block structure. This significantly reduces computational complexity and makes them more suitable for integration into real-time systems.

[0004] While compressed domain technology has proven advantageous in tasks such as video object tracking and behavior recognition, there is currently a lack of effective methods for extracting rPPG signals from compressed domain information. Therefore, a method is needed that can utilize the frequency domain data features from the video encoding process to obtain rPPG signals through only partial decoding, enabling low-resource-consumption and high-efficiency non-contact heart rate detection. Summary of the Invention

[0005] The purpose of this invention is to provide a method for directly extracting rPPG signals in the video compression domain. This method utilizes the characteristics of frequency domain data in video encoding, effectively avoiding the significant time and storage resource consumption during full video decoding, thereby improving the efficiency of rPPG signal extraction. The method includes the following steps:

[0006] S01, For an input face video, first divide the video sequence according to Group of Pictures (GOP), and then determine whether the frame type in the GOP is I-frame or P-frame.

[0007] S02, if it is an I-frame, fully decode the I-frame to obtain the face image, and save the reconstructed DC coefficients during the decoding process. Face detection and skin region segmentation are performed on the face image, where the skin region is the region of interest (ROI) for rPPG signal processing. The ROI is mapped to the transform blocks divided in the encoding, and the proportion P of skin pixels in each block is calculated. s A set of ROI blocks (transform blocks corresponding to skin regions) is marked according to a preset threshold, and their corresponding position and weight information is saved. This set of information is then applied to all frames within the current GOP.

[0008] S03, if it is a P-frame, decode the P-frame portion to obtain the frequency domain data of the image, and obtain the residual DC coefficients reconstructed from each transform block. Reconstructed DC value using reference frame The DC value of each transform block in the current P frame is updated by accumulating the residual DC values.

[0009] S04. Based on the position information in the set in step 2, select ROI blocks in the transform blocks of each frame (I frame and P frame) of GOP. Multiply the DC value of each ROI block in the frame by a coefficient to obtain the pixel mean of the reconstructed block. Then, use the weight information in the set in step 2 to perform a weighted average of the pixel mean of all ROI blocks in each frame to obtain the pixel mean time series of the three channels Y, U, and V.

[0010] S05, repeat the above steps for each GOP group until all frames of the video have been processed. Convert the final YUV mean signal to an RGB mean signal, and then obtain the BVP signal using the rPPG extraction algorithm. Attached Figure Description

[0011] Figure 1 The present invention provides a flowchart of a method for extracting rPPG signals in the compressed domain.

[0012] Figure 2 : Schematic diagram of the selected ROI block and the percentage of marked skin

[0013] Figure 3 Comparison chart of the original video and the BVP signal extracted from the compressed domain. Detailed Implementation

[0014] like Figure 1As shown, this invention proposes a method for extracting rPPG signals in the compressed domain. By utilizing the mathematical equivalence between the DC component in the DCT domain of video coding and the pixel mean, the original signal required for rPPG signal extraction can be obtained without completely decoding the video to the pixel domain, reducing computational complexity and improving the efficiency of physiological signal monitoring. The specific implementation steps are as follows:

[0015] For an input video of a face, first parse the bitstream file and extract each group of pictures (GOPs), then iterate through the frames within each GOP and determine the type F of the current frame. type ∈{I,P};

[0016] When F type When the frame is an I-frame, it is fully decoded to the pixel domain. During decoding, the DC coefficients in the inverse-quantized DCT domain are retained. Face detection is performed on the decoded complete face image, and the facial skin regions are segmented as regions of interest (ROIs). The ROI regions are mapped onto an N×N image of the same size as the encoded transform block, and the ratio P of the number of skin pixels to the total number of pixels in each block is calculated. s Set a threshold T s =0.5, when P s >T s Then mark the transform block as a ROI block, such as Figure 2 As shown, each ROI block is selected and its corresponding position and weight information is saved; otherwise, it is discarded. The information of the final ROI block set will be used as the basis for calculating the average skin pixel value of all frames in the current GOP.

[0017] When F type When the frame is P-frame, the DCT coefficients of each residual block are obtained by decoding the P-frame portion, and the DC coefficients are extracted from them. Based on the principle of inter-frame prediction, the reconstructed DC value of the reference frame (I-frame or P-frame) is used. With the current frame By accumulating the values, the DC value of the transform block can be updated. The accumulation formula is as follows:

[0018]

[0019] In the formula This is the updated DC value for each transform block in the current P-frame.

[0020] According to the principle of traditional two-dimensional discrete cosine transform (DCT), the DC value differs from the actual pixel mean by a scaling factor α. Let the DC value of the ROI block be... The average pixel value of this block can then be expressed as: Therefore, multiplying the DC values ​​obtained from all frames by a coefficient yields the pixel mean of the reconstructed image patch. The pixel mean of all ROI patches in each frame is then weighted using the weight information from the saved set. Thus, the pixel mean of the skin region in image t can be defined as:

[0021]

[0022] In the formula μ c c∈[Y,U,V] represents the pixel mean of color channel c, K is the total number of blocks in the ROI block set, and P s,i The percentage of skin pixels in the i-th ROI block. Let be the reconstructed DC value of the i-th ROI block in the c-channel of the t-th frame. Convert the pixel mean value from the YUV color space to the RGB color space. Finally, use the rPPG algorithm to obtain the BVP signal from the RGB mean signal.

[0023] The comparison results between the original video and the BVP signal extracted from the compressed domain are as follows: Figure 3 As shown, the red line represents the BVP signal extracted from the original video, while the blue line represents the BVP signal extracted from the compressed domain after video compression. It can be observed that the fluctuations and periods of the two waveforms are highly consistent, reflecting that a valid BVP signal can be obtained through only partial decoding.

Claims

1. A method for extracting rPPG signals in a compressed domain, characterized in that, Includes the following steps: S01, For an input face video, first divide the video sequence according to Group of Pictures (GOP), and then determine whether the frame type in the GOP is I-frame or P-frame. S02, if it is an I-frame, fully decode the I-frame to obtain the face image, and save the reconstructed DC coefficients during the decoding process. Face detection and skin region segmentation are performed on the face image, where the skin region is the region of interest (ROI) for rPPG signal processing. The ROI is mapped to the transform blocks divided in the encoding, and the proportion P of skin pixels in each block is calculated. s A set of ROI blocks (transform blocks corresponding to skin regions) is marked according to a preset threshold, and their corresponding position and weight information is saved. This set of information is then applied to all frames within the current GOP. S03, if it is a P-frame, decode the P-frame portion to obtain the frequency domain data of the image, and obtain the residual DC coefficients reconstructed from each transform block. Reconstructed DC value using reference frame The DC value of each transform block in the current P-frame is updated by accumulating the residual DC values. S04. Based on the position information in the set in step 2, select ROI blocks in the transform blocks of each frame (I frame and P frame) of GOP, multiply the DC value of each ROI block in the frame by a coefficient to obtain the pixel mean of the reconstructed block, and then use the weight information in the set in step 2 to perform a weighted average of the pixel mean of all ROI blocks in each frame to obtain the pixel mean time series of the three channels Y, U, and V. S05, repeat the above steps for each GOP group until all frames of the video have been processed. Convert the final YUV mean signal to an RGB mean signal, and then obtain the BVP signal using the rPPG extraction algorithm.

2. The method for extracting rPPG signals in the compressed domain according to claim 1, characterized in that, In step S02, the process of marking ROI blocks includes: mapping the ROI region onto an N×N image with the same size as the encoded transform block, and calculating the ratio P of the number of skin pixels to the total number of pixels in each block. s Set a threshold T s =0.5, when P s >T s If the transformation block is valid, it is marked as a valid ROI block, and the position and weight information of the set are retained; otherwise, it is discarded.

3. The method for extracting rPPG signals in the compressed domain according to claim 1, characterized in that, In step S04, the process of weighted averaging of the pixel mean of the ROI block is as follows: According to the principle of traditional two-dimensional discrete cosine transform (DCT), the DC value differs from the actual pixel mean by a scaling factor α. Let the DC value of the ROI block be... The average pixel value of this block can then be expressed as: Therefore, multiplying the DC value of all ROI blocks in all frames by a coefficient yields the pixel mean of the reconstructed image block. The weighted average of all ROI blocks in each frame is then calculated using the weight information stored in the aforementioned set. The pixel mean of the skin region in the t-th image can be defined as: In the formula μ c c∈[Y,U,V] represents the pixel mean of color channel c, K is the total number of blocks in the ROI block set, and P s,i The percentage of skin pixels in the i-th ROI block. Let be the reconstructed DC value of the i-th ROI block in the c-channel of the t-th frame.

Images