Face video compression method for remote photoplethysmography

Abstract

The invention discloses a face video compression method for remote photoplethysmography, which belongs to the field of computer image video processing and multimedia, and comprises the following steps: obtaining an uncompressed target video source; constructing a time domain down-sampling coding module based on uniform sampling, and carrying out down-sampling; constructing a rapid facial region-of-interest extraction module to extract a facial skin-of-interest region; constructing a computing resource redistribution module based on the facial region of interest to carry out computing resource redistribution; constructing a bit resource redistribution module for remote photoplethysmography to carry out bit resource redistribution; and carrying out bidirectional prediction frame interpolation by adopting a bidirectional prediction motion compensation interpolation technology, and finally carrying out time domain up-sampling coding to obtain a compressed and reconstructed video. According to the invention, on the premise of ensuring the visual effect and the high compression rate, the vital sign information contained in the original video is subjected to lossless or near-lossless compression.

Description

Technical Field

[0001] The invention belongs to the field of computer image video processing and multimedia, and in particular relates to a facial video compression method for remote photoplethysmography. Background Art

[0002] In recent years, with the accelerated pace of life and increased work pressure, public health problems have become more frequent. Chronic diseases such as "three highs" and cardiovascular diseases have become a major threat to human health. Therefore, whether it is to deal with chronic cardiovascular and cerebrovascular diseases or daily health monitoring of the elderly, there is a great demand for daily monitoring of physiological signals.

[0003] Remote photoplethysmography (rPPG) measures physiological signals such as heart rate, respiratory rate, blood oxygen saturation and blood pressure by analyzing subtle color changes reflected from the skin surface in human videos, and then performs cardiopulmonary assessment and disease diagnosis. It has attracted widespread attention from many scholars. rPPG technology has important application value in many fields such as disease prediction and screening, patient monitoring, driver monitoring and fitness training. However, rPPG measurement is susceptible to various noises, including changes in ambient light intensity, rigid / non-rigid motion of subjects, subject skin color and camera characteristics. Among the many influences, one factor is particularly critical, that is, the impact of video compression on rPPG measurement. However, research on this issue has not received much attention at present, because most current rPPG studies use uncompressed video datasets collected in laboratory environments to extract blood volume pulses. This can certainly verify the effectiveness of the proposed rPPG extraction algorithm, but it is not conducive to the generation of large-scale public benchmark datasets that can fairly test multiple different algorithms, because uncompressed videos will occupy a large amount of hard disk space. For example, a 1080p / 30fps video with a duration of only 1 minute can occupy up to 11G of space. In addition, there are still many situations where rPPG videos need to be compressed. For example, patient videos need to be transmitted to a remote medical system for diagnostic analysis. The bit rate of an uncompressed 1080p / 30fps video can reach more than 1400Mb / s, which greatly exceeds the transmission capacity of current telecommunications technology. Therefore, video compression is an indispensable part of camera-based physiological signal detection. Most of the existing physiological detection datasets, such as COHFACE and PFF, are the result of compression by traditional video encoding (such as H.264). However, the traditional hybrid video compression framework will destroy the rPPG signal to a certain extent. This is because such traditional video compression schemes are not specifically designed for rPPG extraction problems. On the contrary, video compression is to remove the pulsation information contained in the tiny color changes of the skin that cannot be perceived by the human eye to reduce the required storage space. Previous work has systematically analyzed the adverse effects of video compression on rPPG signals and found that as the video bit rate decreases (i.e., the video compression rate increases), the rPPG signal quality (signal-to-noise ratio) decreases linearly. When the video is compressed to a certain extent, accurate physiological parameters cannot be extracted.

[0004] In order to deal with the adverse effects of video compression on the quality of rPPG signals, one idea is to analyze the characteristics of compressed videos and design algorithms in a targeted manner to separate rPPG signals from video compression artifacts. Although this idea can alleviate the impact of video compression artifacts to a certain extent, since this solution extracts signals from videos that have been corrupted by compression artifacts, the quality of the extracted rPPG signals is not ideal, and these algorithms are often complex to implement. How to find a solution that can be applied to all previous classic rPPG signal extraction algorithms, such as ICA, CHROM, POS, etc., so that these algorithms can obtain more accurate results without any modification, it is necessary to solve the problem from the source, that is, it is necessary to propose a video compression solution specifically for remote photoplethysmography. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a facial video compression method for remote photoplethysmography, including a time-domain downsampling encoding module based on uniform sampling, a computing resource reallocation module based on a facial region of interest (ROI), a bit resource reallocation module for remote photoplethysmography, and a time-domain upsampling encoding module based on motion compensation interpolation. The present invention performs lossless or near-lossless compression on the vital sign information contained in the original video while ensuring visual effects and a high compression rate.

[0006] The technical solution of the present invention is as follows:

[0007] A facial video compression method for remote photoplethysmography comprises the following steps:

[0008] Step 1: Obtain an uncompressed target video source through a capture device;

[0009] Step 2: construct a time-domain downsampling encoding module based on uniform sampling, input the acquired target video source into the time-domain downsampling encoding module, and obtain a downsampled video;

[0010] Step 3: construct a fast facial region of interest extraction module to extract the facial skin region of interest from the downsampled video to obtain a facial mask of interest;

[0011] Step 4: construct a computing resource reallocation module based on the facial region of interest to reallocate computing resources to the facial mask of interest;

[0012] Step 5: construct a bit resource reallocation module for remote photoplethysmography, reallocate the bit resources for the mask of interest, and obtain a reallocated bit stream;

[0013] Step 6: Use bidirectional prediction motion compensation interpolation technology to perform bidirectional prediction interpolation on the reallocated bit stream, and finally perform time domain upsampling encoding to obtain a compressed and reconstructed video.

[0014] Furthermore, in step 1, the acquisition device is any device including a camera, and the shooting needs to cover the human skin area, including the forehead and cheeks of the face; the acquired target video source is represented as a set {1,2,...,n-1,n}, where n is the original frame number.

[0015] Furthermore, the specific process of step 2 is: first, a time domain downsampling operation is performed at the encoding end of the time domain downsampling encoding module, and then the motion compensation interpolation technology is used at the decoding end to restore the discarded frames; the time domain downsampling process adopts uniform sampling, and the number of frames is 1 / k of the original number of frames, where k is the sampling factor; the downsampled video is represented as a set {1,2,...,n / k}.

[0016] Furthermore, in step 3, the process of extracting the facial skin region of interest is: using the MediaPipeFace Mesh framework to extract 468 facial reference points of the face, then finding the outermost reference points from the extracted 468 facial reference points as the facial contour, and finally subtracting the convex hull area related to the eyes and mouth from the contour including the entire face as the facial skin region of interest, and using the facial skin region of interest result as the facial interest mask.

[0017] Furthermore, in step 4, the core of the computing resource reallocation module is macroblock prediction mode adjustment based on the facial interest mask, and the macroblock prediction mode includes an intra-frame prediction mode and an inter-frame prediction mode;

[0018] The specific process of the intra prediction mode is as follows: first, a facial interest mask is received as control information to adjust the prediction mode of the intra compression module; for each macroblock, it is determined whether it is located in the region of interest. If so, all intra prediction modes are executed; if not, only the 16×16 intra block prediction mode is executed, and the 4×4 intra block prediction mode is no longer executed; finally, the costs of the prediction modes involved are analyzed, and the best intra prediction mode is selected by comparison;

[0019] The specific process of the inter-frame prediction mode is as follows: a facial interest mask is received as control information to adjust the prediction mode of the inter-frame compression module; for each macroblock, first determine whether it is located in the region of interest, if so, execute all inter-frame prediction modes; if not, only execute the Skip prediction mode and the 16×16 inter-frame block prediction mode, and no longer execute the remaining inter-frame block prediction modes; finally, compare and analyze the costs of the prediction modes involved and select the best inter-frame prediction mode.

[0020] Furthermore, in step 5, the bit resource reallocation module for remote photoplethysmography includes two aspects: macroblock quantization parameter reallocation based on facial region of interest and chrominance information focus retention strategy;

[0021] The quantization parameter of each macroblock is locally adjusted based on the macroblock quantization parameter reallocation of the facial region of interest, the mask of interest is input into the bitrate controller, and different quantization parameters are assigned to different macroblocks; the quantization parameter of the jth macroblock in the tth frame of the video is assigned by the following formula:

[0022] QP tj =QP t -3log2(ω j );

[0023] Among them, QP tj is the quantization parameter of the jth macroblock in the tth frame of the video; QP t is the global quantization parameter of the tth frame in the video; ω j It is a s-type function, and the formula is as follows:

[0024]

[0025] Among them, a, b, and c are different control parameters of the sigmoid function; represents the average intensity of interest in the tth frame; R j represents the average intensity of interest within the jth macroblock. If the macroblock is completely within the area of ​​interest, then R j A value of 1 indicates that the macroblock is not within the region of interest. j The value is 0;

[0026] The process of the chroma information focus retention strategy is: first convert the input video format to YUV444 format; then add the "chroma-qp-offset" parameter in the compression command line and set its value to a negative number; chroma-qp-offset is the chroma quantization parameter offset.

[0027] Furthermore, in step 6, the bidirectional prediction motion compensation interpolation technology obtains the inserted frame through the motion vector estimation value of the forward prediction frame and the backward prediction frame at the decoding end, which is obtained by the following formula:

[0028]

[0029] Among them, f t-1 (·) and f t (·) represents two adjacent sampling frames; f i (·) is the t-1 (·) and f t(·) The i-th frame reconstructed by interpolation between two sampling frames; (x, y) is the coordinate of the pixel point to be sought; x is the horizontal coordinate value, y is the vertical coordinate value; k is the sampling factor; (v x ,v y ) is used to reconstruct f t (·) motion vector estimate; v x is the component of the motion vector in the x direction; v y is the component of the motion vector in the y direction;

[0030] The reconstructed frames are temporally upsampled and encoded to obtain compressed reconstructed videos.

[0031] The beneficial technical effects brought about by the present invention are as follows: a time domain downsampling encoding module based on uniform sampling and a time domain upsampling encoding module based on motion compensation interpolation are constructed. Through time domain up and down sampling encoding, the amount of data to be encoded can be reduced, thereby achieving the purpose of reducing the encoding bit rate; a computing resource reallocation module based on facial regions of interest is constructed, which can preferentially allocate computing resources to facial skin areas containing physiological signals, thereby ensuring the video encoding quality of the facial regions of interest; a video encoding bit resource reallocation module based on facial regions of interest is constructed, which can allocate more bit resources to facial skin areas containing physiological signals, thereby achieving the retention of vital sign signals in the facial skin area while ensuring visual effects and low bit rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The flowchart of the facial video compression method for remote photoplethysmography in the present invention.

[0033] Figure 2 The figures are comparison diagrams of the remote photoplethysmography waveform, downsampling process and interpolation reconstruction results in the present invention; wherein (a) is the comparison result diagram when the sampling factor k is 2, (b) is the comparison result diagram when the sampling factor k is 5, and (c) is the comparison result diagram when the sampling factor k is 9.

[0034] Figure 3The figure is a result of comparing four facial region of interest extraction algorithms in the present invention; wherein, (a1) and (a2) are one of the benchmark images of the public datasets PFF dataset and COHFACE dataset, respectively, and the white dotted line is the benchmark value of the facial skin region of interest; (b1) and (b2) are comparison graphs of the extraction results of the YCbCr hue threshold skin color detection algorithm on the PFF dataset and the COHFACE dataset, respectively; (c1) and (c2) are comparison graphs of the extraction results of the BiSeNet face semantic segmentation algorithm on the PFF dataset and the COHFACE dataset, respectively; (d1) and (d2) are comparison graphs of the extraction results of the Viola-Jones face detection plus minimum eigenvalue corner detection algorithm on the PFF dataset and the COHFACE dataset, respectively; (e1) and (e2) are comparison graphs of the extraction results of the MediaPipe framework overlay convex hull algorithm on the PFF dataset and the COHFACE dataset, respectively.

[0035] Figure 4 This is a block diagram of a macroblock prediction model based on a region of interest according to the present invention.

[0036] Figure 5 This is a comparison result diagram of the running time before and after the execution of the computing resource reallocation strategy in the present invention and the quality signal-to-noise ratio of the obtained signal on four data sets: UBFC_rPPG, UBFC_PHYS, PFF, and PURE.

[0037] Figure 6 This is a comparison diagram before and after the macroblock quantization parameter reallocation strategy of the present invention is implemented.

[0038] Figure 7 This is a comparison chart of the results under different chroma QP offset parameters of the present invention.

[0039] Figure 8 This is a comparison chart of the results under different sampling factors in the time domain up and down sampling encoding in the present invention.

[0040] Fig. 9 This is the rPPG waveform extracted from the original video with a bit rate of 215775 kbps of a subject in the UBFC dataset of the present invention.

[0041] Fig.10 This is the rPPG waveform extracted from the compressed video with a compressed bit rate of 3500 kbps obtained by the video compression method proposed in the present invention.

[0042] Fig.11 This is the rPPG waveform extracted from the compressed video with a compressed bit rate of 3500 kbps obtained by using the traditional H.264 compression scheme in the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0044] like Figure 1 As shown, the present invention proposes a facial video compression method for remote photoplethysmography, comprising the following steps:

[0045] Step 1, obtain the uncompressed target video source. The video source acquisition device can be any device containing a camera, including but not limited to smartphones, tablets, laptops, monitoring equipment, VR equipment, sports cameras, drones, driving recorders, and special cameras. The human skin area needs to be covered during shooting, especially the face (forehead and cheeks), because these areas contain rich vascular tissue. Since the field of the present invention is a video compression method for retaining physiological information, in order to extract accurate vital signs signals, the subject should avoid large and high-speed head movements during shooting. The target video source obtained is defined as the set {1,2,...,n-1,n}, where n is the original frame number.

[0046] Step 2: construct a time-domain downsampling coding module based on uniform sampling, input the acquired target video source into the time-domain downsampling coding module, and obtain a downsampled video; the downsampled video is a set {1, 2, ..., n / k};

[0047] The human heart rate generally ranges from 50 to 120 beats per minute. According to the Nyquist sampling theorem, the video frame rate only needs to be between 1.7 and 4 to restore the correct heart rate value. Currently, the video frame rate is generally above 25 frames, so video compression for remote photoplethysmography can first perform time domain downsampling operations at the encoding end, and then use motion compensation interpolation technology at the decoding end to restore the discarded frames. Figure 2 The figure shows the rPPG waveform, downsampling process, and interpolation reconstruction results of the green color channel in a video in the UBFC dataset. Figure 2 As shown in (a), when the sampling factor k is 2, that is, one frame is taken every two frames, the waveform reconstructed by interpolation is almost completely consistent with the original waveform. In order to reduce the computational complexity, the time domain downsampling process adopts uniform sampling, and the number of frames is 1 / k of the original frame number n. In order to ensure the effective retention of physiological signals, k is generally 2-4. Figure 2 As shown in (b), k=5 is set. At this time, k is relatively large, and the restored waveform cannot be completely consistent with the original waveform. Figure 2 As shown in (c), k=9 is set. In this case, k is too large, so that the sampling frequency is lower than twice the highest frequency of the signal, resulting in the inability to restore the waveform of the correct frequency.

[0048] The time domain up-down sampling coding module can reduce the amount of data required for coding, thereby achieving the purpose of reducing the coding bit rate. It can also achieve the goal of allocating more coding bits to each frame when encoding under the same bit rate condition.

[0049] Step 3: construct a fast facial region of interest extraction module to extract the facial skin region of interest (ROI) from the downsampled video and obtain the ROI mask;

[0050] At present, there is no literature or invention in the field of remote vital sign signal detection that conducts a comprehensive and systematic comparison of different global facial skin area extraction algorithms. Therefore, the present invention will compare the advantages and disadvantages of the existing four types of global facial skin detection algorithms in this step to select the most suitable skin detection algorithm according to different rPPG video data sets. The present invention uses the public data sets PFF data set and COHFACE data set to conduct comparative experiments on various algorithms. These algorithms include YCbCr hue threshold skin color detection algorithm, BiSeNet face semantic segmentation algorithm, Viola-Jones face detection plus minimum eigenvalue corner detection algorithm and MediaPipe framework superposition convex hull algorithm. The results are shown in Figure 3 middle.

[0051] Figure 3 (a1) and (a2) are benchmark images taken from the public datasets PFF dataset and COHFACE dataset respectively. The white dashed lines in the figures indicate the benchmark values ​​of the facial skin area of ​​interest.

[0052] The YCbCr hue threshold skin color detection algorithm selects pixels within the skin color range by setting a fixed threshold, usually using the following value range: Where Y is the brightness component; Cb and Cr are the blue and red chrominance components respectively. The advantages of the YCbCr hue threshold skin color detection algorithm are low computational complexity and fast running speed. In addition, this method can also detect non-facial skin areas, such as Figure 3 (b2) shows the neck region detected on the COHFACE dataset. However, this method cannot guarantee the extraction of all skin regions, for example Figure 3 (b1) shows the skin region extraction result on the PFF dataset. The extracted skin region is incomplete, such as the position below the mouth indicated by the arrow. Especially in dim light, it is not guaranteed that all skin regions can be extracted. Figure 3 (b2) shows the skin area extraction result on the COHFACE dataset. It is obvious that the right side of the face area circled by the dotted circle is not extracted.

[0053] The BiSeNet-based face semantic segmentation algorithm calculates the semantics of the subject's face (e.g., mouth, eyes, eyebrows, etc.) through a bilateral segmentation network (BiSeNet), then generates pixel-by-pixel label mappings for different semantics, and finally retains only the semantics related to the facial skin area. The BiSeNet network can achieve real-time performance and high accuracy. In addition, this method can also segment semantics other than facial features, such as Figure 3 (c1) shows that the semantics of glasses can be extracted on the PFF dataset, which is difficult to achieve in the following two methods; however, when the resolution of the input video is low or the light is dim, accurate results cannot be extracted, for example Figure 3 (c2) shows the skin area extraction results on the COHFACE dataset. The central and right areas of the face circled by dotted circles are not extracted.

[0054] The Viola Jones face detection algorithm is a face detection algorithm based on Haar features and AdaBoost classifier, with high accuracy and real-time performance. This algorithm can be combined with the minimum eigenvalue corner detection algorithm to determine the facial contour and eye position, and then determine the position of the eyes, mouth and background based on the facial proportions, and then remove these areas. The disadvantage of this method is that each person's face shape is different, which will lead to inaccurate mouth and eye positions based on proportions, especially when the face is not facing the camera in front, for example Figure 3 (d1) shows the extraction result on the PFF dataset. The mouth area pointed by the arrow is not completely removed. In addition, this method easily removes redundant skin areas, such as Figure 3 (d1) and Figure 3 The position circled by the dotted line in (d2) will lead to inaccurate subsequent compression.

[0055] The MediaPipe Face Mesh framework uses a lightweight model architecture and GPU acceleration to provide fast detection and real-time tracking of 468 facial reference points. The present invention uses the convex hull of the extracted 468 facial reference points as the skin area, and then subtracts the convex hull area related to the eyes and mouth from the convex hull including the entire face. The result is as follows Figure 3 As shown in (e1) and (e2). Whether it is a video with lower resolution or a video with lower light, the MediaPipe solution can provide accurate results. In addition, the motion vectors obtained by this scheme can also be shared with the encoder to reduce the computational cost and complexity of the algorithm. Despite the high computational complexity, the scheme can always produce robust global face extraction results. Therefore, the present invention adopts the MediaPipe framework as a facial skin region of interest extraction scheme, and the extracted facial skin ROI result is used as a facial ROI mask, which is input into subsequent steps to reallocate computing resources and bit resources.

[0056] Step 4: construct a computing resource reallocation module based on the facial region of interest, and input the ROI mask as control information into the computing resource reallocation module to adjust the selection of macroblock intra-frame and inter-frame prediction modes in predictive coding to achieve reallocation of computing resources;

[0057] The high computational complexity of the traditional hybrid coding standard makes it face huge challenges in real-time video transmission and when the coding computing resources are limited. If the overall prediction mode is simplified, the coding quality of the video frame will be affected. In the case of limited computing resources, the computing resources are preferentially allocated to the facial skin area containing physiological signals, so that the video coding quality of the facial area of ​​interest can be guaranteed. The present invention inputs the extracted facial interest mask as control information into the coding framework for adjusting the macroblock intra-frame and inter-frame prediction mode selection in predictive coding. Taking advantage of the different levels of interest in different areas in the video frame sequence, the limited computing resources are preferentially allocated to the facial area of ​​interest, reducing the computational complexity of the non-interested area, thereby ensuring the video coding quality of the facial area of ​​interest under limited computing resources, and further ensuring the retention of the rPPG signal.

[0058] In non-contact physiological signal detection videos, the non-interesting parts of each frame of the image are mostly static backgrounds, and the degree of change between video frames is very small. Executing all prediction model traversal algorithms in these areas will seriously waste encoding time and computing resources. Taking intra-frame prediction mode selection as an example, in order to obtain the best prediction mode, a 16×16 macroblock traverses all intra-frame prediction modes, and a total of (16*9+4)*4=592 rate-distortion optimization calculations are required. The complexity of inter-frame prediction is even higher because it needs to process algorithms with high computational complexity, such as motion estimation and compensation of multiple reference frames and pixel-by-pixel interpolation. Based on the above analysis, the present invention constructs a computational resource reallocation scheme based on facial regions of interest to reduce the computational complexity in macroblock prediction and allocate more resources to facial regions of interest.

[0059] The core of the computational resource reallocation module based on facial region of interest is the adjustment of macroblock prediction mode based on ROI mask. The macroblock prediction mode includes intra-frame prediction mode and inter-frame prediction mode. Figure 4As shown. For I-frame macroblock prediction, that is, intra-frame prediction mode, the process is as follows: first, a facial ROI mask is received as control information to adjust the prediction mode of the intra-frame compression module. Specifically, for each macroblock, first determine whether it is located in the ROI. If so, all intra-frame prediction modes are executed; if not, only the 16×16 intra-frame block prediction mode is executed, and the 4×4 intra-frame block prediction mode is no longer executed. Finally, the cost of the prediction modes involved is analyzed, and the best intra-frame prediction mode is selected by comparison. The frame types of inter-frame prediction mode are mostly P frames or B frames. For P frame or B frame macroblock prediction, that is, the specific process of inter-frame prediction mode is: first, a facial ROI mask is received as control information to adjust the prediction mode of the inter-frame compression module. Specifically, for each macroblock, first determine whether it is located in the ROI. If so, all inter-frame prediction modes are executed; if not, only the Skip prediction mode and the 16×16 inter-frame block prediction mode are executed, and the remaining inter-frame block prediction modes are no longer executed. Taking the X264 encoder as an example, the X264_ANALYSE_I4x4, X264_ANALYSE_I8x8, X264_ANALYSE_BSUB16x16, X264_ANALYSE_PSUB16x16 and other flags are defined as 0 in the program. Finally, the costs of the prediction modes involved are compared and analyzed to select the best inter-frame prediction mode.

[0060] UBFC_rPPG, UBFC_PHYS, PFF, and PURE are four public datasets. Figure 5 The results of the run time and signal-to-noise ratio (SNR) of the four datasets UBFC_rPPG, UBFC_PHYS (first 1500 frames), PFF (first 1000 frames), and PURE before and after the implementation of the computing resource reallocation strategy are shown. Figure 5 It can be seen that after adopting this strategy, the running time is reduced by 83.81% on average, while the quality of the resulting signal, that is, the signal-to-noise ratio, has almost no degradation.

[0061] Step 5: construct a bit resource reallocation module for remote photoplethysmography, input the ROI mask as control information into the encoding module, and perform bit resource reallocation, which specifically includes two aspects: macroblock quantization parameter (QP) reallocation based on the facial region of interest and chrominance information focus retention strategy, and finally obtain the bit stream after quantization parameter reallocation for subsequent bit stream transmission or storage;

[0062] Step 5.1, reallocation of macroblock quantization parameters based on facial regions of interest;

[0063] Macroblock quantization parameter reallocation is to indirectly control the number of coded bits generated by residual data and motion vectors by controlling the quantization parameters during video encoding. The purpose is to make the region of interest have more bit resources than the non-region of interest, so as to retain physiological signals as much as possible, while simplifying irrelevant information (background) to keep the frame-level bit rate constant. This process requires local adjustment of the quantization parameters of each macroblock. The mask of interest is input into the bitrate controller, and different quantization parameters are assigned to different macroblocks. Specifically, the quantization parameter of the jth macroblock in the tth frame of the video is assigned by the following formula:

[0064] QP tj =QP t -3log2(ω j );

[0065] Among them, QP tj is the quantization parameter of the jth macroblock in the tth frame of the video; QP t is the global quantization parameter of the tth frame in the video, obtained by the bit rate control algorithm in the original hybrid coding framework. j is a sigmoid function:

[0066]

[0067] Wherein, a, b, and c are different control parameters of the sigmoid function; in the present invention, a=0.5, b=1.8, and c=5 are set. j represents the average intensity of interest within the jth macroblock. If the macroblock is completely within the area of ​​interest, then R j A value of 1 indicates that the macroblock is not within the region of interest. j The value is 0. It represents the average intensity of the entire image of interest. Its value is determined by the area of ​​interest that occupies the entire image size and is generally in the range of 0.1-0.5.

[0068] The bit resource reallocation module for remote photoplethysmography can control the quantization parameters of different macroblock areas through the above formula, and allocate more bit resources to the facial skin area containing physiological signals, thereby retaining the vital sign signals of the facial skin area while ensuring visual effects and low bit rate.

[0069] Step 5.2: Chroma information focus retention strategy;

[0070] Since the human eye is more likely to perceive brightness information than color information, the traditional hybrid video coding framework will compress the color information and focus on retaining the brightness information. However, vital signs information containing subtle color changes is more contained in these color information, especially the green channel contains the strongest light volume pulse signal. Therefore, rPPG-oriented video compression needs to adjust the compression strategy to allocate more bit resources to the color channel rather than the brightness channel.

[0071] In traditional hybrid video coding schemes, such as H.264, the first step is to convert RGB video into YUV420 format, that is, for every 4 brightness channel values ​​(Y), a pair of chrominance values ​​(UV) is stored. This greatly improves the compression rate without losing too much visual quality. However, this also removes a large amount of vital sign information stored in tiny color changes. Therefore, rPPG-oriented video compression must first adjust the input video format to YUV444 format. Secondly, the focus of chrominance information can be achieved by adding a negative offset to the quantization value of the chrominance space. The strategy of the present invention is to add the "chroma-qp-offset" parameter in the compression command line and set its value to a negative number. Chroma-qp-offset is the chroma quantization parameter offset (abbreviated as chroma QP offset)

[0072] Figure 6 and Figure 7 The result diagram after the execution of the bit resource reallocation strategy for rPPG is shown; Figure 6 This is a comparison diagram before and after the implementation of the macroblock quantization parameter reallocation strategy of the present invention. Figure 7 This is a comparison chart of the results under different chroma QP offset parameters of the present invention. Figure 6 It can be seen that after adopting the macroblock quantization parameter reallocation strategy, only 2500kbps bitrate is needed to achieve the signal quality (indicated by the signal-to-noise ratio SNR value) that originally required 27000kbps bitrate. On the other hand, when the bitrate drops to around 100kbps, it does not produce the poor results that would be obtained if this strategy was not used. Therefore, the macroblock QPs reallocation in this step is a key technology for rPPG-oriented video compression. Figure 7 It can be seen that at the same bit rate, the lower the "chroma QP offset" parameter, the better the quality obtained; or more precisely, in order to achieve the same quality, a value with a lower chroma offset requires a lower bit rate. But overall, the improvement is not as significant as step 4.1. In particular, when the bit rate exceeds 1500kbps, the results produced by the three are almost the same, which also shows that for rPPG videos, a bit rate of 1500kbps is sufficient to produce good quality results.

[0073] Step 6: The above steps finally obtain the reallocated bit stream, which is then decoded by the traditional hybrid coding and decoding framework to obtain the estimated motion vector value (v x ,v y ), and then used for bidirectional prediction interpolation, and temporal upsampling encoding to obtain compressed and reconstructed video;

[0074] The bidirectional prediction motion compensation interpolation technology obtains the interpolation frame by the motion vector value of the forward prediction frame and the backward prediction frame at the decoding end, which is obtained by the following formula:

[0075]

[0076] Among them, f t-1 (·) and f t (·) represents two adjacent sampling frames; f i (·) is the t-1 (·) and f t (·) The i-th frame reconstructed by interpolation between two sampling frames; (x, y) is the coordinate of the pixel point to be sought; x is the horizontal coordinate value, y is the vertical coordinate value; k is the sampling factor; (v x ,v y ) is used to reconstruct f t (·) motion vector estimation value obtained during the encoding process of inter-frame prediction mode; v x is the component of the motion vector in the x direction; v y is the component of the motion vector in the y direction; the reconstructed frame is temporally upsampled and encoded to obtain a compressed and reconstructed video.

[0077] Figure 8 The following is a comparison of the results of different sampling factors in temporal up-down sampling encoding. It can be seen from the figure that when the sampling factor k is set to 2 or 4, higher quality results can be obtained compared with the video without up-down sampling encoding at the same bit rate. From another perspective, in order to achieve the same quality, the video with up-down sampling encoding requires less bit rate. However, when the sampling factor reaches a certain value, such as Figure 8 When the sampling factor k is 6, the result is actually not as good as the unsampled video result. Figure 2 This can be explained by the fact that when the sampling frequency is lower than twice the highest frequency of the signal, the original waveform cannot be reconstructed. In practice, due to the presence of various noises, the sampling frequency needs to be higher, that is, the sampling factor value cannot be too large.

[0078] Figure 9-11 The figure shows the final result achieved by the framework proposed by the present invention. Fig. 9 The rPPG waveform extracted from the original video of a subject in the UBFC dataset with a size of 215775kbps is shown; Fig.10 The figure shows the rPPG waveform extracted from the compressed video with a size of 3500kbps obtained by the video compression method proposed by the present invention. Fig. 9 and Fig.10 It can be seen that the framework proposed by the present invention can compress the original video by more than 60 times without affecting the rPPG signal contained in the video, that is, the physiological parameter information. Fig.11 It shows that after the video is compressed to a bit rate of 3500kbps using the traditional H.264 compression scheme, the extracted rPPG waveform contains a large amount of high-frequency noise, which destroys the physiological parameter information it contains.

[0079] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A facial video compression method for remote photoplethysmography, characterized in that: The steps include: Step 1: Obtain an uncompressed target video source through a capture device; Step 2: construct a time-domain downsampling encoding module based on uniform sampling, input the acquired target video source into the time-domain downsampling encoding module, and obtain a downsampled video; Step 3: construct a fast facial region of interest extraction module to extract the facial skin region of interest from the downsampled video to obtain a facial mask of interest; Step 4: construct a computing resource reallocation module based on the facial region of interest to reallocate computing resources to the facial mask of interest; Step 5: construct a bit resource reallocation module for remote photoplethysmography, reallocate the bit resources for the mask of interest, and obtain a reallocated bit stream; Step 6: Use bidirectional prediction motion compensation interpolation technology to perform bidirectional prediction interpolation on the reallocated bit stream, and finally perform time domain upsampling encoding to obtain a compressed and reconstructed video.

2. The facial video compression method for remote photoplethysmography according to claim 1, characterized in that: In step 1, the acquisition device is any device including a camera, and the shooting needs to cover the human skin area, including the forehead and cheeks of the face; the acquired target video source is represented as a set {1,2,...,n-1,n}, where n is the original frame number.

3. The facial video compression method for remote photoplethysmography according to claim 1, characterized in that: The specific process of step 2 is: first, perform a time domain downsampling operation at the encoding end of the time domain downsampling encoding module, and then use the motion compensation interpolation technology at the decoding end to restore the discarded frames; the time domain downsampling process adopts uniform sampling, the number of frames is 1 / k of the original number of frames, and k is the sampling factor; the downsampled video is represented as a set {1,2,...,n / k}.

4. The facial video compression method for remote photoplethysmography according to claim 1, characterized in that: In step 3, the facial skin region of interest extraction process is: using the MediaPipe Face Mesh framework, extracting 468 facial reference points of the face, then finding the outermost reference points from the extracted 468 facial reference points as the facial contour, and finally subtracting the convex hull area related to the eyes and mouth from the contour including the entire face as the facial skin region of interest, and using the facial skin region of interest result as the facial interest mask.

5. The facial video compression method for remote photoplethysmography according to claim 1, characterized in that: In step 4, the core of the computing resource reallocation module is macroblock prediction mode adjustment based on the facial mask of interest, and the macroblock prediction mode includes an intra-frame prediction mode and an inter-frame prediction mode; The specific process of the intra prediction mode is as follows: first, a facial interest mask is received as control information to adjust the prediction mode of the intra compression module; for each macroblock, it is determined whether it is located in the region of interest. If so, all intra prediction modes are executed; if not, only the 16×16 intra block prediction mode is executed, and the 4×4 intra block prediction mode is no longer executed; finally, the costs of the prediction modes involved are analyzed, and the best intra prediction mode is selected by comparison; The specific process of the inter-frame prediction mode is as follows: receiving a facial interest mask as control information to adjust the prediction mode of the inter-frame compression module; for each macroblock, first determine whether it is located in the region of interest, and if so, execute all inter-frame prediction modes; If not, only the Skip prediction mode and the 16×16 inter-frame block prediction mode are executed, and the other inter-frame block prediction modes are no longer executed; finally, the costs of the prediction modes involved are compared and analyzed to select the best inter-frame prediction mode.

6. The facial video compression method for remote photoplethysmography according to claim 1, characterized in that: In step 5, the bit resource reallocation module for remote photoplethysmography includes two aspects: macroblock quantization parameter reallocation based on facial region of interest and chrominance information focus retention strategy; The quantization parameter of each macroblock is locally adjusted based on the macroblock quantization parameter reallocation of the facial region of interest, the mask of interest is input into the bitrate controller, and different quantization parameters are assigned to different macroblocks; the quantization parameter of the jth macroblock in the tth frame of the video is assigned by the following formula: QP tj =QP t -31og2(ω j ); Among them, QP tj is the quantization parameter of the jth macroblock in the tth frame of the video; QP t is the global quantization parameter of the tth frame in the video; ω j It is a s-type function, and the formula is as follows: Among them, a, b, and c are different control parameters of the sigmoid function; represents the average intensity of interest in the tth frame; R j represents the average intensity of interest within the jth macroblock. If the macroblock is completely within the area of ​​interest, then R j A value of 1 indicates that the macroblock is not within the region of interest. j The value is 0; The process of the chroma information retention strategy is as follows: first convert the input video format to YUV444 format; then add the "chroma-qp-offset" parameter in the compression command line and set its value to a negative number; chroma-qp-offset is the chroma quantization parameter offset.

7. The facial video compression method for remote photoplethysmography according to claim 1, characterized in that: In step 6, the bidirectional prediction motion compensation interpolation technology obtains the inserted frame through the motion vector estimation value of the forward prediction frame and the backward prediction frame at the decoding end, which is obtained by the following formula: Among them, f t-1 (·) and f t (·) represents two adjacent sampling frames; f i (·) is the t-1 (·) and f t (·) The i-th frame reconstructed by interpolation between two sampling frames; (x, y) is the coordinate of the pixel point to be sought; x is the horizontal coordinate value, y is the vertical coordinate value; k is the sampling factor; (v x ,v y ) is used to reconstruct f t (·) motion vector estimation value; v x is the component of the motion vector in the x direction; v y is the component of the motion vector in the y direction; The reconstructed frames are temporally upsampled and encoded to obtain compressed reconstructed videos.

Images