A non-contact health monitoring method with privacy protection characteristics

Abstract

The present invention discloses a non-contact health monitoring method with privacy protection characteristics, which belongs to the field of computer image video processing and health monitoring, and uses remote photoplethysmography rPPG to realize non-contact remote health monitoring; specifically includes the following steps: step 1, obtaining a target video source; step 2, extracting a facial region of interest using a face detection and positioning algorithm; step 3, rearranging pixels within the facial region of interest; step 4, encoding and decoding the video based on a hybrid video coding standard to obtain a compressed video; step 5, extracting an rPPG signal from the compressed video using an rPPG extraction algorithm; step 6, calculating a physiological sign signal based on the rPPG signal. The present invention can erase low-level visual feature information to avoid the leakage of facial information, and can retain high-level physiological feature information to achieve accurate acquisition of vital sign signals.

Description

Technical Field

[0001] The present invention belongs to the field of computer image and video processing and health monitoring, and in particular relates to a non-contact health monitoring method with privacy protection characteristics. Background Art

[0002] In traditional medical devices used for health monitoring, the detection of vital signs signals is mostly contact-based, such as electrocardiogram (ECG) acquisition equipment and photoplethysmography (PPG) equipment. These devices can detect a variety of physiological parameters such as heart rate, respiratory rate, heart rate variability, blood oxygen saturation and blood pressure, which can help diagnose chronic diseases such as cardiovascular disease, respiratory diseases, arrhythmias and abnormal blood pressure. However, these contact-based physiological measurement devices have many limitations. This type of equipment will reduce the range of motion of the subject, causing skin discomfort and even skin damage, especially for premature infants who are more sensitive to the skin. In addition, contact-based devices will introduce measurement deviations due to the influence of motion artifacts, which is not conducive to long-term monitoring and remote monitoring.

[0003] Remote photoplethysmography (rPPG) can effectively avoid the above problems and realize non-contact remote health monitoring. rPPG technology uses a camera to obtain human body (mainly facial) video, and analyzes the subtle color changes reflected by the skin surface in the video to measure physiological signals such as heart rate, respiratory rate, blood oxygen saturation and blood pressure, and then conduct cardiopulmonary assessment and disease diagnosis, which has received widespread attention from many scholars. Compared with PPG, rPPG measurement has low cost, high comfort, wide application range, and is easier to promote and popularize. In recent years, research on rPPG has also grown exponentially. However, the acquisition of rPPG requires clear facial videos of the subjects, so it cannot meet the requirements of visual privacy protection well, because if this information is stolen and abused by others, it may seriously threaten personal privacy. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a non-contact health monitoring method with privacy protection characteristics, which can erase low-level visual feature information to avoid the leakage of facial information, and retain high-level physiological feature information to achieve accurate acquisition of vital sign signals.

[0005] The technical solution of the present invention is as follows:

[0006] A non-contact health monitoring method with privacy protection characteristics, using remote photoplethysmography (rPPG) to achieve non-contact remote health monitoring; specifically comprising the following steps:

[0007] Step 1: Get the target video source;

[0008] Step 2: Use face detection and positioning algorithm to extract facial regions of interest;

[0009] Step 3, rearrange the pixels within the facial region of interest;

[0010] Step 4: Encode and decode the video based on the hybrid video coding standard to obtain a compressed video;

[0011] Step 5: extracting rPPG signals from the compressed video using an rPPG extraction algorithm;

[0012] Step 6: Calculate the physiological sign signal based on the rPPG signal.

[0013] Furthermore, in step 1, the video source acquisition device is any device containing a camera, including but not limited to smartphones, tablet computers, laptops, monitoring equipment, VR equipment, sports cameras, drones, driving recorders, and special cameras; the face area needs to be covered during shooting.

[0014] Furthermore, in the step 2, the face detection and positioning algorithm specifically adopts the Haar cascade face detection algorithm, and the specific process is: input an image containing a face, and if the input is a color image, it is first converted into a grayscale image; the image is scaled; in the input image, a fixed-size sliding window is used to slide from left to right and from top to bottom to detect each position in turn; for each window position, the corresponding Haar feature is calculated and input into the adaptive enhancement classifier for classification; if the classifier determines that the current window is a face, the window is marked as a face area; otherwise, continue to detect the next position; finally output the position information of the detected face area, and the position information is represented in the form of a rectangular frame; when multiple rectangular frames are detected, the rectangular frames are merged, the overlapping rectangular frames are removed, and the remaining rectangular frames constitute a complete face area, that is, the facial area of ​​interest; adjust the coordinate position of the upper left corner of each rectangular frame so that the horizontal and vertical coordinates of the starting position point are both multiples of 16; at the same time, adjust the width and height of each rectangular frame so that both are multiples of 16.

[0015] Furthermore, in step 3, three rearrangement rules are designed, namely:

[0016] Rule 1: First, convert the facial region of interest obtained in step 2 to the YCbCr color space, and then arrange all pixel values ​​in the region of interest from large to small according to the Y channel, and then arrange the same values ​​of the Y channel from large to small according to the Cb channel, and then arrange the same values ​​of the Y and Cb channels from large to small according to the Cr channel value; after arrangement, an n×3 two-dimensional matrix is ​​obtained, where n is the number of pixels in the region of interest and 3 is the three color channels; the n×3 two-dimensional matrix is ​​rearranged in a Z shape to obtain an h×w×3 three-dimensional matrix, where h and w are the height and width of the region of interest respectively;

[0017] Rule 2: Change the Z-shaped arrangement according to a single pixel in Rule 1 to a Z-shaped arrangement according to a single macroblock; the specific process is: first, convert the facial region of interest obtained in step 2 to the YCbCr color space, and then arrange all pixel values ​​in the region of interest from large to small according to the Y channel, and then arrange the same values ​​of the Y channel from large to small according to the Cb channel, and then arrange the same values ​​of the Y and Cb channels from large to small according to the Cr channel value; after the arrangement, an n×3 two-dimensional matrix is ​​obtained, and the n×3 two-dimensional matrix is ​​a row of pixels; define the macroblock size as 16×16 pixels, and take 16 pixels from the n×3 two-dimensional matrix as a row of the macroblock until all 16 rows are taken to obtain the first macroblock; the remaining macroblocks are also taken according to this rule, and finally all single macroblocks are arranged in a Z shape;

[0018] Rule 3: Change the arrangement of macroblocks in Rule 2 from zigzag to top to bottom and left to right. The specific process is: first, convert the facial region of interest obtained in step 2 to the YCbCr color space, and then arrange all pixel values ​​in the region of interest from large to small according to the Y channel, and then arrange the same values ​​of the Y channel from large to small according to the Cb channel, and then arrange the same values ​​of the Y and Cb channels from large to small according to the Cr channel value; after arrangement, an n×3 two-dimensional matrix is ​​obtained, and the n×3 two-dimensional matrix is ​​a row of pixels; take 16 pixels as a group every time, until h groups are taken, and the h groups correspond to the height of the region of interest, and take each group as each column of the final result, a total of w / 16 columns;

[0019] When applying the rearrangement rule to the online video compression of the current video, three rearrangement rules are tested in advance using historical data, and the one with the best video compression result is selected as the rearrangement rule for the current video.

[0020] Furthermore, in step 4, the H.264 video coding standard is used to encode and decode the video, and the specific process is: first, a group of uncompressed video frames are input. If the video is in RGB format, it needs to be converted into YUV format; then, the encoder of the encoding module performs video segmentation, intra-frame prediction, inter-frame prediction, transformation and quantization, entropy coding, and stream reorganization, and finally outputs the encoded video code stream for transmission or storage; during the decoding process of the decoding module, the decoder will perform code stream parsing, entropy decoding, inverse quantization and inverse transformation, loop filtering, and frame-by-frame reconstruction, and finally output the reconstructed video, that is, the compressed video.

[0021] Furthermore, in step 5, a plane orthogonal skin algorithm is specifically used to extract rPPG signals. The specific process is: input the compressed video, construct the time series signal of the input video on the three RGB channels, and then perform plane orthogonal calculation, that is, construct a plane orthogonal to the time-normalized skin color, and then project the pixel points in the skin area onto the constructed plane to obtain the projected pixel values, which are output as a one-dimensional sequence of rPPG signals.

[0022] Furthermore, in step 6, the physiological sign signals include heart rate, respiratory rate, blood oxygen saturation and blood pressure;

[0023] The method for obtaining the heart rate from the waveform of the rPPG signal adopts a time domain-based method or a frequency domain-based method. The time domain-based method uses the peak-to-peak value or average value of the heart beat interval to calculate; the frequency domain-based method uses the maximum power spectrum density within the frequency range and then multiplies it by 60 to obtain the heart rate value;

[0024] The method for extracting the respiratory rate from the rPPG signal adopts the integrated empirical mode decomposition method, which is specifically implemented as follows: the waveform of the rPPG signal is input, and the preprocessing stage includes filtering and denoising the rPPG signal; then the integrated empirical mode decomposition method is applied to decompose the signal into several intrinsic mode components; according to the approximate range of the respiratory frequency, the components or modes related to breathing are identified; finally, the respiratory rate is calculated using the peak-to-peak value in the time domain method or the maximum power spectral density in the frequency domain method;

[0025] The ratio model is used to calculate the blood oxygen saturation. The calculation process is as follows: the ratio in the ratio model represents the relationship between the AC component and the DC component of the pulse wave signal collected at two different wavelengths; the correction coefficient is determined by linear regression analysis, and a mapping function between the ratio and the blood oxygen saturation is established, thereby achieving a high-precision estimation of the blood oxygen saturation;

[0026] The specific process of extracting blood pressure from the waveform morphological features of the rPPG signal is as follows: first, measure different parameters of the waveform, including the peaks and troughs, rise time, fall time, peak width, and trough width in the rPPG signal; then analyze the curvature changes of the waveform to extract information about vascular elasticity and blood flow characteristics; based on the preliminary analysis, select the most meaningful waveform features for blood pressure estimation; finally, use a machine learning algorithm to build a blood pressure estimation model, and the training data contains known blood pressure values ​​and the corresponding waveform features of the rPPG signal.

[0027] Beneficial technical effects brought by the present invention: The present invention has important needs in many places, such as nursing homes or hospitals, which require daily monitoring of the vital signs of the subjects. However, in many cases, the subjects do not want to expose personal information, especially sensitive facial video information. Therefore, the present invention has important practical needs. On the other hand, with the development of modern shooting technology, the resolution of the acquired video is increasing. Uncompressed videos will take up a lot of hard disk space and are not conducive to the effective transmission of videos. Therefore, video compression is an indispensable part of many video processing applications. The face privacy protection scheme proposed in the present invention can further promote the effective compression of videos, which is beneficial to the storage and transmission of large-capacity videos, and is expected to promote the generation of large-scale data sets and the development of telemedicine. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a flow chart of a non-contact health monitoring method with privacy protection characteristics.

[0029] Figure 2 The figures are comparison diagrams of the results of three rules for rearranging pixels in the region of interest of the present invention; wherein (a), (b), and (c) are the result diagrams of the rearrangement of pixels in the region of interest of rule one, rule two, and rule three, respectively.

[0030] Figure 3 This is a diagram showing the specific implementation process of Rule 2 for rearranging pixels within the region of interest in the present invention.

[0031] Figure 4 This is a diagram showing a specific implementation process of Rule 3 for rearranging pixels within a region of interest in the present invention.

[0032] Figure 5 This is the result of converting and transforming any macroblock from the original image according to the present invention.

[0033] Figure 6 This is the result of converting and transforming any macroblock from the rearranged image according to the present invention.

[0034] Figure 7 It is a Pearson coefficient curve diagram in the present invention.

[0035] Figure 8 It is a signal-to-noise ratio curve diagram in the present invention.

[0036] Fig. 9 This is a visualization result diagram of face recognition of 10 subjects before pixel rearrangement in the present invention.

[0037] Fig.10 This is a visualization result diagram of face recognition after pixel rearrangement for 10 subjects in the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0039] like Figure 1 As shown, a non-contact health monitoring method with privacy protection characteristics includes the following steps:

[0040] Step 1, obtain the target video source. The video source acquisition device can be any device containing a camera, including but not limited to smart phones, tablet computers, laptops, monitoring equipment, VR equipment, sports cameras, drones, driving recorders, and special cameras. The face area needs to be covered during shooting, because this area contains richer vascular tissue than other skin areas of the human body, which can effectively realize the extraction of vital sign signals. Since the present invention is aimed at the field of non-contact health monitoring, in order to extract accurate vital sign signals, the subject should avoid large and high-speed head movements during shooting.

[0041] Step 2: Extraction of facial region of interest (ROI) can be any face detection and positioning algorithm, including but not limited to feature-based methods, such as Haar cascade face detection algorithm; model-based methods, such as adaptive enhancement (AdaBoost) algorithm; deep learning-based methods, such as MTCNN, CenterFace, YOLO, SSD, etc. The following takes the classic Haar cascade face detection algorithm as an example to introduce the specific process of face detection.

[0042] The input is an image containing a face, which can be a color image or a grayscale image. The image preprocessing stage includes image grayscale (if the input is a color image, it is first converted to a grayscale image) and resizing (to speed up the detection, the image is usually scaled). The specific detection process is as follows: 1) In the input image, use a fixed-size sliding window (usually the average size of a face) to slide from left to right and from top to bottom to detect each position in turn; 2) For each window position, calculate the corresponding Haar feature and input it into the adaptive boost (AdaBoost) classifier for classification; 3) If the classifier determines that the current window is a face, mark the window as a face area; otherwise, continue to detect the next position. The final output is the location information of the detected face area, which is usually expressed in the form of a rectangular box, containing four coordinates (upper left corner and lower right corner coordinates) or the center point coordinates plus width and height. There may be more than one rectangular box detected above. In this case, it is also necessary to merge the rectangular boxes at different scales, remove the overlapping rectangular boxes, and the remaining rectangular boxes constitute a complete face area, that is, the facial area of ​​interest.

[0043] After the face area is detected, in order to facilitate the rearrangement of the macroblocks in 16×16 size in step 3, the size and position of each detected rectangular frame need to be adjusted. That is, the coordinate position of the upper left corner of each rectangular frame is adjusted so that the horizontal and vertical coordinates of the starting point are multiples of 16; secondly, the width and height of each rectangular frame are adjusted so that both are multiples of 16.

[0044] Step 3, pixel rearrangement within the facial region of interest is a key step in privacy protection in the present invention. Previous means of facial privacy protection include facial blurring, pixelation, and various conversions and deformations. Although these methods can protect faces from being recognized, they also erase the physiological information contained in the face video. The present invention adopts a solution of pixel rearrangement within the facial region of interest, and does not modify each pixel value, but only adjusts the position of the pixel. This does not affect the rPPG signal obtained from the video frame, because most rPPG extraction algorithms extract the pixel average value of each frame from each frame of the image as input to calculate the rPPG signal. In order to ensure privacy, the image pixels can be randomly shuffled here, but in order to promote the compression of subsequent videos, the present invention further arranges the pixels from large to small according to the pixel value, and rearranges them according to the rearrangement rules. The present invention provides and tests three rearrangement rules to find an arrangement method that is more conducive to video compression, as follows:

[0045] Rule 1: First, convert the facial region of interest obtained in step 2 to the YCbCr color space, then arrange all pixel values ​​in the region of interest from large to small according to the Y channel, and then arrange the same values ​​in the Y channel from large to small according to the Cb channel, and then arrange the same values ​​in both the Y and Cb channels from large to small according to the Cr channel value; after the above arrangement, an n×3 two-dimensional matrix is ​​obtained, where n is the number of pixels in the region of interest and 3 is the three color channels; finally, rearrange the above arranged pixels in a Z shape to obtain an h×w×3 three-dimensional matrix, where h and w are the height and width of the region of interest obtained in step 2, respectively. The result is as follows Figure 2 As shown in (a).

[0046] Rule 2 is similar to Rule 1, except that the zigzag arrangement according to a single pixel is changed to a zigzag arrangement according to a single "macroblock". In the present invention, the macroblock size is defined as 16×16 pixels (because the macroblock size in video compression is generally defined as 16×16). The specific process is as follows: Figure 3 As shown, according to rule 1, we first get an n×3 two-dimensional matrix (that is, a row of pixels), and then take 16 pixels as a row of macroblocks, until all 16 rows are taken, and the first macroblock is obtained. The remaining macroblocks are also taken according to this rule, and finally the macroblocks are arranged in a Z shape, and the result is as follows Figure 2 As shown in (b).

[0047] Rule 3 is similar to Rule 2, except that the arrangement of macroblocks is changed from zigzag to top to bottom and from left to right. The specific process is as follows: Figure 4 As shown, according to rule 1, we first get an n×3 two-dimensional matrix (that is, a row of pixels), then take 16 pixels as a group, until h groups are taken (h is the height of the region of interest obtained in step 2), and take each group as each column of the final result, a total of w / 16 columns (w is the width of the region of interest obtained in step 2), and the final result is as follows Figure 2 As shown in (c).

[0048] When applying the rearrangement rule to the online video compression of the current video, three rearrangement rules are tested simultaneously using historical data in advance, and the one with the best video compression result is selected as the rearrangement rule for the current video.

[0049] The principle of the present invention using the above rules is to try to make the pixel values ​​between macroblocks have a consistent distribution, preferably making all pixel values ​​in a macroblock exactly the same, so that the entropy of the image can be reduced as much as possible during video compression and information redundancy can be removed. Figure 5 and Figure 6 An example is used to explain why pixel rearrangement within a region of interest is beneficial to video compression. Figure 5First, we randomly select a macroblock from the original image (here we take 8×8 as an example), convert it to the YCbCr color space, and then perform a discrete cosine transform (DCT) operation. To clearly show the result after DCT, only the specific values ​​of the Y channel are shown here. Figure 6 Shown are the results obtained from the rearranged image. Figure 5 and Figure 6 It can be seen that when the pixel values ​​in a macroblock are completely consistent, after DCT, the "energy" of the resulting matrix is ​​all concentrated on the DC component in the upper left corner, and the other positions become 0. This can greatly reduce information entropy and greatly reduce the storage space required after compression. Figure 7 and Figure 8 The superiority and feasibility of ROI rearrangement for video compression are demonstrated, and the video compression results under three different arrangement rules are shown respectively, which will be explained in detail after all the steps are introduced.

[0050] In addition, in order to show that the rearranged results can effectively resist face recognition and achieve privacy protection at the machine vision level, the present invention also performs face recognition on the 10 subject videos before and after the rearrangement. The results are shown in Fig. 9 Specifically, first, 100 frames of images are randomly selected from each subject's video, and each image is preprocessed by normalization, cropping, scaling, etc. to ensure that the images input to the network have consistent size and format; then, for each input image, a 128-dimensional feature vector is extracted using the trained ArcFace model; ArcFace is a face recognition algorithm based on deep learning, which uses an additive angular margin loss function to improve the model's discrimination ability by increasing the angular interval. Then, the feature matrix is ​​reduced in dimension using the t-SNE algorithm (a nonlinear dimensionality reduction algorithm suitable for visualization of high-dimensional data) to obtain a new two-dimensional feature vector; finally, the two-dimensional feature matrix is ​​used to draw a scatter plot, where each point represents a facial feature vector, and the subject to which it belongs is represented in the form of a number, with labels ranging from 1 to 10. Analysis Fig. 9 and Fig.10 From the clustering of the scatter plot in Fig. 9 Before rearrangement, each subject can be well distinguished, while Fig.10 After the rearrangement, the boundaries of each subject are not clear, indicating that effective face recognition cannot be achieved. Therefore, it is finally shown that pixel rearrangement of the video can not only achieve privacy protection for human vision, but also for machine vision.

[0051] Step 4: Video encoding and decoding may adopt any commonly used hybrid video coding standard, including but not limited to H.262 / MPEG-2, H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc. The following takes the most common H.264 video coding standard as an example to introduce its implementation steps.

[0052] First, the input is a set of uncompressed video frames (YUV format). If the video is in RGB format, it needs to be converted to YUV format. Then, the encoder will perform video segmentation (i.e. macroblock division), intra-frame prediction, inter-frame prediction (including motion estimation and motion compensation), transformation and quantization, entropy coding, and stream reorganization, and finally output the encoded video bitstream for transmission or storage. During the decoding process, the decoder will perform bitstream parsing, entropy decoding, inverse quantization and inverse transformation, loop filtering, frame-by-frame reconstruction, and finally output the reconstructed video, i.e. the compressed video.

[0053] Step 5: extract rPPG signals from the compressed video. Any rPPG extraction algorithm can be used, including but not limited to signal processing-based methods, such as ICA (independent component analysis), PCA (principal component analysis), etc.; model-based methods, such as POS (planar orthogonal skin algorithm), CHROM (chromaticity signal algorithm), etc.; and deep learning-based methods, such as DeepPhys (video physiological signal measurement method based on convolutional attention network), pyVHR (multi-stage Python framework for rPPG measurement), PhysNet (rPPG detection based on deep spatiotemporal network), PhysFormer (physiological signal detection algorithm based on time difference transformation), etc. The ROI required in the above rPPG signal extraction method needs to be as consistent as possible with the ROI extracted by the present invention. The specific process of rPPG extraction is introduced below using the POS algorithm as an example.

[0054] The input video is a (compressed or uncompressed) video stream containing the facial skin area, and the video format is RGB format. Then, the time series signal of the input video on the three RGB channels is constructed, and then the plane orthogonal calculation is performed, that is, a plane orthogonal to the time-normalized skin color is constructed, and then the pixel points in the skin area are projected onto the constructed plane to obtain the projected pixel value. The output is a one-dimensional sequence of rPPG signals.

[0055] Step 6: Calculate physiological signs based on rPPG signals, such as heart rate, respiratory rate, blood oxygen saturation, blood pressure and other indicators. The details are as follows:

[0056] Heart rate refers to the number of heartbeats per minute in a normal person at rest. The methods for obtaining heart rate through the waveform of rPPG signals can be divided into time domain-based methods and frequency domain-based methods. The time domain-based method can be calculated using the peak-to-peak value or average value of the interbeat interval (IBI); the frequency domain-based method generally uses the maximum power spectrum density within the frequency range and then multiplies it by 60 to obtain the heart rate value.

[0057] Respiratory rate is the number of breaths per minute, which can be used as a medical indicator for cardiopulmonary diseases. Respiratory rate can also be obtained from the waveform of the rPPG signal, because studies have shown that respiratory frequency can be adjusted to affect the rPPG signal. The method for extracting respiratory rate from rPPG signals can adopt the integrated empirical mode decomposition (EEMD) method, which is specifically implemented as follows: the input signal is the rPPG waveform, and the preprocessing stage includes filtering and denoising the rPPG signal to ensure the effectiveness of subsequent analysis; then the EEMD algorithm is applied to decompose the signal into several intrinsic mode components; according to the approximate range of respiratory frequency, the components or modes related to breathing are identified; finally, the peak-to-peak value in the time domain method or the maximum power spectral density in the frequency domain method can be used to calculate the respiratory rate.

[0058] Blood oxygen saturation (SpO2) represents the ratio of oxygenated hemoglobin to total hemoglobin in the blood. The blood oxygen saturation value of a normal person is above 95%. At present, video-based blood oxygen saturation monitoring can be roughly divided into ratio model-based methods and deep learning-based calculation methods. Deep learning-based methods, such as K-nearest neighbor algorithms and convolutional neural networks, usually require a large amount of training data and are relatively complex to implement. The following takes the ratio model as an example to introduce the solution process of SpO2: The ratio in the ratio model represents the relationship between the AC component and the DC component of the pulse wave signal collected at two different wavelengths. The correction coefficient is determined by linear regression analysis, and a mapping function between the ratio and blood oxygen saturation is established, thereby achieving high-precision estimation of blood oxygen saturation.

[0059] Blood pressure is also a commonly used indicator in video health monitoring systems. Camera-based blood pressure measurement methods can be divided into waveform characterization methods and end-to-end algorithms. End-to-end blood pressure estimation is achieved by inputting rPPG signals into a deep learning network structure. The specific process of extracting blood pressure from the waveform morphological features of rPPG signals is as follows: First, different parameters of the waveform are measured, such as the peak (maximum value) and trough (minimum value) in the rPPG signal, the rise time (time from trough to peak), the fall time (time from peak to next trough), the peak width (width at the peak), the trough width, etc. Then the curvature change of the waveform is analyzed, which can be used to extract information about vascular elasticity and blood flow characteristics. Based on the preliminary analysis, the most meaningful waveform features are selected for blood pressure estimation. Finally, a machine learning algorithm is used to build a blood pressure estimation model, and the training data should contain known blood pressure values ​​and corresponding waveform features of rPPG signals.

[0060] In order to prove the superiority and feasibility of ROI rearrangement for video compression proposed in this invention, the Figure 7 and Figure 8 Pearson coefficient curve and signal-to-noise ratio curve, Figure 7 and Figure 8 The effect of ROI rearrangement on video compression is shown. Figure 7 and Figure 8 The result is compressed using the H.264 standard.

[0061] Figure 7 The Pearson coefficient between the rPPG signal extracted from the compressed video (expanded in step 5) and the rPPG signal extracted from the uncompressed video is shown. The Pearson coefficient is a commonly used statistical method to measure the strength and direction of the linear relationship between two variables. It measures the correlation between two variables by calculating the covariance and standard deviation between them. The value range of the Pearson coefficient is [-1, +1], where -1 indicates a perfect negative correlation, 0 indicates no correlation, and 1 indicates a perfect positive correlation. The larger the value, the stronger the correlation. Figure 7 It can be seen that as the compression bit rate increases, the Pearson coefficient also increases. That is, as the video compression level decreases, the rPPG signal retained in the video is closer to the uncompressed state, which is in line with common sense. However, the impact of bit rate changes on the Pearson coefficient is far less than the impact of modifying the ROI rearrangement rules. Figure 7It can be seen that the Pearson coefficient of the rPPG signal extracted from the compressed video without rearrangement is only between 0.4-0.425, while the Pearson coefficient of the rPPG signal extracted from the compressed video after rearrangement can reach 0.87. The result of rule three is slightly better than that of rule two, and both are better than the rearrangement method of rule one.

[0062] Figure 8 The quality of the rPPG signal extracted from the compressed video (expanded in step 5) is shown, indicated by the signal-to-noise ratio. Previous work has systematically analyzed the adverse effects of video compression on rPPG signals and found that as the video bit rate decreases (i.e., the degree of video compression increases), the rPPG signal quality (signal-to-noise ratio) decreases. Figure 8 This is also verified. In addition, under the same degree of compression (that is, the same bit rate), rearranging the ROI can obtain better rPPG quality (higher signal-to-noise ratio) than not rearranging it. In other words, to achieve the same quality (signal-to-noise ratio), the bit rate required after rearranging the ROI is greatly reduced than the bit rate required without rearranging it, that is, the required storage space is greatly reduced. This is the effect of the rearrangement on video compression proposed by the present invention. It can also be seen from the figure that the results obtained by rules two and three are basically the same, which is much better than rule one. This is because most video compression schemes are based on a series of operations on macroblocks. Comprehensive Figure 7 and Figure 8 It can be concluded that the video pixel rearrangement method proposed in the present invention can not only achieve privacy protection, but also greatly promote the effective compression of the video.

[0063] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A non-contact health monitoring method with privacy protection characteristics, It is characterized in that Remote photoplethysmography (rPPG) is used to achieve non-contact remote health monitoring; the specific steps include: Step 1: Get the target video source; Step 2: Use face detection and positioning algorithm to extract facial regions of interest; Step 3, rearrange the pixels within the facial region of interest; Step 4: Encode and decode the video based on the hybrid video coding standard to obtain a compressed video; Step 5: extracting rPPG signals from the compressed video using an rPPG extraction algorithm; Step 6: Calculate the physiological sign signal based on the rPPG signal.

2. The non-contact health monitoring method with privacy protection characteristics according to claim 1, It is characterized in that In step 1, the video source acquisition device is any device containing a camera, including but not limited to smart phones, tablet computers, laptops, monitoring equipment, VR equipment, sports cameras, drones, driving recorders and special cameras; the face area needs to be covered during shooting.

3. The non-contact health monitoring method with privacy protection characteristics according to claim 1, It is characterized in that In the step 2, the face detection and positioning algorithm specifically adopts the Haar cascade face detection algorithm, and the specific process is: input an image containing a face, and if the input is a color image, it is first converted into a grayscale image; the image is scaled; in the input image, a sliding window of a fixed size is used to slide from left to right and from top to bottom to detect each position in turn; for each window position, the corresponding Haar feature is calculated and input into the adaptive enhancement classifier for classification; if the classifier determines that the current window is a face, the window is marked as a face area; otherwise, the next position is detected; finally, the position information of the detected face area is output, and the position information is represented in the form of a rectangular frame; when multiple rectangular frames are detected, the rectangular frames are merged, the overlapping rectangular frames are removed, and the remaining rectangular frames constitute a complete face area, that is, the facial area of ​​interest; the coordinate position of the upper left corner of each rectangular frame is adjusted so that the horizontal and vertical coordinates of the starting position point are both multiples of 16; at the same time, the width and height of each rectangular frame are adjusted so that both are multiples of 16.

4. The non-contact health monitoring method with privacy protection characteristics according to claim 3, It is characterized in that In step 3, three rearrangement rules are designed, namely: Rule 1: First, convert the facial region of interest obtained in step 2 to the YCbCr color space, and then arrange all pixel values ​​in the region of interest from large to small according to the Y channel, and then arrange the same values ​​of the Y channel from large to small according to the Cb channel, and then arrange the same values ​​of the Y and Cb channels from large to small according to the Cr channel value; after arrangement, an n×3 two-dimensional matrix is ​​obtained, where n is the number of pixels in the region of interest and 3 is the three color channels; the n×3 two-dimensional matrix is ​​rearranged in a Z shape to obtain an h×w×3 three-dimensional matrix, where h and w are the height and width of the region of interest respectively; Rule 2: Change the Z-shaped arrangement according to a single pixel in Rule 1 to a Z-shaped arrangement according to a single macroblock; the specific process is: first, convert the facial region of interest obtained in step 2 to the YCbCr color space, and then arrange all pixel values ​​in the region of interest from large to small according to the Y channel, and then arrange the same values ​​of the Y channel from large to small according to the Cb channel, and then arrange the same values ​​of the Y and Cb channels from large to small according to the Cr channel value; after the arrangement, an n×3 two-dimensional matrix is ​​obtained, and the n×3 two-dimensional matrix is ​​a row of pixels; define the macroblock size as 16×16 pixels, and take 16 pixels from the n×3 two-dimensional matrix as a row of the macroblock until all 16 rows are taken to obtain the first macroblock; the remaining macroblocks are also taken according to this rule, and finally all single macroblocks are arranged in a Z shape; Rule 3: Change the arrangement of macroblocks in Rule 2 from zigzag to top to bottom and left to right. The specific process is: first, convert the facial region of interest obtained in step 2 to the YCbCr color space, and then arrange all pixel values ​​in the region of interest from large to small according to the Y channel, and then arrange the same values ​​of the Y channel from large to small according to the Cb channel, and then arrange the same values ​​of the Y and Cb channels from large to small according to the Cr channel value; after arrangement, an n×3 two-dimensional matrix is ​​obtained, and the n×3 two-dimensional matrix is ​​a row of pixels; take 16 pixels as a group every time, until h groups are taken, and the h groups correspond to the height of the region of interest, and take each group as each column of the final result, a total of w / 16 columns; When applying the rearrangement rule to the online video compression of the current video, three rearrangement rules are tested in advance using historical data, and the one with the best video compression result is selected as the rearrangement rule for the current video.

5. The non-contact health monitoring method with privacy protection characteristics according to claim 1, It is characterized in that In step 4, the H.264 video coding standard is used to encode and decode the video. The specific process is: first, a group of uncompressed video frames are input. If the video is in RGB format, it needs to be converted into YUV format; then, the encoder of the encoding module performs video segmentation, intra-frame prediction, inter-frame prediction, transformation and quantization, entropy coding, and stream reorganization, and finally outputs the encoded video code stream for transmission or storage; during the decoding process of the decoding module, the decoder will perform code stream analysis, entropy decoding, inverse quantization and inverse transformation, loop filtering, and frame-by-frame reconstruction, and finally output the reconstructed video, that is, the compressed video.

6. The non-contact health monitoring method with privacy protection characteristics according to claim 1, It is characterized in that In step 5, a plane orthogonal skin algorithm is specifically used to extract rPPG signals. The specific process is: input the compressed video, construct the time series signal of the input video on the three RGB channels, and then perform plane orthogonal calculation, that is, construct a plane orthogonal to the time-normalized skin color, and then project the pixel points in the skin area onto the constructed plane to obtain the projected pixel values, which are output as a one-dimensional sequence of rPPG signals.

7. The non-contact health monitoring method with privacy protection characteristics according to claim 1, It is characterized in that In step 6, the physiological sign signals include heart rate, respiratory rate, blood oxygen saturation and blood pressure; The method for obtaining the heart rate from the waveform of the rPPG signal adopts a time domain-based method or a frequency domain-based method. The time domain-based method uses the peak-to-peak value or average value of the heart beat interval to calculate; the frequency domain-based method uses the maximum power spectrum density within the frequency range and then multiplies it by 60 to obtain the heart rate value; The method for extracting the respiratory rate from the rPPG signal adopts the integrated empirical mode decomposition method, which is specifically implemented as follows: the waveform of the rPPG signal is input, and the preprocessing stage includes filtering and denoising the rPPG signal; then the integrated empirical mode decomposition method is applied to decompose the signal into several intrinsic mode components; according to the approximate range of the respiratory frequency, the components or modes related to breathing are identified; finally, the respiratory rate is calculated using the peak-to-peak value in the time domain method or the maximum power spectral density in the frequency domain method; The ratio model is used to calculate the blood oxygen saturation. The calculation process is as follows: the ratio in the ratio model represents the relationship between the AC component and the DC component of the pulse wave signal collected at two different wavelengths; the correction coefficient is determined by linear regression analysis, and a mapping function between the ratio and the blood oxygen saturation is established, thereby achieving a high-precision estimation of the blood oxygen saturation; The specific process of extracting blood pressure from the waveform morphological features of the rPPG signal is as follows: first, measure different parameters of the waveform, including the peaks and troughs, rise time, fall time, peak width, and trough width in the rPPG signal; then analyze the curvature changes of the waveform to extract information about vascular elasticity and blood flow characteristics; based on the preliminary analysis, select the most meaningful waveform features for blood pressure estimation; finally, use a machine learning algorithm to build a blood pressure estimation model, and the training data contains known blood pressure values ​​and the corresponding waveform features of the rPPG signal.

Images