Remote contactless heart rate estimation method, system and device based on sparse structure representation
By segmenting and representing face videos in a sparse structure, the problem of unconsidered signal pulsation and correlation between sub-regions is solved, achieving high accuracy and robustness in remote heart rate estimation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUIZHOU NORMAL UNIVERSITY
- Filing Date
- 2022-03-29
- Publication Date
- 2026-05-01
AI Technical Summary
Existing long-range heart rate estimation methods based on sparse representations fail to effectively consider the pulsation and correlation of signals between sub-regions, resulting in inaccurate heart rate signal estimation.
A sparse structure representation-based approach is adopted. Face detection is divided into multiple sub-regions, the signal-to-noise ratio of each sub-region is calculated, high-quality sub-regions are selected, and the heart rate signal is reconstructed using a hybrid structure dictionary and sparse coding. The final heart rate is obtained by combining power spectrum analysis.
It improves the accuracy and robustness of heart rate signal estimation, especially the detection performance in different scenarios, achieving an accuracy of up to 96% and a Pearson correlation coefficient of 98%.
Smart Images
Figure CN114596963B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of biometrics research and application technology, and more specifically to the field of video-based remote heart rate estimation technology. Background Technology
[0002] Heart rate is an important physiological signal used to calculate physiological parameters such as heart rate, heart rate variability, respiratory rate, and blood pressure. Traditional heart rate detection methods require skin contact (such as electrocardiograms), which can lead to discomfort and hygiene issues. Remote heart rate estimation based on facial video is a non-contact method for heart rate measurement that is convenient to use. The beating of the heart causes subtle changes in skin color, especially on the face. Based on this, remote photoplethysmography (rPPG) has been proposed, utilizing computer vision, signal processing, and other techniques to extract pure heart rate signals from these subtle color changes. Remote non-contact heart rate estimation methods fall into three main categories: blind source separation, color subspace projection, and deep learning.
[0003] With the successful applications of deep learning in fields such as computer vision and natural language processing, end-to-end deep learning network methods have been introduced into the field of remote heart rate estimation based on facial videos. Spetlik et al. proposed the HR-CNN network, which acquires pulse signals through a two-dimensional convolutional network based on signal-to-noise ratio constraints. Considering the influence of abnormal lighting and motion, Chen et al. introduced an attention mechanism and proposed a deep physics method by constructing feature maps that learn spatiotemporal information. To better represent color change information, Niu et al. converted RGB signals to YUV space, extracted heart rate change information, and converted it into a spatiotemporal map for heart rate detection. Deep learning methods show better results on datasets, but the models are complex and have weak generalization. However, the challenges of deep learning methods are that they require large amounts of labeled datasets to train the network, are computationally slow, have many parameters, and require pre-trained network models.
[0004] Traditional methods such as blind source separation and color subspace decomposition offer better interpretability and faster execution speeds. Sparse representation has wide applications in computer vision, and its introduction into heart rate signal reconstruction has yielded promising results. Liu et al. proposed a sparse representation method for adaptive interference orthogonal matching pursuit (AOP), which uses an AOP algorithm to construct a discrete sine dictionary to recover unstable and disrupted heart rate signals from faces. Heart rate, as a physiological signal, contains both periodic and pulsating signals. While Liu et al.'s method achieved good results, it did not consider the pulsation and correlation between sub-regions, leading to inconsistent and inaccurate heart rate estimations. Summary of the Invention
[0005] The purpose of this invention is to address the technical problem that the above-mentioned sparse signal representation methods, due to the lack of consideration for the pulsation and correlation between sub-regions, lead to inconsistent estimated heart rate signals and inaccurate heart rate estimation. This invention provides a remote non-contact heart rate estimation method, system, and device based on sparse structure representation.
[0006] To achieve the above objectives, the present invention specifically adopts the following technical solution:
[0007] A remote, non-contact heart rate estimation method based on sparse structure representation includes the following steps:
[0008] Step 1: Input a face video. Select the forehead and cheeks as the Region of Interest (ROI) of the detected face, and divide it into blocks to obtain multiple sub-regions. (In remote video heart rate detection, face detection is required first. In different scenarios, faces can be processed in terms of movement and rotation, so the face detection method should be timely and accurate. Considering the real-time performance and positioning tracking effect of the detection method, this invention uses the PCN deep network face recognition algorithm for face detection and tracking to obtain the coordinates and width of the face. Since the forehead and cheek areas of the face contain rich heart rate information, based on the above analysis, this invention uses the PCN algorithm to select the forehead and cheeks as the ROI, and divides the forehead and cheeks into blocks to obtain multiple sub-regions.)
[0009] Step 2: Construct the average RGB pulse signal for each sub-region, perform motion compensation on the RGB pulse signal (eliminate sudden pixel changes and generate a preliminary denoised color signal), and calculate the chromaticity signal S using a linear combination of different RGB channels. Based on the chromaticity signal S, calculate the signal-to-noise ratio (SNR) of each region (due to the presence of foreign objects on the face occlusion surface, uneven light distribution, motion, etc., the chromaticity signal quality of some sub-regions is poor. Therefore, sub-regions rich in heart rate information are selected by calculating the SNR of the chromaticity signal). Sub-regions with an SNR greater than the overall SNR are defined as high-quality sub-regions, and the corresponding chromaticity signal is defined as a high-quality chromaticity signal, generating a high-quality sub-region chromaticity signal M.
[0010] Step 3: Analyze the actual heart rate signal M pulse The heart rate signal was reconstructed by performing sparse structure representation. The reconstructed heart rate signal was further calculated from the high-quality sub-region chroma signal M, which consists of the chroma signals of sub-regions with high signal-to-noise ratios. in, The chromaticity signal representing the selected high-quality sub-region (M is the high-quality sub-region chromaticity signal M obtained by linear calculation from the average RGB pulse signal) is derived from the actual heart rate signal (M). pulse ) and noise signal (Mnoise Composed of, i.e., M = M pulse +M noise The sparse structure representation reconstructs the heart rate signal. (as an estimate of the heart rate signal), The reconstruction of the heart rate signal from the selected high-quality sub-region, X, is obtained as follows: A hybrid structure dictionary D is constructed using cosine bases of different frequencies and wavelet bases of different scales (the periodic characteristics of the heart rate signal are reconstructed using different cosine bases, and the fluctuation characteristics of the heart rate signal are reconstructed using wavelet bases of different scales). The coefficient matrix A is solved. The heart rate signal is represented by appropriate dictionary atoms from the hybrid dictionary using a sparse structure method. The heart rate signal is sparsely encoded using a greedy algorithm to form the coefficient matrix A. The heart rate signal is reconstructed by the inner product of the hybrid structure dictionary D and the coefficient matrix A, i.e., X = D·A.
[0011] Step 4: Average the reconstructed heart rate signals from each sub-region, then perform power spectrum analysis to obtain the heart rate (HR).
[0012] In this technical solution, a video containing a human face is input. The detected face's forehead and cheeks are extracted as regions of interest (ROIs). Within each ROI, the forehead and cheeks are divided into multiple sub-regions. The original color signal is extracted and analyzed. Motion compensation is used to eliminate sudden pixel changes caused by rigid facial movements and unstable lighting conditions. The chromaticity signal and signal-to-noise ratio (SNR) of each sub-region are calculated. Sub-regions with an SNR greater than the overall SNR are defined as high-quality sub-regions, and their corresponding chromaticity signals are defined as high-quality chromaticity signals, thus generating high-quality chromaticity signals. A structure dictionary composed of cosine bases of different frequencies and wavelet bases of different scales is used to efficiently reconstruct the period and amplitude of the heart rate signal. A sparse matrix is solved, and a more suitable dictionary atom is selected from the chosen dictionary to represent the heart rate signal using a sparse structure method. The heart rate signal is reconstructed using the inner product method of the dictionary and the sparse matrix. The reconstructed heart rate signal matrix is averaged. Finally, power spectrum analysis is performed on the averaged heart rate signals from each sub-region to obtain the final heart rate. This application divides the face into multiple sub-regions, extracts high-quality chroma signals from these sub-regions, constructs a hybrid structure dictionary, searches for the optimal sparse representation based on the hybrid structure dictionary, and reconstructs the heart rate signal using the hybrid structure dictionary and sparse coding, representing the most realistic heart rate signal with as few dictionary atoms as possible. This helps improve the accuracy of the algorithm.
[0013] Further, in step 2, the average value, absolute deviation of the average value, and standard deviation of the RGB pulse signal for each sub-region are calculated (the standard deviation reflects the degree to which the signal deviates from the average value). Then, the magnitude of deviation of each atom from the average value is calculated. Atoms exceeding both the average value and the standard deviation are calculated by adding the average value and the standard deviation; atoms below the average value and the standard deviation are calculated by subtracting the standard deviation from the average value for motion compensation (color signals contain a large amount of noise, and heart rate signals are extremely weak, with no significant changes in heart rate over a short period. Therefore, this invention analyzes the original color signal and eliminates sudden pixel changes caused by rigid facial movements and unstable lighting conditions. The average value of the color signal reflects the DC component of the signal), specifically defined as follows:
[0014]
[0015]
[0016]
[0017]
[0018] Where i is the channel number, i = 1, 2, 3 represent the RGB channel signals respectively, specifically i = 1 for the R channel; i = 2 for the G channel; i = 3 for the B channel, l for the signal length, and C for the signal length. i (t) is the signal length of the i-th channel at time t, C i_avg For the average signal of channel i, C i_Me C represents the average signal deviation of channel i. i_SD This represents the standard deviation of the signal in channel i.
[0019] Furthermore, in step 2, the chromaticity signal S is derived from S = X f -αY f It is calculated that α is the sum of two signals X. f and Y f The ratio of the standard deviations of X, i.e., α = σ(X) f ) / σ(Y f ), X s and Y s It is a fusion of normalized RGB pulse signals after motion compensation, X s and Y s The calculation function is as follows:
[0020] X s =3R n -2G n
[0021] Y s =1.5R n +G n -1.5Bn
[0022] Among them, R n G n and B n The color signal for each sub-region;
[0023] X f and Y f They are X s and Y s The bandpass filter version is used to calculate X. s and Y s Afterwards, by X s and Y s After passing through a bandpass filter, X is obtained. f and Y f Finally, the S-chromaticity signal is calculated;
[0024] The signal-to-noise ratio function for each sub-region is as follows:
[0025]
[0026] Among them, S f Let S represent the spectral values after the Fourier transform of the chrominance signal S, f represent the frequency limiting the heart rate, and the range of f is [0.6, 4] (f = [0.4, 6]). W t This indicates the size of the sliding window used for calculation.
[0027] Furthermore, in step 3, X is obtained as follows: using the cosine basis D c And Xiaobo D w A hybrid structure dictionary D is constructed (since any periodic discrete signal can be represented as a linear combination of cosine-based signals of different frequencies, and heart rate signals have different periodicity and amplitudes, discrete cosine-based sparse representation and wavelet-based sparse representation are used. To reduce the dictionary size and improve algorithm performance, the frequency of the cosine-based signals is limited to the heart rate frequency range (0.6-4 Hz). Based on this, a hybrid structure dictionary containing cosine and wavelet bases is constructed).
[0028]
[0029] D w = WaveDict(Short3, N b (j, b)
[0030] Where, k i For the i-th element in the discrete cosine dictionary th The frequency of an atom, k i With k i+1 The difference is F rFor video frame rate, N b Let j be the length of the generated signal, b be the level vector, and f be the shift factor. L The length of the generated signal sequence,
[0031] The hybrid dictionary model is as follows:
[0032]
[0033] Furthermore, in step 3, the coefficient matrix is solved using a sparse structure method (the sparse structure representation method is based on the true heart rate signal (M... pulse A sparse structure representation was performed to obtain an estimate of the heart rate signal. Although the heart rate information contained in each region is different, their basic heart rate components should be the same, and the heart rate signals in different regions should be similar. Therefore, for the heart rate signals reconstructed in different sub-regions, the same dictionary atoms in the signal base dictionary are selected by minimizing the L1 and L2 parameterization of the coefficient matrix, i.e., min||A|| 1，2 Furthermore, the reconstructed heart rate signal X and the true heart rate signal M... pulse There should be a small error between them. This problem can be bounded by minimizing the L2 parameterization between the true heart rate signal and the reconstructed heart rate signal, i.e., minimizing... Based on this, the reconstruction of the heart rate signal can be represented as solving the following model: Selecting suitable dictionary atoms to represent the pulse signal from the mixed dictionary, performing sparse coding using a greedy algorithm (to find the most suitable dictionary atoms), and solving the coefficient matrix specifically involves solving the following model:
[0034] The sparse coding is obtained, forming the coefficient matrix A.
[0035] Where λ is used to balance the signal reconstruction accuracy. And penalty item ||A|| 1，2 The regularization parameter;
[0036] Furthermore, in step 3, the heart rate signal is reconstructed using the inner product of the mixed dictionary D and the coefficient matrix A. The inner product model is as follows:
[0037] X = D·A.
[0038] An ideal PPG signal should contain both periodicity and volatility. Periodicity can be represented as a linear combination of a small number of cosine bases, while amplitude can be represented as a linear combination of a small number of wavelets. Therefore, an ideal PPG signal can be represented by a small number of mixed structure dictionaries, i.e., A is a sparse matrix. Furthermore, the heart rate signal is formed by changes in heartbeat, and due to interference from noise such as light and motion, the facial skin undergoes different color changes.
[0039] Furthermore, the reconstructed heart rate signals from each sub-region are averaged to obtain the final heart rate signal, specifically:
[0040]
[0041] Where N is the total number of selected sub-regions, x n This represents the heart rate signal for the selected sub-region.
[0042] Furthermore, in step 4, a power spectrum analysis is performed on the averaged heart rate signal, and the component with the highest peak value is selected as the main component of the heart rate signal, that is, the frequency f corresponding to the component with the highest power is used. HR Finally, the time-frequency conversion is performed to obtain the final heart rate HR, specifically: HR = f HR *60.
[0043] A remote, non-contact heart rate estimation system based on sparse structure representation, characterized in that it includes:
[0044] The sub-region acquisition module is used to input a face video, fix the forehead and cheek parts of the detected face as the region of interest, and divide it into blocks to obtain multiple sub-regions;
[0045] The high-quality sub-region chroma signal generation module is used to construct the average RGB pulse signal of each sub-region, perform motion compensation on the RGB pulse signal, and calculate the chroma signal S using the linear combination of different RGB channels. Based on the chroma signal S, the signal-to-noise ratio of each sub-region is calculated. Sub-regions with a signal-to-noise ratio greater than the overall signal-to-noise ratio are defined as high-quality sub-regions, and the corresponding chroma signal is defined as a high-quality chroma signal, generating a high-quality sub-region chroma signal M.
[0046] The heart rate signal reconstruction module is used to reconstruct the real heart rate signal M. pulse The heart rate signal was reconstructed by performing sparse structure representation. A high-quality sub-region chroma signal M consists of chroma signals from sub-regions with high signal-to-noise ratios, i.e. in, This represents the chroma signal of the selected high-quality sub-region. The reconstruction of the heart rate signal from the selected high-quality sub-region is obtained as follows: A hybrid structure dictionary D is formed using cosine bases of different frequencies and wavelet bases of different scales. The coefficient matrix A is solved. The heart rate signal is represented by appropriate dictionary atoms from the hybrid dictionary using a sparse structure method. The heart rate signal is sparsely encoded using a greedy algorithm to form the coefficient matrix A. The heart rate signal is reconstructed by the inner product of the hybrid structure dictionary D and the coefficient matrix A, i.e., X = D·A.
[0047] The heart rate acquisition module is used to average the reconstructed heart rate signals from each sub-region, and then perform power spectrum analysis to obtain the heart rate (HR).
[0048] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the remote non-contact heart rate estimation method based on sparse structure representation.
[0049] The beneficial effects of this invention are as follows:
[0050] 1. Divide the face into multiple sub-regions, extract high-quality chroma signals from the sub-regions, construct a hybrid structure dictionary, find the optimal sparse representation based on the hybrid structure dictionary, and reconstruct the heart rate signal using the hybrid structure dictionary and sparse coding. Using as few dictionary atoms as possible to represent the most realistic heart rate signal helps improve the accuracy of the algorithm.
[0051] 2. At a frame length of 1200, this application achieves the best results when MAE and RMSE are below 5 bpm, the accuracy of the detection method is greater than 96%, and the Pearson correlation between detected heart rates is greater than 97%.
[0052] 3. Under steady-state, speaking, slow translation, slow rotation, moderate rotation, and rapid translation conditions, the overall accuracy of the detection method in this application exceeds 95%, and the Pearson correlation coefficient between the detected heart rate and the true value exceeds 98%. Under steady-state, speaking, slow translation, slow rotation, and moderate rotation conditions, the Pearson correlation coefficient is above 99%, indicating that the detection method performs well under all of these conditions.
[0053] 4. In comparison with other methods on the UBFC dataset, the method of this invention has a MAE and RMSE of less than 5 bpm, and an Acc and PCC of more than 96%, indicating that the method of this invention has strong applicability. Compared with the best-performing POS method, the MAE is improved by 1.697 bpm, the RMSE is improved by 6.075 bpm, and the Pearson correlation coefficient is improved by 11.9%, indicating that the method of this invention has high algorithm robustness under stable light and heart rate change conditions.
[0054] 5. The detection method in this invention detects a high degree of consistency between the heart rate and the true HR in all states of the entire dataset. Attached Figure Description
[0055] Figure 1 This is a block diagram of the remote non-contact heart rate estimation method based on sparse structure representation of the present invention;
[0056] Figure 2 This is a flowchart of the remote non-contact heart rate estimation method based on sparse structure representation of the present invention;
[0057] Figure 3 This is a flowchart of the sparse structure representation of the present invention. Detailed Implementation
[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0059] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.
[0060] Example 1
[0061] like Figures 1 to 3 As shown, this embodiment provides a remote non-contact heart rate estimation method based on sparse structure representation, including the following steps:
[0062] Step 1: Input a face video. Select the forehead and cheeks as the Region of Interest (ROI) of the detected face, and divide it into blocks to obtain multiple sub-regions. (In remote video heart rate detection, face detection is required first. In different scenarios, faces can be processed in terms of movement and rotation, so the face detection method should be timely and accurate. Considering the real-time performance and positioning tracking effect of the detection method, this invention uses the PCN deep network face recognition algorithm for face detection and tracking to obtain the coordinates and width of the face. Since the forehead and cheek areas of the face contain rich heart rate information, based on the above analysis, this invention uses the PCN algorithm to select the forehead and cheeks as the ROI, and divides the forehead and cheeks into blocks to obtain multiple sub-regions.)
[0063] Step 2: Construct the average RGB pulse signal for each sub-region, perform motion compensation on the RGB pulse signal (eliminate sudden pixel changes and generate a preliminary denoised color signal), and calculate the chromaticity signal S using a linear combination of different RGB channels. Based on the chromaticity signal S, calculate the signal-to-noise ratio (SNR) of each region (due to the presence of foreign objects on the face occlusion surface, uneven light distribution, motion, etc., the chromaticity signal quality of some sub-regions is poor. Therefore, sub-regions rich in heart rate information are selected by calculating the SNR of the chromaticity signal). Sub-regions with an SNR greater than the overall SNR are defined as high-quality sub-regions, and the corresponding chromaticity signal is defined as a high-quality chromaticity signal, generating a high-quality sub-region chromaticity signal M.
[0064] The average value, absolute deviation, and standard deviation of the RGB pulse signal for each sub-region are calculated (the standard deviation reflects the degree to which the signal deviates from the average value). Then, the magnitude of deviation of each atom from the average value is calculated. Atoms exceeding both the average value and the standard deviation are calculated by adding the average value and the standard deviation; atoms below the average value and the standard deviation are calculated by subtracting the standard deviation from the average value for motion compensation (color signals contain a large amount of noise, and heart rate signals are extremely weak, with no significant changes in heart rate over a short period. Therefore, this invention analyzes the original color signal and eliminates sudden pixel changes caused by rigid facial movements and unstable lighting conditions. The average value of the color signal reflects the DC component of the signal), specifically defined as follows:
[0065]
[0066]
[0067]
[0068]
[0069] Where i is the channel number, i = 1, 2, 3 represent the RGB channel signals respectively, specifically i = 1 for the R channel; i = 2 for the G channel; i = 3 for the B channel, l for the signal length, and C for the signal length. i (t) is the signal length of the i-th channel at time t, C i_avg For the average signal of channel i, C i_Me C represents the average signal deviation of channel i. i_SD The standard deviation of the i-channel signal;
[0070] The chromaticity signal S is derived from S = X f -αY f It is calculated that α is the sum of two signals X. f and Y f The ratio of the standard deviations of X, i.e., α = σ(X) f ) / σ(Y f ), X s and Y s It is a fusion of normalized RGB pulse signals after motion compensation, X s and Y s The calculation function is as follows:
[0071] X s =3R n -2G n
[0072] Y s =1.5R n +G n -1.5B n
[0073] Among them, R n G n and B n The color signal for each sub-region;
[0074] X f and Y f They are X s and Y s The bandpass filter version is used to calculate X. s and Y s Afterwards, by X s and Y s After passing through a bandpass filter, X is obtained. f and Y f Finally, the S-chromaticity signal is calculated;
[0075] The signal-to-noise ratio function for each sub-region is as follows:
[0076]
[0077] Among them, S f Let S represent the spectral values after the Fourier transform of the chrominance signal S, f represent the frequency limiting the heart rate, and the range of f is [0.6, 4] (f = [0.4, 6]). W t This indicates the size of the sliding window used for calculation;
[0078] Step 3: Analyze the actual heart rate signal M pulse The heart rate signal was reconstructed by performing sparse structure representation. The reconstructed heart rate signal was further calculated from the high-quality sub-region chroma signal M, which consists of the chroma signals of sub-regions with high signal-to-noise ratios. in, The chromaticity signal representing the selected high-quality sub-region (M is the high-quality sub-region chromaticity signal M obtained by linear calculation from the average RGB pulse signal) is derived from the actual heart rate signal (M). pulse ) and noise signal (M noise Composed of, i.e., M = M pulse +M noise The sparse structure representation reconstructs the heart rate signal. (as an estimate of the heart rate signal), The reconstruction of the heart rate signal from the selected high-quality sub-region, X, is obtained as follows: A hybrid structure dictionary D is constructed using cosine bases of different frequencies and wavelet bases of different scales (the periodic characteristics of the heart rate signal are reconstructed using different cosine bases, and the fluctuation characteristics of the heart rate signal are reconstructed using wavelet bases of different scales). The coefficient matrix A is solved. The heart rate signal is represented by appropriate dictionary atoms from the hybrid dictionary using a sparse structure method. The heart rate signal is sparsely encoded using a greedy algorithm to form the coefficient matrix A. The heart rate signal is reconstructed by the inner product of the hybrid structure dictionary D and the coefficient matrix A, i.e., X = D·A.
[0079] The method for obtaining X is as follows: using the cosine basis D c And Xiaobo D w A hybrid structure dictionary D is constructed (since any periodic discrete signal can be represented as a linear combination of cosine-based signals of different frequencies, and heart rate signals have different periodicity and amplitudes, discrete cosine-based sparse representation and wavelet-based sparse representation are used. To reduce the dictionary size and improve algorithm performance, the frequency of the cosine-based signals is limited to the heart rate frequency range (0.6-4 Hz). Based on this, a hybrid structure dictionary containing cosine and wavelet bases is constructed).
[0080]
[0081] D w = WaveDict(Short3, N b (j, b)
[0082] Where, k i For the i-th element in the discrete cosine dictionary th The frequency of an atom, k i With k i+1 The difference is F r For video frame rate, N b Let j be the length of the generated signal, b be the level vector, and f be the shift factor. L The length of the generated signal sequence,
[0083] The hybrid dictionary model is as follows:
[0084]
[0085] Solving for the coefficient matrix using the sparse structure method (the sparse structure representation method is based on the true heart rate signal (M...) pulseA sparse structure representation was performed to obtain an estimate of the heart rate signal. Although the heart rate information contained in each region is different, their basic heart rate components should be the same, and the heart rate signals in different regions should be similar. Therefore, for the heart rate signals reconstructed in different sub-regions, the same dictionary atoms in the signal base dictionary are selected by minimizing the L1 and L2 parameterization of the coefficient matrix, i.e., min||A|| 1，2 Furthermore, the reconstructed heart rate signal X and the true heart rate signal M... pulse There should be a small error between them. This problem can be bounded by minimizing the L2 parameterization between the true heart rate signal and the reconstructed heart rate signal, i.e., minimizing... Based on this, the reconstruction of the heart rate signal can be represented as solving the following model: Selecting suitable dictionary atoms to represent the pulse signal from the mixed dictionary, performing sparse coding using a greedy algorithm (to find the most suitable dictionary atoms), and solving the coefficient matrix specifically involves solving the following model:
[0086] The sparse coding is obtained, forming the coefficient matrix A.
[0087] Where λ is used to balance the signal reconstruction accuracy. And penalty item ||A|| 1，2 The regularization parameter;
[0088] The heart rate signal is reconstructed using the inner product of a mixed dictionary D and a coefficient matrix A. The inner product model is as follows:
[0089] X = D·A.
[0090] An ideal PPG signal should contain both periodicity and volatility. Periodicity can be represented as a linear combination of a small number of cosine bases, while amplitude can be represented as a linear combination of a small number of wavelets. Therefore, an ideal PPG signal can be represented by a small number of mixed structure dictionaries, i.e., A is a sparse matrix. Furthermore, the heart rate signal is formed by changes in heartbeat, and due to interference from noise such as light and motion, the facial skin undergoes different color changes.
[0091] The heart rate signal is obtained by averaging the reconstructed heart rate signals from each sub-region, specifically as follows:
[0092]
[0093] Where N is the total number of selected sub-regions, x n The heart rate signal for the selected sub-region;
[0094] Step 4: Average the reconstructed heart rate signals from each sub-region, then perform power spectrum analysis to obtain the heart rate (HR). Perform power spectrum analysis on the averaged heart rate signal and select the component with the highest peak value as the principal component of the heart rate signal, i.e., use the frequency f corresponding to the component with the highest power. HR Finally, the time-frequency conversion is performed to obtain the final heart rate HR, specifically: HR = f HR *60.
[0095] in, Figure 2 In this diagram, I represents face detection and ROI selection; II represents chroma signal extraction and motion compensation; III represents the original RGB pulse signal; IV represents sparse structure representation; V represents the average reconstructed heart rate signal; and VI represents heart rate calculation. Figure 3 In the diagram, a represents the hybrid structure dictionary (D); b represents the matching coefficient matrix (A); c represents the reconstructed sub-region heart rate signal; and d represents the average reconstructed heart rate signal of each sub-region.
[0096] To verify the superiority of the method of this invention in video-based remote heart rate estimation, experiments were conducted on the publicly available datasets PURE and UBFC. To evaluate the overall performance of the algorithm, the mean absolute error (MAE), root mean square error (RMSE), accuracy (Acc), and Pearson correlation coefficient (PCC) of heart rate were selected as evaluation metrics. The heart rate error of the nth video sample is defined as follows:
[0097]
[0098] Where n represents the nth th One test sample, This indicates the heart rate signal extracted from the video. This represents the heart rate detected from the PPG, i.e., the true signal.
[0099] The mean absolute error of heart rate is defined as follows:
[0100]
[0101] N represents the total number of test samples.
[0102] The root mean square error of heart rate is defined as follows:
[0103]
[0104] The accuracy of heart rate is defined as follows:
[0105]
[0106] The Pearson correlation coefficients between the estimated and actual values are as follows:
[0107]
[0108] Where μ represents the average estimated heart rate, This represents the true average value.
[0109] (1) Results analysis of the present invention on the UBFC dataset
[0110] Table 1 Performance at different video lengths
[0111] Frame length MAE(bpm) RMSE (bpm) Acc(%) PCC 300 8.756 12.515 91.315 0.782 450 7.091 10.764 92.979 0.824 600 5.843 8.949 94.199 0.869 750 4.017 5.711 95.925 0.940 900 4.217 6.011 95.790 0.939 1200 3.467 4.071 96.526 0.976
[0112] All video segments were played at 30fps, and the videos were divided into different lengths to achieve heart rate signal recovery. The performance of the detection method at different video lengths is shown in Table 1. At a length of 1200 frames, the detection method achieved the best results with MAE and RMSE below 5 bpm, an accuracy greater than 96%, and a Pearson correlation greater than 97% between detected heart rates. Below 450 frames, the performance of the detection method dropped sharply. This is because short videos in natural conditions cannot provide sufficiently stable information for heart rate signal recovery.
[0113] (2) Results analysis of the present invention on the PURE dataset
[0114] The performance of this invention was tested in the PURE dataset under different states, including six different states. The results are shown in Table 2. The overall accuracy of the detection method exceeds 95%, and the Pearson correlation coefficient between the detected heart rate and the true value exceeds 98%. In steady state, speaking, slow translation, slow rotation, and moderate rotation states, the Pearson correlation coefficient is above 99%, indicating that the detection method performs well in all these states.
[0115] The detection method exhibits fluctuations in error during speaking and slow translation modes because facial skin wrinkles during speech, and motion artifacts form during slow translation, all of which interfere with chroma signal extraction. During fast translation, the problem arises because face detection struggles to capture rapid head rotations, potentially leading to inaccurate ROI selection. Rapid rotation also causes image blurring in the video, further obscuring subtle heart rate information.
[0116] The detection method in this invention detects a high degree of consistency between the heart rate and the actual ground heart rate in all states of the entire dataset.
[0117] Table 2 Performance of the algorithm under different states
[0118] Evaluation criteria 01 02 03 04 05 06 MAE(bpm) 2.577 3.046 2.678 2.738 2.246 2.396 RMSE (bpm) 3.282 3.621 3.748 4.163 2.904 2.949 Acc(%) 95.951 95.060 96.708 95.891 96.629 96.365 PCC 0.996 0.991 0.991 0.985 0.996 0.994
[0119] 01: Stable state; 02: Speaking; 03: Slow translation; 04: Fast translation; 05: Slow rotation; 06: Medium speed rotation.
[0120] (3) Experimental comparison of the present invention
[0121] This invention was compared with other methods on the UBFC dataset. Table 3 shows the experimental results of six different estimation methods on the UBFC dataset. The MAE and RMSE of the method of this invention are below 5 bpm, and the Acc and PCC are above 96%, indicating that the method of this invention has strong applicability. Compared with the best-performing POS method, the MAE is improved by 1.697 bpm, the RMSE is improved by 6.075 bpm, and the Pearson correlation coefficient is improved by 11.9%. The results show that the method of this invention has high robustness under stable light conditions and heart rate changes.
[0122] DAOMP also uses sparse representation. Due to its simple dictionary, it does not consider the heart rate correlation between sub-regions, resulting in a large error between the reconstructed heart rate signal and the original pulse signal.
[0123] POS, as a subspace decomposition method, has also yielded good results.
[0124] Table 3. Comparison results of six comparison methods on the UBFC dataset.
[0125] Methods MAE(bpm) RMSE (bpm) Acc(%) PCC ICA 28.455 36.858 72.420 0.277 CHROM 7.627 15.693 92.542 0.714 POS 5.164 10.146 94.872 0.857 LGI 10.570 20.715 89.841 0.553 DAOMP 14.580 22.804 85.850 0.484 Ours 3.467 4.071 96.526 0.976
[0126] Example 2
[0127] A remote, non-contact heart rate estimation system based on sparse structure representation includes:
[0128] The sub-region acquisition module is used to input a face video, fix the forehead and cheek parts of the detected face as the region of interest, and divide it into blocks to obtain multiple sub-regions;
[0129] The high-quality sub-region chroma signal generation module is used to construct the average RGB pulse signal of each sub-region, perform motion compensation on the RGB pulse signal, and calculate the chroma signal S using the linear combination of different RGB channels. Based on the chroma signal S, the signal-to-noise ratio of each sub-region is calculated. Sub-regions with a signal-to-noise ratio greater than the overall signal-to-noise ratio are defined as high-quality sub-regions, and the corresponding chroma signal is defined as a high-quality chroma signal, generating a high-quality sub-region chroma signal M.
[0130] The heart rate signal reconstruction module is used to reconstruct the real heart rate signal M. pulse The heart rate signal was reconstructed by performing sparse structure representation. A high-quality sub-region chroma signal M consists of chroma signals from sub-regions with high signal-to-noise ratios, i.e. in, This represents the chroma signal of the selected high-quality sub-region. The reconstruction of the heart rate signal from a selected high-quality sub-region, X, is obtained as follows: A hybrid structure dictionary D is constructed using cosine bases of different frequencies and wavelet bases of different scales. The coefficient matrix A is solved. Appropriate dictionary atoms are selected from the hybrid dictionary using a sparse structure method to represent the heart rate signal. A greedy algorithm is used for sparse coding to construct the coefficient matrix A. The heart rate signal is reconstructed using the inner product of the hybrid structure dictionary D and the coefficient matrix A.
[0131] X = D·A;
[0132] The heart rate acquisition module is used to average the reconstructed heart rate signals from each sub-region, and then perform power spectrum analysis to obtain the heart rate (HR).
[0133] In the high-quality sub-region chromaticity signal generation module:
[0134] Calculate the average value, absolute deviation, and standard deviation of the RGB pulse signal for each sub-region. Then, calculate the magnitude of each atom's deviation from the average value. Atoms exceeding both the average value and the standard deviation are compensated by adding the average value and the standard deviation; atoms below the average value and the standard deviation are compensated by subtracting the standard deviation from the average value. The specific definitions are as follows:
[0135]
[0136]
[0137]
[0138]
[0139] Where l is the number of channels, l = 1, 2, 3 represent RGB channel signals respectively, specifically l = 1 for the R channel; i = 2 for the G channel; i = 3 for the B channel, l is the signal length, C i (t) is the signal length of the i-th channel at time t, C i_avg For the average signal of channel i, C i_Me C represents the average signal deviation of channel i. i_SD The standard deviation of the i-channel signal;
[0140] The chromaticity signal S is derived from S = X f -αY f It is calculated that α is the sum of two signals X. f and Y f The ratio of the standard deviations of X, i.e., α = σ(X) f ) / σ(Y f ), X s and Ys It is a fusion of normalized RGB pulse signals after motion compensation, X s and Y s The calculation function is as follows:
[0141] X s =3R n -2G n
[0142] Y s =1.5R n +G n -1.5B n
[0143] Among them, R n G n and B n The color signal for each sub-region;
[0144] The signal-to-noise ratio function for each sub-region is as follows:
[0145]
[0146] Among them, S f Let S represent the spectral values after the Fourier transform of the chrominance signal S, f represent the frequency limiting the heart rate, and the range of f is [0.6, 4]. t This indicates the size of the sliding window used for calculation.
[0147] In the heart rate signal reconstruction module:
[0148] The method for obtaining X is as follows: using the cosine basis D c And Xiaobo D w Construct a mixed dictionary D,
[0149]
[0150] D w = WaveDict(Short3, N b (j, b)
[0151] Where, k i For the i-th element in the discrete cosine dictionary th The frequency of an atom, k i With k i+1 The difference is F r For video frame rate, N b Let j be the length of the generated signal, b be the level vector, and f be the shift factor. L The length of the generated signal sequence,
[0152] The hybrid dictionary model is as follows:
[0153]
[0154] To solve for the coefficient matrix, a sparse structure method is used to select appropriate dictionary atoms from the mixed dictionary to represent the pulse signal. A greedy algorithm is then used for sparse coding. Specifically, solving for the coefficient matrix involves solving the following model:
[0155] The sparse coding is obtained, forming the coefficient matrix A.
[0156] Where λ is used to balance the signal reconstruction accuracy. And penalty item ||A|| 1，2 The regularization parameter;
[0157] The heart rate signal is reconstructed using the inner product of a mixed dictionary D and a coefficient matrix A. The inner product model is as follows:
[0158] X = D·A;
[0159] The heart rate signal is obtained by averaging the reconstructed heart rate signals from each sub-region, specifically as follows:
[0160]
[0161] Where N is the total number of selected sub-regions, x n This represents the heart rate signal for the selected sub-region.
[0162] In the heart rate acquisition module:
[0163] Power spectrum analysis was performed on the averaged heart rate signal, and the component with the highest peak value was selected as the main component of the heart rate signal, i.e., the frequency f corresponding to the component with the highest power was used. HR Finally, the time-frequency conversion is performed to obtain the final heart rate HR, specifically: HR = f HR *60.
[0164] Example 3
[0165] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the remote non-contact heart rate estimation method based on sparse structure representation.
Claims
1. A remote non-contact heart rate estimation method based on sparse structure representation, characterized in that, Includes the following steps: Step 1: Input a face video, fix the forehead and cheeks of the detected face as the region of interest, and divide it into blocks to obtain multiple sub-regions; Step 2: Construct the average RGB pulse signal for each sub-region, perform motion compensation on the RGB pulse signal, and calculate the chromaticity signal using a linear combination of different RGB channels. Based on this chromaticity signal Calculate the signal-to-noise ratio (SNR) of each sub-region. Sub-regions with an SNR greater than the overall SNR are defined as high-quality sub-regions, and their corresponding chroma signals are defined as high-quality chroma signals. Generate high-quality sub-region chroma signals. ; Step 3: Analyze the actual heart rate signal. The heart rate signal was reconstructed by performing sparse structure representation. High-quality sub-region chroma signal It consists of chroma signals from sub-regions with high signal-to-noise ratio, i.e. ,in, This represents the chroma signal of the selected high-quality sub-region. This represents the reconstruction of the heart rate signal from a selected high-quality sub-region. The method for obtaining it is as follows: a hybrid structure dictionary is constructed using cosine bases of different frequencies and wavelet bases of different scales. Solve the coefficient matrix The heart rate signal is represented by appropriate dictionary atoms from a mixed dictionary using a sparse structure method, and sparse encoding is performed using a greedy algorithm to construct a coefficient matrix. A dictionary with a hybrid structure is used. With coefficient matrix The inner product method reconstructs the heart rate signal, i.e. ; In step 3, The method for obtaining it is as follows: using the cosine basis And Xiaobo Constructing a mixed dictionary , ； ； in, For the first in the discrete cosine dictionary The frequency of atoms, and The difference is , For video frame rate, The length of the generated signal, For level vectors, The translation factor is... The length of the generated signal sequence, The hybrid dictionary model is as follows: ； To solve for the coefficient matrix, a sparse structure method is used to select appropriate dictionary atoms from the mixed dictionary to represent the pulse signal. A greedy algorithm is then used for sparse coding. Specifically, solving for the coefficient matrix involves solving the following model: This yields sparse coding and forms a coefficient matrix. , in, To balance signal reconstruction accuracy and penalty items The regularization parameter; Step 4: Average the reconstructed heart rate signals from each sub-region, then perform power spectrum analysis to obtain the heart rate. .
2. The remote non-contact heart rate estimation method based on sparse structure representation according to claim 1, characterized in that, In step 2, the average value, absolute deviation, and standard deviation of the RGB pulse signal for each sub-region are calculated. Then, the deviation of each atom from the average value is calculated. Atoms that exceed both the average value and the standard deviation are compensated by adding the average value and the standard deviation; atoms that are below the average value and the standard deviation are compensated by subtracting the standard deviation from the average value. The specific definitions are as follows: ； ； ； ； in For the number of channels, =1, 2, and 3 represent the RGB channel signals, specifically... =1, which represents the R channel; =2, which represents the G channel; =3 is channel B. For signal length, for The first moment The signal length of the channel, for The channel signal is averaged. for The average signal deviation of the channel. for The standard deviation of the channel signal.
3. The remote non-contact heart rate estimation method based on sparse structure representation according to claim 1, characterized in that, In step 2, the chromaticity signal Depend on Calculated, of which There are two signals. and The ratio of the standard deviations of , i.e. , and It is a fusion of normalized RGB pulse signals after motion compensation. and The calculation function is as follows: ； ； in, , and The color signal for each sub-region; The signal-to-noise ratio function for each sub-region is as follows: ； in, Represents chromaticity signal The spectrum value after Fourier transform, This indicates the restriction on heart rate. The range is [0.6, 4]. This indicates the size of the sliding window used for calculation.
4. The remote non-contact heart rate estimation method based on sparse structure representation according to claim 1, characterized in that, In step 3, the heart rate signal is reconstructed using a hybrid dictionary. With coefficient matrix The inner product method reconstructs the heart rate signal; the inner product model is as follows: 。 5. The remote non-contact heart rate estimation method based on sparse structure representation according to claim 4, characterized in that, The heart rate signal is obtained by averaging the reconstructed heart rate signals from each sub-region, specifically as follows: ； in, The total number of selected sub-regions. This represents the heart rate signal for the selected sub-region.
6. The remote non-contact heart rate estimation method based on sparse structure representation according to claim 1, characterized in that, In step 4, power spectrum analysis is performed on the averaged heart rate signal, and the component with the highest peak value is selected as the main component of the heart rate signal, that is, the frequency corresponding to the component with the highest power is used. Finally, the time-frequency data is converted to the time domain to obtain the final heart rate. Specifically: .
7. A remote non-contact heart rate estimation system based on sparse structure representation, characterized in that, include: The sub-region acquisition module is used to input a face video, fix the forehead and cheek parts of the detected face as the region of interest, and divide it into blocks to obtain multiple sub-regions; A high-quality sub-region chromaticity signal generation module is used to construct the average RGB pulse signal for each sub-region, perform motion compensation on the RGB pulse signal, and calculate the chromaticity signal using a linear combination of different RGB channels. Based on this chromaticity signal Calculate the signal-to-noise ratio (SNR) of each sub-region. Sub-regions with an SNR greater than the overall SNR are defined as high-quality sub-regions, and their corresponding chroma signals are defined as high-quality chroma signals. Generate high-quality sub-region chroma signals. ; The heart rate signal reconstruction module is used to reconstruct the actual heart rate signal. The heart rate signal was reconstructed by performing sparse structure representation. High-quality sub-region chroma signal It consists of chroma signals from sub-regions with high signal-to-noise ratio, i.e. ,in, This represents the chroma signal of the selected high-quality sub-region. This represents the reconstruction of the heart rate signal from a selected high-quality sub-region. The method for obtaining it is as follows: a hybrid structure dictionary is constructed using cosine bases of different frequencies and wavelet bases of different scales. Solve the coefficient matrix The heart rate signal is represented by appropriate dictionary atoms from a mixed dictionary using a sparse structure method, and sparse encoding is performed using a greedy algorithm to construct a coefficient matrix. A dictionary with a hybrid structure is used. With coefficient matrix The inner product method reconstructs the heart rate signal, i.e. ; In the heart rate signal reconstruction module, The method for obtaining it is as follows: using the cosine basis And Xiaobo Constructing a mixed dictionary , ； ； in, For the first in the discrete cosine dictionary The frequency of atoms, and The difference is , For video frame rate, The length of the generated signal, For level vectors, The translation factor is... The length of the generated signal sequence, The hybrid dictionary model is as follows: ； To solve for the coefficient matrix, a sparse structure method is used to select appropriate dictionary atoms from the mixed dictionary to represent the pulse signal. A greedy algorithm is then used for sparse coding. Specifically, solving for the coefficient matrix involves solving the following model: This yields sparse coding and forms a coefficient matrix. , in, To balance signal reconstruction accuracy and penalty items The regularization parameter; The heart rate acquisition module is used to average the reconstructed heart rate signals from each sub-region, followed by power spectrum analysis to obtain the heart rate. .
8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the remote non-contact heart rate estimation method based on sparse structure representation as described in any one of claims 1-6.