Non-contact heart rate measurement method based on frequency domain learning
By employing frequency domain learning methods and utilizing three-dimensional convolutional neural networks and spectral adaptive learning, the inconvenience of contact heart rate monitoring and the accuracy issues of existing non-contact methods in complex environments are resolved, achieving highly accurate and robust non-contact heart rate measurement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-25
- Publication Date
- 2026-04-14
AI Technical Summary
Existing contact heart rate monitoring technologies are complex to operate, uncomfortable to wear, and have poor accuracy and robustness in complex environments. Non-contact heart rate monitoring methods have decreased accuracy when there are changes in light and when the user is moving. End-to-end deep learning models have difficulty robustly identifying the periodic patterns of heart rate.
A frequency domain learning-based approach is adopted to extract spatiotemporal feature maps through a three-dimensional convolutional neural network and perform spectral adaptive learning. Frequency domain analysis is used to enhance heart rate-related features and suppress irrelevant noise, thereby achieving non-contact heart rate measurement.
It improves the accuracy and robustness of heart rate measurement and reduces computational complexity in complex scenarios such as changes in lighting and user movement. It is suitable for mobile devices and has good cross-scenario generalization ability.
Smart Images
Figure CN121845547A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a non-contact heart rate measurement method based on frequency domain learning. Background Technology
[0002] Traditional heart rate monitoring technologies heavily rely on contact sensors. For example, electrocardiograms require multiple electrodes to be deployed on the skin to capture the heart's electrophysiological activity, which is complex and severely restricts user movement. Contact heart rate monitoring has the following drawbacks: it must be in close contact with the skin, which can cause pressure discomfort with prolonged wear, making it unsuitable for people with fragile skin (such as newborns or burn patients); it is also extremely sensitive to motion artifacts, as user limb movements can easily cause sensor displacement or pressure changes, resulting in strong noise interference that severely reduces signal quality and measurement reliability.
[0003] The heart's pumping activity is periodic, causing corresponding periodic changes in blood volume in the blood vessels under the skin. These optical changes caused by blood volume pulses (BVP) result in extremely subtle periodic color variations in skin areas such as the face and palms. These changes are imperceptible to the naked eye but can be captured by consumer-grade cameras (such as smartphone cameras). Remote photoplethysmography (rPPG) extracts and amplifies these subtle color changes in skin pixels from the video stream through image processing and signal analysis, reconstructing the blood volume pulses. Furthermore, by analyzing the spectrum or detecting the peak value of the blood volume pulse signal, physiological indicators such as heart rate can be accurately calculated, enabling non-contact heart rate monitoring.
[0004] Early rPPG methods primarily relied on signal processing techniques to separate the BVP signal from a mixed signal. Representative techniques included methods employing Principal Component Analysis (PCA) or Independent Component Analysis (ICA), as well as CHROM and POS methods utilizing the optical properties of skin. The performance of these traditional methods heavily depended on strict prior assumptions, such as constant illumination, static background, and essentially stationary head. In the complex environments of the real world, factors such as changes in illumination and minute user movements can severely interfere with the signal, leading to a significant decrease in the accuracy of these methods and greatly limiting their robustness and generalization ability.
[0005] Currently, end-to-end deep learning models have become the most cutting-edge and mainstream technology in the rPPG field. These models take the raw video pixels as input and automatically learn to reconstruct the BVP signal using structures such as convolutional neural networks (CNN), recurrent neural networks (RNN), or more advanced Transformers.
[0006] However, current models are insufficient in modeling the "quasi-periodic" nature of BVP signals. CNN-based models, due to the limitations of their local receptive fields in convolutional operations, can only learn local morphological features of the BVP waveform, making it difficult to establish long-range temporal dependencies between multiple cardiac cycles from a global perspective. This results in an inability to adequately model the rhythmicity of the signal, leading to poor accuracy in heart rate estimation. Transformer-based models, on the other hand, suffer from challenges. Firstly, their quadratic complexity is insufficient for practical applications. Secondly, their global attention in the temporal domain is easily interfered with by non-periodic noise in the video (such as instantaneous facial expressions and lighting fluctuations), resulting in "dispersed and unstable" temporal correlations learned. This makes it difficult for the model to robustly identify true periodic patterns from mixed signals, thus affecting its generalization ability. Summary of the Invention
[0007] This invention proposes a non-contact heart rate measurement method based on frequency domain learning. Starting from the physiological rhythmic characteristics of the blood volume pulse signal, this method performs adaptive spectral energy learning on the spatiotemporal feature map extracted using a three-dimensional convolutional neural network, and adaptive spectral learning on the extracted spatiotemporal feature map. This makes the model more focused on heart rate-related features, achieving more accurate and better generalization performance in heart rate estimation.
[0008] The present invention proposes a non-contact heart rate measurement method based on frequency domain learning, the specific steps of which are as follows:
[0009] (1) Collect human skin video containing physiological information through a consumer-grade camera; preprocess the video to obtain differential frame segments of the video image: use a detection algorithm to detect regions of interest (ROI) rich in physiological information frame by frame and crop out the ROI; then, obtain differential frame sequence through differential operation and data standardization to suppress static noise introduced by factors such as lighting and skin color; finally, segment the differential frame sequence to obtain several differential frame segments, each of which can be directly input into the trained end-to-end frequency domain learning deep network neural model;
[0010] (2) Feed the differential frame fragments obtained in step (1) into the trained end-to-end frequency domain learning deep neural network model to obtain differential blood volume pulse (dBVP) fragments: The model includes a feature extraction network, a spatial feature redistribution module, a spectrum adaptive learning network, and a differential blood volume pulse (dBVP) signal prediction network.
[0011] The feature extraction network extracts a three-dimensional spatiotemporal feature map containing physiological information from the differential frame fragments;
[0012] The spatial feature redistribution module performs spatial redistribution, embedding the spatial information of the three-dimensional spatiotemporal feature map into the channel dimension to generate a time-channel two-dimensional feature map rich in channel semantics.
[0013] The spectrum adaptive learning network is used to enhance the features of the time-channel two-dimensional feature map: by utilizing the quasi-periodic prior of the blood volume pulse (BVP) signal, the time-domain features are transformed to the frequency domain for adaptive adjustment, enhancing the frequency components related to the true heart rate and suppressing irrelevant noise, and then inversely transformed back to the time domain to obtain the enhanced feature map.
[0014] The differential blood volume pulse (dBVP) signal prediction network estimates the dBVP segment corresponding to the differential frame segment based on the enhanced feature map;
[0015] (3) Based on the time order of the differential frame segments, the dBVP segments obtained in step (2) are spliced together sequentially to obtain the dBVP signal of the corresponding video; finally, the final blood volume pulse (BVP) signal is obtained by performing time-domain integration on the signal.
[0016] (4) After bandpass filtering the blood volume pulse (BVP) signal obtained in step (3), non-contact heart rate measurement of an individual is achieved through signal analysis methods.
[0017] In this invention, step (1) uses a consumer-grade camera to capture the face region, the video capture is greater than 20 frames / second, and the recording duration is greater than 15 seconds; the video is a video containing the human face region.
[0018] In this invention, the detection algorithm in step (1) is selected from any one of the Haar Cascade algorithm, YOLO algorithm, or key point detection algorithm to crop out the facial region rich in physiological information; the difference operation calculates the pixel difference between the next frame image and the current frame image to highlight the temporal change information between adjacent frames; the data standardization process adopts any one of mean-variance normalization (Z-score Normalization), maximum-minimum normalization (Min-MaxNormalization), or robust normalization; the frame sequence segmentation refers to dividing a continuous video frame sequence into segments of fixed length to adapt to the input dimension of the model while preserving its temporal order.
[0019] In this invention, the feature extraction network described in step (2) is composed of at least one stacked basic processing unit, which includes a three-dimensional convolutional layer (3D Conv), a batch normalization layer (Batch Normalization), and a nonlinear activation function layer; wherein, only spatial pooling is used between network layers for downsampling to gradually reduce the spatial resolution of the feature map.
[0020] In this invention, the spatial feature redistribution module in step (2) keeps the temporal dimension of the feature map unchanged, flattens and merges the channel dimension, height dimension, and width dimension, and reorganizes them into a new channel dimension; that is, it extracts features to obtain a spatiotemporal feature map of size T×C×H×W, where: T is the number of frames in the time dimension, C is the original number of channels, H×W is the spatial dimension, and the transformation relationship satisfies:
[0021]
[0022] Where D = C × H × W is the channel dimension of the new feature map, which integrates the control and channel semantic information of the original features.
[0023] In this invention, the spectrum adaptive learning network described in step (2) includes at least one spectrum adaptive learning module, and a spectrum adaptive learning module includes a spectrum decoupling unit and an energy constraint unit.
[0024] The spectrum decoupling unit includes a fully connected neural network layer with a multi-branch architecture to process the input two-dimensional feature map; the multi-branch architecture uses periodic activation functions and nonlinear activation functions in parallel, wherein: the periodic activation functions are specifically sine (sin) and cosine (cos) functions, which model the periodic characteristics of the BVP signal and realize the fusion of channel features;
[0025] The energy constraint unit first transforms the processed feature map into the frequency domain in the time dimension using Discrete Fourier Transform (DFT); then it learns energy constraint coefficients using at least two linear layers in the frequency domain, which are multiplied with the input frequency domain features as weights to adaptively enhance heart rate-related features and suppress irrelevant noise; finally, it transforms the adjusted frequency domain feature map back into the time domain using Inverse Discrete Fourier Transform (IDFT) to obtain the enhanced feature map.
[0026] The frequency domain linear layer is designed for complex number features. Following the rules for complex number calculation, the frequency domain linear layer contains two weight matrices and two biases, corresponding to the calculation of the real and imaginary parts of the complex number weights, respectively. The calculation method is as follows:
[0027] H = ZW + B = (z r w r -z i wi +b r )+j(z r w i +z i w r +b i )
[0028] Where H is the output complex feature of the frequency domain linear layer, Z is the input complex feature of the frequency domain linear layer, and W and B are the complex weight matrix and complex bias of the frequency domain linear layer, respectively. The ModReLU activation function is used between the frequency domain linear layers to suppress low-energy irrelevant frequency bands. The ModReLU activation function is a modulus-length linear rectified function applied to the complex domain to calculate the modulus of the complex number. When the modulus value is less than a certain threshold, the complex number is set to zero. The calculation method is as follows:
[0029]
[0030] Where K is a complex feature in the frequency domain, and b is a learnable parameter representing the threshold for suppressing the modulus.
[0031] In this invention, the bandpass filter in step (4) is a Butterworth filter, and its passband cutoff frequency is set to 0.7-3.3 Hz; the signal analysis method is selected from one or more of Fourier transform, power spectral density or peak detection.
[0032] The present invention has the following beneficial effects:
[0033] 1. This invention transforms complex spatiotemporal feature maps into two-dimensional feature maps focused on the time dimension through unique spatial feature redistribution, effectively reducing the complexity of model processing and enabling global frequency domain analysis to focus more on extracting core physiological rhythm information;
[0034] 2. By shifting the core of signal processing from the time domain to the frequency domain and introducing a spectral decoupling unit containing a periodic activation function, this invention facilitates a more explicit and fundamental modeling of the quasi-periodic characteristics of blood volume pulse signals.
[0035] 3. This invention utilizes an energy-constrained unit to adaptively enhance heart rate-related features and suppress irrelevant noise in the frequency domain, significantly improving the signal-to-noise ratio of the signal, thereby ensuring high accuracy and robustness of measurement results in complex real-world scenarios such as changes in lighting and slight movements;
[0036] 4. This invention achieves an operation equivalent to global convolution in the time domain with low computational complexity by learning in the frequency domain space. It can efficiently capture the long-range temporal dependencies of signals, which is beneficial for deployment on mobile and edge computing devices and ensures high performance.
[0037] 5. This invention focuses on the essential characteristics of signals in the frequency domain, and the trained model is not easily affected by domain offset, ensuring generalization ability across datasets and scenarios, and providing reliable measurement results;
[0038] 6. The method of the present invention provides a completely non-contact physiological signal measurement scheme that relies solely on ordinary consumer-grade cameras. It can be easily integrated into various platforms such as smartphones, computers, and smart homes, which is conducive to achieving continuous and unobtrusive health status monitoring and has broad application prospects in the fields of telemedicine and personalized health management. Attached Figure Description
[0039] Figure 1 This is a schematic diagram of the original image and the preprocessed difference image. (a) The original image of a certain frame of the video; (b) The image of the face region cropped out using the face detection algorithm; (c) The difference image obtained after "differential-standardization", that is, the pixel difference between the next frame image and the current frame image, and standardized using the mean-variance method.
[0040] Figure 2 This is a flowchart of non-contact heart rate monitoring based on the frequency domain in Embodiment 1 of the present invention.
[0041] Figure 3 The specific structure of the frequency domain adaptive learning module in Embodiment 1 of the present invention includes a frequency domain decoupling unit and an energy constraint unit.
[0042] Figure 4 Examples of the predicted and actual BVP signal values in the time and frequency domains in Embodiment 1 of this invention are shown below. Solid lines represent predicted values, and dashed lines represent actual values. (a) Frequency domain comparison of predicted and actual values in Example 1 of Embodiment 1; (b) Time domain comparison of predicted and actual values in Example 1 of Embodiment 1; (c) Frequency domain comparison of predicted and actual values in Example 2 of Embodiment 1; (d) Time domain comparison of predicted and actual values in Example 2 of Embodiment 1. Detailed Implementation
[0043] The method and its application of the present invention will be further described below with reference to the accompanying drawings and embodiments. These embodiments do not limit the present invention; structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are all included within the scope of protection of the present invention.
[0044] Example 1: The performance of the non-contact heart rate measurement method based on frequency domain learning of the present invention was tested on a public database. This example selected two internationally published rPPG standard datasets: UBFC-rPPG and PURE. These datasets contain facial videos of different subjects in different scenarios (e.g., still, speaking, head movement), and simultaneously record physiological signals measured by contact sensors as the gold standard. The specific steps of the non-contact heart rate measurement method based on frequency domain learning of the present invention are as follows:
[0045] (1) As Figure 1 (a) is the original image of the video capture. The video was captured at 30 frames per second. The video capture scene was close to natural light to avoid being too bright or too dark. The video image mainly contains the face area. The video sampling time is about one minute.
[0046] (2) Figure 1 As shown in (b), the Haar Cascade algorithm is first used to detect faces, and the face region is cropped and its resolution adjusted to 72×72 pixels. Then, a difference operation is performed by subtracting the previous frame from the next frame of the video, and the differenced image is standardized using mean-variance, resulting in the image shown below. Figure 1 (c) shows that the normalized differential frame sequence is divided into fixed-length spatiotemporal segments, each segment having T frames and a size of T×3×72×72 (corresponding to the number of frames, number of channels, height and width respectively).
[0047] (3) Input the differential frame fragments obtained in step (2) into the neural network to predict the differential BVP signal. As shown in Figure (2), the processing flow first goes through a feature extraction network, which is composed of a basic processing unit consisting of a three-dimensional convolutional layer, a normalization layer, an activation function layer, and a stacked pooling layer. The convolution kernel of the three-dimensional convolutional layer is (3, 3, 3), the activation function layer uses the hyperbolic tangent function, and spatial pooling is used to gradually reduce the spatial dimension to obtain a spatiotemporal feature map with a size of T×64×4×4. Then, through the spatial feature redistribution module, the spatial dimension information of the feature map is embedded into the channel dimension to obtain a feature map with a size of T×1024. Next, the feature map is input into the spectrum adaptive learning network, and through the stacking of multiple frequency domain adaptive learning modules, an enhanced feature map with a size of T×64 is output. Finally, the dBVP signal prediction network outputs a dBVP signal of length T based on the enhanced feature map.
[0048] (4) As shown in Figure (3), the frequency domain adaptive learning module includes a frequency domain decoupling unit and an energy constraint unit. The frequency domain decoupling unit uses periodic activation function branches and aperiodic activation function branches. The features of each branch channel focus on different frequencies, which helps with frequency domain decoupling. The periodic activation function uses sine and cosine functions, and the aperiodic activation function uses hyperbolic tangent (tanh) function. The energy constraint unit uses two frequency domain linear layers to learn the energy constraint coefficients. The ModReLU function is used as the activation function between the two layers to suppress low-energy frequency bands and enhance heart rate-related frequency bands. The conversion between the time domain and the frequency domain is achieved through discrete Fourier transform and inverse discrete Fourier transform.
[0049] (5) The predicted differential BVP signals are spliced together in time order and the blood volume pulse (BVP) signal is reconstructed through integration. Then, a Butterworth bandpass filter with a cutoff frequency of 0.7Hz-3.3Hz is applied for noise reduction. Then, the signal is divided into 10-second windows and the discrete Fourier transform is applied for spectrum analysis. The frequency corresponding to the point with the highest energy in the spectrum is extracted as the dominant frequency f of the time period. According to the formula HR=f×60, the signal is converted into heart rate value (BPM).
[0050] (6) In this embodiment, heart rate estimation experiments were conducted on two public datasets, UBFC-rPPG and PURE. Specific performance metrics are shown in Table 1. The metrics used are mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE). Lower values for MAE, RMSE, and MAPE indicate better performance. MAE and RMSE are expressed in beats per minute (bpm). The table mainly includes four experiments: two in-dataset experiments to measure the model's fitting performance, and two cross-dataset experiments to measure the model's generalization performance. The in-dataset experiments used a 7:3 ratio to divide the training and test sets, while the cross-dataset experiments used one dataset as the training set and the other as the test set. The results show that the model in this embodiment has a mean absolute error of less than 1 bpm in both the in-dataset and cross-dataset experiments, achieving high-precision non-contact heart rate measurement and demonstrating excellent cross-scenario generalization ability.
[0051] Table 1 shows the performance of the non-contact heart rate measurement method based on frequency domain learning on two public datasets, both within and across datasets. The evaluation metrics are mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE).
[0052] Table 1:
[0053]
[0054] Figure 4This section presents the comparison results between the predicted BVP signal and the actual value in the frequency and time domains. Specifically, (a) and (b) correspond to Example 1, and (c) and (d) correspond to Example 2; (a) and (c) are frequency domain comparison graphs, and (b) and (d) are time domain comparison graphs. The visualization results show that the predicted curve and the actual curve are in high agreement, indicating that the frequency domain learning model constructed in this example can accurately extract the blood volume pulse signal from the video and achieve high-precision non-contact heart rate estimation.
Claims
1. A non-contact heart rate measurement method based on frequency domain learning, characterized in that, The specific steps are as follows: (1) Collect human skin video containing physiological information through a consumer-grade camera; preprocess the video to obtain differential frame segments of the video image: use a detection algorithm to detect regions of interest (ROI) rich in physiological information frame by frame and crop out the ROI; then, obtain differential frame sequence through differential operation and data standardization to suppress static noise introduced by factors such as lighting and skin color; finally, segment the differential frame sequence to obtain several differential frame segments, each of which can be directly input into the trained end-to-end frequency domain learning deep network neural model; (2) Feed the differential frame fragments obtained in step (1) into the trained end-to-end frequency domain learning deep neural network model to obtain differential blood volume pulse (dBVP) fragments: The model includes a feature extraction network, a spatial feature redistribution module, a spectrum adaptive learning network, and a differential blood volume pulse (dBVP) signal prediction network. The feature extraction network extracts a three-dimensional spatiotemporal feature map containing physiological information from the differential frame fragments; The spatial feature redistribution module performs spatial redistribution, embedding the spatial information of the three-dimensional spatiotemporal feature map into the channel dimension to generate a time-channel two-dimensional feature map rich in channel semantics. The spectrum adaptive learning network is used to enhance the features of the time-channel two-dimensional feature map: by utilizing the quasi-periodic prior of the blood volume pulse (BVP) signal, the time-domain features are transformed to the frequency domain for adaptive adjustment, enhancing the frequency components related to the true heart rate and suppressing irrelevant noise, and then inversely transformed back to the time domain to obtain the enhanced feature map. The differential blood volume pulse (dBVP) signal prediction network estimates the dBVP segment corresponding to the differential frame segment based on the enhanced feature map; (3) Based on the time order of the differential frame segments, the dBVP segments obtained in step (2) are spliced together sequentially to obtain the dBVP signal of the corresponding video; finally, the final blood volume pulse (BVP) signal is obtained by performing time-domain integration on the signal. (4) After bandpass filtering the blood volume pulse (BVP) signal obtained in step (3), non-contact heart rate measurement of an individual is achieved through signal analysis methods.
2. The method according to claim 1, characterized in that, Step (1) Use a consumer-grade camera to capture the face area. The video capture rate is greater than 20 frames per second and the recording duration is greater than 15 seconds. The video is a video that includes the human face area.
3. The method according to claim 1, characterized in that, The detection algorithm in step (1) is selected from any one of the Haar Cascade algorithm, YOLO algorithm, or key point detection algorithm to crop out the facial region rich in physiological information; the difference operation calculates the pixel difference between the next frame image and the current frame image to highlight the temporal change information between adjacent frames; the data standardization process adopts any one of mean-variance normalization (Z-score Normalization), maximum-minimum normalization (Min-Max Normalization), or robust normalization; the frame sequence segmentation refers to dividing the continuous video frame sequence into segments of fixed length to adapt to the input dimension of the model, while preserving its temporal order.
4. The method according to claim 1, characterized in that, The feature extraction network described in step (2) consists of at least one stacked basic processing unit, which includes a three-dimensional convolutional layer (3DConv), a batch normalization layer, and a non-linear activation function layer; wherein, only spatial pooling is used between network layers for downsampling to gradually reduce the spatial resolution of the feature map.
5. The method according to claim 1, characterized in that, The spatial feature redistribution module in step (2) keeps the temporal dimension of the feature map unchanged, flattens and merges the channel dimension, height dimension, and width dimension, and reorganizes them into a new channel dimension; that is, it extracts features to obtain a spatiotemporal feature map of size T×C×H×W, where: T is the number of frames in the time dimension, C is the original number of channels, H×W is the spatial dimension, and the transformation relationship satisfies: Where D = C × H × W is the channel dimension of the new feature map, which integrates the control and channel semantic information of the original features.
6. The method according to claim 1, characterized in that, The spectrum adaptive learning network described in step (2) includes at least one spectrum adaptive learning module. A spectrum adaptive learning module includes a spectrum decoupling unit and an energy constraint unit. The spectrum decoupling unit includes a fully connected neural network layer with a multi-branch architecture to process the input two-dimensional feature map; the multi-branch architecture uses periodic activation functions and nonlinear activation functions in parallel, wherein: the periodic activation functions are specifically sine (sin) and cosine (cos) functions, which model the periodic characteristics of the BVP signal and realize the fusion of channel features; The energy constraint unit first transforms the processed feature map into the frequency domain in the time dimension using Discrete Fourier Transform (DFT); then it learns energy constraint coefficients using at least two linear layers in the frequency domain, which are multiplied with the input frequency domain features as weights to adaptively enhance heart rate-related features and suppress irrelevant noise; finally, it transforms the adjusted frequency domain feature map back into the time domain using Inverse Discrete Fourier Transform (IDFT) to obtain the enhanced feature map. The frequency domain linear layer is designed for complex number features. Following the rules for complex number calculation, the frequency domain linear layer contains two weight matrices and two biases, corresponding to the calculation of the real and imaginary parts of the complex number weights, respectively. The calculation method is as follows: H=ZW+B=(z r w r -z i w i +b r )+j(z r w i +z i w r +b i ) Where H is the output complex feature of the frequency domain linear layer, Z is the input complex feature of the frequency domain linear layer, and W and B are the complex weight matrix and complex bias of the frequency domain linear layer, respectively. The ModReLU activation function is used between the frequency domain linear layers to suppress low-energy irrelevant frequency bands. The ModReLU activation function is a modulus-length linear rectified function applied to the complex domain to calculate the modulus of the complex number. When the modulus value is less than a certain threshold, the complex number is set to zero. The calculation method is as follows: Where K is a complex feature in the frequency domain, and b is a learnable parameter representing the threshold for suppressing the modulus.
7. The method according to claim 1, characterized in that, The bandpass filter mentioned in step (4) is a Butterworth filter, and its passband cutoff frequency is set to 0.7-3.3 Hz; the signal analysis method is selected from one or more of Fourier transform, power spectral density or peak detection.