A lightweight real-time heart rate monitoring model and heart rate monitoring method

Abstract

The present invention relates to the technical field of the intersection of computer vision and biomedical sensing, and specifically provides a lightweight real-time heart rate monitoring model and method. The model includes a video preprocessing module for standardizing input video preprocessing, a lightweight dual-branch network module for parallel processing of spatiotemporal features and motion compensation branches on video images, a heart rate calculation module that receives heart rate prediction-related features output by the lightweight dual-branch network module and calculates heart rate, and a multi-task loss function module that considers cascade optimization of multiple losses. The model is trained using training data. After training, lightweight real-time heart rate monitoring can be implemented using the model, and heart rate can be predicted based on real-time images captured by a camera. Based on rPPG technology, the present invention achieves high-precision real-time heart rate estimation through a deep learning model.

Description

Technical Field

[0001] The present invention belongs to the technical field of intersection of computer vision and biomedical sensing, and in particular relates to a lightweight real-time heart rate monitoring model and a heart rate monitoring method. Background Art

[0002] Traditional heart rate monitoring relies primarily on contact sensors, such as electrocardiogram (ECG) devices and pulse oximeters. While these devices can provide relatively accurate measurements in clinical settings, they have significant drawbacks. First, they are invasive and can cause skin discomfort when worn for extended periods. For example, after wearing ECG electrodes continuously for several hours, skin irritation and redness may occur, impacting the user experience and the continuity of monitoring. Second, their use cases are limited. During exercise, body movement and sweating can easily interfere with the signal, leading to inaccurate measurements. In environments with frequent lighting changes, such as transitioning from bright indoor lighting to natural outdoor lighting, pulse oximeter accuracy can be severely affected. Third, the high cost of specialized medical equipment makes it difficult to widely adopt in ordinary homes, schools, gyms, and other settings, limiting the widespread adoption of heart rate monitoring technology.

[0003] Camera-based rPPG technology has garnered significant attention in recent years. It extracts heart rate signals by analyzing changes in light reflected from the skin, opening up a new path for contactless heart rate monitoring. However, the technology currently faces numerous challenges. For one thing, facial micro-movements (such as talking, chewing, and changes in facial expression) and ambient light fluctuations (such as flickering indoor lights and varying outdoor sunlight intensity) can easily distort the extracted heart rate signal. These micro-movements can interfere with changes in light reflected from the skin, making it difficult to accurately isolate the heart rate-related signal. Ambient light fluctuations can also alter the light intensity captured by the camera, affecting signal quality. Furthermore, traditional rPPG algorithms suffer from poor generalization. Methods such as independent component analysis (ICA) and CHROM rely heavily on handcrafted features and are unable to adapt to complex and changing real-world scenarios, such as those involving varying skin tones, lighting conditions, and facial poses, resulting in a significant decrease in monitoring accuracy. Furthermore, while some complex deep learning-based models (such as 3D-CNN) demonstrate high accuracy on specific datasets, they are computationally intensive, making them difficult to run in real time on resource-constrained devices like mobile devices, thus failing to meet user needs for real-time monitoring.

[0004] Currently, existing camera-based rPPG technologies mainly include signal processing methods, supervised learning methods and related open source tools.

[0005] The POS algorithm is a representative signal processing method. Based on the principles of physiological optics, this algorithm constructs a projection plane orthogonal to skin tone and projects the spatially normalized and averaged pixel values ​​onto this plane to recover the photoplethysmography (PPG) waveform and extract the heart rate signal. However, in practical applications, when the face undergoes dramatic movements, such as rapid head turns or drastic changes in facial expression, the POS algorithm struggles to accurately track changes in skin-reflected light, resulting in significant errors in heart rate monitoring.

[0006] In terms of supervised learning, PhysNet uses a 3D convolutional network to model the spatiotemporal features of videos. Through multiple layers of 3D convolutional and pooling layers, it automatically learns heart rate-related features from videos. However, this model has a high parameter count of 10M, requiring significant computational resources and time for heart rate prediction. For example, running the PhysNet model on a common mobile device for heart rate monitoring can take hundreds of milliseconds or even longer to process a single frame, far from meeting the requirements of real-time monitoring (which typically requires a frame rate of 30 FPS or higher, meaning a processing time of less than 33 milliseconds per frame).

[0007] Regarding open-source tools, the rPPG-Toolbox integrates six supervised and unsupervised algorithms, providing a convenient platform for the research and application of rPPG technology. It supports preprocessing, model training, and evaluation of a variety of public datasets, advancing the rPPG field. However, the tool lacks lightweight models optimized specifically for mobile devices. When running on mobile devices, it cannot fully utilize device performance, making it difficult to achieve efficient real-time heart rate monitoring.

[0008] Computational efficiency is a key factor limiting the application of existing rPPG technology. Deep learning models like PhysNet have enormous parameters and complex structures. When running on mobile devices, they experience significant latency due to limited computing power and memory. This not only fails to meet the requirements of real-time heart rate monitoring but can also lead to data loss or inaccurate results, significantly limiting its practical application.

[0009] At the feature extraction level, traditional methods rely heavily on manually designed color space transformations and feature engineering, making them difficult to adapt to complex and changing real-world scenarios. Differences in skin color and varying lighting conditions in different environments can render manually designed features ineffective, affecting the accuracy of heart rate monitoring.

[0010] Cross-scenario generalization is also a shortcoming of existing rPPG technology. Existing rPPG models are often trained on specific datasets, which have limitations in terms of skin color, lighting, facial motion, and other aspects. Performance degrades significantly when applied to scenes that differ significantly from the training set. For example, a model trained on a dataset primarily composed of indoor Caucasian facial videos will experience significantly increased error when used to monitor heart rate on videos of Asian faces collected outdoors.

[0011] Therefore, there is an urgent need to design a lightweight end-to-end heart rate detection method to overcome the limitations of traditional technologies in computational efficiency, feature extraction, and cross-scenario generalization applications. Summary of the Invention

[0012] In view of this, the present invention aims to provide a lightweight real-time heart rate monitoring model and heart rate monitoring method. By adopting remote photoplethysmography (rPPG), facial videos are collected with the help of an ordinary camera, and high-precision, real-time heart rate estimation is achieved through a deep learning model. A lightweight dual-branch architecture is adopted, and the spatiotemporal features and motion compensation branches are processed in parallel. While ensuring the accuracy of heart rate monitoring, the computational complexity of the model is greatly reduced. The real-time heart rate monitoring frame rate is ≥30FPS, which can meet the user's demand for instant acquisition of heart rate data.

[0013] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0014] The present invention provides a lightweight real-time heart rate monitoring model, comprising:

[0015] The video preprocessing module is used to preprocess the input video, detect and align faces in the input video, and perform spatiotemporal segmentation on the input video to obtain standardized video blocks;

[0016] A lightweight two-branch network module includes a spatiotemporal feature branch and a motion compensation branch, both of which are input to video blocks, and a feature fusion unit. The spatiotemporal feature branch includes multiple layers of separable 3D convolution, each layer of which is used to perform depth convolution on each channel of the input data, and then combine the results of the depth convolution in the channel dimension through spatial convolution to obtain the spatiotemporal features in the video block; the motion compensation branch is used to dynamically capture the motion information of the face, obtain the motion offset field, and perform motion compensation on the features in the video block; the feature fusion unit is used to fuse the outputs of the spatiotemporal feature branch and the motion compensation branch to obtain relevant features for heart rate prediction;

[0017] A heart rate calculation module calculates the heart rate based on the heart rate prediction-related features output by the lightweight dual-branch network module and outputs the predicted heart rate;

[0018] A multi-task loss function module, which includes heart rate loss, noise suppression loss, and regularization loss, and optimizes the model based on these three losses.

[0019] Preferably, the video preprocessing module uses the MTCNN algorithm to perform face detection and alignment on the input video, and crops the facial area into a region of interest with a fixed resolution.

[0020] Preferably, after acquiring the region of interest, the video preprocessing module segments the input video into non-overlapping T-frame sequences in a spatiotemporal manner, and divides each frame image into an N×N grid to extract local spatiotemporal features.

[0021] Preferably, after extracting the local spatiotemporal features in the input video, the method further includes performing dynamic normalization processing on the RGB channels of the input video. The calculation formula for the dynamic normalization processing is:

[0022] ;

[0023] in, represents the normalized pixel value of channel c in the t-th frame, Represents any channel in RGB channels, represents the pixel value of channel c in the t-th frame, represents the pixel mean of channel c in the t-th frame, represents the pixel standard deviation of channel c in the t-th frame, is a constant used to prevent division by zero errors during calculations.

[0024] Preferably, the spatiotemporal feature branch includes 4 layers of separable 3D convolution, the kernel size of each layer of separable 3D convolution is 3×3×3, the step size is 2×2×2, and the operation of the separable 3D convolution is:

[0025] ;

[0026] in, Indicates the Layer separable 3D convolution, Indicates the The layer can separate the output of 3D convolution, Represents a separable 3D convolution operation.

[0027] Preferably, the motion compensation branch includes at least two layers of deformable convolution, which adaptively adjusts the sampling position of the convolution kernel according to the video block to obtain facial motion information. The operation of each layer of deformable convolution is:

[0028] ;

[0029] in, represents the output of deformable convolution, represents the position on the output feature map, Indicates the fixed position of the convolution kernel, that is, in the absence of offset, the first The sampling points should be located at the coordinates on the input feature map, Indicates the location What we learned is relative to The offset, represents the weight of the convolution kernel, Represents the total number of sampling points in the convolution kernel, Represents the input feature map. The feature map of the first layer of deformable convolution input is the standardized video block X, and the feature map of the second layer of deformable convolution input is the output result of the first layer of deformable convolution.

[0030] Preferably, the motion compensation branch uses optical flow estimation to obtain a motion offset field, thereby achieving motion compensation.

[0031] Preferably, the feature fusion unit outputs the final result of the spatiotemporal feature branch The final output of the motion compensation branch The images are concatenated and dimensionality reduced through 1×1 convolution to obtain and output relevant features for heart rate prediction.

[0032] Preferably, the heart rate calculation module obtains a preliminary heart rate prediction value based on the heart rate prediction related features output by the lightweight dual-branch network module, filters the preliminary heart rate prediction value through a second-order Butterworth low-pass filter, and performs peak detection on the filtered signal, calculates the heart rate based on the time interval between adjacent peaks, and obtains a readable and usable predicted heart rate.

[0033] Preferably, the heart rate loss uses the negative Pearson correlation coefficient to measure the correlation between the final predicted heart rate and the actual heart rate. The expression is:

[0034] ;

[0035] in, Indicates the The true heart rate of the training samples, Indicates the The predicted heart rate of training samples, Indicates the number of samples for model training, Indicates the average value of the true heart rate, represents the average value of the predicted heart rate;

[0036] Noise suppression loss The expression is:

[0037] ;

[0038] in, Indicates the predicted heart rate at frequency The energy of

[0039] Regularization loss uses L2 regularization term, regularization loss The expression is:

[0040] ;

[0041] in, represents the regularization coefficient, represents the set of parameters of the model, represents any parameter in the model.

[0042] Another aspect of the present invention is to provide a lightweight real-time heart rate monitoring method, which uses a lightweight real-time heart rate monitoring model to perform heart rate detection and obtain a predicted heart rate.

[0043] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0044] Based on rPPG technology, the present invention designs and implements a lightweight end-to-end neural network model that can predict the heart rate of people in videos with high precision and in real time. During the feature extraction process, a lightweight dual-branch architecture is adopted to design the spatiotemporal feature extraction and motion compensation branches in parallel. This architecture innovatively separates the spatiotemporal feature extraction and motion compensation, allowing the model to more focusedly learn different aspects of information. The spatiotemporal feature branch uses separable 3D convolution to efficiently extract the spatiotemporal features of the video, greatly reducing the computational complexity of the model and the amount of calculation, so that the model can run efficiently on mobile devices, realize real-time heart rate monitoring, and meet users' needs for instant access to heart rate data. Compared with traditional complex models, such as PhysNet, which are difficult to deploy locally on mobile terminals due to computational issues, the present invention greatly improves computational efficiency while ensuring the accuracy of heart rate monitoring, reduces the time to process each frame of video, and can achieve a smooth real-time monitoring effect, achieving a real-time heart rate monitoring frame rate of ≥30FPS.

[0045] The present invention also adds a motion compensation branch to the model. This branch employs an adaptive motion compensation mechanism and utilizes techniques such as deformable convolution to dynamically capture facial motion information, effectively reducing the interference of facial motion on heart rate monitoring and providing support for accurate extraction of heart rate signals. Compared to traditional POS algorithms, deformable convolution overcomes the limitations of the traditional convolution kernel's fixed sampling position when facing intense facial movement. It adaptively adjusts the sampling position based on image content, enabling the model to better adapt to changes in facial movement and more accurately track changes in skin-reflected light, resulting in more stable and accurate extraction of heart rate signals and improving the accuracy and stability of heart rate monitoring results.

[0046] The present invention introduces noise suppression loss to suppress noise in non-heart rate frequency bands by designing a multi-task loss function, and takes heart rate estimation and noise suppression as joint optimization tasks, making full use of the correlation between the two. During the training process, the heart rate loss guides the model to learn accurate heart rate features, the noise suppression loss constrains the model to reduce noise in non-heart rate frequency bands, and the regularization loss prevents the model from overfitting, thereby improving the model's generalization ability in different data sets and complex scenarios. The model of the present invention can better adapt to changes in different skin colors, lighting conditions, and facial movements, and can stably and accurately monitor heart rate in various practical application scenarios, overcoming the problem of significant performance degradation of existing models when applied across scenarios.

[0047] During the heart rate calculation process, the present invention processes the preliminary heart rate prediction value output by the lightweight two-branch network through a second-order Butterworth low-pass filter combined with a peak detection algorithm, effectively removing noise interference and further improving the accuracy and stability of the final output heart rate.

[0048] This invention significantly reduces the amount of model computation required, with model parameters designed to be close to 0.8MB. Peak memory consumption during model operation is estimated to be ≤50MB, placing low demands on mobile device resources, allowing it to be deployed and run locally on mobile devices. Compared to existing models with large parameter counts and high memory usage, this approach significantly reduces the impact on other mobile device functions when running on mobile devices, ensuring smooth device operation and broadening the technology's application scope. It is particularly suitable for resource-constrained mobile devices and embedded systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 This is a modular flow chart of the overall architecture of a lightweight real-time heart rate monitoring model provided according to an embodiment of the present invention;

[0051] Figure 2 is a workflow diagram of a video preprocessing module provided according to an embodiment of the present invention;

[0052] Figure 3 is an architectural diagram of a lightweight dual-branch network module provided according to an embodiment of the present invention;

[0053] Figure 4 is a flowchart of a heart rate calculation module according to an embodiment of the present invention;

[0054] Figure 5 4 is a flowchart of a multi-task loss function module provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention. Similar elements in different embodiments use associated similar element numbers. In the following embodiments, many detailed descriptions are intended to enable the present invention to be better understood. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other elements, materials, or methods. In some cases, some operations related to the present invention are not shown or described in the specification. This is to avoid the core part of the present invention being overwhelmed by too much description. For those skilled in the art, it is not necessary to describe these related operations in detail. They can fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0056] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other to form various implementation methods. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various orders in the description and the drawings are only for the purpose of clearly describing a certain embodiment and are not intended to be a required order, unless otherwise specified that a certain order must be followed.

[0057] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0058] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art can understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0059] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0060] See also Figure 1 In one embodiment of the present invention, a lightweight, real-time heart rate monitoring model is provided to address the problems of traditional heart rate prediction models, such as poor effectiveness, severe noise interference, high computational complexity, and the inability to be deployed on mobile devices with general computing capabilities. The model architecture primarily comprises a video preprocessing module, a lightweight two-branch network module, a heart rate calculation module, and a multi-task loss function module.

[0061] See also Figure 2The video preprocessing module performs standardized preprocessing on the input video. Specifically, the model is deployed on a smartphone (a typical device with general-purpose computing power). 30Hz video is captured via the smartphone camera and used as the input for the heart rate monitoring model. After the 30Hz video captured by the camera is fed into the video preprocessing module, the original input video resolution can vary, as it varies depending on the device, camera, and phone camera parameters. To facilitate subsequent processing, it must be converted to a uniform resolution. Assuming the original input video resolution is a common 1280×720, it is converted to a preset uniform resolution. After the uniform resolution is achieved, the MTCNN (Multi-Task Cascaded Convolutional Neural Network) algorithm is used to perform face detection and alignment on each frame of the input video. The MTCNN algorithm, through a cascade of convolutional neural networks, detects the facial region in each frame of the input video and quickly and accurately locates multiple facial key points, such as the eyes, nose, and mouth. Based on these key points, the facial region is cropped into a fixed-resolution ROI (Region of Interest), typically 128×128 or 256×256. The cropping process typically centers around a key facial point, completely encompassing the key facial parts within the cropped area. This focuses attention on the face, removes background interference, and provides cleaner data for subsequent heart rate signal extraction. This process unifies the diverse resolutions of the original input video and accurately crops the facial region of interest. MTCNN then performs detection and alignment on the cropped facial image, providing high-quality data for subsequent video processing tasks.

[0062] After facial detection and alignment, the cropped image is segmented. Specifically, the cropped and aligned video is segmented into multiple T-frame image sequences, where T=180. That is, each 180 frames (6 seconds) is divided into an input sequence. In addition, during the image sequence segmentation process, overlapping segmentation or non-overlapping segmentation can be used. Non-overlapping segmentation means that each image sequence has no overlapping time. The first sequence: frame 1 to frame 180; the second sequence: frame 181 to frame 360, and so on. This segmentation method ensures that each sequence is independent in the time dimension, facilitating subsequent analysis of the dynamic changes of the heart rate signal. Overlapping segmentation means that each image sequence has overlapping time. In this embodiment of the present invention, the overlap rate of adjacent image sequences is set to 50%. This design can fully utilize the information in the video while ensuring data continuity, avoiding information loss due to segmentation, which makes it difficult for the model to learn the complete trend of heart rate signal changes.

[0063] After obtaining the image sequence through segmentation, each 180-frame sequence is further divided into an N×N grid, where N=8. Each frame is divided into 64 grid regions, achieving spatiotemporal segmentation of the video image. In the temporal dimension, the continuous 180 frames can capture the dynamic changes in the heart rate signal. In the spatial dimension, dividing each frame into an 8×8 grid can capture subtle changes in facial features across different regions. For example, different grid regions may correspond to different parts of the face, such as the forehead and cheeks. Changes in light reflection in these regions may vary in correlation with heart rate, and segmentation allows for more detailed analysis of these features.

[0064] After the spatiotemporal segmentation is completed, the RGB channels of each frame of the video are dynamically normalized. For any t-th frame image, the sum of the pixel values ​​of the R, G, and B channels is calculated respectively, and then the sum is divided by the number of pixels to obtain the mean value of any channel c of any t-th frame image. Then, according to the pixel value of any channel c of each pixel in any t-th frame image and the corresponding mean The sum of the squares of the differences is divided by the total number of pixels to obtain the pixel variance of channel c in the tth frame , further square root to obtain the standard deviation , thus achieving dynamic normalization processing and obtaining standardized video blocks. The calculation formula for dynamic normalization processing is:

[0065] ;

[0066] in, represents the normalized pixel value of channel c in the t-th frame, Represents any channel in RGB channels, represents the pixel value of channel c in the t-th frame, represents the pixel mean of channel c in the t-th frame, represents the pixel standard deviation of channel c in the t-th frame, A very small constant used to prevent division by zero errors during calculations.

[0067] The dynamic normalization process described above can bring the RGB channel data of different videos to a unified scale, eliminating the impact of factors such as light intensity differences on the data, improving the stability and convergence speed of model training, and ensuring the robustness of the model during application.

[0068] See also Figure 3 , the standardized video blocks output by the video preprocessing module Input a lightweight dual-branch network module to perform spatiotemporal feature extraction and motion compensation on the video block, where R represents a set of numbers, H represents the height of each image frame of the video block, W represents the width of each image frame of the video block, T=180, H=128, W=128, and 3 represents the three RGB channels. Traditional designs usually only use 3D convolution for feature extraction, and lack a motion compensation process, which causes the model's predicted heart rate to be severely interfered with by noise such as facial movement. Alternatively, spatiotemporal feature extraction and motion compensation are combined for processing, and traditional 3D convolution is an integrated calculation method with large computational complexity and low computational efficiency. It cannot be applied to devices with small computational complexity, and has high requirements for deployed devices. Therefore, the embodiment of the present invention designs the spatiotemporal feature extraction and motion compensation branches in parallel, and adopts a lightweight dual-branch architecture. Specifically, the video block The input is simultaneously fed into the parallel spatiotemporal feature branch and motion compensation branch. The spatiotemporal feature branch decomposes the traditional 3D convolution into two steps: spatial convolution and depthwise convolution. In terms of branch structure, the spatiotemporal feature branch includes four layers of progressive separable 3D convolution, each with a kernel size of 3×3×3 and a stride of 2×2×2. This design decomposes the traditional 3D convolution into two steps: spatial convolution and depthwise convolution, significantly reducing the number of model parameters and computation without sacrificing performance.

[0069] For each layer of separable 3D convolution, the operation is:

[0070] ;

[0071] in, Indicates the Layer separable 3D convolution, Indicates the The layer can separate the output of 3D convolution, , Represents a separable 3D convolution operation.

[0072] For any layer of separable 3D convolution, first perform the image processing on the input data (i.e., the output of the previous layer of separable 3D convolution, the input of the first layer of separable 3D convolution is the video block ) to perform deep convolution, and perform 3×3×3 convolution operations on each channel to extract the local spatiotemporal features of each channel. The results of the deep convolution are then combined in the channel dimension to achieve cross-channel information fusion. This decomposition method can not only significantly reduce the amount of calculation and parameters, but also effectively extract the spatiotemporal features of the video block. After 4 layers of separable 3D convolution processing, according to the input video block Output convolution feature map (the final output of the spatiotemporal feature branch), and .

[0073] The motion compensation branch includes two layers of deformable convolution (Deformable Conv). By introducing a learnable offset, the convolution kernel sampling position is dynamically and adaptively adjusted based on the content of each frame, thus better adapting to changes in image content. Traditional motion compensation uses fixed sampling positions (usually a regular grid of points), which has limited ability to compensate for complex dynamic changes.

[0074] In actual calculation, by inputting feature maps at position Sampling is performed and with weights Multiply and sum to get the output feature map In position This operation mechanism enables the convolution kernel to dynamically adjust the sampling position according to the facial movement and better capture the motion characteristics. The specific operation for each layer of deformable convolution is:

[0075] ;

[0076] in, represents the output of deformable convolution, represents the position on the output feature map, Indicates the fixed position of the convolution kernel, that is, in the absence of offset, the first The sampling points should be located at the coordinates on the input feature map, Indicates the location What we learned is relative to The offset, represents the weight of the convolution kernel, Represents the total number of sampling points in the convolution kernel, Represents the input feature map. The feature map of the first layer of deformable convolution input is the standardized video block , the feature map of the second layer of deformable convolution input is the output result of the first layer of deformable convolution.

[0077] After two layers of deformable convolution, the motion offset field can be output , where 2 represents the offset in the x and y directions. Based on motion offset field After making corresponding offset adjustments, the motion-compensated features are obtained. (The final output result of the dynamic compensation branch).

[0078] After the spatiotemporal feature branch and motion compensation branch perform feature extraction and motion compensation, the corresponding output results and Input feature fusion unit for splicing, the splicing operation is about to and Merge the features by channel dimension to form a high-dimensional feature map. This concatenated high-dimensional feature map typically has a high number of channels, which increases the complexity of subsequent computations. To reduce feature dimensionality and fuse information, a 1×1 convolution can be used for dimensionality reduction. This 1×1 convolution adjusts the number of channels without changing the spatial size of the feature map, achieving feature fusion and dimensionality reduction. This allows the model to more effectively utilize these features for subsequent heart rate calculations. The concatenated high-dimensional features are mapped to a low-dimensional space, and the output is the relevant features for heart rate prediction.

[0079] See also Figure 4 , the heart rate prediction related features output by the lightweight two-branch network module are input into the heart rate calculation module. The heart rate calculation module calculates the preliminary heart rate prediction value based on the video spatiotemporal features and motion compensation information related features learned by the lightweight two-branch network module. However, the preliminary heart rate prediction value still has noise and cannot be directly read out. Therefore, a second-order Butterworth low-pass filter is further used to filter the preliminary heart rate prediction value. By setting a suitable cutoff frequency to filter out high-frequency noise, the prediction signal is smoother and the interference of noise on the heart rate calculation is reduced. In practical applications, according to the frequency characteristics of the heart rate signal, selecting a suitable cutoff frequency range can better retain the characteristics of the heart rate signal while removing most other interfering frequency components. According to actual human physiological parameters, the cutoff frequency range is usually selected from 0.75-2.5Hz. Therefore, the transfer function of the second-order Butterworth low-pass filter is It can be expressed as:

[0080] ;

[0081] in, is the pole of the second-order Butterworth low-pass filter, is a complex frequency variable, is the order of the denominator polynomial, indicating that A extreme point.

[0082] After filtering, peak detection is performed on the filtered preliminary heart rate prediction signal. Local peak points are found in the filtered signal. These peak points correspond to the fluctuations of the heart rate signal. The heart rate can be calculated based on the time interval between adjacent peaks. The calculation formula for heart rate is:

[0083] ;

[0084] in, Indicates heart rate (beats / minute), is the average time interval (in seconds) between adjacent peaks. After the above processing, the preliminary heart rate prediction value can be converted into a readable and usable predicted heart rate. At this point, the predicted heart rate of the person in the video can be obtained and output based on the input video.

[0085] In addition, there is a multi-task loss function module for model training and optimization, please refer to Figure 5 This embodiment of the present invention improves model performance by jointly optimizing heart rate estimation (the primary task) and frequency-domain noise suppression (the auxiliary task). The model's total loss function includes heart rate loss, noise suppression loss, and regularization loss. Feedback optimization is performed on the model based on these three losses. This multi-task learning strategy optimizes the primary task (heart rate estimation) while leveraging the auxiliary task (noise suppression) to provide additional supervision information. Regularization prevents overfitting, thereby improving the model's overall performance and generalization capabilities.

[0086] Among them, the heart rate loss uses the negative Pearson correlation coefficient to measure the correlation between the final predicted heart rate and the true heart rate. The final predicted heart rate is the predicted heart rate value output by the model. The true heart rate is the actual heart rate of the person in the training sample during the training process. After the model training is completed and the model is applied, there is no actual heart rate input, so the loss function will not be used to optimize the model. However, after the model training is completed, training samples and test samples can still be further supplemented to verify and optimize the model. The expression is:

[0087] ;

[0088] in, Indicates the The true heart rate of the training samples, Indicates the The predicted heart rate of training samples, Indicates the number of samples for model training, Indicates the average value of the true heart rate, Represents the average of the predicted heart rates. The Pearson correlation coefficient measures the linear correlation between two variables and ranges from −1 to +1. This embodiment of the present invention uses a negative Pearson correlation coefficient as the loss function to maximize the correlation between the predicted heart rate and the true heart rate. Specifically, the smaller the loss function value, the closer the model-predicted heart rate is to the true value.

[0089] The noise suppression loss is used to constrain the energy of the prediction signal outside the heart rate frequency band (0.7-2.5Hz). The expression is:

[0090] ;

[0091] in, Indicates the predicted heart rate at frequency The energy of the place.

[0092] Pass-through noise suppression loss The energy outside the heart rate frequency band in the predicted heart rate can be constrained, noise signals not related to the heart rate can be suppressed, and the quality of the heart rate signal can be improved. During the training process, the noise filtering effect of the evaluation model can be reflected to ensure that when predicting the heart rate, the model focuses on the signal within the heart rate frequency band and ignores the noise interference in other frequency bands.

[0093] Regularization loss uses L2 regularization term to prevent overfitting in model training. The expression is:

[0094] ;

[0095] in, represents the regularization coefficient, represents the set of parameters of the model, represents any parameter in the model. L2 regularization penalizes the sum of the squares of the model parameters to prevent overfitting. During training, large parameter values ​​can lead to overcomplexity and overfitting. L2 regularization keeps model parameters within a reasonable range, thereby improving the model's generalization ability.

[0096] By aggregating the three aforementioned loss calculations into the model's total loss calculation module and feeding the total loss back to the model, model optimization training can be achieved. When the total loss meets the preset conditions, the model training is confirmed to be complete. The total loss can be calculated as a weighted sum of the three aforementioned losses, and the weight coefficients of the three losses can be adaptively adjusted based on actual requirements. By jointly optimizing the heart rate loss, noise suppression loss, and regularization loss, the model can improve the accuracy of heart rate estimation while suppressing noise interference and preventing overfitting, thereby significantly improving the model's performance and generalization ability.

[0097] As an optional embodiment, the model proposed in this embodiment can be deployed on mobile device processors, including but not limited to mobile phones, PCs, the cloud, and cores with performance comparable to current mainstream mobile processors. In actual deployment, the model needs to be adapted and optimized based on the architectural characteristics of the deployment platform, for example, by leveraging its specific instruction set to accelerate the computation process and improve inference efficiency.

[0098] As an optional embodiment, optical flow estimation (such as RAFT) can be used instead of deformable convolution for motion compensation. Optical flow estimation aims to calculate the motion vector of pixels between adjacent frames in a video. Its core principles are based on the assumptions of constant brightness, small motion, and spatial consistency. Taking the variational method as an example, the basic calculation formula is:

[0099] ;

[0100] in, is the energy function for optical flow calculation, is the image intensity, is the optical flow vector, which represents the motion displacement of the pixel in the x and y directions. Represents the image intensity, that is, the image at position and time The pixel value at is a regularization parameter used to balance the weights of the data term and the smooth term.

[0101] Optical flow estimation can more accurately capture large-scale motion. By calculating the motion vector of each pixel, it can describe changes in facial motion in more detail. However, since optical flow estimation requires complex calculations for each pixel to solve the optical flow equation, the computational complexity is higher than that of deformable convolution. Therefore, the disadvantage of optical flow estimation is that the amount of calculation increases and the computational complexity is higher. In practical applications, if the accuracy of motion compensation is extremely high and the computing resources of the device are relatively sufficient, optical flow estimation can be considered instead of deformable convolution.

[0102] As an optional embodiment, for the design of the loss function during training, adversarial training (GAN) can be used to replace the original frequency domain loss calculation. The training situation is evaluated and fed back by introducing a generator and a discriminator. The goal of the generator is to output a predicted signal that is as close as possible to the real heart rate signal, while the discriminator is used to distinguish the spectrum of the predicted signal output by the generator from the real heart rate signal. Specifically, the generator Try to minimize , discriminator Trying to maximize , the objective function of adversarial training can be expressed as:

[0103] ;

[0104] ;

[0105] in, is the loss function of the generator, which represents the goal that the generator tries to minimize. is the loss function of the discriminator, which represents the goal that the discriminator tries to maximize. is the real heart rate signal, that is, the real heart rate signal sampled in the training data set, is the input noise of the generator, usually a random vector, is the distribution of real heart rate signal data, is the distribution of noise, is the output of the generator, which is represented by the input noise The generated heart rate signal, The discriminator is the real heart rate signal The judgment of is the probability of a true signal, Represents the discriminator's loss function for the true signal in the generator judgment, Represents the discriminator's loss function for the generator's generated signal judgment, Represents the discriminator's loss function for the true signal Judgment Represents the discriminator's loss function for generating signals judgment.

[0106] Through adversarial training, the generator generates realistic heart rate signals, and the discriminator distinguishes between the generated signals and the real ones. This enhances the model's ability to learn the spectral characteristics of heart rate signals, allowing the generator to produce predictions that are closer to the real heart rate signals. However, adversarial training is relatively complex, requiring careful tuning of hyperparameters, and is prone to problems such as mode collapse, where the generator can only generate a limited number of samples, resulting in unstable training.

[0107] As an optional embodiment, the model quantization can use floating point numbers (such as FP32) or integers (such as INT8). Among them, the FP32 weights of the model are converted to INT8, and the space occupied by each weight value is reduced from 32 bits to 8 bits. This can further compress the model volume and reduce the model storage requirements and computational complexity. The principle of model quantization is to reduce the representation accuracy of weights and activation values, thereby reducing the storage and computational requirements of the model while maintaining a small loss in model accuracy. During the conversion process, the model needs to be calibrated. By running the model on a small amount of calibration data, the distribution of activation values ​​is statistically analyzed, and the quantization parameters are determined to ensure that the performance loss of the quantized model is within an acceptable range. Compression of the model volume can not only reduce storage overhead, but also speed up the loading speed of the model and improve inference efficiency.

[0108] As an optional embodiment, during the model deployment process, a specialized hardware acceleration framework (such as TensorRT) can be used to deploy the model to a specific hardware platform (such as the NVIDIA Jetson series development board). This hardware acceleration framework can optimize the model, such as performing layer fusion and quantization, to improve the model's inference speed on the specific hardware. During deployment, the model configuration must be optimized based on hardware characteristics, such as the number of GPU cores and memory bandwidth, to fully utilize the hardware's performance. Furthermore, the data transmission efficiency between the model and the hardware must be considered to reduce data transmission time and further improve overall inference speed.

[0109] While the effectiveness of this invention has not been verified, the optimized model, when run on a mobile device such as a smartphone, can maintain an average latency of less than 40ms, meeting the 30 FPS requirement for real-time processing and enabling users to obtain heart rate monitoring results almost instantly. The model parameter size is designed to be approximately 0.8MB, and peak memory consumption during operation is expected to be ≤50MB. This ensures that running on a mobile device does not significantly impact other device functions and ensures smooth operation.

[0110] After the training of the above lightweight real-time heart rate monitoring model is completed, the trained model can be used to detect the heart rate of the characters in the video and output the corresponding predicted heart rate.

[0111] In short, the above description is only a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included in the scope of protection of this specification.

[0112] The systems, devices, modules, or units described in one or more of the above embodiments may be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0113] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0114] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0115] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A lightweight real-time heart rate monitoring model, characterized in that: include: A video preprocessing module is used to preprocess the input video, detect and align faces in the input video, and perform spatiotemporal segmentation on the input video to obtain standardized video blocks; A lightweight dual-branch network module includes a spatiotemporal feature branch and a motion compensation branch, both of which are input to the video block, and a feature fusion unit, wherein the spatiotemporal feature branch includes multiple layers of separable 3D convolution, each layer of separable 3D convolution is used to perform depth convolution on each channel of the input data, and then combine the results of the depth convolution in the channel dimension through spatial convolution to obtain the spatiotemporal features in the video block; the motion compensation branch is used to dynamically capture facial motion information, obtain a motion offset field, and perform motion compensation on the features in the video block; the feature fusion unit is used to perform feature fusion on the outputs of the spatiotemporal feature branch and the motion compensation branch to obtain relevant features for heart rate prediction; a heart rate calculation module, which calculates the heart rate based on the heart rate prediction related features output by the lightweight dual-branch network module and outputs the predicted heart rate; A multi-task loss function module includes heart rate loss, noise suppression loss and regularization loss, and optimizes the model based on the heart rate loss, noise suppression loss and regularization loss.

2. The lightweight real-time heart rate monitoring model according to claim 1, characterized in that: The video preprocessing module uses the MTCNN algorithm to perform face detection and alignment on the input video, and crops the facial area into a region of interest with a fixed resolution.

3. The lightweight real-time heart rate monitoring model according to claim 2, characterized in that: After acquiring the region of interest, the video preprocessing module segments the input video into non-overlapping T-frame sequences in a spatiotemporal manner, and divides each frame image into an N×N grid to extract local spatiotemporal features.

4. The lightweight real-time heart rate monitoring model according to claim 1, characterized in that: After extracting the local spatiotemporal features in the input video, the method further includes performing dynamic normalization processing on the RGB channels of the input video. The calculation formula for dynamic normalization processing is: ； in, represents the normalized pixel value of channel c in the t-th frame, Represents any channel in RGB channels, represents the pixel value of channel c in the t-th frame, represents the pixel mean of channel c in the t-th frame, represents the pixel standard deviation of channel c in the t-th frame, is a constant used to prevent division by zero errors during calculations.

5. The lightweight real-time heart rate monitoring model according to claim 1, characterized in that: The spatiotemporal feature branch includes four layers of separable 3D convolutions. The kernel size of each layer of separable 3D convolution is 3×3×3, the step size is 2×2×2, and the operation of the separable 3D convolution is: ； in, Indicates the The layer can separate the output of 3D convolution, Represents a separable 3D convolution operation.

6. The lightweight real-time heart rate monitoring model according to claim 1, characterized in that: The motion compensation branch includes at least two layers of deformable convolution, which adaptively adjusts the sampling position of the convolution kernel according to the video block to obtain facial motion information. The operation of each layer of deformable convolution is as follows: ； in, represents the output of deformable convolution, represents the position on the output feature map, Indicates the fixed position of the convolution kernel, that is, in the absence of offset, the first The sampling points should be located at the coordinates on the input feature map, Indicates the location What we learned is relative to The offset, represents the weight of the convolution kernel, Represents the total number of sampling points in the convolution kernel, Represents the input feature map. The feature map of the first layer of deformable convolution input is the standardized video block X, and the feature map of the second layer of deformable convolution input is the output result of the first layer of deformable convolution.

7. The lightweight real-time heart rate monitoring model according to claim 1, characterized in that: The motion compensation branch uses optical flow estimation to obtain a motion offset field, thereby achieving motion compensation.

8. The lightweight real-time heart rate monitoring model according to claim 1, characterized in that: The feature fusion unit finally outputs the spatiotemporal feature branch The final output result of the motion compensation branch The images are concatenated and dimensionality reduced through 1×1 convolution to obtain and output relevant features for heart rate prediction.

9. The lightweight real-time heart rate monitoring model according to claim 1, characterized in that: The heart rate calculation module obtains a preliminary heart rate prediction value based on the heart rate prediction related features output by the lightweight dual-branch network module, filters the preliminary heart rate prediction value through a second-order Butterworth low-pass filter, performs peak detection on the filtered signal, calculates the heart rate based on the time interval between adjacent peaks, and obtains a readable and usable predicted heart rate.

10. The lightweight real-time heart rate monitoring model according to claim 1, characterized in that: The heart rate loss uses a negative Pearson correlation coefficient to measure the correlation between the final predicted heart rate and the actual heart rate. The expression is: ； in, Indicates the The true heart rate of the training samples, Indicates the The predicted heart rate of training samples, Indicates the number of samples for model training, Indicates the average value of the true heart rate, represents the average value of the predicted heart rate; The noise suppression loss The expression is: ； in, Indicates the predicted heart rate at frequency The energy of The regularization loss adopts L2 regularization term, and the regularization loss The expression is: ； in, represents the regularization coefficient, represents the set of parameters of the model, represents any parameter in the model.

11. A heart rate monitoring method, characterized in that: Heart rate detection is performed using the lightweight real-time heart rate monitoring model as described in any one of claims 1 to 10 to obtain a predicted heart rate.

Images