Contactless heart rate estimation method and system based on multimodal signal fusion

Abstract

The present invention discloses a non-contact heart rate estimation method and system based on multimodal signal fusion, which relates to the field of image processing technology. The method includes: locating the region of interest in a face video, extracting and fusing HSV and Lab color space signals; suppressing ambient light interference through background brightness compensation to obtain a color conversion signal and normalizing it; using historical heart rate to adjust the bandwidth of a bandpass filter to filter the signal to generate an rPPG signal; performing time domain and frequency domain analysis on the rPPG signal, combining the two results and taking a weighted average to obtain a heart rate estimate; and using a historical data smoothing strategy to correct the heart rate estimation result to ensure accuracy. The present invention achieves high-precision non-contact heart rate estimation by extracting and fusing HSV and Lab color space signals, background brightness compensation, signal normalization, and processing the face video signal based on the historical heart rate adjustment bandpass filter.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a non-contact heart rate estimation method and system based on multimodal signal fusion. Background Art

[0002] In spatial domain enhancement, directly manipulating pixels is the most intuitive approach. Point processing methods such as contrast stretching or gamma correction can be used to nonlinearly adjust grayscale distribution. For example, logarithmic transformation can highlight dark details, while histogram equalization improves global contrast by redistributing pixel grayscale values. However, global equalization can lead to local overexposure or noise amplification. Therefore, adaptive histogram equalization (CLAHE) has been proposed to avoid distortion by limiting contrast changes in local areas. In addition, spatial filtering techniques such as Gaussian smoothing and median filtering can effectively suppress noise but may blur edges. Sharpening filters (such as the Laplacian operator) emphasize texture by enhancing high-frequency components, but they must be handled with caution to avoid noise interference.

[0003] Frequency domain enhancement converts the image into the frequency domain based on the Fourier transform, achieving the desired effect by adjusting different frequency components. Low-pass filters preserve low-frequency information to smooth the image and are often used for denoising. High-pass filters enhance high-frequency edge information, but require careful noise control. Bandpass or band-stop filters target specific frequency ranges, for example, to eliminate periodic noise. This method has the advantage of separating structural and random components in the image, but the frequency domain conversion is computationally expensive and requires addressing errors between the frequency and spatial domains. For color images, color space conversion is a key step in enhancement. Adjusting each channel independently in RGB space can result in color distortion, so images are often converted to HSV or HSL space, adjusting only the brightness (V or L component) while maintaining hue and saturation. Retinex theory further simulates the human eye's adaptation to illumination, improving uneven illumination by separating the illumination and reflection components.

[0004] Remote photoplethysmography (rPPG), a non-contact physiological signal detection technology, estimates heart rate by analyzing subtle fluctuations in facial skin color as blood volume changes in videos. Its core principle is similar to traditional contact photoplethysmography (PPG), but without the need for direct skin contact. Therefore, it has broad application prospects in areas such as health monitoring and emotion recognition. However, existing technologies face significant challenges in practical applications, such as noise interference, motion artifacts, and lighting variations, which limit the accuracy of heart rate estimation.

[0005] Traditional rPPG algorithms primarily rely on signal processing techniques. For example, the GREEN algorithm extracts pulse waves from the green channel signal, which is related to the absorption characteristics of hemoglobin; the POS algorithm uses linear projection to separate noise components from skin-reflected signals; and the CHROM algorithm suppresses motion interference by constructing chrominance signals. However, these methods generally rely on fixed-bandwidth filtering and a single color space (such as RGB or the green channel), making them difficult to dynamically adapt to complex environments. Furthermore, traditional algorithms are highly sensitive to skin color differences. Individuals with darker skin are susceptible to interference from ambient light due to the smaller signal amplitude, resulting in insufficient robustness. Image sharpening techniques are used in remote photoplethysmography (rPPG) to highlight weak physiological signals in video. Its core principle is to use spatial convolution operations to enhance high-frequency details, thereby highlighting the periodic changes in skin surface color caused by cardiac pulsation. These changes arise from periodic fluctuations in blood volume. During systole, subcutaneous capillaries dilate, increasing blood absorption of specific wavelengths, resulting in a decrease in the intensity of reflected light. The opposite occurs during diastole. This optical phenomenon manifests as tiny periodic fluctuations in pixel values in video, but their amplitude is extremely small and easily masked by ambient light, motion artifacts, and sensor noise. Sharpening achieves signal enhancement through the design of a specific convolution kernel. A typical implementation uses a Laplacian kernel, essentially a high-pass filter that amplifies high-frequency components in an image by calculating the differences between local pixels. For example, a 3×3 convolution kernel enhances local contrast by assigning a higher positive weight to the central pixel and a negative weight to surrounding pixels. This operation is mathematically equivalent to taking a second-order derivative of the image and can significantly enhance high-frequency details such as edges and textures, which often include subtle color variations associated with blood flow. Smaller convolution kernels focus on microscopic pixel neighborhoods, enabling more detailed capture of subtle color variations caused by blood flow. Fluctuations in skin-reflected light caused by heartbeats typically affect only localized groups of pixels. Large convolution kernels can easily introduce signal mixing from irrelevant areas, obscuring valid physiological features. Smaller weights, however, reduce nonlinear distortion of the original signal, ensuring that the fundamental frequency and harmonic components of the heart rate signal more closely resemble true physiological characteristics. Furthermore, local pixel shifts caused by small head motions are significantly amplified by large-weight kernels, while gentle enhancement by small-weight kernels can reduce the impact of such disturbances.

[0006] In recent years, deep learning-based methods have become a research hotspot. For example, ST-ASENet enhances facial feature extraction through spatiotemporal attention mechanisms, and rPPG-MAE exploits the self-similarity of physiological signals using masked autoencoders. These methods automatically learn spatiotemporal features through neural networks, demonstrating higher accuracy in complex backgrounds and low-light conditions. However, deep learning models rely heavily on large-scale labeled data for training and require high computational resources, making them difficult to deploy in real-time monitoring devices. Furthermore, the models' limited generalization to extreme scenarios, such as sudden changes in lighting and intense exercise, and their lack of interpretability limit their practical application in healthcare. For example, while Transformer-based fusion frameworks can improve long-range feature modeling, they suffer from high computational latency and cannot meet real-time requirements.

[0007] The core problem with existing technologies is that traditional algorithms, due to their fixed parameter design, are difficult to adapt to dynamic environmental changes, while deep learning methods improve accuracy but sacrifice real-time performance and computational efficiency. In addition, most solutions do not fully integrate multimodal signals or dynamically adjust parameters, resulting in insufficient signal extraction stability. For example, a single color space is easily affected by the wavelength of light, while a fixed bandwidth filter cannot track dynamic changes in heart rate, especially during exercise or physiological state fluctuations, where error accumulation is significant. Therefore, an optimization solution that balances accuracy, robustness, and real-time performance is urgently needed to improve heart rate estimation performance in complex environments through multimodal signal fusion and dynamic parameter adjustment mechanisms. Summary of the Invention

[0008] In order to solve the above problems, the present invention proposes a non-contact heart rate estimation method and system based on multimodal signal fusion. By combining image sharpening, multi-color space signal fusion, background brightness compensation, color change enhancement, adaptive normalization and bandpass filtering technology, as well as time-frequency domain joint estimation and historical data smoothing strategy, more accurate and stable face video heart rate detection can be achieved.

[0009] The specific plan is as follows:

[0010] On the one hand, a non-contact heart rate estimation method based on multimodal signal fusion includes:

[0011] S1, obtain sample face video;

[0012] S2, using image sharpening technology to highlight the subtle color changes in the sample face video and obtain the sharpened face video image;

[0013] S3, locates the region of interest based on facial feature points in the sharpened face video;

[0014] S4, extracting signals of the HSV and Lab color spaces from the region of interest, and fusing the signals of the two color spaces to obtain a mixed signal containing information of the two color spaces;

[0015] S5, using a background brightness compensation strategy to suppress ambient light interference on the mixed signal to obtain an original brightness compensated signal;

[0016] S6, performing median filtering on the original brightness compensation signal to obtain a brightness compensation signal after median filtering; subtracting the brightness compensation signal after median filtering from the original brightness compensation signal to obtain a color conversion signal;

[0017] S7, normalizing the color conversion signal by taking the maximum and minimum values of the color conversion signal as a signal normalization range to obtain a normalized signal;

[0018] S8, adjusting the bandwidth of the original band-pass filter based on the historical heart rate, and band-pass filtering the normalized signal through the adjusted band-pass filter to obtain the rPPG signal;

[0019] S9, performing time domain analysis and frequency domain analysis on the rPPG signal, and weighting the heart rate value estimated by the time domain analysis and the heart rate value estimated by the frequency domain analysis to obtain a heart rate estimation result;

[0020] S10, correcting the heart rate estimation result by using a historical data smoothing strategy to obtain a final heart rate estimation value.

[0021] Furthermore, the S4 specifically includes:

[0022] Color signal extraction is performed on the HSV color space and the Lab color space to obtain color signal component H, color signal component S, color signal component V, color signal component L, color signal component a, and color signal component b. The color signal component H, color signal component S, and color signal component V are weighted to obtain a mixed signal of the HSV color space; the color signal component L, color signal component a, and color signal component b are weighted to obtain a mixed signal of the Lab color space.

[0023] Furthermore, the S5 specifically includes: extracting a brightness signal outside the area of interest as a reference reference signal, subtracting the reference reference signal from the mixed signal of the HSV color space and the mixed signal of the Lab color space to obtain brightness compensation, and performing a weighted addition operation on the brightness-compensated mixed signal of the HSV color space and the mixed signal of the Lab color space to obtain an original brightness-compensated signal.

[0024] Furthermore, in S8, adjusting the bandwidth of the original bandpass filter based on the historical heart rate specifically includes:

[0025] Get the preset bandwidth range of the original band-pass filter, obtain the heart rate mean within the specified time period based on the original band-pass filter, convert the heart rate mean into heart rate frequency, and adaptively adjust the bandwidth range of the original band-pass filter based on the heart rate frequency to obtain the adjusted band-pass filter. The calculation formula is as follows:

[0026] ;

[0027] ;

[0028] in, Indicates the bandpass range to the left of the adjusted bandpass filter reference frequency; Indicates the bandpass range to the right of the adjusted bandpass filter reference frequency; Indicates the heart rate frequency corresponding to the mean heart rate during a specified period; and Represents the range of the adjusted bandpass filter, which is used to filter out frequency components in [ , ] part outside the scope.

[0029] Furthermore, the S9 specifically includes: using FFT to extract frequency domain information in the rPPG signal, calculating the frequency component with the largest amplitude in the spectrum based on the frequency domain information, and obtaining a heart rate value estimated by frequency domain analysis; calculating the heart rate value estimated by time domain analysis based on the peak interval in the rPPG signal; combining the heart rate value estimated by time domain analysis and the heart rate value estimated by frequency domain analysis by weighted averaging to obtain a heart rate estimation result; the heart rate estimation result is calculated by the following formula:

[0030] ;

[0031] in, represents the heart rate value estimated by frequency domain analysis; represents the heart rate value estimated by time domain analysis; Indicates the heart rate estimation result; is the weighting coefficient with a value range of (0,1), which is used to control the weight of frequency domain and time domain information in heart rate estimation.

[0032] Furthermore, the S10 specifically includes: calculating the average of the previous five heart rates based on the most recent five heart rate values, and the calculation formula is as follows:

[0033] ;

[0034] in, 、 、 、 and They are the heart rate values of the previous five times; Indicates the average of the previous five heart rates;

[0035] The heart rate estimation result and the mean of the previous five times Calculate the difference, if the difference exceeds the preset threshold ,Right now:

[0036] ;

[0037] This indicates that there is an abnormal fluctuation between the current heart rate estimation result and the historical estimation result. The current heart rate estimation result and the average of the previous five times are weighted and fused to eliminate the influence of the abnormal fluctuation. The weighted fusion formula is as follows:

[0038] ;

[0039] in, represents the weight of the current heart rate estimation result, and , used to ensure that the current heart rate estimation result has a larger proportion during fusion; Indicates the final heart rate estimate.

[0040] On the other hand, the non-contact heart rate estimation system based on multimodal signal fusion includes:

[0041] Face video acquisition module, used to obtain sample face videos;

[0042] A sharpening module is used to highlight subtle color changes in the sample face video through image sharpening technology to obtain a sharpened face video image;

[0043] A region of interest positioning module is used to locate the region of interest based on facial feature points in the sharpened face video;

[0044] A mixed signal acquisition module is used to extract signals of the HSV and Lab color spaces from the region of interest, fuse the signals of the two color spaces, and obtain a mixed signal containing information of the two color spaces;

[0045] The original brightness compensation signal acquisition module is used to suppress the ambient light interference by using the background brightness compensation strategy on the mixed signal to obtain the original brightness compensation signal;

[0046] A color conversion signal acquisition module is used to perform median filtering on the original brightness compensation signal to obtain a brightness compensation signal after median filtering; and to subtract the brightness compensation signal after median filtering from the original brightness compensation signal to obtain a color conversion signal;

[0047] a normalized signal acquisition module, configured to normalize the color conversion signal by taking the maximum and minimum values of the color conversion signal as a signal normalization range to obtain a normalized signal;

[0048] An rPPG signal acquisition module is used to adjust the bandwidth of the original bandpass filter based on the historical heart rate, and to perform bandpass filtering on the normalized signal through the adjusted bandpass filter to obtain the rPPG signal;

[0049] A heart rate estimation module is used to perform time domain analysis and frequency domain analysis on the rPPG signal, and to weight the heart rate value estimated by the time domain analysis and the heart rate value estimated by the frequency domain analysis to obtain a heart rate estimation result;

[0050] The final heart rate estimation value acquisition module corrects the heart rate estimation result through the historical data smoothing strategy to obtain the final heart rate estimation value.

[0051] The present invention adopts the above technical solution and has the following beneficial effects:

[0052] (1) The present invention uses image sharpening technology and extracts and fuses signals from HSV and Lab color spaces to highlight subtle color changes in face videos, and adopts background brightness compensation strategy to suppress ambient light interference, thereby improving the accuracy and stability of color change information under complex lighting conditions;

[0053] (2) The present invention achieves effective purification and accurate analysis of heart rate signals by subtracting the median filtered signal from the unfiltered signal to highlight color change information, combining normalization processing based on the maximum and minimum values of the signal and bandpass filtering technology that dynamically adjusts the bandwidth according to the historical heart rate;

[0054] (3) The present invention corrects the estimated heart rate value through a historical data smoothing strategy, effectively reducing the impact of accidental errors and ensuring the stability and reliability of the final output heart rate estimation result, so that continuous and accurate heart rate monitoring can be provided even in the presence of interference factors such as exercise or environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a non-contact heart rate estimation method based on multimodal signal fusion according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of a module detecting 468 key points on a face according to an embodiment of the present invention;

[0057] Figure 3 4 is a diagram of a non-contact heart rate estimation system based on multimodal signal fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0059] like Figure 1 As shown, the present invention provides a non-contact heart rate estimation method based on multimodal signal fusion, comprising:

[0060] S1, obtain sample face video.

[0061] Specifically, in this embodiment, facial images are extracted frame by frame from the input video, and the entire video is divided into pictures of corresponding frame lengths. The entire video image can be regarded as consisting of N frames, and subsequent processing is also based on corresponding operations for each frame image.

[0062] S2, through image sharpening technology, highlights the subtle color changes in the sample face video and obtains the sharpened face video image.

[0063] Specifically, in this embodiment, each frame of the image is sharpened using image sharpening technology to highlight subtle color changes in the facial area. During the image sharpening process, the convolution kernel exists in the form of a matrix. It performs a convolution operation with each pixel in the image and its surrounding neighborhood to change the value of the pixel in the image to achieve the sharpening effect. The formula for the convolution operation is as follows:

[0064] ;

[0065] Where I(x,y) is the pixel value of the input image, K(i,j) is the weight in the convolution kernel, and I' (x,y) is the pixel value of the output image after convolution. Sharpening uses a 3×3 convolution kernel design with high weights at the center and negative weights at the periphery. This convolution operation emphasizes the high-frequency components of the image, making skin color fluctuations caused by changes in blood volume more pronounced.

[0066] S3, locates the region of interest based on the facial feature points in the sharpened face video.

[0067] Specifically, such as Figure 2As shown, this embodiment uses the Mediapipe Face Mesh module to detect 468 3D facial keypoints in real time. By assigning two sets of keypoints to the cheeks, the algorithm accurately locates the cheeks. The cheeks are chosen because they are relatively flat, less affected by head movement, and have dense subcutaneous capillaries, resulting in stronger blood flow signals. This allows the algorithm to delineate two independent ROIs (left and right cheeks) to avoid background noise interference. Face Mesh, part of the Mediapipe framework, is designed to provide high-precision facial keypoint localization for each facial image, supporting both 2D and 3D facial analysis. The model takes an image or video stream as input and outputs the coordinates of 468 facial keypoints. Each keypoint has a position in 3D space, and depth information can be extracted for that facial region. Face Mesh uses a deep learning model to detect key points in facial images and accurately locate facial feature areas such as eyes, eyebrows, nose, mouth, and facial contours. The algorithm selects some feature points on the left and right cheeks respectively, uses these feature points as the boundaries of the ROI area, and uses the area surrounded by these feature points as the ROI area required by the algorithm. Signals are then extracted from these two ROI areas.

[0068] S4, extracting signals of the HSV and Lab color spaces from the region of interest, and fusing the signals of the two color spaces to obtain a mixed signal containing information of the two color spaces.

[0069] Specifically, the S4 includes:

[0070] Color signal extraction is performed on the HSV color space and the Lab color space to obtain color signal component H, color signal component S, color signal component V, color signal component L, color signal component a, and color signal component b. The color signal component H, color signal component S, and color signal component V are weighted to obtain a mixed signal of the HSV color space; the color signal component L, color signal component a, and color signal component b are weighted to obtain a mixed signal of the Lab color space.

[0071] Specifically, HSV color space and Lab color space are extracted from the two ROI areas for color signals respectively, and different color components in the two color spaces are extracted one by one and then weighted added to obtain a mixed signal. The mixed signal contains the color change information of the two color spaces, that is, it contains more features, which helps to extract a signal that is closer to the real pulse signal.

[0072] Specifically, in this embodiment, the HSV color space and the Lab color space are selected for color signal extraction. The color signals from each color space are extracted one by one and then weightedly added to produce a mixed signal from each color space. This mixed signal contains color variation information from both color spaces. Weighted fusion of the mixed signals from the two color spaces, which means it contains more features, helps extract a signal closer to the true pulse signal. Unlike the commonly used RGB color space, the HSV color space is closer to how humans perceive color. It describes color using three components: hue, saturation, and value. These three components are the extraction targets in the HSV color space. Since OpenCV's default color space is BGR, each image frame must be converted to the HSV color space before extracting these three components. Given a pixel in the BGR color space, B, G, and R are the pixel values of the blue, green, and red channels, respectively. Before conversion, the BGR values must be normalized to the range [0, 1]. This step is for convenience in subsequent calculations:

[0073] ;

[0074] Then for each pixel, calculate the maximum value Cmax and minimum value Cmin:

[0075] ;

[0076] The next step is to calculate the color difference (i.e. the difference between the maximum and minimum values):

[0077] ;

[0078] The three components can then be calculated, with the hue H being calculated based on the color difference of the pixels. There are different formulas for calculating hue, depending on the color channel where the maximum value Cmax is located. :

[0079] ;

[0080] :

[0081] ;

[0082] :

[0083] ;

[0084] Saturation Describes the purity of the color. The calculation formula for saturation is:

[0085] ;

[0086] if , indicating that the pixel is gray (no color), Brightness is the maximum value , which is the brightness value of the pixel:

[0087] ;

[0088] The Lab color space (CIE 1976 (L*, a*, b*) color space) is a color space defined by the International Commission on Illumination (CIE) in 1976. It aims to provide a device-independent color representation that closely simulates the human eye's perception of color. The Lab color space also consists of three components. L* represents the lightness of a color, ranging from 0 to 100, with 0 representing black and 100 representing white. It describes the lightness or darkness of a color. a* (the red-green axis) represents the color's transition from green to red, with negative values representing green and positive values representing red, ranging from approximately -128 to 127. b* (the blue-yellow axis) represents the color's transition from blue to yellow, with negative values representing blue and positive values representing yellow, ranging from approximately -128 to 127.

[0089] To convert an image from BGR color space to Lab color space, you first need to convert from BGR color space to XYZ color space, and then convert from XYZ color space to Lab color space. The converted X, Y, and Z are the components of the XYZ color space. Next, each pixel in the XYZ color space is converted to Lab color space. First, the XYZ values need to be normalized. The D65 white point is usually used as a reference:

[0090] ;

[0091] Normalize each pixel using a reference value:

[0092] ;

[0093] Next, the L*, a*, and b* values can be calculated using the following formulas:

[0094] First calculate L* (lightness):

[0095] ;

[0096] in, is a piecewise function:

[0097] ;

[0098] The next step is to calculate the values of a* and b* respectively:

[0099] ;

[0100] ;

[0101] The different components of the two color spaces are weighted and added according to certain weights to obtain the mixed signals of their respective color spaces.

[0102] S5, a background brightness compensation strategy is used on the mixed signal to suppress ambient light interference and obtain an original brightness compensated signal.

[0103] Specifically, S5 includes: extracting a brightness signal outside the area of interest as a reference reference signal, subtracting the reference reference signal from the mixed signal of the HSV color space and the mixed signal of the Lab color space to obtain brightness compensation, and performing a weighted addition operation on the brightness-compensated mixed signal of the HSV color space and the mixed signal of the Lab color space to obtain an original brightness-compensated signal.

[0104] Specifically, to mitigate the effects of ambient lighting, the algorithm extracts the background brightness (V in HSV and L in Lab) outside the ROI and subtracts it from the mixed signal in the corresponding color space to achieve brightness compensation. Finally, the compensated signals from the two color spaces are weighted and fused to generate a mixed signal containing multi-dimensional features, enhancing robustness to lighting changes.

[0105] S6, performing median filtering on the original brightness compensation signal to obtain a brightness compensation signal after median filtering; subtracting the brightness compensation signal after median filtering from the original brightness compensation signal to obtain a color conversion signal.

[0106] Specifically, baseline drift was removed by median filtering, the mixed signal was median filtered (window length 300 frames), and the filtered result was subtracted from the original signal to obtain a detrended signal. This step can highlight the periodic characteristics of blood volume changes while suppressing low-frequency noise.

[0107] S7 , normalizing the color conversion signal by taking the maximum and minimum values of the color conversion signal as a signal normalization range to obtain a normalized signal.

[0108] Specifically, the mixed signal needs to be normalized to eliminate the influence of individual skin color differences. Traditional methods use global normalization, but this can compress the dynamic range of the signal for individuals with darker skin. To address this, the algorithm proposes local adaptive normalization: the signal is divided into sliding windows of 300 frames, and the maximum and minimum values are independently calculated for each window. Based on these values, the signal is linearly mapped to the range [0, 1]. This method ensures that signal variations for individuals with different skin colors are fully expressed within their own dynamic range.

[0109] S8, adjusting the bandwidth of the original bandpass filter based on the historical heart rate, and performing bandpass filtering on the normalized signal through the adjusted bandpass filter to obtain the rPPG signal.

[0110] Specifically, in S8, adjusting the bandwidth of the original pass filter based on the historical heart rate specifically includes:

[0111] Get the preset bandwidth range of the original band-pass filter, obtain the heart rate mean within the specified time period based on the original band-pass filter, convert the heart rate mean into heart rate frequency, and adaptively adjust the bandwidth range of the original band-pass filter based on the heart rate frequency to obtain the adjusted band-pass filter. The calculation formula is as follows:

[0112] ;

[0113] ;

[0114] in, Indicates the bandpass range to the left of the adjusted bandpass filter reference frequency; Indicates the bandpass range to the right of the adjusted bandpass filter reference frequency; Indicates the heart rate frequency corresponding to the mean heart rate during a specified period; and Represents the range of the adjusted bandpass filter, which is used to filter out frequency components in [ , ] part outside the scope.

[0115] Specifically, in this embodiment, the original bandpass filter bandwidth is set to [0.7, 3] Hz, corresponding to the normal heart rate range of [42, 180]. After normalization, the signal is then subjected to adaptive bandpass filtering. Traditional rPPG algorithms typically use fixed-bandwidth bandpass filters (e.g., 0.7, 3 Hz) to filter out non-heart rate-related noise. However, in dynamic scenarios (such as motion or sudden changes in illumination), fixed bandwidths struggle to adapt to the range of heart rate fluctuations and the frequency band shifts of interfering signals. To address this, the present invention proposes an adaptive bandpass filtering strategy based on historical heart rate estimates. The core of this strategy lies in dynamically adjusting the filter's upper and lower frequency limits to balance noise suppression with effective signal preservation. The application of the adaptive bandpass filter in heart rate signal extraction not only effectively avoids sudden noise interference and signal fluctuations caused by head movement or facial expressions, but also ensures that the heart rate signal remains stable within the normal heart rate range, significantly improving the accuracy and stability of signal extraction. When the heart rate is within a stable range, the bandpass filter bandwidth automatically decreases, more accurately focusing on the characteristic frequency band of the heart rate signal. This dynamic adjustment mechanism enables the filter to flexibly adapt to different motion states and physiological changes, making the extracted heart rate signal smoother while avoiding the loss of effective information caused by over-filtering or too narrow a bandwidth. This adaptive bandpass filter method not only effectively reduces interference from external factors such as the environment, facial expressions, and head movement, but also ensures accurate extraction of heart rate signals under different physiological states, making the extraction of heart rate signals more consistent with actual physiological changes rather than following fixed filtering standards.

[0116] S9, performing time domain analysis and frequency domain analysis on the rPPG signal, and weighting the heart rate value estimated by the time domain analysis and the heart rate value estimated by the frequency domain analysis to obtain a heart rate estimation result.

[0117] Specifically, S9 includes: extracting frequency domain information from the rPPG signal using FFT, calculating the frequency component with the largest amplitude in the spectrum based on the frequency domain information, and obtaining a heart rate value estimated by frequency domain analysis; calculating the heart rate value estimated by time domain analysis based on the peak interval in the rPPG signal; and combining the heart rate value estimated by time domain analysis and the heart rate value estimated by frequency domain analysis by weighted averaging to obtain a heart rate estimation result; the heart rate estimation result is calculated by the following formula:

[0118] ;

[0119] in, represents the heart rate value estimated by frequency domain analysis; represents the heart rate value estimated by time domain analysis; Indicates the heart rate estimation result; is the weighting coefficient with a value range of (0,1), which is used to control the weight of frequency domain and time domain information in heart rate estimation.

[0120] Specifically, traditional heart rate estimation methods typically perform a fast Fourier transform (FFT) on a signal, then find the frequency component with the largest amplitude in the spectrum. This frequency component is then multiplied by 60 to convert the signal to beats per minute (bpm). This method's core assumption is that the peaks in the signal spectrum correspond to the primary components of the heart rate, allowing heart rate to be estimated from the dominant frequencies in the spectrum. In the time domain, heart rate signals fluctuate periodically, with each heartbeat generating a peak. The interval between two adjacent peaks in a heart rate signal reflects the duration of a cycle, which is directly related to the heart rate. Heart rate can be directly estimated by measuring the peak interval in the signal. This method does not rely on frequency domain information, but instead directly utilizes the signal's time domain characteristics to estimate heart rate. While simple and intuitive, this method can be affected by noise and motion artifacts, resulting in inaccurate peak identification and, consequently, the accuracy of heart rate estimation. To overcome the limitations of relying solely on time or frequency domain information, this paper proposes a weighted approach that combines both. First, an FFT is used to extract frequency domain information from the signal, and the frequency component with the largest amplitude in the spectrum is calculated to obtain a preliminary heart rate estimate (in bpm). The signal's peak intervals are then further analyzed to calculate a heart rate estimate in the time domain. These two estimates are then combined using a weighted average to produce a more accurate heart rate estimate.

[0121] Specifically, because relying solely on frequency domain or time domain analysis has limitations: frequency domain methods (such as FFT main peak detection) are susceptible to periodic noise interference, while time domain methods (such as peak interval calculation) are sensitive to instantaneous noise (such as facial micro-movements). Therefore, the advantages of both are combined to achieve a more robust heart rate estimation. In frequency domain analysis, the algorithm performs a fast Fourier transform (FFT) on the filtered signal and extracts the frequency component with the largest amplitude in the power spectrum. , and convert to heart rate value:

[0122] ;

[0123] In the time domain analysis, the algorithm detects the signal peak interval (PPI), removes outliers (such as PPI < 0.3 s or PPI > 1.5 s), and calculates the median interval to obtain the time domain heart rate:

[0124] ;

[0125] To balance the contributions of the two, the algorithm introduces a dynamic weighting factor α. When the amplitude of the main peak in the frequency domain is significantly higher than that of the secondary peak (e.g., the main peak amplitude > 2 times the secondary peak amplitude), α is set to a larger value, prioritizing the frequency domain result. When the time domain peak interval is highly stable (e.g., the PPI standard deviation < 0.1 s), α is set to a smaller value to enhance the time domain contribution. The final weighted heart rate estimate can be expressed as follows:

[0126] .

[0127] S10, correcting the heart rate estimation result by using a historical data smoothing strategy to obtain a final heart rate estimation value.

[0128] Specifically, the step S10 includes calculating the average of the previous five heart rates based on the five most recent heart rate values, using the following formula:

[0129] ;

[0130] in, 、 、 、 and They are the heart rate values of the previous five times; Indicates the average of the previous five heart rates;

[0131] The heart rate estimation result and the mean of the previous five times Calculate the difference, if the difference exceeds the preset threshold ,Right now:

[0132] ;

[0133] This indicates that there is an abnormal fluctuation between the current heart rate estimation result and the historical estimation result. The current heart rate estimation result and the average of the previous five times are weighted and fused to eliminate the influence of the abnormal fluctuation. The weighted fusion formula is as follows:

[0134] ;

[0135] in, represents the weight of the current heart rate estimation result, and , used to ensure that the current heart rate estimation result has a larger proportion during fusion; Indicates the final heart rate estimate.

[0136] Specifically, the average heart rate within a specified time period is mainly used as the reference frequency of the new band-pass filter to adjust the band-pass range. It pays more attention to the current heart rate and generally takes the average of the previous three heart rates. The average heart rate here is mainly used for historical data smoothing, so the average of the previous five heart rates is taken, which is more biased towards historical heart rate data.

[0137] Specifically, under normal circumstances, the heart rate value will not fluctuate violently in a short period of time. Therefore, the heart rate estimation results of the most recent times can be used as a reference to "smooth" the current heart rate estimate, avoid unnecessary impact of sudden abnormal signal fluctuations on the final result, and reduce estimation errors caused by instantaneous noise, motion artifacts or other sudden interference. Based on this idea, the heart rate estimation smoothing strategy refers to introducing the average of the previous five heart rate estimation results as a reference to analyze the difference between the current estimated heart rate and the average of the previous five times. If the difference between the current heart rate value and the average of the previous five heart rates exceeds the preset threshold, the two are weightedly averaged to correct the current heart rate estimate and achieve a more stable heart rate estimate. The specific implementation can be divided into three steps: outlier detection: calculate the average of the previous five heart rate estimates, if the current heart rate estimate The average of the previous five heart rate estimates If the difference between them exceeds the threshold, it is determined to be an abnormal value (assuming that this may be caused by noise or other sudden interference). If it is less than the threshold, it is determined to be a normal value and no smoothing is required. Weighted correction: Smoothing operation is performed on abnormal values. Usually, The value of can be adjusted according to the reliability of the signal. When the current heart rate signal is more stable and accurate, It will tend to a larger value, and when it exceeds the threshold continuously, it is considered that the current heart rate signal is disturbed. The final page will display the final heart rate estimate after S10 processing and the PPG estimated waveform after S8 filtering.

[0138] like Figure 3 As shown, this embodiment also discloses a non-contact heart rate estimation system based on multimodal signal fusion, including:

[0139] A face video acquisition module 31 is used to acquire a sample face video;

[0140] A sharpening module 32 is used to highlight subtle color changes in the sample face video using image sharpening technology to obtain a sharpened face video image;

[0141] A region of interest positioning module 33 is used to locate the region of interest based on facial feature points in the sharpened face video;

[0142] A mixed signal acquisition module 34 is used to extract signals of the HSV and Lab color spaces from the region of interest, and fuse the signals of the two color spaces to obtain a mixed signal containing information of the two color spaces;

[0143] The original brightness compensation signal acquisition module 35 is used to suppress ambient light interference by applying a background brightness compensation strategy to the mixed signal to obtain an original brightness compensation signal;

[0144] The color conversion signal acquisition module 36 is configured to perform median filtering on the original brightness compensation signal to obtain a brightness compensation signal after median filtering; and to subtract the brightness compensation signal after median filtering from the original brightness compensation signal to obtain a color conversion signal.

[0145] a normalized signal acquisition module 37 for normalizing the color conversion signal by taking the maximum and minimum values of the color conversion signal as a signal normalization range to obtain a normalized signal;

[0146] an rPPG signal acquisition module 38, configured to adjust the bandwidth of the original bandpass filter based on the historical heart rate, and perform bandpass filtering on the normalized signal through the adjusted bandpass filter to obtain the rPPG signal;

[0147] a heart rate estimation module 39 for performing time domain analysis and frequency domain analysis on the rPPG signal, and weighting the heart rate value estimated by the time domain analysis and the heart rate value estimated by the frequency domain analysis to obtain a heart rate estimation result;

[0148] The final heart rate estimation value acquisition module 310 corrects the heart rate estimation result through a historical data smoothing strategy to obtain a final heart rate estimation value.

[0149] The specific implementation of the non-contact heart rate estimation system based on multimodal signal fusion is the same as the non-contact heart rate estimation method based on multimodal signal fusion, and will not be repeated in this embodiment.

[0150] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. A non-contact heart rate estimation method based on multimodal signal fusion, characterized in that: include: S1, obtain sample face video; S2, using image sharpening technology to highlight the subtle color changes in the sample face video and obtain the sharpened face video image; S3, locates the region of interest based on facial feature points in the sharpened face video; S4, extracting signals of the HSV and Lab color spaces from the region of interest, and fusing the signals of the two color spaces to obtain a mixed signal containing information of the two color spaces; S5, using a background brightness compensation strategy to suppress ambient light interference on the mixed signal to obtain an original brightness compensated signal; The step S5 specifically includes: extracting a luminance signal outside the region of interest as a reference signal, subtracting the reference signal from the mixed signal in the HSV color space and the mixed signal in the Lab color space to obtain luminance compensation, and performing a weighted addition operation on the luminance-compensated mixed signal in the HSV color space and the mixed signal in the Lab color space to obtain an original luminance-compensated signal; S6, performing median filtering on the original brightness compensation signal to obtain a brightness compensation signal after median filtering; subtracting the brightness compensation signal after median filtering from the original brightness compensation signal to obtain a color conversion signal; S7, normalizing the color conversion signal by taking the maximum and minimum values of the color conversion signal as a signal normalization range to obtain a normalized signal; S8, adjusting the bandwidth of the original band-pass filter based on the historical heart rate, and band-pass filtering the normalized signal through the adjusted band-pass filter to obtain the rPPG signal; S9, performing time domain analysis and frequency domain analysis on the rPPG signal, and weighting the heart rate value estimated by the time domain analysis and the heart rate value estimated by the frequency domain analysis to obtain a heart rate estimation result; S10, correcting the heart rate estimation result by using a historical data smoothing strategy to obtain a final heart rate estimation value; The step S10 specifically includes calculating the average of the previous five heart rates based on the five most recent heart rate values, using the following formula: Among them, H1, H2, H3, H4 and H5 are the heart rate values of the first five times; Indicates the average of the previous five heart rates; The heart rate estimation result H current and the average of the previous five heart rates Calculate the difference, if the difference exceeds the preset threshold ΔH threshold ,Right now: This indicates that there is an abnormal fluctuation between the current heart rate estimation result and the historical estimation result. The current heart rate estimation result and the average of the previous five times are weighted and fused to eliminate the influence of the abnormal fluctuation. The weighted fusion formula is as follows: Among them, β represents the weight of the current heart rate estimation result, and β>0.5 is used to ensure that the current heart rate estimation result has a larger proportion during fusion; H final Indicates the final heart rate estimate.

2. The non-contact heart rate estimation method based on multimodal signal fusion according to claim 1, characterized in that: Said S4 specifically includes: Color signal extraction is performed on the HSV color space and the Lab color space to obtain color signal component H, color signal component S, color signal component V, color signal component L, color signal component a, and color signal component b. The color signal component H, color signal component S, and color signal component V are weighted to obtain a mixed signal of the HSV color space; the color signal component L, color signal component a, and color signal component b are weighted to obtain a mixed signal of the Lab color space.

3. The non-contact heart rate estimation method based on multimodal signal fusion according to claim 1, characterized in that: In S8, adjusting the bandwidth of the original bandpass filter based on the historical heart rate specifically includes: Get the preset bandwidth range of the original band-pass filter, obtain the heart rate mean within the specified time period based on the original band-pass filter, convert the heart rate mean into heart rate frequency, and adaptively adjust the bandwidth range of the original band-pass filter based on the heart rate frequency to obtain the adjusted band-pass filter. The calculation formula is as follows: f low,t =f HR,t -Δf low ; f high,t =f HR,t +Δf high ; Where Δf low Indicates the bandpass range to the left of the adjusted bandpass filter reference frequency; Δf high Indicates the bandpass range to the right of the adjusted bandpass filter reference frequency; f HR,t Indicates the heart rate frequency corresponding to the mean heart rate during a specified period; f low,t and f high,t Indicates the range of the adjusted bandpass filter, which is used to filter out frequency components in [f low,t , f high,t ] part outside the scope.

4. The non-contact heart rate estimation method based on multimodal signal fusion according to claim 1, characterized in that: S9 specifically includes: extracting frequency domain information from the rPPG signal using FFT, calculating the frequency component with the largest amplitude in the spectrum based on the frequency domain information, and obtaining a heart rate value estimated by frequency domain analysis; calculating the heart rate value estimated by time domain analysis based on the peak interval in the rPPG signal; and combining the heart rate value estimated by time domain analysis and the heart rate value estimated by frequency domain analysis by weighted averaging to obtain a heart rate estimation result; the heart rate estimation result is calculated using the following formula: HR=α×HR freq +(1-α)×HR time ; Among them, HR freq Represents the heart rate value estimated by frequency domain analysis; HR time represents the heart rate value estimated by time domain analysis; HR represents the heart rate estimation result; α is the weighting coefficient with a value range of (0,1), which is used to control the weight of frequency domain and time domain information in heart rate estimation.

5. A non-contact heart rate estimation system based on multimodal signal fusion, characterized in that: include: Face video acquisition module, used to obtain sample face videos; A sharpening module is used to highlight subtle color changes in the sample face video through image sharpening technology to obtain a sharpened face video image; A region of interest positioning module is used to locate the region of interest based on facial feature points in the sharpened face video; A mixed signal acquisition module is used to extract signals of the HSV and Lab color spaces from the region of interest, fuse the signals of the two color spaces, and obtain a mixed signal containing information of the two color spaces; The original brightness compensation signal acquisition module is used to suppress the ambient light interference by using the background brightness compensation strategy on the mixed signal to obtain the original brightness compensation signal; The original brightness compensation signal acquisition module is specifically used to: extract the brightness signal outside the region of interest as a reference reference signal, subtract the reference reference signal from the mixed signal of the HSV color space and the mixed signal of the Lab color space to obtain brightness compensation, and perform a weighted addition operation on the brightness compensated mixed signal of the HSV color space and the mixed signal of the Lab color space to obtain the original brightness compensation signal; A color conversion signal acquisition module is used to perform median filtering on the original brightness compensation signal to obtain a brightness compensation signal after median filtering; and to subtract the brightness compensation signal after median filtering from the original brightness compensation signal to obtain a color conversion signal; a normalized signal acquisition module, configured to normalize the color conversion signal by taking the maximum and minimum values of the color conversion signal as a signal normalization range to obtain a normalized signal; An rPPG signal acquisition module is used to adjust the bandwidth of the original bandpass filter based on the historical heart rate, and to perform bandpass filtering on the normalized signal through the adjusted bandpass filter to obtain the rPPG signal; A heart rate estimation module is used to perform time domain analysis and frequency domain analysis on the rPPG signal, and to weight the heart rate value estimated by the time domain analysis and the heart rate value estimated by the frequency domain analysis to obtain a heart rate estimation result; The final heart rate estimation value acquisition module corrects the heart rate estimation result through the historical data smoothing strategy to obtain the final heart rate estimation value; The final heart rate estimation value acquisition module is specifically used to calculate the average of the previous five heart rates based on the most recent five heart rate values. The calculation formula is as follows: Among them, H1, H2, H3, H4 and H5 are the heart rate values of the first five times; Indicates the average of the previous five heart rates; The heart rate estimation result H 2urrent and the average of the previous five heart rates Calculate the difference, if the difference exceeds the preset threshold ΔH threshold ,Right now: This indicates that there is an abnormal fluctuation between the current heart rate estimation result and the historical estimation result. The current heart rate estimation result and the average of the previous five times are weighted and fused to eliminate the influence of the abnormal fluctuation. The weighted fusion formula is as follows: Among them, β represents the weight of the current heart rate estimation result, and β>0.5 is used to ensure that the current heart rate estimation result has a larger proportion during fusion; H final Indicates the final heart rate estimate.

Images