Remote heart rate measurement method and system based on physical information guide signal reconstruction

Abstract

The invention discloses a remote heart rate measurement method and system based on physical information guide signal reconstruction. Physical model prior and a data-driven deep network are fused, and a time-frequency joint supervision strategy is supplemented, so that high-precision and high-robustness extraction of rPPG signals is realized. By constructing a deep learning framework including feature enhancement, double-branch decomposition, physical guidance reconstruction and mixed domain optimization, weak physiological pulse signals and strong interference artifacts are effectively decoupled and separated from a face video including complex motion and illumination noise, so that the waveform fidelity of the extracted signals in a time domain is ensured, and the extraction accuracy is improved. And the periodic purity in the frequency domain is more directly optimized, so that the quality and the reliability of a final output signal are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of remote physiological signal monitoring technology, and more specifically, to a remote heart rate measurement method and system based on physical information-guided signal reconstruction. Background Technology

[0002] Remote photoplethysmography (rPPG) is a non-contact physiological signal measurement technique that measures vital signs such as heart rate by analyzing subtle changes in skin color caused by heartbeats in facial videos captured by ordinary cameras. Due to its non-contact, low-cost, and convenient characteristics, it shows great potential for application in fields such as telemedicine, health monitoring, and driver fatigue detection.

[0003] However, the widespread application of rPPG technology faces a core challenge: the pulse signal it extracts (i.e., the optical changes caused by the periodic variations in blood volume) is extremely weak and easily obscured by various noise sources. The main noise sources include: 1) motion artifacts, caused by minute movements of the subject's head or body, facial expressions, etc.; and 2) lighting fluctuations, caused by changes or flickering of ambient light. These noises introduce interference into the video signal that is far stronger than the actual pulse signal, leading to a significant decrease in the signal-to-noise ratio (SNR) of the extracted signal, and compromising accuracy.

[0004] To address this issue, researchers have proposed several approaches. Early signal processing methods, such as Independent Component Analysis (ICA) or Chromaticity-Based Techniques (CHROM), perform well in static or controlled environments, but their performance degrades sharply in real dynamic scenes filled with motion and changing lighting.

[0005] In recent years, deep learning techniques, especially convolutional neural networks (CNNs), have been widely applied to rPPG signal extraction. Models such as PhysNet have achieved significant progress compared to traditional methods by constructing spatiotemporal networks to learn pulse features directly from videos. However, current deep learning methods still have two key limitations:

[0006] First, there is a lack of explicit physical constraints on signal decomposition. Most existing models operate in an "end-to-end" black-box mode. Although they can output a waveform, they lack clear physical guidance for the decomposition process of pulse signals and noise signals. This makes the models prone to confusion when faced with complex noise, misclassifying some noise as pulse signals, or vice versa, thus limiting their robustness.

[0007] Second, the optimization objective of the loss function is one-sided. Existing models primarily focus on optimizing the time-domain similarity between the output waveform and the true waveform (e.g., using mean squared error or negative Pearson correlation coefficient). This approach neglects the direct optimization of the signal's frequency domain quality. However, the periodicity of the signal, i.e., its energy concentration in the frequency domain, is a key factor determining the signal-to-noise ratio (SNR). The lack of direct supervision in the frequency domain makes it difficult for the model to learn pulse signals with high periodic purity.

[0008] Therefore, there is an urgent need for a new technical solution that can combine the prior knowledge of physical models with the powerful feature learning capabilities of deep networks, and jointly optimize signal quality from both time and frequency dimensions, thereby achieving robust decomposition of noise and accurate extraction of high signal-to-noise ratio rPPG signals. Summary of the Invention

[0009] The purpose of this invention is to overcome the technical shortcomings of existing remote photoplethysmography (rPPG) signal extraction methods, which suffer from weak noise resistance and low signal-to-noise ratio in real dynamic scenarios due to the lack of physical constraints and single time-domain optimization. To address this, this invention provides a remote heart rate measurement method and system based on physically guided signal reconstruction. This method integrates a prior physical model with a data-driven deep network and employs a time-frequency joint supervision strategy to achieve high-precision and robust extraction of rPPG signals. By constructing a unified deep learning framework that includes feature enhancement, bi-branch decomposition, physically guided reconstruction, and hybrid domain optimization, it can effectively decouple and separate weak physiological pulse signals from strong interference artifacts in facial videos containing complex motion and lighting noise. This approach not only ensures the waveform fidelity of the extracted signal in the time domain but also directly optimizes its periodic purity in the frequency domain, thereby significantly improving the quality and reliability of the final output signal.

[0010] In a first aspect, the present invention provides a remote heart rate measurement method based on physical information-guided signal reconstruction, the method comprising:

[0011] Acquire facial video segments, and perform inter-frame differencing on the complete video sequence to enhance dynamic change features related to pulse and suppress static background noise, generating a feature-enhanced video stream;

[0012] The enhanced video stream is input into a dual-branch core feature extraction network to extract physiological pulse wave waveforms and non-pulse noise artifacts.

[0013] A signal reconstruction module guided by physical information reconstructs physiological pulse wave waveforms and non-pulse noise artifacts, obtains the reconstructed RGB time series signal, and calculates the physical reconstruction loss. The physical reconstruction loss is then introduced into a hybrid time-frequency domain loss function for backpropagation to optimize network parameters.

[0014] The pulse signals extracted from the local path and the global path in the pulse signal branch are weighted and fused to obtain the remote photoplethysmography pulse wave signal.

[0015] The extracted remote photoplethysmography (PPG) signal is further processed to obtain the heart rate;

[0016] The dual-branch core feature extraction network includes an initial feature extractor, a pulse signal branch, and a noise signal branch.

[0017] The initial feature extractor extracts initial features from the feature-enhanced video stream;

[0018] The pulse signal branch learns the initial features and extracts the physiological pulse waveform. The pulse signal branch includes a local path and a global path, which interact through a bidirectional information flow. The local path operates at high spatial resolution and focuses on learning fine textures and short-range temporal dynamics. The global path captures global contextual information and long-range temporal dependencies by adaptively performing spatial average pooling on the feature maps of the local path.

[0019] The noise signal branch explicitly learns the initial features and models non-pulse noise artifacts caused by motion and illumination variations.

[0020] Preferably, the temporal difference operation specifically includes: processing a complete facial video segment... Perform an overall temporal shift to obtain a video sequence that is one frame ahead. A video sequence delayed by one frame Calculate the first difference video Second difference video :

[0021]

[0022] The first and second difference videos are spliced ​​together and then processed through a fusion gate to obtain a feature-enhanced video stream.

[0023] Preferably, the local path in the pulse signal branch includes multiple cascaded Texture-aware Short-term Temporal Feature Blocks (TSTF-Blocks). Each TSTF-Block includes a first convolutional module, a second convolutional module, and a first 3D max-pooling layer, all cascaded in sequence. The implementation process is as follows: the initial input features are fed into the first TSTF-Block for processing to obtain features, and the intermediate features processed by the first convolutional module in the first TSTF-Block are input into the first global long-term temporal feature block. Other TSTF-Blocks satisfy the following: the i-th TSTF-Block receives the weighted result of the features output by the previous TSTF-Block and the intermediate features of the i-th global long-term temporal feature block, processes it to obtain features, and inputs the intermediate features processed by the first convolutional module in the current TSTF-Block into the i-th global long-term temporal feature block.

[0024] More preferably, the global path in the pulse signal branch includes multiple cascaded Global Long-term Temporal Feature Blocks (GLTF-Blocks). The first GLTF-Block includes a third convolutional module and a second 3D max pooling layer, which are cascaded in sequence. Other GLTF-Blocks include a three-dimensional adaptive average pooling layer, a fourth convolutional module, a fifth convolutional module, and a second 3D max pooling layer, which are cascaded in sequence. The implementation process is as follows: the first GLTF-Block receives the intermediate features output by the first texture short-term temporal feature block and processes them to obtain features. Other GLTF-Blocks all satisfy the following: the i-th GLTF-Block receives the features output by the previous GLTF-Block, which are processed sequentially by the three-dimensional adaptive average pooling layer and the fourth convolutional module to obtain intermediate features. The current intermediate features are input to the i-th texture short-term temporal feature block, and the result of the weighted sum of the intermediate features of the i-th texture short-term temporal feature block and the result of the weighted sum are input to the fifth convolutional module and the second 3D max pooling layer for feature extraction.

[0025] Most preferably, each of the three outputs—the local path, the global path, and the noise signal branch—is connected in series with a signal generation module that incorporates spatiotemporal attention.

[0026] The signal generation module with spatiotemporal attention includes a spatiotemporal attention module and a signal generator connected in series.

[0027] The spatiotemporal attention module includes a spatial attention unit and a temporal attention unit: the spatial attention unit generates a spatial weight map by applying convolution operations in the spatial dimension and combining it with the Sigmoid function, thereby highlighting key parts of the facial skin region related to the pulse and suppressing background or irrelevant areas; the temporal attention unit performs convolution operations in the temporal dimension and generates a temporal weight map by combining it with the Sigmoid function, in order to emphasize dynamic changes consistent with the heartbeat cycle and weaken non-periodic interference.

[0028] The signal generator is used to convert the feature map processed by the spatiotemporal attention module into a one-dimensional time series signal.

[0029] Preferably, the hybrid time-frequency domain loss function Including physical reconstruction loss Time domain loss, frequency domain loss The time-domain loss includes the negative Pearson correlation coefficient loss. and mean square error loss ;

[0030]

[0031] in, , , , They are respectively The weighting coefficients.

[0032] More preferably, the physical reconstruction loss Represented as:

[0033]

[0034] in This represents the reconstructed RGB time series signal. This represents the true, original average RGB time series signal.

[0035] In a second aspect, the present invention provides a remote heart rate measurement system, comprising:

[0036] The feature enhancement module is used to perform temporal difference processing on the input facial video signal to highlight dynamic physiological features and obtain a feature-enhanced video stream.

[0037] The feature extraction module is used to extract physiological pulse wave waveforms and non-pulse noise artifacts from the feature-enhanced video stream;

[0038] The physical information-guided signal reconstruction module is used to reconstruct physiological pulse wave waveforms and non-pulse noise artifacts, obtain the reconstructed RGB time series signal, and calculate the physical reconstruction loss. The physical reconstruction loss is then introduced into a hybrid time-frequency domain loss function for backpropagation to optimize network parameters.

[0039] The pulse wave signal extraction module is used to perform weighted fusion of the pulse signals extracted from the local path and the global path in the pulse signal branch, and finally output the remote photoplethysmography pulse wave signal.

[0040] The physiological signal extraction module is used to further process the extracted remote photoplethysmography (PPG) pulse wave signal to obtain the heart rate.

[0041] Thirdly, the present invention provides an electronic device, characterized in that it includes a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the method.

[0042] Fourthly, the present invention provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the method.

[0043] The beneficial effects of this invention are at least as follows:

[0044] This invention utilizes a novel hybrid deep learning architecture that combines physically-guided signal reconstruction with a time-frequency joint optimization strategy to effectively decouple weak physiological pulse signals from strong motion and illumination noise. The physical reconstruction loss introduced by this method provides strong physical constraints for signal decomposition, while the hybrid time-frequency domain loss function simultaneously ensures waveform fidelity and periodic purity. This allows the model to focus more on color change characteristics caused by heartbeats and consistent with the physical model of skin reflection, thereby significantly improving the robustness and signal-to-noise ratio (SNR) of rPPG signal extraction, achieving state-of-the-art performance on multiple public datasets. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the overall architecture of the method of the present invention;

[0046] Figure 2 This is a flowchart illustrating the architecture of the pulse signal branch in this invention. Detailed Implementation

[0047] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operating procedures.

[0048] Experimental Datasets and Data Preprocessing

[0049] To verify the effectiveness and robustness of the method of the present invention, this embodiment uses two large-scale public rPPG datasets that are widely used internationally: the PURE dataset and the UBFC-rPPG dataset.

[0050] 1) PURE Dataset: This dataset is specifically designed to evaluate the robustness of the rPPG algorithm in the presence of significant motion interference. It contains 60 videos from 10 subjects. During the data collection process, subjects were asked to perform six different head movement scenarios, including standing still, small-amplitude translation, large-amplitude translation, small-amplitude head turning, large-amplitude head turning, and speaking. These carefully designed scenarios comprehensively simulate various motion artifacts that may be encountered in real-world applications.

[0051] 2) UBFC-rPPG Dataset: This dataset is designed to simulate more natural, uncontrolled application scenarios. It contains videos of 42 subjects. During the recording process, subjects were asked to play a game that did not require high mathematical skills while being filmed, accompanied by natural, slight head movements and changes in lighting caused by screen reflections.

[0052] For the videos in the two datasets mentioned above, this invention employs a unified data preprocessing workflow, the specific steps of which are as follows:

[0053] Facial Region Detection and Tracking: First, in the first frame of the video, a traditional Haar feature cascade classifier (i.e., the Viola-Jones algorithm) is used to locate the facial region and determine its initial bounding box. Then, for all subsequent frames of the facial video segment, the KLT tracking algorithm is used to continuously track the facial region within the bounding box.

[0054] Region of Interest (ROI) Determination and Cropping: To improve the signal-to-noise ratio and reduce background interference, the facial bounding box was enlarged by 1.2 times in both height and width to ensure that the face is fully contained even during head movement. Subsequently, each frame of the image was cropped based on the enlarged bounding box.

[0055] Size normalization: All cropped video frames are scaled to a fixed resolution of 192×128 pixels. This step ensures that the data input to subsequent deep learning networks is of consistent size.

[0056] Ground-truth signal alignment: The ground-truth PPG signal provided by the dataset, which is synchronously collected by a contact device (pulse oximeter), is interpolated or downsampled to keep its sampling rate consistent with the video frame rate (30Hz) to ensure that the predicted signal is strictly aligned with the real signal in time during training and evaluation.

[0057] See Figure 1 As shown, this embodiment provides a remote heart rate measurement method based on physical information-guided signal reconstruction, including the following steps:

[0058] Step 1: Obtain facial video clips The facial video clips are processed using temporal difference operations to enhance dynamic features related to the pulse and suppress static background noise, generating a feature-enhanced video stream. ;in This represents the number of channels in the video. For common RGB color videos, this value is usually 3, corresponding to the three color channels: red, green, and blue. The time dimension of the video; and These are the height and width of each frame in the video, in pixels.

[0059] Step 2: Input the enhanced video stream into a dual-branch core feature extraction network to extract physiological pulse wave waveforms and non-pulse noise artifacts;

[0060] The dual-branch core feature extraction network includes an initial feature extractor, a pulse signal branch, and a noise signal branch:

[0061] The initial feature extractor extracts initial features from the feature-enhanced video stream.

[0062] As an example, the initial feature extractor consists of multiple three-dimensional convolutional layers and spatial pooling layers for learning general spatiotemporal representations.

[0063] The pulse signal branch learns the initial features and extracts physiological rPPG pulse waveforms. See Appendix for details. Figure 2 The pulse signal branch employs a multi-scale feature interaction network structure, including a local path and a global path, which interact through bidirectional information flow to learn and extract physiological rPPG pulse waveforms. and The local path operates at high spatial resolution, focusing on learning fine textures and short-range temporal dynamics, while the global path captures global contextual information and long-range temporal dependencies by adaptively performing spatial average pooling on the feature maps of the local path.

[0064] Furthermore, the Local Path comprises multiple concatenated Texture Short-Range Temporal Feature Extraction Blocks (TSTF-Blocks). Each TSTF-Block includes a first convolutional module, a second convolutional module, and a first 3D max-pooling layer, all cascaded sequentially. The implementation process involves feeding the initial input features into the first TSTF-Block for processing to obtain the features. Simultaneously, the intermediate features processed by the first convolutional module in the first texture short-range temporal feature extraction block are input into the first global long-range temporal feature extraction block; other texture short-range temporal feature extraction blocks all satisfy the following: the i-th texture short-range temporal feature extraction block receives the features output by the previous texture short-range temporal feature extraction block. The result of weighting the intermediate features of the i-th global long-range time feature extraction block is processed to obtain the feature. Simultaneously, the intermediate features processed by the first convolutional module in the current texture short-range temporal feature extraction block are input into the i-th global long-range temporal feature extraction block. As an example, this embodiment uses 3 texture short-range temporal feature extraction blocks.

[0065] Furthermore, the Global Path comprises multiple concatenated Global Long-Range Temporal Feature Extraction Blocks (GLTF-Blocks). The first GLTF-Block includes a third convolutional module and a second 3D max-pooling layer, all cascaded sequentially. Other GLTF-Blocks include a 3D adaptive average pooling layer (AdaptiveAvgPool3D), a fourth convolutional module, a fifth convolutional module, and a second 3D max-pooling layer, all cascaded sequentially. The implementation process is as follows: the first GLTF-Block receives intermediate features output from the first texture short-range temporal feature extraction block, processes them, and obtains the features. All other global long-range time feature extraction blocks satisfy the following condition: the i-th global long-range time feature extraction block receives the features output by the previous global long-range time feature extraction block. The three-dimensional adaptive average pooling layer and the fourth convolutional module process the data sequentially to obtain intermediate features. The current intermediate features are input to the i-th texture short-range temporal feature extraction block. Simultaneously, the intermediate features of the i-th texture short-range temporal feature extraction block are weighted and the result is input to the fifth convolutional module and the second 3D max pooling layer for feature extraction. As an example, this embodiment uses three global long-range temporal feature extraction blocks.

[0066] As an example, the first, second, third, fourth, and fifth convolutional modules have the same structure, all including a 3D convolutional layer, batch normalization (BN), and the ReLU activation function.

[0067] The noise signal branch explicitly learns the initial features and models non-pulse noise artifacts caused by motion and illumination variations. This branch is structurally designed to complement the pulse branch, comprising a cascaded spatial convolution module, a temporal difference convolution module, multiple max-pooling layers, a spatial adaptive pooling module, and a convolutional layer. Specifically, this branch's architecture prioritizes spatial convolution to capture skin texture information within each frame, followed immediately by a temporal difference convolution to explicitly model the drastic inter-frame changes caused by motion. In subsequent network layers, this branch progressively reduces spatial resolution through multiple max-pooling layers while maintaining the full temporal dimension. This design ensures that this branch can focus on learning and extracting artifacts with spatial and temporal noise characteristics without confusing it with the fine pulsation signals that the pulse branch focuses on. Finally, a one-dimensional noise signal is generated through a spatial adaptive pooling layer and a final convolutional layer. .

[0068] In one implementation, each of the outputs of the local path, the global path, and the noise signal branch is connected in series with a signal generation module with spatiotemporal attention.

[0069] The signal generation module with spatiotemporal attention includes a spatiotemporal attention module and a signal generator connected in series.

[0070] The spatiotemporal attention module is used to simultaneously model the spatial and temporal importance distributions on a 3D feature map. Its structure includes a spatial attention unit and a temporal attention unit: the spatial attention unit generates a spatial weight map by applying convolution operations in the spatial dimension and combining it with a sigmoid function, thereby highlighting key pulse-related areas in the facial skin region and suppressing background or irrelevant areas; the temporal attention unit performs convolution operations in the temporal dimension and combines it with a sigmoid function to generate a temporal weight map, emphasizing dynamic changes consistent with the heartbeat cycle and weakening non-periodic interference caused by movement, blinking, or light flickering. The spatial and temporal weight maps are applied to the input features and jointly weighted, thus achieving spatiotemporal dual recalibration of the features. This module is applied in both the local and global path outputs of the pulse signal branch. By strengthening effective features related to the physiological pulse in the short-range dynamics of local textures and the long-range dependencies of the global pulse, it improves the accuracy and robustness of pulse signal extraction. Simultaneously, in the noise signal branch, this module is used to highlight salient patterns of motion and lighting artifacts, avoiding confusion with the pulse signal. Under the constraints of physical model reconstruction and time-frequency hybrid loss, the spatiotemporal attention module further ensures the spatial positioning accuracy and temporal dynamic consistency of the final output signal, thereby significantly enhancing the signal-to-noise ratio and reliability of the remote photoplethysmography pulse wave signal.

[0071] The signal generator (rPPGSignalGenerator) internally performs the following operations to convert a four-dimensional spatiotemporal feature map into a one-dimensional time series signal:

[0072] (1) Feature Refinement: The input feature map (i.e., the output feature of the last texture short-range temporal feature extraction block or the output feature of the last global long-range temporal feature extraction block) is further refined and integrated through a three-dimensional convolutional layer (3D Convolution), batch normalization (BN) and ReLU activation function.

[0073] (2) Spatiotemporal Dimension Transformation: A three-dimensional adaptive average pooling layer (AdaptiveAvgPool3d) is applied. The key function of this pooling layer is to compress the spatial dimension (height H and width W) of the feature map to 1x1 while fully preserving its temporal dimension (T). This operation effectively aggregates the spatial features at each time point into a single representative value, thereby transforming the four-dimensional spatiotemporal feature volume into a three-dimensional temporal feature stream (dimensions of B×C×T×1×1).

[0074] (3) Signal shaping: The channel dimension (C) is reduced to 1 through a 1×1×1 convolution, and then a tensor reshape operation is performed to finally obtain a one-dimensional corresponding pulse signal (i.e., or ).

[0075] Among them, the three signal generators, which are loaded from the local path, the global path, and the noise signal branch respectively, have the same structure but independent weights.

[0076] Step 3: The physical information-guided signal reconstruction module utilizes a preset pulse basis vector representing the optical properties of blood. A learnable motion basis vector for adaptive modeling of motion noise. The physiological pulse wave waveform and non-pulse noise artifacts are reconstructed to obtain the reconstructed RGB time series signal, and the physical reconstruction loss is calculated. Physical reconstruction loss A hybrid time-frequency domain loss function is introduced for backpropagation to optimize network parameters;

[0077] As an example, the pulse basis vector preset in this embodiment It was obtained through offline analysis of a facial video dataset containing multiple subjects in a static state. The specific calculation process includes the following sub-steps:

[0078] 1) Extract each video segment from the dataset. The original average RGB timing signal is obtained by averaging the spatial dimensions (height H and width W) of each frame. .

[0079] 2) To eliminate non-pulse-related interference, the signal of each channel... The signal is then processed. First, the signal is mean-reduced to eliminate the DC component. Then, singular spectrum analysis (SSA) is used to detrend the mean-reduced signal, removing low-frequency artifacts unrelated to the pulse caused by weak head movements or slow changes in illumination, thus obtaining a pure pulsating signal component. .

[0080] 3) The pure pulsating signal component obtained after processing all video clips. The data is aggregated along the time dimension, treating the three-channel RGB values ​​at each time point as a three-dimensional spatial point, thus constructing a large RGB three-dimensional spatial point cloud. Principal Component Analysis (PCA) is applied to this point cloud data to calculate the eigenvectors of its covariance matrix. Since the pulse signal is the most significant physiological factor causing skin color changes, its vibration direction corresponds to the direction of maximum data variance. Therefore, the first principal component of the point cloud data covariance matrix is ​​used as the pulse basis vector. The initial estimate.

[0081] 4) To ensure the consistency of vector direction and convert it into a unit vector, the obtained pulse basis vectors are subjected to direction alignment and L2 norm normalization to obtain the final preset pulse basis vectors. :

[0082]

[0083] Obtain the preset pulse basis vector and learnable motion basis vectors ( After optimization via backpropagation using a hybrid time-frequency domain loss function, the signal reconstruction module calculates the reconstructed average RGB signal. The calculation formula is as follows:

[0084]

[0085] in, The final fused pulse signal, and Let be a learnable scalar scaling factor. Based on this reconstruction process, a physical reconstruction loss is defined. The loss is defined as the average RGB signal after reconstruction. Compared to the true original average RGB signal The L1 distance between them is used as a regularization term to constrain the physical rationality of the signal decomposition process.

[0086]

[0087] The physically-guided signal reconstruction module provides a strong physical prior constraint for deep learning networks. Without this module, the network's decomposition of pulse and noise might be a purely "black box" mathematical process; while the decomposition result may be optimal in terms of the loss function, it may not accurately reflect the actual physical process. This invention introduces this module to enforce the decomposed pulse signal... and noise signals The original observed signal must be linearly reconstructed from a well-defined physical model (i.e., the skin reflex model) to ensure the physical interpretability and realistic plausibility of the signal decomposition. This mechanism greatly improves the model's ability to correctly decouple the true pulse signal when faced with complex and intense noise interference.

[0088] The hybrid time-frequency domain loss function includes physical reconstruction loss. Time domain loss, frequency domain loss The time-domain loss includes the negative Pearson correlation coefficient loss. Mean Square Error (MSE) Loss ;

[0089]

[0090] in, , , , They are respectively Weighting coefficients;

[0091] Pearson correlation coefficient:

[0092]

[0093] Therefore, the loss function Defined as:

[0094]

[0095] Mean square error:

[0096]

[0097] In the formula, A time series representing the ground-truth PPG signal; The time series of the rPPG signal predicted by the method of the present invention is represented; t represents the index of a time point in the signal; T represents the total number of time points in the signal; and Let represent the amplitudes of the true signal and the predicted signal at time t, respectively. and These represent the time averages of the entire real signal and the entire predicted signal, respectively.

[0098] The frequency domain loss employs power spectral density (PSD) loss, applied to the predicted signal respectively. and true signal Calculate their power spectral density (PSD). ) and PSD ( Then, the mean square error between the two power spectral density distributions is calculated as the final PSD loss.

[0099]

[0100] In the formula, and They represent the true signal respectively. and prediction signal The frequency domain energy distribution obtained after power spectral density analysis; Indicates the index of the frequency component in the frequency domain; This represents the total number of frequency components obtained after performing a Fourier transform. and Represent the true signal and the predicted signal in the frequency components, respectively. The power spectral density value at that location.

[0101] Step 4: Extract the pulse signals from the local path and global path in the pulse signal branch. , Weighted fusion is performed to ultimately output a high signal-to-noise ratio and high-precision rPPG signal. .

[0102]

[0103] in, This is a learnable fusion weight. This means that the network can autonomously learn and decide how much weight to assign to global path information in the final signal based on the training data.

[0104] In this way, the present invention achieves complementary advantages of two types of information: It provides fine waveform details from high-resolution features, while It provides periodic information with global consistency and greater robustness to local noise. Learnable fusion weights. This ensures that this complementarity can adaptively achieve an optimal balance, thereby improving the final output rPPG signal. It combines the advantages of high waveform fidelity and strong noise robustness.

[0105] Step 5: Process the extracted rPPG signal Further processing is performed to calculate and obtain the final physiological signal.

[0106] As an example, the physiological signal in this embodiment can be heart rate (HR), and the extracted rPPG signal can be analyzed using frequency domain analysis. Further processing is required:

[0107] First, for one-dimensional time series rPPG signals Apply Fast Fourier Transform (FFT) to transform it from the time domain to the frequency domain, and calculate its power spectrum to obtain the energy distribution of the signal at different frequencies;

[0108] Subsequently, in order to filter out noise interference that exceeds the normal physiological range, the dominant frequency with the largest energy value in the power spectrum was searched and determined only within a predefined frequency range that conforms to human physiological characteristics, namely 0.8 Hz to 2.2 Hz (corresponding to a heart rate range of 48 to 132 beats per minute).

[0109] Finally, the identified dominant frequency value (in Hertz Hz) is multiplied by 60 to convert it into a final heart rate value in beats per minute (BPM), thus completing the measurement of the entire physiological signal.

[0110] To verify the effectiveness of this invention, comprehensive experimental verification was conducted on two internationally recognized rPPG datasets (PURE and UBFC-rPPG), and quantitative comparisons were made with various current state-of-the-art methods (such as PhysNet, GLISNet, etc.).

[0111] Experimental results demonstrate that the proposed method exhibits state-of-the-art performance across all key evaluation metrics. Particularly noteworthy is its superior noise resistance on the PURE dataset, specifically designed for evaluating motion robustness. For instance, compared to the previous leading method GLISNet, the proposed method improves the signal-to-noise ratio (SNR) by over 2.5 dB; simultaneously, in terms of heart rate estimation accuracy, the mean absolute error (MAE) is reduced from 1.45 BPM to 0.97 BPM, and the root mean square error (RMSE) is significantly reduced from 2.99 BPM to 1.45 BPM.

[0112] On the UBFC-rPPG dataset, which simulates natural scenes, this invention also achieved top performance, with its two key accuracy metrics, MAE (1.10 BPM) and RMSE (2.46 BPM), both outperforming all comparable methods.

[0113] This invention provides an electronic device, specifically, the electronic device includes a memory and a processor, the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the embodiments.

[0114] The memory may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk drive. Communication between this system network element and at least one other network element is achieved through at least one communication interface (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0115] The bus can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0116] The memory is used to store programs. After receiving an execution instruction, the processor executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor or implemented by the processor.

[0117] The processor may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0118] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0119] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A remote heart rate measurement method based on physical information-guided signal reconstruction, characterized in that, The method includes: Acquire facial video segments, and perform inter-frame differencing on the complete video sequence to enhance dynamic change features related to pulse and suppress static background noise, generating a feature-enhanced video stream; The enhanced video stream is input into a dual-branch core feature extraction network to extract physiological pulse wave waveforms and non-pulse noise artifacts. A signal reconstruction module guided by physical information reconstructs physiological pulse wave waveforms and non-pulse noise artifacts, obtains the reconstructed RGB time series signal, and calculates the physical reconstruction loss. The physical reconstruction loss is then introduced into a hybrid time-frequency domain loss function for backpropagation to optimize network parameters. The pulse signals extracted from the local path and the global path in the pulse signal branch are weighted and fused to obtain the remote photoplethysmography pulse wave signal. The extracted remote photoplethysmography (PPG) signal is further processed to obtain the heart rate; The dual-branch core feature extraction network includes an initial feature extractor, a pulse signal branch, and a noise signal branch. The initial feature extractor extracts initial features from the feature-enhanced video stream; The pulse signal branch learns the initial features and extracts the physiological pulse waveform. The pulse signal branch includes a local path and a global path, which interact through a bidirectional information flow. The local path operates at high spatial resolution and focuses on learning fine textures and short-range temporal dynamics. The global path captures global contextual information and long-range temporal dependencies by adaptively performing spatial average pooling on the feature maps of the local path. The noise signal branch explicitly learns the initial features and models non-pulse noise artifacts caused by motion and illumination variations.

2. The method according to claim 1, characterized in that, The time difference operation specifically includes: processing complete facial video segments... Perform an overall temporal shift to obtain a video sequence that is one frame ahead. A video sequence delayed by one frame Calculate the first difference video Second difference video : ； The first and second difference videos are spliced ​​together and then processed through a fusion gate to obtain a feature-enhanced video stream.

3. The method according to claim 1, characterized in that, The local path in the pulse signal branch includes multiple cascaded texture short-range temporal feature extraction blocks, each texture short-range temporal feature extraction block including a first convolutional module, a second convolutional module, and a first 3D max pooling layer cascaded in sequence. The implementation process is as follows: the initial input features are fed into the first texture short-range temporal feature extraction block for processing to obtain features. At the same time, the intermediate features processed by the first convolution module in the first texture short-range temporal feature extraction block are input into the first global long-range temporal feature extraction block. Other texture short-range temporal feature extraction blocks all satisfy the following: the i-th texture short-range temporal feature extraction block receives the weighted result of the features output by the previous texture short-range temporal feature extraction block and the intermediate features of the i-th global long-range temporal feature extraction block, processes it to obtain features, and at the same time, the intermediate features processed by the first convolution module in the current texture short-range temporal feature extraction block are input into the i-th global long-range temporal feature extraction block.

4. The method according to claim 3, characterized in that, The global path in the pulse signal branch includes multiple cascaded global long-range time feature extraction blocks. The first global long-range time feature extraction block includes a third convolutional module and a second 3D max pooling layer, which are cascaded in sequence. Other global long-range time feature extraction blocks include a three-dimensional adaptive average pooling layer, a fourth convolutional module, a fifth convolutional module, and a second 3D max pooling layer, which are cascaded in sequence. The implementation process is as follows: The first global long-range temporal feature extraction block receives the intermediate features output by the first texture short-range temporal feature extraction block, and then processes them to obtain features; other global long-range temporal feature extraction blocks all satisfy the following: the i-th global long-range temporal feature extraction block receives the features output by the previous global long-range temporal feature extraction block, and the 3D adaptive average pooling layer and the fourth convolutional module process them in sequence to obtain intermediate features. The current intermediate features are input to the i-th texture short-range temporal feature extraction block, and at the same time, the result of the weighted sum of the intermediate features of the i-th texture short-range temporal feature extraction block is input to the fifth convolutional module and the second 3D max pooling layer for feature extraction.

5. The method according to claim 4, characterized in that, Each of the three outputs—the local path, the global path, and the noise signal branch—is connected in series with a signal generation module that incorporates spatiotemporal attention. The signal generation module with spatiotemporal attention includes a spatiotemporal attention module and a signal generator connected in series. The spatiotemporal attention module includes a spatial attention unit and a temporal attention unit: the spatial attention unit generates a spatial weight map by applying convolution operations in the spatial dimension and combining it with the Sigmoid function, thereby highlighting key parts of the facial skin region related to the pulse and suppressing background or irrelevant areas; the temporal attention unit performs convolution operations in the temporal dimension and generates a temporal weight map by combining it with the Sigmoid function, in order to emphasize dynamic changes consistent with the heartbeat cycle and weaken non-periodic interference. The signal generator is used to convert the feature map processed by the spatiotemporal attention module into a one-dimensional time series signal.

6. The method according to claim 1, characterized in that, The hybrid time-frequency domain loss function Including physical reconstruction loss Time domain loss, frequency domain loss The time-domain loss includes the negative Pearson correlation coefficient loss. and mean square error loss ; ； in, , , , They are respectively The weighting coefficients.

7. The method according to claim 6, characterized in that, The physical reconstruction loss Represented as: ； in This represents the reconstructed RGB time series signal. This represents the true, original average RGB time series signal.

8. A remote heart rate measurement system implementing the method of any one of claims 1-7, characterized in that, include: The feature enhancement module performs temporal difference processing on the input facial video signal to highlight dynamic physiological features. Obtain the feature-enhanced video stream; A shared feature module is used to extract physiological pulse wave waveforms and non-pulse noise artifacts from the feature-enhanced video stream; The physical information-guided signal reconstruction module is used to reconstruct physiological pulse wave waveforms and non-pulse noise artifacts, obtain the reconstructed RGB time series signal, and calculate the physical reconstruction loss. The physical reconstruction loss is then introduced into a hybrid time-frequency domain loss function for backpropagation to optimize network parameters. The pulse wave signal extraction module is used to perform weighted fusion of the pulse signals extracted from the local path and the global path in the pulse signal branch, and finally output the remote photoplethysmography pulse wave signal. The physiological signal extraction module is used to further process the extracted remote photoplethysmography (PPG) pulse wave signal to obtain the heart rate.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method of any one of claims 1-7.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the method of any one of claims 1-7.

Images