A non-contact heart rate detection method, system and related equipment

Abstract

The present invention discloses a non-contact heart rate detection method, system and related equipment. The detection method includes: obtaining the original face video and performing preprocessing to obtain a video frame sequence; calculating the pixel-by-pixel difference between adjacent frames based on the video frame sequence to extract a differential video frame sequence; performing convolution operations and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and applying 3D convolution to obtain spatiotemporal features; using time differential convolution to project the spatiotemporal features into query vectors, key vectors and value vectors, and calculating the correlation between vectors in the frequency domain. By designing two paths to process the original video frame and the differential video frame sequence respectively, by extracting time-frequency features, the time-frequency information related to the heart rate in the video frame is effectively captured. The features of the two paths are spliced ​​in the channel dimension, and finally spatial averaging, upsampling and 1D convolution are performed to finally obtain accurate rPPG signal prediction results.

Description

Technical Field

[0001] The present application relates to the field of heart rate detection technology, and specifically to a non-contact heart rate detection method, system and related equipment. Background Art

[0002] Photoplethysmography (PPG) is a non-invasive optical technology that is widely used to monitor vital signs such as heart rate (HR) and heart rate variability (HRV), providing an important basis for physiological and psychological state analysis. Traditional PPG signals are usually collected through contact sensors (such as smart watches, pulse oximeters, etc.). Although these devices are effective in clinical and daily monitoring, their reliance on skin contact may cause user discomfort and are prone to introducing artifacts due to movement or differences in skin characteristics, thus limiting their practicality in continuous monitoring and real-world scenarios.

[0003] To solve the above problems, remote photoplethysmography (rPPG) came into being. rPPG achieves non-contact heart rate monitoring by analyzing subtle color changes in face videos, which has significant advantages in comfort and practicality. However, rPPG technology still faces many challenges in practical applications, such as sensitivity to changes in lighting conditions, motion artifact interference, and low signal-to-noise ratio, which seriously affect its detection accuracy and robustness.

[0004] Early rPPG methods mainly rely on traditional signal processing techniques, such as color space transformation and spectral analysis. Although these methods are feasible in controlled environments, they rely on manually designed features and have limited adaptability, making it difficult to cope with dynamic and complex environments. With the rapid development of deep learning, data-driven deep learning methods have gradually become mainstream. Some studies have designed non-end-to-end deep learning models based on prior knowledge of traditional methods, which usually require complex preprocessing steps (such as extracting spatiotemporal maps based on regions of interest (ROIs)) to capture rPPG features. However, the applicability of these methods is limited by strict preprocessing processes and is difficult to meet the needs of practical applications.

[0005] In recent years, end-to-end deep learning methods have emerged, which can predict rPPG signals directly from facial videos, reducing the reliance on preprocessing. However, these methods lack in feature diversity and are sensitive to real-world challenges such as head motion and illumination changes, resulting in poor performance in complex real-world scenarios. Recently, Transformer-based architectures have shown significant potential in capturing long-term spatiotemporal dependencies, but the high computational cost of their standard self-attention mechanism limits their deployment in real-time applications. Summary of the invention

[0006] In this embodiment, a non-contact heart rate detection method, system, electronic device and storage medium are provided to solve the problems of poor robustness and high computational complexity of the non-contact heart rate detection method in the prior art.

[0007] In a first aspect, an embodiment of the present invention provides a non-contact heart rate detection method, the non-contact heart rate detection method comprising:

[0008] Obtain the original face video and perform preprocessing to obtain a video frame sequence;

[0009] Calculating pixel-by-pixel differences between adjacent frames based on the video frame sequence to extract a differential video frame sequence;

[0010] Perform convolution operations and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and apply 3D convolution to obtain spatiotemporal features;

[0011] Use time difference convolution to project the spatiotemporal features into query vector, key vector and value vector, and calculate the correlation between vectors in the frequency domain;

[0012] A learnable frequency domain filter is used to selectively enhance relevant frequencies and suppress noise, and then converted back to the time domain via inverse Fourier transform.

[0013] Perform nonlinear feature enhancement on the processed feature map and add it to the input feature map to generate the final output;

[0014] The optimized time-frequency features are extracted layer by layer, the optimized feature representations are concatenated, and the predicted rPPG signals are generated. The heart rate mixed loss function is used to guide the adjustment of model parameters.

[0015] In an optional embodiment, the original face video is obtained and preprocessed to obtain a video frame sequence, including:

[0016] The original face video is segmented according to the predetermined time sequence length;

[0017] Use the face detection tool to detect the face area and crop the video frame based on the area;

[0018] The cropped video frames are uniformly adjusted to a fixed size to obtain a video frame sequence.

[0019] In an optional embodiment, a convolution operation and processing are performed on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and a 3D convolution is applied to obtain spatiotemporal features, including:

[0020] Performing a convolution operation on the video frame sequence and the differential video frame sequence, and sequentially performing batch normalization, ReLU activation, and maximum pooling to extract preliminary features;

[0021] Apply 3D convolution to segment it into non-overlapping spatiotemporal tags and obtain video frame spatiotemporal features and differential frame spatiotemporal features.

[0022] In an optional embodiment, time-differential convolution is used to project the spatiotemporal features into a query vector, a key vector, and a value vector, and the correlation between the vectors is calculated in the frequency domain, including:

[0023] A two-stage optimization of frequency domain features is performed, which includes optimizing the frequency components in the spatial dimension using learnable spatial-frequency weights;

[0024] In the temporal dimension, one-dimensional Fourier transform is applied to capture periodic patterns and temporal dependencies to enhance the focusing ability of frequency components related to physiological rhythms.

[0025] In an optional embodiment, a learnable frequency domain filter is used to selectively enhance relevant frequencies and suppress noise, and then converted back to the time domain by inverse Fourier transform, including:

[0026] The frequency domain representation is filtered using a learnable frequency domain filter to obtain filtered frequency domain features;

[0027] The filtered frequency domain features are converted back to the time domain through inverse Fourier transform to obtain the time domain features.

[0028] In an optional embodiment, nonlinear feature enhancement is performed on the processed feature map and added to the input feature map to generate a final output, including:

[0029] The GEGLU function is used to perform nonlinear feature enhancement on the time domain features;

[0030] A convolutional layer is used to project the enhanced features back to the original dimension and added to the input feature map to obtain the final output.

[0031] In an optional embodiment, the optimized time-frequency features are extracted layer by layer, the optimized feature representations are concatenated, and the predicted rPPG signals are generated by processing, and the heart rate mixed loss function is used to guide the model parameter adjustment, including:

[0032] Use multiple layers of frequency domain Transformer encoder layers to extract and optimize time-frequency features layer by layer;

[0033] splicing the optimized feature representations, and generating a predicted rPPG signal based on the optimized feature representations;

[0034] The loss is calculated using the heart rate hybrid loss function based on the predicted rPPG signal and the true rPPG signal;

[0035] The back-propagation algorithm is used to update the model parameters based on the calculated loss value.

[0036] Compared with the prior art, the non-contact heart rate detection method of the present invention has the following beneficial effects:

[0037] The present invention processes the original video frame and the differential video frame sequence respectively by designing two paths. First, the input of each path is shallowly processed, and then the time-frequency information related to the heart rate in the video frame is effectively captured by extracting the time-frequency features. By enhancing the complementary information between the original frame path and the differential frame path, the representation ability of dynamic changes and spatiotemporal patterns is further improved, and the visual information that may be lost in the differential video frame features in the differential video frame path can be effectively processed and the influence of motion artifacts can be reduced. After the above processing, the features of the two paths are spliced ​​in the channel dimension, and finally spatial averaging, upsampling and 1D convolution are performed to finally obtain accurate rPPG signal prediction results.

[0038] In a second aspect, an embodiment of the present invention provides a non-contact heart rate detection system, including:

[0039] A data preprocessing module is used to obtain the original face video and perform preprocessing to obtain a video frame sequence;

[0040] A differential frame sequence calculation module, used for calculating the pixel-by-pixel difference between adjacent frames based on the video frame sequence to extract a differential video frame sequence;

[0041] A shallow feature extraction module is used to perform convolution operations and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and apply 3D convolution to obtain spatiotemporal features;

[0042] The frequency-domain spatiotemporal self-attention module and the frequency-domain spatiotemporal cross-attention module are used to project the spatiotemporal features into query vectors, key vectors, and value vectors using temporal difference convolution, and calculate the correlation between vectors in the frequency domain;

[0043] A frequency-domain discriminative feed-forward network module is used to selectively enhance relevant frequencies and suppress noise using a learnable frequency-domain filter, and then transform back to the time domain via an inverse Fourier transform;

[0044] Multi-layer frequency domain Transformer encoder layer, which performs nonlinear feature enhancement on the processed feature map and adds it to the input feature map to generate the final output;

[0045] The rPPG signal prediction module is used to extract and optimize time-frequency features layer by layer, concatenate the optimized feature representations, and process the generated predicted rPPG signals, using the heart rate mixed loss function to guide the adjustment of model parameters.

[0046] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0047] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the non-contact heart rate detection method as described in the first aspect.

[0048] Compared with the prior art, the beneficial effects of the non-contact heart rate detection system, electronic device and storage medium of the present invention are the same as those of the non-contact heart rate detection method described in the first aspect, so they will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 is a flow chart of a non-contact heart rate detection method according to an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of a non-contact heart rate detection method in an embodiment of the present invention;

[0052] Figure 3 It is a structural block diagram of a non-contact heart rate detection system in an embodiment of the present invention;

[0053] Figure 4 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] In an embodiment of the present invention, a non-contact heart rate detection method is provided. Figure 1 is a flow chart of the non-contact heart rate detection method of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0057] S100, obtaining an original face video, and performing preprocessing to obtain a video frame sequence;

[0058] In this embodiment, the original face video is obtained and preprocessed to obtain a video frame sequence, including:

[0059] The original face video is segmented according to the predetermined time sequence length;

[0060] Use the face detection tool to detect the face area and crop the video frame based on the area;

[0061] The cropped video frames are uniformly adjusted to a fixed size to obtain a video frame sequence.

[0062] Exemplarily, in the data preprocessing stage, the original face video is first segmented according to the time series length T (T=160) to obtain multiple video samples. For each sliced ​​video sample, the YOLO5Face face detection tool is used to detect the face area, and the video frame is cropped based on the area. To ensure consistency, all subsequent video frames are cropped according to the face area detected in the first frame. Finally, the cropped video frames are uniformly adjusted to a fixed size H×W (H=128, W=128), thereby obtaining an input video frame sequence X with a shape of N×3×T×H×W. Among them, N represents the batch size, T is the time series length, and H and W are the height and width of the video frame, respectively.

[0063] S200, calculating pixel-by-pixel differences between adjacent frames based on the video frame sequence to extract a differential video frame sequence;

[0064] Specifically, firstly, the original input video frame sequence , by calculating the pixel-by-pixel difference between adjacent frames, extract the differential video frame sequence , the specific calculation method is as follows:

[0065] ;

[0066] ;

[0067] in, , Represents an absolute value operation.

[0068] S300, performing convolution operation and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and applying 3D convolution to obtain spatiotemporal features;

[0069] In this embodiment, the video frame sequence and differential video frame sequence Convolution operations and processing are performed to reduce the spatial size and extract preliminary features, and 3D convolution is applied to obtain spatiotemporal features, including:

[0070] For the video frame sequence and the differential video frame sequence Perform convolution operation, batch normalization, ReLU activation and maximum pooling in sequence to extract preliminary features;

[0071] Apply 3D convolution to segment it into non-overlapping spatiotemporal tags and obtain video frame spatiotemporal features and differential frame spatiotemporal features.

[0072] For example, for an input video frame sequence and differential video frame sequence First, they are processed by three shallow feature extraction modules, which are composed of 3D convolution layer, batch normalization layer, ReLU activation layer and maximum pooling layer. The convolution kernel sizes of the 3D convolution layers of these shallow feature extraction modules are 1×5×5, 3×3×3 and 3×3×3 respectively. For each shallow feature extraction module, after the convolution operation, batch normalization, ReLU activation and maximum pooling are performed in turn to reduce the spatial size by half. After these processes, the output features and Shape , where C is the number of channels after convolution, which is 48 here. and Apply 3D convolution to split it into layers of size (4×4×4) non-overlapping spatiotemporal labels to obtain the spatiotemporal features of video frames and differential frame temporal features , the shapes are ,in , , .

[0073] S400, using time difference convolution to project the spatiotemporal features into a query vector, a key vector and a value vector, and calculating the correlation between the vectors in the frequency domain;

[0074] It should be noted that the essence of step S400 is to construct a frequency-domain spatiotemporal self-attention module, and complete the following steps through the frequency-domain spatiotemporal self-attention module:

[0075] Specifically, time-difference convolution is used to project the spatiotemporal features into query vectors, key vectors, and value vectors, and the correlation between vectors is calculated in the frequency domain, including:

[0076] A two-stage optimization of frequency domain features is performed, which includes optimizing the frequency components in the spatial dimension using learnable spatial-frequency weights;

[0077] In the temporal dimension, one-dimensional Fourier transform is applied to capture periodic patterns and temporal dependencies to enhance the focusing ability of frequency components related to physiological rhythms.

[0078] For example, the input features of the module First, use temporal difference convolution (TDC) to project it into query (Query, Q) vector, key (Key, K) vector, and value (Value, V) vector. The shapes of Q, K, and V are , where. The time difference convolution calculation method is as follows:

[0079] ;

[0080] in is a learnable weight parameter, represents the current spatial and temporal position relative to the center point, and R represents the local sampling Neighborhood, R' represents the local spatial region in adjacent time steps, and the hyperparameter θ∈[0, 1] is used to adjust the weight of spatiotemporal difference, which is specifically set to 0.5. The calculation methods of Q, K, and V are as follows:

[0081] ;

[0082] Then, fast Fourier transform is applied to feature Q and feature K, and the correlation is calculated using the dot product method in the frequency domain. The specific calculation method is as follows:

[0083] ;

[0084] in , , Represent the two-dimensional Fourier transform, the two-dimensional inverse Fourier transform, and the conjugate transpose operation respectively. In order to further optimize the features related to physiological signals and enhance the modeling ability of spatiotemporal dependencies, the present invention proposes a two-stage frequency domain feature optimization process. In the spatial dimension, the attention map Perform a 2D Fourier transform and use learnable spatial-frequency weights The frequency components are optimized, and the specific calculation method is as follows:

[0085] ;

[0086] Subsequently, a one-dimensional Fourier transform is applied in the time dimension to capture periodic patterns and time dependencies, thereby enhancing the model's ability to focus on frequency components related to physiological rhythms and suppress irrelevant noise. Time-frequency optimization is accomplished by the following formula:

[0087] ;

[0088] in are learnable time-frequency domain weights, and Represent the one-dimensional Fourier transform and its inverse transform respectively. Then, the following formula is used to obtain the final aggregated output features:

[0089] ;

[0090] in Indicates the size The convolution operation is used to map the channel dimension back to dimension C, Represents the attention map Perform layer normalization operation.

[0091] Then we build a frequency-domain spatiotemporal cross attention module, which receives two input features and ,in represents the spatiotemporal characteristics of video frames, represents the temporal and spatial characteristics of the differential frame. Use time difference convolution (TDC) to get the query (Query, Q) vector and key (Key, K) vector. Use time difference convolution (TDC) to get the value (Value, V) vector, the shape of Q, K, V are The calculation method is as follows:

[0092] ;

[0093] The subsequent processing steps are the same as those of the frequency-domain spatiotemporal self-attention module, i.e., applying fast Fourier transform to feature Q and feature K, and calculating correlations using the dot product method in the frequency domain, etc.

[0094] S500, using a learnable frequency domain filter to selectively enhance relevant frequencies and suppress noise, and converting back to the time domain through an inverse Fourier transform;

[0095] S600, performing nonlinear feature enhancement on the processed feature map, and adding it to the input feature map to generate a final output;

[0096] It should be noted that the essence of step S500 is to construct a frequency domain discrimination feedforward network module, and complete the following steps through the frequency domain discrimination feedforward network module.

[0097] Use learnable frequency domain filters to selectively enhance relevant frequencies and suppress noise, and convert back to the time domain through inverse Fourier transform, including:

[0098] The frequency domain representation is filtered using a learnable frequency domain filter to obtain filtered frequency domain features;

[0099] The filtered frequency domain features are converted back to the time domain through inverse Fourier transform to obtain the time domain features.

[0100] Nonlinear feature enhancement is performed on the processed feature map and added to the input feature map to generate the final output, including:

[0101] The GEGLU function is used to perform nonlinear feature enhancement on the time domain features;

[0102] A convolutional layer is used to project the enhanced features back to the original dimension and added to the input feature map to obtain the final output.

[0103] First, the module input feature map shape Through a Convolution projects to a higher dimensional space to generate feature maps ,shape For which is the expanded hidden dimension; then, Perform a one-dimensional Fourier transform (1D FFT) along the time dimension to get a frequency domain representation , Shape ,in represents the frequency dimension; then, using a learnable frequency domain filter , the shape is The frequency domain representation is filtered to selectively enhance the relevant frequencies and suppress the noise; then, the filtered features are converted back to the time domain by inverse one-dimensional Fourier transform to obtain Next, use the GEGLU function to Perform nonlinear feature enhancement; finally, Convolution projects the enhanced features back to the original dimension , and add it to the input feature map X to get the final output of the module , the shape is The specific calculation method is as follows:

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] in is the GEGLU function, and Represent one-dimensional Fourier transform and its inverse transform respectively.

[0109] Then the obtained spatiotemporal features of the video frame and differential frame temporal features The two parallel processing paths are used for processing. The processing path is input into the N-layer frequency domain Transformer encoder layer composed of the frequency domain spatiotemporal self-attention module and the frequency domain discriminant feedforward network module for processing, where N is 12. For the i-th layer, the calculation method is as follows:

[0110] ;

[0111] ;

[0112] in represents the frequency-domain spatiotemporal self-attention module, represents the frequency domain discriminant feedforward network module, .

[0113] For the difference frame spatiotemporal features The processing path is input into the N-layer frequency domain Transformer encoder layer composed of the frequency domain spatiotemporal cross attention module and the frequency domain discriminant feedforward network module for processing, where N is 12. For the i-th layer, the calculation method is as follows:

[0114] ;

[0115] ;

[0116] in represents the frequency-domain spatiotemporal cross attention module, represents the frequency domain discriminant feedforward network module, .

[0117] S700, extract and optimize time-frequency features layer by layer, concatenate the optimized feature representations, process and generate predicted rPPG signals, and use the heart rate mixed loss function to guide model parameter adjustment.

[0118] Specifically, the optimized time-frequency features are extracted layer by layer, the optimized feature representations are concatenated, and the predicted rPPG signals are processed and generated. The heart rate mixed loss function is used to guide the model parameter adjustment, including:

[0119] Use multiple layers of frequency domain Transformer encoder layers to extract and optimize time-frequency features layer by layer;

[0120] splicing the optimized feature representations, and generating a predicted rPPG signal based on the optimized feature representations;

[0121] The loss is calculated using the heart rate hybrid loss function based on the predicted rPPG signal and the true rPPG signal;

[0122] The back-propagation algorithm is used to update the model parameters based on the calculated loss value.

[0123] Based on the feature representation of the two branches obtained above, and Feature concatenation along the channel dimension to obtain feature representation , the shape is Then the concatenated features are represented as Input to the rPPG signal prediction module. First, the feature map is averaged in the spatial domain through the spatial averaging operation to generate a time series signal , the shape is , and then upsample in the time dimension to the length of the original video frame sequence to obtain , the shape is Then, a one-dimensional convolutional layer with a kernel size of 3 and a stride of 1 is applied to the time series signal. To perform time domain filtering and map the channel dimension to 1, the predicted rPPG signal is finally generated , the shape is .

[0124] Based on the obtained and data of real rPPG signal , the heart rate mixed loss function is used to calculate the loss. The heart rate mixed loss function mainly consists of three parts: time domain loss , frequency domain loss and heart rate distance loss . Time Domain Loss The negative Pearson correlation coefficient (NPCC) is used to measure the trend and amplitude consistency between the predicted rPPG signal and the true signal. The calculation method is as follows:

[0125] ;

[0126] in is the length of the signal. The frequency domain loss is calculated based on the power spectral density (PSD) of the signal, and the cross entropy is used to measure the difference between the PSD of the predicted signal and the PSD distribution derived from the true heart rate signal. The specific calculation method is as follows:

[0127] ;

[0128] in Represents the real signal In frequency The power spectral density distribution at Represents the prediction signal In frequency Power spectral density distribution at . Heart rate distance loss The difference between the heart rate distribution of the predicted signal and the true signal is measured by KL divergence. The heart rate is derived from the PSD peak frequency. The specific calculation method is as follows:

[0129] ;

[0130] The overall heart rate hybrid loss function is calculated by weighted summation of three contrastive losses:

[0131] ;

[0132] specific Take 0.2, Take 1.0, Take 1.0.

[0133] It can be seen from the technical solution provided by the present invention that, specifically, by designing two paths to process the original video frame and the differential video frame sequence respectively, the input of each path is firstly shallowly processed, and then the time-frequency features are extracted through a series of frequency domain Transformer modules. The frequency domain Transformer module of the original video frame path consists of a frequency domain spatiotemporal self-attention (FDSTSA) module and a frequency domain discriminant feedforward (FDFF) module, which can effectively capture the time-frequency information related to the heart rate in the video frame. The frequency domain Transformer in the differential video frame path consists of a frequency domain spatiotemporal cross attention (FDSTCA) module and a frequency domain discriminant feedforward (FDFF) module. By introducing the frequency domain spatiotemporal cross attention module, it further enhances the representation ability of dynamic changes and spatiotemporal patterns by enhancing the complementary information between the original frame path and the differential frame path, and can effectively process the visual information that may be lost in the differential video frame features in the differential video frame path and reduce the influence of motion artifacts. After the above processing, the features of the two paths are spliced ​​in the channel dimension, and finally spatially averaged, upsampled and 1D convolved through the rPPG prediction module to obtain accurate rPPG signal prediction results.

[0134] The public dataset UBFC-rPPG dataset was used for verification. The UBFC-rPPG dataset is a dataset for studying video-based remote photoplethysmography (rPPG) signal extraction and heart rate estimation. The dataset contains 42 uncompressed videos recorded by the Logitech C920 HD Pro camera, which are mainly used to simulate rPPG signal estimation under different ambient lighting conditions. We divide it according to the ratio of training set: validation set = 4:1. We train 10 epochs from S100 to S500 and then take the model weights obtained from the last epoch training for testing on the test set. Finally, the MAE (Mean Absolute Error) and RMSE (Root Mean Squared Error) of heart rate estimation on the UBFC-rPPG dataset test set are statistically analyzed. The statistical results are shown in Table 1:

[0135] Table 1

[0136]

[0137] The embodiment of the present invention also provides a non-contact heart rate detection system, which is used to implement the above method embodiment, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0138] like Figure 3 As shown, Figure 3 : is a structural block diagram of the non-contact heart rate detection system of the present invention, the system comprises:

[0139] The data preprocessing module 101 is used to obtain the original face video and perform preprocessing to obtain a video frame sequence;

[0140] A differential frame sequence calculation module 102, configured to calculate pixel-by-pixel differences between adjacent frames based on the video frame sequence to extract a differential video frame sequence;

[0141] A shallow feature extraction module 103 is used to perform convolution operations and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and apply 3D convolution to obtain spatiotemporal features;

[0142] The frequency-domain spatiotemporal self-attention module 104 and the frequency-domain spatiotemporal cross-attention module 105 are used to project the spatiotemporal features into a query vector, a key vector and a value vector using a temporal difference convolution, and calculate the correlation between the vectors in the frequency domain;

[0143] A frequency domain discrimination feed-forward network module 106, for selectively enhancing relevant frequencies and suppressing noise using a learnable frequency domain filter, and converting back to the time domain through an inverse Fourier transform;

[0144] A multi-layer frequency domain Transformer encoder layer 107, which is used to perform nonlinear feature enhancement on the processed feature map and add it to the input feature map to generate the final output;

[0145] The rPPG signal prediction module 108 is used to extract optimized time-frequency features layer by layer, concatenate the optimized feature representations, and process and generate predicted rPPG signals, using the heart rate mixed loss function to guide model parameter adjustment.

[0146] By designing two paths to process the original video frames and differential video frame sequences respectively, the input of each path is first processed shallowly, and then the time-frequency features are extracted through a series of frequency domain Transformer modules. The frequency domain Transformer module of the original video frame path consists of a frequency domain spatiotemporal self-attention (FDSTSA) module and a frequency domain discriminative feedforward (FDFF) module, which can effectively capture the time-frequency information related to heart rate in the video frame. The frequency domain Transformer in the differential video frame path consists of a frequency domain spatiotemporal cross attention (FDSTCA) module and a frequency domain discriminative feedforward (FDFF) module. By introducing the frequency domain spatiotemporal cross attention module, it further enhances the representation ability of dynamic changes and spatiotemporal patterns by enhancing the complementary information between the original frame path and the differential frame path, and can effectively process the visual information that may be lost in the differential video frame features in the differential video frame path and reduce the influence of motion artifacts. After the above processing, the features of the two paths are spliced ​​in the channel dimension, and finally the rPPG prediction module is used for spatial averaging, upsampling and 1D convolution to obtain accurate rPPG signal prediction results.

[0147] Figure 4 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method:

[0148] Obtain the original face video and perform preprocessing to obtain a video frame sequence;

[0149] Calculating pixel-by-pixel differences between adjacent frames based on the video frame sequence to extract a differential video frame sequence;

[0150] Perform convolution operations and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and apply 3D convolution to obtain spatiotemporal features;

[0151] Use time difference convolution to project the spatiotemporal features into query vector, key vector and value vector, and calculate the correlation between vectors in the frequency domain;

[0152] A learnable frequency domain filter is used to selectively enhance relevant frequencies and suppress noise, and then converted back to the time domain via inverse Fourier transform.

[0153] Perform nonlinear feature enhancement on the processed feature map and add it to the input feature map to generate the final output;

[0154] The optimized time-frequency features are extracted layer by layer, the optimized feature representations are concatenated, and the predicted rPPG signals are generated. The heart rate mixed loss function is used to guide the adjustment of model parameters.

[0155] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0156] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0157] Obtain the original face video and perform preprocessing to obtain a video frame sequence;

[0158] Calculating pixel-by-pixel differences between adjacent frames based on the video frame sequence to extract a differential video frame sequence;

[0159] Perform convolution operations and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and apply 3D convolution to obtain spatiotemporal features;

[0160] Use time difference convolution to project the spatiotemporal features into query vector, key vector and value vector, and calculate the correlation between vectors in the frequency domain;

[0161] A learnable frequency domain filter is used to selectively enhance relevant frequencies and suppress noise, and then converted back to the time domain via inverse Fourier transform.

[0162] Perform nonlinear feature enhancement on the processed feature map and add it to the input feature map to generate the final output;

[0163] The optimized time-frequency features are extracted layer by layer, the optimized feature representations are concatenated, and the predicted rPPG signals are generated. The heart rate mixed loss function is used to guide the adjustment of model parameters.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact heart rate detection method, characterized in that: The non-contact heart rate detection method comprises: Obtain the original face video and perform preprocessing to obtain a video frame sequence; Calculating pixel-by-pixel differences between adjacent frames based on the video frame sequence to extract a differential video frame sequence; Perform convolution operations and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and apply 3D convolution to obtain spatiotemporal features; The spatiotemporal features of the video frame are input into the frequency domain spatiotemporal self-attention module, and the spatiotemporal features of the video frame and the differential frame spatiotemporal features are input into the frequency domain spatiotemporal cross attention module; the spatiotemporal features are projected into query vectors, key vectors, and value vectors using temporal differential convolution, and the correlation between vectors is calculated in the frequency domain, including: Performing a two-dimensional Fourier transform on the query vector and the key vector to obtain frequency domain features; In the spatial dimension, the frequency components are optimized using learnable spatial frequency domain weights; In the time dimension, one-dimensional Fourier transform is applied to capture periodic patterns and time dependencies to enhance the focusing ability of frequency components related to physiological rhythms. A learnable frequency domain filter is used to selectively enhance relevant frequencies and suppress noise, and then converted back to the time domain via inverse Fourier transform. Perform nonlinear feature enhancement on the processed feature map and add it to the input feature map to generate the final output; The optimized time-frequency features are extracted layer by layer, the optimized feature representations are concatenated, and the predicted rPPG signals are generated. The heart rate mixed loss function is used to guide the adjustment of model parameters.

2. The non-contact heart rate detection method according to claim 1, characterized in that: Get the original face video and preprocess it to get the video frame sequence, including: The original face video is segmented according to the predetermined time sequence length; Use the face detection tool to detect the face area and crop the video frame based on the area; The cropped video frames are uniformly adjusted to a fixed size to obtain a video frame sequence.

3. The non-contact heart rate detection method according to claim 1, characterized in that: Convolution operations and processing are performed on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and 3D convolution is applied to obtain spatiotemporal features, including: Performing a convolution operation on the video frame sequence and the differential video frame sequence, and sequentially performing batch normalization, ReLU activation, and maximum pooling to extract preliminary features; Apply 3D convolution to segment it into non-overlapping spatiotemporal tags and obtain video frame spatiotemporal features and differential frame spatiotemporal features.

4. The non-contact heart rate detection method according to claim 1, characterized in that: Use time difference convolution to project spatiotemporal features into query vectors, key vectors, and value vectors, and calculate the correlation between vectors in the frequency domain, including: A two-stage optimization of frequency domain features is performed, which includes optimizing the frequency components in the spatial dimension using learnable spatial-frequency weights; In the temporal dimension, one-dimensional Fourier transform is applied to capture periodic patterns and temporal dependencies to enhance the focusing ability of frequency components related to physiological rhythms.

5. The non-contact heart rate detection method according to claim 1, characterized in that: Use learnable frequency domain filters to selectively enhance relevant frequencies and suppress noise, and convert back to the time domain through inverse Fourier transform, including: The frequency domain representation is filtered using a learnable frequency domain filter to obtain filtered frequency domain features; The filtered frequency domain features are converted back to the time domain through inverse Fourier transform to obtain the time domain features.

6. The non-contact heart rate detection method according to claim 5, characterized in that: Nonlinear feature enhancement is performed on the processed feature map and added to the input feature map to generate the final output, including: The GEGLU function is used to perform nonlinear feature enhancement on the time domain features; A convolutional layer is used to project the enhanced features back to the original dimension and added to the input feature map to obtain the final output.

7. The non-contact heart rate detection method according to claim 1, characterized in that: Extract optimized time-frequency features layer by layer, concatenate the optimized feature representations, and process to generate predicted rPPG signals. Use the heart rate mixed loss function to guide model parameter adjustment, including: Use multiple layers of frequency domain Transformer encoder layers to extract and optimize time-frequency features layer by layer; splicing the optimized feature representations, and generating a predicted rPPG signal based on the optimized feature representations; The loss is calculated using the heart rate hybrid loss function based on the predicted rPPG signal and the true rPPG signal; The back-propagation algorithm is used to update the model parameters based on the calculated loss value.

8. A non-contact heart rate detection system, characterized in that: include: A data preprocessing module is used to obtain the original face video and perform preprocessing to obtain a video frame sequence; A differential frame sequence calculation module, used for calculating the pixel-by-pixel difference between adjacent frames based on the video frame sequence to extract a differential video frame sequence; A shallow feature extraction module is used to perform convolution operations and processing on the video frame sequence and the differential video frame sequence to reduce the spatial size and extract preliminary features, and apply 3D convolution to obtain spatiotemporal features; A frequency-domain spatiotemporal self-attention module and a frequency-domain spatiotemporal cross-attention module. The frequency-domain spatiotemporal self-attention module is used to receive the spatiotemporal features of video frames, and the frequency-domain spatiotemporal cross-attention module is used to receive the spatiotemporal features of video frames and the differential frame spatiotemporal features; Use time difference convolution to project spatiotemporal features into query vectors, key vectors, and value vectors, and calculate the correlation between vectors in the frequency domain, including: Performing a two-dimensional Fourier transform on the query vector and the key vector to obtain frequency domain features; In the spatial dimension, the frequency components are optimized using learnable spatial frequency domain weights; In the time dimension, one-dimensional Fourier transform is applied to capture periodic patterns and time dependencies to enhance the focusing ability of frequency components related to physiological rhythms. A frequency-domain discriminative feed-forward network module is used to selectively enhance relevant frequencies and suppress noise using a learnable frequency-domain filter, and then transform back to the time domain via an inverse Fourier transform; Multi-layer frequency domain Transformer encoder layer, which performs nonlinear feature enhancement on the processed feature map and adds it to the input feature map to generate the final output; The rPPG signal prediction module is used to extract and optimize time-frequency features layer by layer, concatenate the optimized feature representations, and process the generated predicted rPPG signals, using the heart rate mixed loss function to guide the adjustment of model parameters.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the non-contact heart rate detection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the non-contact heart rate detection method according to any one of claims 1 to 7 are implemented.

Images