Video pulse wave extraction method based on progressive self-supervised learning

Abstract

The invention discloses a video pulse wave extraction method based on progressive self-supervised learning, and the method comprises the steps: 1, preprocessing a face video clip, and obtaining an original signal; 2, processing the original signal by using a signal enhancement strategy to obtain an enhanced signal, and forming a positive sample pair with the original signal; 3, constructing a deep learning model capable of adaptively quantifying the contribution weight of the global domain of interest; 4, decomposing a video pulse wave extraction task into a signal extraction sub-task and a noise suppression sub-task, constructing a two-stage time-frequency domain joint loss function, and performing progressive self-supervised training; and 5, calling the pre-trained model to recover a high-quality pulse wave signal from the to-be-detected original signal to realize heart rate monitoring. According to the method, the deep learning model can be self-supervised and trained without a true value signal label, video pulse wave extraction is achieved, meanwhile, complex environment noise is restrained, the generalization performance of the model in a real scene is improved, and development of a non-contact health monitoring technology is promoted.

Description

Technical Field

[0001] This invention belongs to the technical field of biomedical signal processing, specifically relating to a video pulse wave extraction method based on progressive self-supervised learning. Background Technology

[0002] Remote photoplethysmography (rPPG) based on video captures the subtle color changes in facial skin caused by a heartbeat using consumer-grade cameras, thus enabling contactless extraction of pulse wave signals. However, because the skin color changes caused by a heartbeat are extremely small and easily affected by various noises such as head movements and changes in lighting, accurately recovering high-quality pulse wave signals from video presents a significant challenge.

[0003] With the development of deep learning technology, supervised learning-based rPPG methods have made significant progress in improving the quality of pulse wave signals. However, such methods rely on synchronously acquired contact sensor signals as real label signals. On the one hand, the data acquisition process is complex and costly, and may even pose safety hazards in special scenarios such as driving monitoring. On the other hand, due to individual differences in pulse transit time (PTT), the model is difficult to train.

[0004] To reduce reliance on labeled data, self-supervised learning (SSL) has been introduced into the rPPG field. It learns effective feature representations from unlabeled data by constructing predefined tasks, avoiding the limitations of real-label acquisition. Existing SSL methods mostly employ contrastive learning, constructing positive and negative sample pairs and optimizing the contrastive loss function to improve feature discrimination. However, in pulse wave signal recovery tasks, current negative sample construction methods are imprecise, and general image enhancement methods cannot effectively simulate real noise patterns in skin color changes. Furthermore, while some non-contrastive SSL methods have explored aspects such as signal periodic feature extraction, they generally neglect the fine-grained dynamic changes in the contribution of different facial regions of interest to signal recovery in complex noise environments, resulting in weak generalization performance in practical applications.

[0005] In summary, existing rPPG techniques for pulse wave signal recovery under label-free conditions still suffer from problems such as unreasonable self-supervised framework design, insufficient utilization of dynamic contribution of the region of interest, and poor generalization performance in real-world noise scenarios. Therefore, there is an urgent need to propose a self-supervised rPPG method that can achieve high-quality pulse wave signal extraction in complex environments and has stronger generalization performance. Summary of the Invention

[0006] To address the shortcomings of the aforementioned technologies, this invention proposes a video pulse wave extraction method based on progressive self-supervised learning. The aim is to utilize a self-supervised learning approach, progressing from simple to complex tasks, to train a deep learning model that adaptively quantifies the contribution weights of the global region of interest, thereby recovering high-quality pulse wave signals from facial videos and achieving accurate heart rate monitoring even under complex noise conditions.

[0007] The present invention adopts the following solution to solve the technical problem: The present invention provides a video pulse wave extraction method based on progressive self-supervised learning, characterized by the following steps: Step 1: Process the captured facial video clips Each video frame After segmenting each region of interest, then processing each video frame separately... Signal extraction is performed on each region of interest to obtain the original rPPG signal set. Then, normalization is performed to obtain the normalized original rPPG signal set. ;in, C Indicates channel dimension, H Indicates the height of the video frame. W Indicates the width of the video frame. T Indicates the time dimension. Spatial dimension; Step 2: Employ signal-level data augmentation methods to... The process yielded four enhanced rPPG signal sets, which were then normalized to obtain four normalized enhanced rPPG signal sets, including: a normalized weakly enhanced rPPG signal set. Normalized strongly enhanced rPPG signal set Normalized periodically enhanced rPPG signal set and normalized local dropout enhanced rPPG signal set and will And any rPPG signal set from the four normalized enhanced rPPG signal sets is denoted as , ; Step 3: Construct a multi-scale region-of-interest (ROI) attention network, including: an ROI-independent temporal encoder, a multi-scale stitching module, a spatial weight adaptive quantization encoder, and a pulse wave decoding module, and then... The pulse wave signal set is obtained through processing. ; Step 4: Construct the joint time-frequency domain loss function, including: the local search narrowband signal-to-noise ratio loss in stage 1. The time-frequency domain loss of the global search in the second stage This is used to train a multi-scale domain of interest attention network, resulting in a pre-trained multi-scale domain of interest attention model. Step 5: Use the pre-trained multi-scale region of interest attention model to process the original rPPG signal set to be tested, and obtain the optimal pulse wave signal set to realize heart rate monitoring.

[0008] The video pulse wave extraction method based on progressive self-supervised learning described in this invention is characterized in that step 2 is performed according to the following steps: Step 2.1, in Spatial Dimensions Random selection above A domain of interest, and in Add the mean value to the rPPG original signal sequence corresponding to any channel dimension in any region of interest. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being... The upper limit of the amplitude is Gaussian noise, thus affecting Different Gaussian noises are added to the original rPPG signal corresponding to each channel dimension in each region of interest to obtain a weakly enhanced rPPG signal sequence, which is then normalized to obtain... ; Step 2.2, in Spatial Dimensions Random selection above A domain of interest, and in Add the mean value to the rPPG original signal sequence corresponding to any channel dimension in any region of interest. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being... The upper limit of the amplitude is Gaussian noise, thus affecting Different Gaussian noises are added to the original rPPG signal sequence corresponding to each channel dimension in each region of interest to obtain a strongly enhanced rPPG signal sequence, which is then normalized to obtain... ; Step 2.3, in Spatial Dimensions Random selection above A domain of interest, and in For each channel dimension within a region of interest, the original rPPG signal sequence is augmented with frequencies ranging from [frequency range]. Random sampled values ​​within the range, with amplitude in the interval The cosine signal of the random sampled values ​​within the range is used to obtain the periodically enhanced rPPG signal sequence, which is then normalized to obtain... ; Step 2.4, in Spatial Dimensions Random selection above One domain of interest, and The original rPPG signal sequence corresponding to each channel dimension in each region of interest is replaced with a signal having a mean of 1. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being... The upper limit of the amplitude is Gaussian noise is removed to obtain the locally dropped enhanced rPPG signal, which is then normalized to obtain... .

[0009] Furthermore, step 3 includes: Step 3.1, constructing a region-of-interest independent temporal encoder is... It consists of cascaded time blocks, and... Processing yields the first... Cascaded spatially independent temporal feature maps , among which, the Each cascaded time block is generated by the convolution kernel. Two-dimensional convolutional layer Normalization layer and nonlinear activation layer composition; and They represent the first Two-dimensional convolutional layers in cascaded time blocks The number of rows and columns of the convolution kernel, for Channel dimension; Step 3.2: Constructing a multi-scale stitching module consists of average pooling blocks and feature stitching blocks, and then sequentially processing them... The process is performed to obtain multi-scale temporal feature maps. Among them, the average pooling block is composed of The average pooling layers are connected in parallel, making the first... The width of the pooling window for each average pooling layer is Height is ; for Channel dimension; Step 3.3, constructing the spatial weight adaptive quantization encoder is... It is composed of cascaded adaptive interest domain weighted modules, and sequentially applied to... Processing yields the first... Level spatial weighted feature map sequence , among which, the The level-adaptive domain-of-interest weighting module consists of the first... Level multi-scale temporal self-attention module and the first Composed of a first-level feedforward module, the first The multi-scale temporal self-attention module consists of the first level. Multi-head self-attention layer and the Level attention layer normalization layer Composition, the first The first-level feedforward module consists of the first-level feedforward module. Level feedforward layer and the Level feedforward normalization composition; express The first moment Level-weighted feature map; Step 3.4: Constructing the pulse wave decoding module is done using the spatial compression module. and It is composed of cascaded decoding modules, and sequentially... The pulse wave signal set is obtained through processing. Among them, the space compression module The average pooling window is A single average pooling layer, and These represent the width and height of the average pooling window, respectively; The level decoding module consists of convolutional kernels with a size of The Level 1 one-dimensional convolutional layer , No. Level batch normalization layer and the Level Activation Layer composition; Furthermore, step 3.1 includes: when At that time, the rPPG signal set Enter the number In a series of cascaded time blocks, and through a two-dimensional convolutional layer After expanding the channel dimensions, batch normalization layers are then performed sequentially. and nonlinear activation The processing yields the first Cascaded spatially independent temporal feature maps ,in, For the first Channel dimension of a cascaded spatially independent temporal feature map; when At that time, the first Cascaded spatially independent temporal feature maps Enter the first The cascaded time blocks yield the first... Cascaded spatially independent temporal feature maps Ultimately, by the first The output of the cascaded time block is... Cascaded spatially independent temporal feature maps .

[0010] Furthermore, step 3.2 includes: Each of the parallel average pooling layers respectively... Perform time-series-independent average pooling based on the region of interest to obtain Temporal feature maps at various scales ;in, Indicates the first Temporal feature maps at various scales; Feature splicing blocks along the channel dimension The data is concatenated to obtain multi-scale temporal feature maps. .

[0011] Furthermore, step 3.3 includes: when At that time, According to the time dimension After splitting, the first one is obtained. Multi-scale feature map sequence ,in, express The first moment Multi-scale feature maps in spatial dimensions; Will Enter the first In the adaptive interest domain weighting module, the first... The first level of multi-scale temporal self-attention module Multi-head self-attention layer right The process yields a spatially quantized weighted feature map based on the correlation of temporal features. Then by the first Level attention layer normalization layer right Process and obtain The first moment Level-space linear weighted feature map Then proceed through the first... Level feedforward layer and the Level feedforward normalization Perform nonlinear feature mapping to output The first moment Level spatial weighted feature map ; when At that time, The first moment Level spatial weighted feature map Enter the first The process is performed in an adaptive domain-of-interest weighted module, and the output is... The first moment Level spatial weighted feature map Thus, by the first Level Adaptive Region of Interest Weighted Module Output The first moment Level spatial weighted feature map , .

[0012] Furthermore, step 3.4 includes: Will The first moment Level spatial weighted feature map Input space compression module In the middle, and on By performing summation and averaging operations along the spatial dimension, we obtain... Spatial compression characteristics of time This will compress the spatial feature sequence. After concatenation along the time dimension, a spatially compressed temporal feature map is obtained. ; when At that time, Enter the first In the level decoding module, by the first Level 1 one-dimensional convolutional layer right After feature dimension compression, it then goes through the first... Level batch normalization layer and the Level Activation Layer The processing yields the first Timing characteristics of multi-channel compression ,in, for Channel dimension; when At that time, the first Timing characteristics of multi-channel compression Enter the first The process is performed in the first-level decoding module to obtain the second-level decoding module. Timing compression characteristics of multi-channel compression Ultimately, by the first The level decoding module outputs a pulse wave signal set with a channel dimension of 1. , .

[0013] Furthermore, step 4 is performed according to the following steps: Step 4.1: Construct the local search narrowband signal-to-noise ratio loss for the first stage using equation (1). : (1) In equation (1), for The power spectrum, Represents the frequency independent variable. For pulse wave signal set Perform a local search for the pulse center frequency, with a value range of [value range missing]. , express The lower boundary, express The upper boundary, It is a narrow frequency band; Step 4.2: Construct the global search time-frequency domain loss for the second stage using equation (2). : (2) In equation (2), For balance parameters; The loss represents the temporal similarity of positive samples, and is obtained from equation (3); The value represents the global search narrowband signal-to-noise ratio loss, which is obtained from equation (4); (3) (4) In equation (4), For pulse wave signal set Perform a global search for the pulse center frequency, with a value range of [range missing]. ; express The lower boundary, express The lower boundary, of which, , ; Represents the original pulse wave signal. This indicates an enhanced pulse wave signal set. ; In equation (3), Representing the negative Pearson-related loss, and obtained from equation (5), The mean squared error loss is represented by equation (6), which is obtained from the equation: (5) (6) In equations (5) and (6), Indicates to The result obtained by averaging over the time dimension Indicates to The result obtained by averaging over the time dimension; Step 4.3: Use the AdamW optimizer to perform the first stage of self-supervised training of the multi-scale region-of-interest attention network on the video pulse wave, and calculate... To update network parameters when the number of training iterations reaches a threshold. At this point, the first phase of self-supervised training ends, resulting in a pre-trained multi-scale domain-of-interest attention network. The AdamW optimizer was used to perform the second stage of self-supervised training on video pulse waves for the initially trained multi-scale region-of-interest attention network, and the calculation was performed. To update network parameters when the number of training iterations reaches a threshold. At this point, the second phase of self-supervised training ends, and a pre-trained multi-scale domain of interest attention model is obtained.

[0014] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program that supports the processor in executing the video pulse wave extraction method, and the processor is configured to execute the program stored in the memory.

[0015] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the video pulse wave extraction method.

[0016] Compared with existing technologies, the beneficial effects of this invention are reflected in: 1. This invention provides a progressive self-supervised learning framework for video pulse wave extraction, decomposing the task into two sub-tasks: pulse signal extraction and complex noise suppression. It also designs a two-stage progressive time-frequency joint loss. The first stage uses prior heart rate distribution to train the model, searching for pulse wave signals within a narrow spectral range. The second stage broadens the search range to accept extreme heart rate conditions, while simultaneously reducing the similarity of positive sample pairs in the time domain, effectively suppressing complex noise. This invention alleviates the training difficulty and model collapse risk of self-supervised learning, ensuring that the deep learning model can accurately extract weak-amplitude pulse wave signals from unlabeled facial videos, and effectively improving the model's generalization performance.

[0017] 2. This invention provides four signal-level enhancement methods for rPPG raw signals, including weak enhancement, strong enhancement, periodic enhancement, and local dropout enhancement. These methods form positive sample pairs with the raw signal and do not rely on negative sample pairs. The parameters of each enhancement method are highly randomized, enabling a more comprehensive simulation of latent noise modes. Compared to existing image-level enhancement methods, this invention directly acts on the rPPG raw signal at the color channel level, more comprehensively and accurately simulating latent noise modes, thereby enabling the model to better suppress complex noise in real-world situations.

[0018] 3. This invention designs a deep learning model capable of adaptively quantifying the contribution weights of the global region of interest. It employs an independent temporal encoder to extract key temporal features and eliminate redundant information. Multi-scale feature concatenation improves stability and reliability, and a spatial encoder is combined to adaptively evaluate the weights of each region at fine-grained temporal intervals. This structure achieves more accurate pulse wave signal recovery while maintaining a lightweight design. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram illustrating the facial video preprocessing steps of the present invention; Figure 3 This is a diagram of the multi-scale domain-of-interest attention network structure of the present invention; Figure 4a is a structural diagram of the domain-of-interest independent timing encoder of the present invention; Figure 4b is a structural diagram of the multi-scale splicing module of the present invention; Figure 4c is a structural diagram of the spatial weight adaptive quantization encoder of the present invention; Figure 4d is a structural diagram of the pulse wave decoding module of the present invention. Detailed Implementation

[0020] In this example, a video pulse wave extraction method based on progressive self-supervised learning is presented, such as... Figure 1 As shown, the procedure is as follows: Step 1, as follows Figure 2 As shown, for the captured facial video clips Preprocessing was performed, and facial video was detected using FAN facial landmark detection technology to obtain 68 facial landmark points. Based on these facial landmark points, each video frame was analyzed... After segmenting each region of interest, then processing each video frame separately... Signal extraction is performed on each region of interest to obtain the original rPPG signal set. Then, normalization is performed to obtain the normalized original rPPG signal set. ;in, C Indicates channel dimension, HIndicates the height of the video frame. W Indicates the width of the video frame. T Indicates the time dimension. For spatial dimensions; in this example, T =450, H =480, W =640, C =3, M =6.

[0021] Step 2: Employ signal-level data augmentation methods to... The process yielded four enhanced rPPG signal sets, which were then normalized to obtain four normalized enhanced rPPG signal sets, including: a normalized weakly enhanced rPPG signal set. Normalized strongly enhanced rPPG signal set Normalized periodically enhanced rPPG signal set and normalized local dropout enhanced rPPG signal set and will And any rPPG signal set from the four normalized enhanced rPPG signal sets is denoted as , .

[0022] Step 2.1, in Spatial Dimensions Random selection above A domain of interest, and in Add the mean value to the rPPG original signal sequence corresponding to any channel dimension in any region of interest. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being... The upper limit of the amplitude is Gaussian noise, thus affecting Different Gaussian noises are added to the original rPPG signal corresponding to each channel dimension in each region of interest to obtain a weakly enhanced rPPG signal sequence, which is then normalized to obtain... In this example, For interval Random sampled values ​​between, , , , , .

[0023] Step 2.2, in Spatial Dimensions Random selection above A domain of interest, and in Add the mean value to the rPPG original signal sequence corresponding to any channel dimension in any region of interest. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being... The upper limit of the amplitude is Gaussian noise, thus affecting Different Gaussian noises are added to the original rPPG signal sequence corresponding to each channel dimension in each region of interest to obtain a strongly enhanced rPPG signal sequence, which is then normalized to obtain... In this example, For interval Internal random sample values, , , , , .

[0024] Step 2.3, in Spatial Dimensions Random selection above A domain of interest, and in For each channel dimension within a region of interest, the original rPPG signal sequence is augmented with frequencies ranging from [frequency range]. Random sampled values ​​within the range, with amplitude in the interval The cosine signal of the random sampled values ​​within the range is used to obtain the periodically enhanced rPPG signal sequence, which is then normalized to obtain... In this example, for Random sampled values ​​between, , , , .

[0025] Step 2.4, in Spatial Dimensions Random selection above One domain of interest, and The original rPPG signal sequence corresponding to each channel dimension in each region of interest is replaced with a signal having a mean of 1. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being... The upper limit of the amplitude is Gaussian noise is removed to obtain the locally dropped enhanced rPPG signal, which is then normalized to obtain... In this example, for Uniformly random sampled values ​​between , , , , .

[0026] Step 3: Construct a multi-scale domain-of-interest attention network, with the structure as follows: Figure 3 As shown, it includes: a region-of-interest independent temporal encoder, a multi-scale stitching module, a spatial weight adaptive quantization encoder, and a pulse wave decoding module, and performs... The pulse wave signal set is obtained through processing. ; Step 3.1, as shown in Figure 4a, involves constructing a domain-of-interest independent temporal encoder. It consists of cascaded time blocks, and... Processing yields the first... Cascaded spatially independent temporal feature maps , among which, the Each cascaded time block is generated by the convolution kernel. Two-dimensional convolutional layer Normalization layer and nonlinear activation layer Composition; and on Processing yields the first... Cascaded spatially independent temporal feature maps ,in, and They represent the first Two-dimensional convolutional layers in cascaded time blocks The number of rows and columns of the convolution kernel, for The channel dimension; in this example, In each time block, the kernel size of the two-dimensional convolutional layer is the same, that is... , , .

[0027] when At that time, the rPPG signal set Enter the number In a series of cascaded time blocks, and through 3D convolutional layers After expanding the channel dimensions, batch normalization layers are then performed sequentially. and nonlinear activation The processing yields the first Cascaded spatially independent temporal feature maps ,in, For the first In this example, the channel dimension of the cascaded spatially independent temporal feature maps is... ; when At that time, the first Cascaded spatially independent temporal feature maps Enter the first The cascaded time blocks yield the first... Cascaded spatially independent temporal feature maps Ultimately, by the first The output of the cascaded time block is... Cascaded spatially independent temporal feature maps .

[0028] Step 3.2, as shown in Figure 4b, constructing a multi-scale stitching module consists of average pooling blocks and feature stitching blocks, and then sequentially processing them... The process is performed to obtain multi-scale temporal feature maps. Among them, the average pooling block is composed of The average pooling layers are connected in parallel, making the first... The width of the pooling window for each average pooling layer is and height are ; for The channel dimension; in this example, Average pooling layer Pooling window , Average pooling layer Pooling window , Average pooling layer Pooling window , .

[0029] Each of the parallel average pooling layers respectively... Perform time-series-independent average pooling based on the region of interest to obtain Temporal feature maps at various scales ;in, Indicates the first Temporal feature maps at various scales; Feature splicing blocks along the channel dimension The data is concatenated to obtain multi-scale temporal feature maps. .

[0030] Step 3.3, as shown in Figure 4c, involves constructing a spatial weighted adaptive quantization encoder. It is composed of cascaded adaptive interest domain weighted modules, and sequentially applied to... Processing yields the first... Level spatial weighted feature map sequence , among which, the The level-adaptive domain-of-interest weighting module consists of the first... Level multi-scale temporal self-attention module and the first Composed of a first-level feedforward module, the first The multi-scale temporal self-attention module consists of the first level. Multi-head self-attention layer and the Level attention layer normalization layer Composition, the first The first-level feedforward module consists of the first-level feedforward module. Level feedforward layer and the Level feedforward normalization composition; express The first moment Level-weighted feature map; in this example .

[0031] when At that time, According to the time dimension After splitting, the first one is obtained. Multi-scale feature map sequence ,in, express The first moment Multi-scale feature maps in spatial dimensions; Will Enter the first In the adaptive interest domain weighting module, the first... The first level of multi-scale temporal self-attention module Multi-head self-attention layer right The process yields a spatially quantized weighted feature map based on the correlation of temporal features. Then by the first Level attention layer normalization layer right Process and obtain The first moment Level-space linear weighted feature map Then proceed through the first... Level feedforward layer and the Level feedforward normalization Perform nonlinear feature mapping to output The first moment Level spatial weighted feature map ; when At that time, The first moment Level spatial weighted feature map Enter the first The process is performed in an adaptive domain-of-interest weighted module, and the output is... The first moment Level spatial weighted feature map Thus, by the first Level Adaptive Region of Interest Weighted Module Output The first moment Level spatial weighted feature map , .

[0032] Step 3.4, as shown in Figure 4d, constructs the pulse wave decoding module using a spatial compression module. and It is composed of cascaded decoding modules, and sequentially... The pulse wave signal is obtained through processing. Among them, the space compression module The average pooling window is A single average pooling layer, and These represent the width and height of the average pooling window, respectively; The level decoding module consists of convolutional kernels with a size of The Level 1 one-dimensional convolutional layer , No. Level batch normalization layer and the Level Activation Layer Composition; in this example, , , , .

[0033] Will The first moment Level spatial weighted feature map Input space compression module In the middle, and on By performing summation and averaging operations along the spatial dimension, we obtain... Spatial compression characteristics of time This will compress the spatial feature sequence. After concatenation along the time dimension, a spatially compressed temporal feature map is obtained. ; when At that time, Enter the first In the level decoding module, by the first Level 1 one-dimensional convolutional layer right After feature dimension compression, it then goes through the first... Level batch normalization layer and the Level Activation Layer The processing yields the first Timing characteristics of multi-channel compression ,in, for Channel dimension; when At that time, the first Timing characteristics of multi-channel compression Enter the first The process is performed in the first-level decoding module to obtain the second-level decoding module. Timing compression characteristics of multi-channel compression Ultimately, by the first The level decoding module outputs a pulse wave signal with a channel dimension of 1. , .

[0034] Step 4: Based on pulse wave extraction and complex noise suppression targets, construct a joint time-frequency domain loss function, including: local search narrowband signal-to-noise ratio loss in the first stage. The time-frequency domain loss of the global search in the second stage This is used to train a multi-scale domain of interest attention network, resulting in a pre-trained multi-scale domain of interest attention model. Step 4.1: Construct the local search narrowband signal-to-noise ratio loss for the first stage using equation (1). : (1) In equation (1), for The power spectrum, Represents the frequency independent variable. For pulse wave signal set Perform a local search for the pulse center frequency, with a value range of [range missing]. , express The lower boundary, express The upper boundary, This refers to a narrow frequency band; in this example, , .

[0035] Step 4.2: Construct the global search time-frequency domain loss for the second stage using equation (2). : (2) In equation (2), To balance the parameters, in this example, ; The loss represents the temporal similarity of positive samples, and is obtained from equation (3); The value represents the global search narrowband signal-to-noise ratio loss, which is obtained from equation (4); (3) (4) In equation (4), For pulse wave signal set Perform a global search for the pulse center frequency, with a value range of [range missing]. ; express The lower boundary, express The lower boundary, of which Less than , Greater than ; Represents the original pulse wave signal. This indicates an enhanced pulse wave signal set. In this example, , .

[0036] In equation (3), Representing the negative Pearson-related loss, and obtained from equation (5), The mean squared error loss is represented by equation (6), which is obtained from the equation: (5) (6) In equations (5) and (6), Indicates to The result obtained by averaging over the time dimension Indicates to The result is obtained by averaging over the time dimension.

[0037] Step 4.3: Use the AdamW optimizer to perform the first stage of self-supervised training of the multi-scale region-of-interest attention network on the video pulse wave, and calculate... To update network parameters when the number of training iterations reaches a threshold. At this point, the first phase of self-supervised training ends, resulting in a pre-trained multi-scale domain-of-interest attention network. The AdamW optimizer was used to perform the second stage of self-supervised training on video pulse waves for the initially trained multi-scale region-of-interest attention network, and the calculation was performed. To update network parameters when the number of training iterations reaches a threshold. At this point, the second phase of self-supervised training ends, and a pre-trained multi-scale domain-of-interest attention model is obtained; in this example, , .

[0038] Step 5: Use the pre-trained multi-scale region of interest attention model to process the original rPPG signal set to be tested, and obtain the optimal pulse wave signal set to realize heart rate monitoring.

[0039] In this example, the test facial video clips are preprocessed according to step 1 above. Obtain the original test signal ,right Perform normalization and map to The interval is denoted as the normalized original test signal. The normalized original test signal is input into the pre-trained multi-scale region-of-interest attention model, which outputs the test pulse wave signal. For testing pulse wave signals The heart rate can be obtained by using the Fast Fourier Transform to determine the dominant frequency. .

[0040] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0041] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0042] To verify the effectiveness of the progressive self-supervised method proposed in this invention, we use the public datasets PURE and UBFC-RPPG as examples. During training, no ground truth labels are used, and no fine-tuning is performed; the pre-trained model is directly used for validation. This example employs a challenging PURE dataset with in-set testing (subject-independent five-fold cross-validation) and cross-dataset testing (UBFC-RPPG training, PURE testing) to analyze the experimental results. Heart rate is evaluated using mean absolute error (MAE), root mean square error (RMSE), and Pearson's correlation coefficient (r). The proposed method is compared with state-of-the-art traditional methods, supervised learning methods, and self-supervised learning methods; the results are shown in Tables 1 and 2.

[0043] Table 1. Analysis of Heart Rate Assessment Results in the PURE Dataset

[0044] Table 2 Analysis of Heart Rate Assessment Results from PURE Cross-Dataset Tests (UBFC-RPPG Training)

[0045] Of the methods mentioned above, the Green method is from "Remote Plethysmographic Imaging Using Ambient Light," and the CHROM method is from "Robust Pulse Rate From Chrominance-based RPPG."》, The POS method is from "Algorithmic Principles of Remote PPG", the HR-CNN method is from "Visual HeartRate Estimation with Convolutional Neural Network", the RPNet method is from "UnifyingFrame Rate and Temporal Dilations for Improved Remote Pulse Detection", the PhysNet method is from "Remote Photoplethysmograph Signal Measurement from FacialVideos Using Spatio-temporal Networks", Dual-GAN from "Dual-GAN: Joint BVP and NoiseModeling for Remote Physiological Measurement", the Contrast-Phys+ method is from "Contrast-phys+: Unsupervised and Weakly-supervised Video-based RemotePhysiological Measurement via Spatiotemporal contrast", the SiNC method is from "Non-contrastive Unsupervised Learning of Physiological Signals from Video", the DeepPhys method is from "DeepPhys: Video-based Physiological Measurement usingConvolutional Attention Networks", the TS-CAN method is from "Multi-task Temporal ShiftAttention Networks for On-device Contactless Vitals Measurement", the Physformer method is from "Physformer: Facial Video-based Physiological Measurement with TemporalDifference Transformer", As shown in Table 1, the present invention achieves the lowest mean absolute error of 0.59 bpm in heart rate measurement. As shown in Table 2, deep learning-based methods generally exhibit performance degradation and weak generalization capabilities. This is due to the complex noise in the test set PURE, while the training set UBFC-RPPG has a relatively simple noise mode. The method proposed in this invention achieves better results than state-of-the-art methods in all three metrics of heart rate measurement.

[0046] In summary, the video pulse wave extraction method based on progressive self-supervised learning proposed in this invention can accurately extract pulse waves from videos without relying on ground truth signal labels, and achieves robust generalization to complex noise.

Claims

1. A video pulse wave extraction method based on progressive self-supervised learning, characterized in that, The procedure is as follows: Step 1: Process the captured facial video clips Each video frame After segmenting each region of interest, then processing each video frame separately... Signal extraction is performed on each region of interest to obtain the original rPPG signal set. Then, normalization is performed to obtain the normalized original rPPG signal set. ;in, C Indicates channel dimension, H Indicates the height of the video frame. W Indicates the width of the video frame. T Indicates the time dimension. Spatial dimension; Step 2: Employ signal-level data augmentation methods to... The process yielded four enhanced rPPG signal sets, which were then normalized to obtain four normalized enhanced rPPG signal sets, including: a normalized weakly enhanced rPPG signal set. Normalized strongly enhanced rPPG signal set Normalized periodically enhanced rPPG signal set and normalized local dropout enhanced rPPG signal set and will And any rPPG signal set from the four normalized enhanced rPPG signal sets is denoted as , ; Step 3: Construct a multi-scale region-of-interest (ROI) attention network, including: an ROI-independent temporal encoder, a multi-scale stitching module, a spatial weight adaptive quantization encoder, and a pulse wave decoding module, and then... The pulse wave signal set is obtained through processing. ; Step 4: Construct the joint time-frequency domain loss function, including: the local search narrowband signal-to-noise ratio loss in stage 1. The time-frequency domain loss of the global search in the second stage This is used to train a multi-scale domain of interest attention network, resulting in a pre-trained multi-scale domain of interest attention model. Step 5: Use the pre-trained multi-scale region of interest attention model to process the original rPPG signal set to be tested, and obtain the optimal pulse wave signal set to realize heart rate monitoring.

2. The video pulse wave extraction method based on progressive self-supervised learning according to claim 1, characterized in that, Step 2 is performed as follows: Step 2.1, in Spatial Dimensions Random selection above A domain of interest, and in Add the mean value to the rPPG original signal sequence corresponding to any channel dimension in any region of interest. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being The upper limit of the amplitude is Gaussian noise, thus affecting Different Gaussian noises are added to the original rPPG signal corresponding to each channel dimension in each region of interest to obtain a weakly enhanced rPPG signal sequence, which is then normalized to obtain... ; Step 2.2, in Spatial Dimensions Random selection above A domain of interest, and in Add the mean value to the rPPG original signal sequence corresponding to any channel dimension in any region of interest. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being The upper limit of the amplitude is Gaussian noise, thus affecting Different Gaussian noises are added to the original rPPG signal sequence corresponding to each channel dimension in each region of interest to obtain a strongly enhanced rPPG signal sequence, which is then normalized to obtain... ; Step 2.3, in Spatial Dimensions Random selection above A domain of interest, and in For each channel dimension within a region of interest, the original rPPG signal sequence is augmented with frequencies ranging from [frequency range]. Random sampled values ​​within the range, with amplitude within the interval The cosine signal of the random sampled values ​​within the range is used to obtain the periodically enhanced rPPG signal sequence, which is then normalized to obtain... ; Step 2.4, in Spatial Dimensions Random selection above One domain of interest, and The original rPPG signal sequence corresponding to each channel dimension in each region of interest is replaced with a signal having a mean of 1. The variance is the interval Random sampled values ​​within the range, with the lower bound of the amplitude being The upper limit of the amplitude is Gaussian noise is used to obtain the locally dropped enhanced rPPG signal, which is then normalized to obtain... .

3. The video pulse wave extraction method based on progressive self-supervised learning according to claim 2, characterized in that, Step 3 includes: Step 3.1, constructing a region-of-interest independent temporal encoder is... It consists of cascaded time blocks, and for Processing is performed to obtain the first... Cascaded spatially independent temporal feature maps , among which, the Each cascaded time block is generated by the convolution kernel. Two-dimensional convolutional layer Batch Normalization Layer and nonlinear activation layer composition; and They represent the first Two-dimensional convolutional layers in cascaded time blocks The number of rows and columns of the convolution kernel, for Channel dimension; Step 3.2: Constructing a multi-scale stitching module consists of average pooling blocks and feature stitching blocks, and then sequentially processing them... The process is performed to obtain multi-scale temporal feature maps. Among them, the average pooling block is composed of The average pooling layers are connected in parallel, making the first... The width of the pooling window for each average pooling layer is Height is ; for Channel dimension; Step 3.3, constructing the spatial weight adaptive quantization encoder is... It is composed of cascaded adaptive interest domain weighted modules, and sequentially applied to... Processing is performed to obtain the first... Level spatial weighted feature map sequence , among which, the The level-adaptive domain-of-interest weighting module consists of the first... Level multi-scale temporal self-attention module and the first Composed of a first-level feedforward module, the first The multi-scale temporal self-attention module consists of the first level. Multi-head self-attention layer and the Level attention layer normalization layer Composition, the first The first-level feedforward module consists of the first-level feedforward module. Level feedforward layer and the Level feedforward normalization composition; express The first moment Level-weighted feature map; Step 3.4: Constructing the pulse wave decoding module is done using the spatial compression module. and It is composed of cascaded decoding modules, and sequentially... The pulse wave signal set is obtained through processing. Among them, the space compression module The average pooling window is A single average pooling layer, and These represent the width and height of the average pooling window, respectively; The level decoding module consists of convolutional kernels with a size of The Level 1 one-dimensional convolutional layer , No. Level batch normalization layer and the Level Activation Layer composition.

4. The video pulse wave extraction method based on progressive self-supervised learning according to claim 3, characterized in that, Step 3.1 includes: when At that time, the rPPG signal set Enter the number In a series of cascaded time blocks, and through a two-dimensional convolutional layer After expanding the channel dimensions, batch normalization layers are then performed sequentially. and nonlinear activation The processing yields the first Cascaded spatially independent temporal feature maps ,in, For the first Channel dimension of a cascaded spatially independent temporal feature map; when At that time, the first Cascaded spatially independent temporal feature maps Enter the first The cascaded time blocks yield the first... Cascaded spatially independent temporal feature maps Ultimately, it was the first The output of the cascaded time block is... Cascaded spatially independent temporal feature maps .

5. The video pulse wave extraction method based on progressive self-supervised learning according to claim 3, characterized in that, Step 3.2 includes: Each of the parallel average pooling layers respectively... Perform time-series-independent average pooling based on the region of interest to obtain Temporal feature maps at various scales ;in, Indicates the first Temporal feature maps at various scales; Feature splicing blocks along the channel dimension By concatenating the data, multi-scale temporal feature maps can be obtained. .

6. The video pulse wave extraction method based on progressive self-supervised learning according to claim 3, characterized in that, Step 3.3 includes: when At that time, According to the time dimension After splitting, the first one is obtained. Multi-scale feature map sequence ,in, express The first moment Multi-scale feature maps in spatial dimensions; Will Enter the first In the adaptive interest domain weighting module, the first... The first level of multi-scale temporal self-attention module Multi-head self-attention layer right The process yields a spatially quantized weighted feature map based on the correlation of temporal features. Then by the first Level attention layer normalization layer right Process and obtain The first moment Level-space linear weighted feature map Then proceed through the first... Level feedforward layer and the Level feedforward normalization Perform nonlinear feature mapping to output The first moment Level spatial weighted feature map ; when At that time, The first moment Level spatial weighted feature map Enter the first The process is performed in an adaptive domain-of-interest weighted module, and the output is... The first moment Level spatial weighted feature map Thus, by the first Level Adaptive Region of Interest Weighted Module Output The first moment Level spatial weighted feature map , .

7. The video pulse wave extraction method based on progressive self-supervised learning according to claim 3, characterized in that, Step 3.4 includes: Will The first moment Level spatial weighted feature map Input space compression module In the middle, and on By performing summation and averaging operations along the spatial dimension, we obtain... Spatial compression characteristics of time This will compress the spatial feature sequence. After concatenation along the time dimension, a spatially compressed temporal feature map is obtained. ; when At that time, Enter the first In the level decoding module, by the first Level 1 one-dimensional convolutional layer right After feature dimension compression, it then goes through the first... Level batch normalization layer and the Level Activation Layer The processing yields the first Timing characteristics of multi-channel compression ,in, for Channel dimension; when At that time, the first Timing characteristics of multi-channel compression Enter the first The process is performed in the first-level decoding module to obtain the second-level decoding module. Timing compression characteristics of multi-channel compression Ultimately, it was the first The level decoding module outputs a pulse wave signal set with a channel dimension of 1. , .

8. The video pulse wave extraction method based on progressive self-supervised learning according to claim 1, characterized in that, Step 4 is performed as follows: Step 4.1: Construct the local search narrowband signal-to-noise ratio loss for the first stage using equation (1). : (1) In equation (1), for The power spectrum, Represents the frequency independent variable. For pulse wave signal set Perform a local search for the pulse center frequency, with a value range of [value range missing]. , express The lower boundary, express The upper boundary, It is a narrow frequency band; Step 4.2: Construct the global search time-frequency domain loss for the second stage using equation (2). : (2) In equation (2), For balance parameters; The loss represents the temporal similarity of positive samples, and is obtained from equation (3); The value represents the global search narrowband signal-to-noise ratio loss, which is obtained from equation (4); (3) (4) In equation (4), For pulse wave signal set Perform a global search for the pulse center frequency, with a value range of [range missing]. ; express The lower boundary, express The lower boundary, of which, , ; Represents the original pulse wave signal. This indicates an enhanced pulse wave signal set. ; In equation (3), Representing the negative Pearson-related loss, and obtained from equation (5), The mean squared error loss is represented by equation (6), which is obtained from equation (6): (5) (6) In equations (5) and (6), Indicates to The result obtained by averaging over the time dimension Indicates to The result obtained by averaging over the time dimension; Step 4.3: Use the AdamW optimizer to perform the first stage of self-supervised training of the multi-scale region-of-interest attention network on the video pulse wave, and calculate... To update network parameters when the number of training iterations reaches a threshold. At this point, the first phase of self-supervised training ends, resulting in a pre-trained multi-scale domain-of-interest attention network. The AdamW optimizer was used to perform the second stage of self-supervised training on the pre-trained multi-scale region-of-interest attention network, and the video pulse wave was calculated. To update network parameters when the number of training iterations reaches a threshold. At this point, the second phase of self-supervised training ends, and a pre-trained multi-scale domain of interest attention model is obtained.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the video pulse wave extraction method according to any one of claims 1-8, the processor being configured to execute the program stored in the memory.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the video pulse wave extraction method according to any one of claims 1-8.

Images