Remote heart rate detection method based on domain self-adaption

Abstract

The invention discloses a remote heart rate detection method based on domain self-adaption, and relates to the technical field of deep learning and remote heart rate detection.A source domain data set is used for training a domain self-adaption Teamer network, and the Teamer network is used for outputting a predicted rPPG signal according to an input noise rPPG signal and a face video; generating a prediction rPPG signal corresponding to a target domain face video by using the trained Teamer network, taking the prediction rPPG signal as a target domain pseudo tag, and constructing a target domain data set; the target domain data set is used for training a domain-adaptive Student network, and the Student network is used for outputting a predicted rPPG signal according to an input noise rPPG signal and a face video; and performing remote heart rate detection by using the trained Student network, and generating a predicted rPPG signal of the to-be-detected face video. The cross-domain heart rate detection method has better adaptability, accuracy and efficiency in cross-domain heart rate detection.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and remote heart rate detection, and in particular to a remote heart rate detection method based on domain adaptation. Background Art

[0002] In the application of modern technology, remote heart rate monitoring technology plays an indispensable role in healthcare and sports monitoring. Traditional heart rate monitoring relies primarily on contact devices, such as ECG electrodes and pulse sensors. In conventional ECG monitoring scenarios in hospitals, patients need to connect multiple wires to the ECG electrodes, which not only restricts their freedom of movement but can also cause psychological stress and physical discomfort. In sports scenarios, such as running and fitness, wearing such contact devices can severely impact the exercise experience and can even lead to monitoring interruptions due to device shaking or falling off.

[0003] With the rapid development of computer vision technology, video-based remote heart rate detection has gradually emerged as a research hotspot. This technology analyzes subtle facial changes in videos to detect heart rate, freeing itself from the constraints of contact devices. Theoretically, it can provide real-time, non-invasive heart rate monitoring in a variety of scenarios. However, in practical applications, this technology faces the severe challenge of domain differences.

[0004] Video data collected by different environments and devices exhibit significant differences in multiple dimensions. In terms of lighting, the uniform and soft lighting indoors and the strong, changeable sunlight and dappled shade outdoors can lead to significant differences in the presentation of faces in videos. In low-light indoors, facial details may be blurred; while in direct strong sunlight outdoors, the face may experience reflections and shadows, making key features difficult to identify. In terms of equipment, videos collected by cameras of different resolutions have different clarity. Low-resolution videos may lose important facial details, while high-resolution videos may put pressure on subsequent processing due to the large amount of data. In addition, frame rate differences between different devices can also affect the accuracy of capturing facial changes in the video. In terms of human posture, the position and angle of the face in the video constantly change when viewed from the front, side, lowering the head, raising the head, and in various dynamic postures, which brings great difficulties to video-based heart rate detection.

[0005] This domain difference results in significant differences in the performance of video-based heart rate detection models across different datasets. Source domain video data collected in a laboratory environment typically features stable lighting, a fixed resolution, and relatively standard person poses. Heart rate detection models trained on this basis may perform well in this environment. However, when applied to actual outdoor scenes (the target domain), the model's detection accuracy may be significantly reduced due to variations in the aforementioned environmental, device, and pose factors. For example, in strong outdoor light, facial reflections may make some features difficult to identify, causing the model to misjudge heart rate. When a person moves rapidly, frequent changes in pose may prevent the model from accurately tracking facial features, thus affecting the accuracy of heart rate detection.

[0006] Existing heart rate detection methods have many shortcomings when dealing with domain differences. Traditional methods based on manual features and machine learning, such as those based on skin color models and signal processing algorithms, have poor adaptability to environmental and equipment changes and have difficulty accurately extracting heart rate-related features in complex scenarios. While deep learning-based methods can automatically learn features to a certain extent, deep neural networks are sensitive to data distribution. When the domain difference between test data and training data is large, the model is prone to overfitting or underfitting, resulting in performance degradation. For example, some heart rate detection models based on convolutional neural networks (CNNs) may not be able to effectively adjust the learning of color and texture features when faced with video data under different lighting conditions, thereby affecting the accuracy of heart rate detection.

[0007] Furthermore, in related video analysis fields, such as physiological measurements based on facial videos, remote photoplethysmography (rPPG) technology aims to non-contactly measure cardiac activity and physiological signals using facial videos, offering significant potential applications in healthcare monitoring and other fields. Early research primarily employed classical signal processing methods, analyzing subtle color changes within facial regions of interest (ROIs) to obtain heart rate information. Other studies have utilized color subspace transformations to measure rPPG using all skin pixels. In recent years, with the rapid development of deep learning technology, deep learning-based rPPG measurement methods have gradually become dominant. However, traditional methods often focus solely on spatiotemporal rPPG features between adjacent frames, ignoring the interrelationships between quasi-periodic rPPG features over longer timescales. While purely end-to-end deep learning methods directly use facial video frames as input to predict rPPG and other physiological signals, these methods are susceptible to interference from complex scenes and struggle to effectively exclude features unrelated to rPPG, resulting in significant performance degradation on real-world datasets.

[0008] In summary, in areas such as remote heart rate detection and facial video-based physiological measurements, existing technical methods have difficulty effectively addressing issues such as domain differences and are unable to meet the stringent accuracy, robustness, and adaptability requirements of practical applications. Therefore, developing new methods that can effectively address these challenges is of great practical significance. This will not only promote technological progress in these fields, but also provide more reliable technical support for related practical applications. In remote heart rate detection technology, addressing domain differences and improving model performance in different scenarios are crucial for achieving accurate and convenient heart rate monitoring and promoting the development of industries such as healthcare and sports monitoring. Summary of the Invention

[0009] In order to overcome the defects in the above-mentioned prior art, the present invention provides a remote heart rate detection method based on domain adaptation, which has better adaptability, accuracy and efficiency in cross-domain heart rate detection, while maintaining low annotation data requirements, and is suitable for remote health monitoring in actual scenarios.

[0010] To achieve the above object, the present invention adopts the following technical solutions, including:

[0011] A remote heart rate detection method based on domain adaptation includes the following steps:

[0012] S1, using the source domain dataset to train the domain-adaptive Teacher network. The Teacher network is used to output the predicted rPPG signal based on the input noisy rPPG signal and face video. The source domain dataset includes the source domain face video and its corresponding source domain true label, i.e., the source domain true rPPG signal.

[0013] S2, using the trained Teacher network to generate the predicted rPPG signal corresponding to the target domain face video as the target domain pseudo label, and constructing the target domain dataset; the target domain dataset includes the target domain face video and its corresponding target domain pseudo label;

[0014] S3, using the target domain dataset to train the domain-adaptive Student network, which is used to output the predicted rPPG signal based on the input noisy rPPG signal and face video;

[0015] S4 uses the trained Student network to perform remote heart rate detection and generate the predicted rPPG signal of the face video to be detected.

[0016] Preferably, in step S1, the training process of the Teacher network is specifically as follows:

[0017] S11, the i-th source domain face video x i s The corresponding source domain true label yi s Denote it as the initial signal y0, add noise in time steps, and obtain the noise signal after noise addition in each time step;

[0018] S12, for the i-th source domain face video x i s Perform feature extraction and process to obtain a multi-scale spatiotemporal graph;

[0019] S13, fusing the noise signal with the multi-scale spatiotemporal graph to obtain fused data;

[0020] S14, the fused data is input into the denoiser P conditioned on the initial signal y0 and the time step to reconstruct the noise-free rPPG signal

[0021] S15, reconstruct the noise-free rPPG signal After embedding E s After processing, we get the i-th source domain face video x i s The corresponding multiple predicted rPPG signals are multiple pseudo labels First, based on the mean square error loss of power spectral density PSD, the pseudo label closest to the initial signal y0 is selected. Reconstruct the source domain embedding E s For the i-th source domain face video x i s The true source domain label y i s and predicted rPPG signals Negative Pearson correlation loss via negative Pearson correlation loss Inversely optimize the denoiser P and source domain embedding E s .

[0022] Preferably, for any given data X, embed E through the source domain s Generate a deviation B bias , the generated deviation B bias The output data of the denoiser is used to reconstruct the noise-free rPPG signal Add together to get the source domain embedding E s Output data

[0023] The mean square error loss of the power spectral density PSD is specifically calculated as follows:

[0024]

[0025] Among them, PSD pred [i] and PSD true[i] are respectively The i-th element in the predicted PSD signal and the true PSD signal obtained after processing y0, l represents the length of the signal, and N represents the number of elements.

[0026] Preferably, in step S2, the reasoning process of the Teacher network, i.e., using the trained Teacher network to generate target domain pseudo labels corresponding to the target domain face video, is specifically as follows:

[0027] S21, randomly generated noise signal;

[0028] S22, for the j-th target domain face video x j t Perform feature extraction and process to obtain a multi-scale spatiotemporal graph;

[0029] S23, fusing the randomly generated noise signal with the multi-scale spatiotemporal graph to obtain fused data;

[0030] S24, the fused data is embedded into the source domain E through the denoiser P. s After processing, the j-th target domain face video x is obtained j t The corresponding predicted rPPG signal is the target domain pseudo label

[0031] Preferably, in step S3, the training process of the Student network is as follows:

[0032] S31, the target domain face video x j t Corresponding target domain pseudo labels Noise is added in time steps to obtain the noise signal after noise addition in each time step;

[0033] S32, for the j-th target domain face video x j t Perform feature extraction and process to obtain a multi-scale spatiotemporal graph;

[0034] S33, fusing the noise signal with the multi-scale spatiotemporal graph to obtain fused data;

[0035] S34, the fused data is input into the denoiser P, and then embedded into the target domain E t After processing, we get the j-th target domain face video x j t Corresponding predicted rPPG signal

[0036] S35, build target domain embedding E t For the jth target domain face video The target domain pseudo labels and predicted rPPG signals Negative Pearson correlation loss via negative Pearson correlation loss Reverse optimization of the target domain embedding E in the Student network t .

[0037] Preferably, the linear layer in the denoiser P is embedded with a low-rank adaptation mechanism LoRA, which is as follows:

[0038]

[0039] Among them, W represents the updated weight matrix, W0 represents the fixed original weight matrix, α is the scaling factor, and the low-rank matrix A∈R m×r , B∈R r×n , r is the rank of the low-rank matrix and r≤min(m,n), m represents the input dimension of the weight matrix, and n represents the output dimension of the weight matrix;

[0040] In the training process, two pairs of low-rank matrix groups (A tea , B tea ) and (A stu , B stu ), two pairs of low-rank matrix groups (A tea , B tea ) and (A stu , B stu ) are introduced into the Teacher network and the Student network as trainable parameters respectively. During the training process, the weight matrix W0 corresponding to the key part of the denoiser P is decomposed and updated with low rank, while the other parts remain unchanged. In the process of forward propagation, i.e., adding noise, the updated weight matrix W is calculated according to the low rank adaptation mechanism; in the process of back propagation, i.e., removing noise, the low rank matrix group (A tea , B tea ) and (A stu , B stu ) and updates them according to the gradients, while the original weight matrix W0 remains unchanged.

[0041] Preferably, in step S4, the inference process of the Student network, i.e., using the trained Student network to perform remote heart rate detection and generate a predicted rPPG signal of the face video to be detected, is specifically as follows:

[0042] S41, sampling noise signals from Gaussian distribution to obtain H initial heart rate data assumptions y 0:H,0|T ;

[0043] S42, extracting features from the face video to be detected and processing it to obtain a multi-scale spatiotemporal graph;

[0044] S43, assume that H initial heart rate data are y 0:H,0|T Fuse with the multi-scale spatiotemporal graph to obtain H fused data;

[0045] S44, input H fused data into the denoiser P, and predict feasible heart rate data hypotheses

[0046] S45, the feasible heart rate data is assumed Generating noisy heart rate data via a diffusion model As the input of the denoiser P for the next time step, the process is expressed as:

[0047]

[0048] Where t and t′ are the current and next time steps respectively, initially t=T,∈~Ν(0,I) is y 0:H,t Independent standard Gaussian noise;

[0049]

[0050] Among them, ∈ t is the noise at time step t; σ t Used to control the randomness of the diffusion process; is the cumulative noise scaling factor from time step 0 to t;

[0051] The process of step S45 is iterated K times, starting from T, and the time step of each iteration is expressed as:

[0052]

[0053] Preferably, during the training process of the Teacher network and the Student network, a denoiser P is used to denoise the fused data. The denoiser P adopts STHD, i.e., a spatiotemporal hybrid denoiser, as shown below:

[0054] The multi-scale spatiotemporal graph and the noisy rPPG signal are concatenated into a matrix for dimension alignment; the resulting matrix is ​​then projected into a high-dimensional embedding through an embedding layer. middle;

[0055] Mapping high-dimensional embedding x to query key Sum Then they are divided into spatial groups along the channel dimension and time group Where n represents the number of facial regions of interest (ROIs), J represents the number of frames of video data, C represents the number of channels, and D represents the dimension of the high-dimensional embedding after projection by the embedding layer.

[0056] Spatial correlation Z between facial regions of interest ROI in the same frame S Calculated using the following formula:

[0057] Z S =MSA S (Q S ,K S ,V S )

[0058] Among them, MSA S It is a spatial multi-head self-attention mechanism;

[0059] Temporal correlation Z between facial regions of interest ROIs in the same frame T Calculated using the following formula:

[0060] Z T =MSA T (Q T ,K T ,V T )

[0061] Among them, MSA T It is a temporal multi-head self-attention mechanism.

[0062] The present invention also provides a readable storage medium having a computer program stored thereon, and when the computer program is executed, the remote heart rate detection method based on domain adaptation is implemented.

[0063] The present invention also provides a computer program product, which includes a computer program / instruction, which, when executed by a processor, implements the remote heart rate detection method based on domain adaptation.

[0064] The advantages of the present invention are:

[0065] (1) This paper improves the domain adaptation capability. Through the Teacher-Student framework and pseudo-label generation, it effectively reduces the distribution difference between the source domain and the target domain, and improves the performance of the model in the new domain (target domain) without the need for a large amount of labeled data.

[0066] (2) Noise is added during the training process and the signal is reconstructed through a denoiser, making the model more robust to input noise and adaptable to various interferences that may exist in practical applications.

[0067] (3) During the training phase, noise is added step by step and the signal is reconstructed to adapt the model to complex noises such as illumination changes and motion blur in real scenes. During the inference phase, the diffusion model is iteratively denoised (gradually refined from time step T to 0) and multiple candidate signals are generated in combination with the Gaussian initial hypothesis to improve the stability of heart rate prediction.

[0068] (4) Multi-scale spatiotemporal graphs are used to capture the spatiotemporal changes of the face at different levels to improve the accuracy of rPPG signal extraction, especially in complex environments or when changes occur between different individuals.

[0069] (5) The LoRA mechanism is introduced to reduce the number of training parameters, speed up training, and maintain model performance, which is suitable for situations with limited resources.

[0070] (6) PSD loss and Pearson correlation loss are combined to optimize the signal simultaneously in the frequency domain and time domain, ensuring that the predicted heart rate signal is close to the true value in frequency and waveform. The Teacher network uses PSD mean square error loss to filter pseudo labels and combines it with negative Pearson correlation loss to optimize signal correlation, ensuring that the generated pseudo labels are close to the true signal in both the frequency domain and time domain, reducing the interference of noise on target domain training.

[0071] (7) In the inference stage, the diffusion model is used to gradually denoise the data to generate more reliable heart rate data and improve the stability and accuracy of the detection results.

[0072] (8) The present invention combines the diffusion model with domain adaptation. The source domain and target domain embedding modules learn domain-specific features respectively, and achieve implicit feature alignment through pseudo-labels to solve the problem of cross-domain data distribution offset and break through the traditional method's reliance on strongly labeled data.

[0073] (9) A spatiotemporal hybrid denoiser (STHD) is used to model the spatial correlation (the relationship between different ROIs in the same frame) and temporal dynamic changes (temporal evolution across frames) of facial ROI regions through spatial and temporal attention grouping mechanisms, thereby improving feature fusion efficiency and optimizing signal reconstruction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a flow chart of a remote heart rate detection method based on domain adaptation of the present invention.

[0075] Figure 2 This is a process diagram of a remote heart rate detection method based on domain adaptation of the present invention. DETAILED DESCRIPTION

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0077] Depend on Figure 1 and Figure 2 As shown, the present invention provides a remote heart rate detection method based on domain adaptation, and the specific process is as follows:

[0078] S1, the training of the domain-adaptive Teacher network is as follows:

[0079] The domain-adaptive Teacher network is trained using a source domain dataset. The Teacher network is used to output a predicted rPPG signal based on the input noisy rPPG signal and face video. The source domain dataset includes the source domain face video and its corresponding source domain ground truth label, i.e., the source domain ground truth rPPG signal.

[0080] S2, use the trained Teacher network to generate the predicted rPPG signal corresponding to the target domain face video as the target domain pseudo label, and construct the target domain dataset; the target domain dataset includes the target domain face video and its corresponding target domain pseudo label.

[0081] S3, domain-adaptive Student network training, is as follows:

[0082] The target domain dataset is used to train a domain-adaptive Student network, which is used to output a predicted rPPG signal based on the input noisy rPPG signal and face video.

[0083] S4 uses the trained Student network to perform remote heart rate detection and generate the predicted rPPG signal of the face video to be detected.

[0084] In this embodiment, the data set is specifically as follows:

[0085] The source domain dataset is S = {(x i s ,y i s )|i=1,2,...,n s}, the target domain dataset is T={(x j t )|j=1,2,...,n t}. Among them, x i srepresents the i-th source domain face video, y i s Represents x i s The corresponding source domain true label (i.e. the source domain true rPPG signal), n s represents the total number of samples in the source domain dataset S, x j t represents the jth target domain face video, n t Represents the total number of samples in the target domain dataset T.

[0086] In this embodiment, the training process of the Teacher-Student network in steps S1-S3 is as follows:

[0087] S1, the training process of the Teacher network.

[0088] S11, the i-th source domain face video x i s The corresponding source domain true label y i s Denote it as the initial signal y0, add Gaussian noise in time steps, and obtain the noise signal after noise addition at each time step. In the forward process (noise addition process), gradually add a variance of β at each time step t t ∈[0,1] Gaussian noise ∈~Ν(0,I), and the noise signal y after adding noise at each time step t is obtained t , the formula is as follows:

[0089]

[0090] In the formula, q(y t |y t-1 ) represents the probability distribution of the forward diffusion process, describing the noise signal y from time step t-1 t-1 The noise signal y at time step t t The noise adding process; Ν(·) is the symbol of Gaussian distribution, and its mean is The covariance matrix is ​​β t I; I is the identity matrix, indicating that the noise is independent in each dimension and has the same variance.

[0091] Using the properties of the Markov chain, y in the above formula t Sampling can be performed directly based on the original signal y0 as the condition, and the formula is as follows:

[0092]

[0093] In the formula, q(y t |y0) represents the probability distribution of the forward diffusion process, describing the noise signal y from the original signal y0 to the time step tt The noise adding process; is the cumulative noise scaling factor from time step 0 to t, defined as is the mean of the Gaussian distribution, which represents the mean shift of the original signal y0 after t steps of noise are added; is the covariance matrix of the Gaussian distribution, which represents the variance of the cumulative noise, where I is the identity matrix, indicating that the noise is independent in each dimension and has the same variance.

[0094] S12, for the i-th source domain face video x i s Perform feature extraction and process to obtain a multi-scale spatiotemporal graph.

[0095] S13, fusing the noise signal with the multi-scale spatiotemporal graph to obtain fused data.

[0096] S14, during the remote heart rate detection process, the fused data will be input into the denoiser P conditioned on the initial signal y0 and time step t to reconstruct the noise-free rPPG signal

[0097]

[0098] S15, reconstruct the noise-free rPPG signal After embedding E s After processing, we get the i-th source domain video x i s The corresponding multiple predicted rPPG signals are multiple pseudo labels First, based on the mean square error loss of power spectral density PSD, the pseudo label closest to the initial signal y0 is selected. Reconstruct the source domain embedding E s For the i-th source domain face video x i s The true source domain label y i s and predicted rPPG signals Negative Pearson correlation loss via negative Pearson correlation loss Inversely optimize the denoiser P and source domain embedding E s .

[0099] Reconstructed noise-free rPPG signal After embedding E s The specific process is as follows:

[0100] In the training phase of the Teacher network, for any given data X, the source domain is embedded into E sA deviation B with a dimension determined by the input data will be generated bias ; The generated deviation B bias and the intermediate output y of the denoiser middle (Here is the reconstructed noise-free rPPG signal ) are added together to get the final output data result:

[0101] The mean square error loss based on power spectral density PSD is calculated as follows:

[0102]

[0103] Among them, PSD pred [i] and PSD true [i] are respectively The i-th element in the predicted PSD signal and the true PSD signal obtained after processing y0, l represents the length of the signal, and N represents the number of elements.

[0104] The specific calculation of power spectral density PSD is as follows:

[0105]

[0106] Among them, the one-dimensional signal The sampling rate is f Hz, l represents the length of the signal; PSD(f,y n ) is the power spectral density at frequency f; y n is the value of the nth sampling point of y; f s is the sampling frequency; n is the index of the signal on the time axis, ranging from 0 to l-1;

[0107] S2, the reasoning process of the Teacher network, that is, using the trained Teacher network to generate the target domain pseudo label corresponding to the target domain face video.

[0108] S21, collects random noise signals from Gaussian distribution.

[0109] S22, for the j-th target domain face video x j t ,First, a feature extraction operation is performed on it, and then the extracted features are processed into a multi-scale spatiotemporal graph.

[0110] S23, fusing the randomly generated noise signal with the multi-scale spatiotemporal graph to obtain fused data.

[0111] S24, the fused data is input into the Teacher network, and the Teacher network embeds the fused data into the source domain E through the denoiser P. s Processing, get the j-th target domain face video xj t The corresponding predicted rPPG signal is the target domain pseudo label At the same time, the parameters in the Teacher network are fixed so that they do not participate in gradient updates during the subsequent Student network training process.

[0112] S3, the training process of the Student network.

[0113] S31, the target domain face video The corresponding pseudo labels The data is passed into the Student network, and Gaussian noise is added in time steps to obtain the noise signal after adding noise in each time step.

[0114] S32, for the j-th target domain face video x j t Perform feature extraction and process to obtain a multi-scale spatiotemporal graph.

[0115] S33, fusing the noise signal with the multi-scale spatiotemporal graph to obtain fused data.

[0116] S34, the fused data is input into the denoiser P, and then embedded into the target domain E t After processing, the target domain embedding E is obtained t For the jth target domain face video Predicted rPPG signal

[0117] Target domain embedding E t The processing process is as follows:

[0118] In the training phase of the Student network, for any given data X, the target domain is embedded into E s A deviation B with a dimension determined by the input data will be generated bias ; The generated deviation B bias and the intermediate output y of the denoiser middle (Here is the jth target domain face video in the target domain The target domain pseudo labels ) are added together to get the final output data result:

[0119] S35, build target domain embedding E t For the jth target domain face video The target domain pseudo labels and predicted rPPG signals Negative Pearson correlation loss via negative Pearson correlation loss Reverse optimization of the target domain embedding E in the Student networkt , and embed the low-rank adaptation mechanism LoRA in the linear layer in the denoiser P to speed up the calculation process and optimize the denoiser P with fixed parameters.

[0120] Negative Pearson correlation loss The calculation is as follows:

[0121]

[0122] Where Cov(x,y) represents the covariance of variables x and y.

[0123] The low-rank adaptation mechanism is as follows:

[0124]

[0125] Among them, W represents the updated weight matrix, W0 represents the fixed original weight matrix, α is a scaling factor, and the low-rank matrix A∈R m×r , B∈R r×n , r is the rank of the low-rank matrix and r≤min(m,n), m represents the input dimension of the weight matrix (such as the dimension of the query or key), and n represents the output dimension of the weight matrix (such as the dimension of the value).

[0126] In the training process, two pairs of low-rank matrix groups (A tea , B tea ) and (A stu , B stu ), two pairs of low-rank matrix groups (A tea , B tea ) and (A stu , B stu ) are introduced as trainable parameters into the Teacher network model and the Student network model respectively. During the training process, the weight matrix W0 corresponding to some key components of the denoiser P is decomposed and updated with low rank, while the other parts remain unchanged. In the forward propagation (adding noise), the updated weight matrix W is calculated according to the low rank adaptation mechanism; in the back propagation (removing noise), only the low rank matrix group (A tea , B tea ) and (A stu , B stu ) and update their values ​​according to the gradient, while the original weight matrix W0 remains unchanged.

[0127] In this embodiment, the inference process of the Student network in step S4, i.e., using the trained Student network to perform remote heart rate detection and generate a predicted rPPG signal for the face video to be detected, is specifically as follows:

[0128] S41, sample noise from the unit Gaussian distribution to obtain H initial heart rate data, assuming y 0:H,0|T .

[0129] S42, extracting features from the face video to be detected, and processing it to obtain a multi-scale spatiotemporal graph.

[0130] S43, assume that H initial heart rate data y 0:H,0|T Fuse with the multi-scale spatiotemporal graph to obtain H fused data.

[0131] S44, input H fused data into the denoiser P, and predict feasible heart rate data hypotheses

[0132] By adding y 0:H,T Input to the denoiser to predict plausible hypotheses for heart rate data

[0133] For each hypothesis y 0:H,T The scores of each hypothesis are calculated as the sum of the differences in PSD values ​​between adjacent segments. H , the specific operations are as follows:

[0134]

[0135] Among them, score H The score of the hypothesis; w represents the fragment number; ||·|| represents the L2 norm.

[0136] The highest-scoring hypothesis is selected as the feasible hypothesis for the heart rate data.

[0137] S45, the feasible heart rate data is assumed Generate noisy heart rate data through a diffusion model (specifically the DDIM model) As the input of the denoiser for the next time step, the process is expressed as:

[0138]

[0139] Where t and t′ are the current and next time steps respectively, initially t=T,∈~Ν(0,I) is y 0:H,t Independent standard Gaussian noise;

[0140]

[0141] Among them, ∈ t is the noise at time step t; σ t Used to control the randomness of the diffusion process; is the cumulative noise scaling factor from time step 0 to t;

[0142] The process of step S45 is iterated K times, starting from T, and the time step of each iteration is expressed as:

[0143]

[0144] In this embodiment, during the training process of the Teacher-Student network, a denoiser P is used to perform denoising processing on the fused data, i.e., backward diffusion. The denoiser P adopts STHD, i.e., a spatiotemporal hybrid denoiser. The specific denoising steps are as follows:

[0145] The multi-scale spatiotemporal graph and the noisy rPPG signal are concatenated into a matrix for dimension alignment; the resulting matrix is ​​then projected into a high-dimensional embedding through an embedding layer. middle.

[0146] Mapping high-dimensional embedding x to query key Sum Then they are divided into spatial groups along the channel dimension and time group Where n represents the number of facial regions of interest (ROIs), J represents the number of frames of video data, C represents the number of channels, and D represents the dimension of the high-dimensional embedding after projection by the embedding layer.

[0147] Spatial correlation Z between facial regions of interest ROI in the same frame S Calculated using the following formula:

[0148] Z S =MSA S (Q S ,K S ,V S )

[0149] Among them, MSA S It is a spatial multi-head self-attention mechanism;

[0150] Temporal correlation Z between facial regions of interest ROIs in the same frame T Calculated using the following formula:

[0151] Z T =MSA T (Q T ,K T ,V T )

[0152] Among them, MSA T It is a temporal multi-head self-attention mechanism.

[0153] During the inference process of the Student network, the fused data is denoised by the denoiser P. After K iterations of denoising, the denoised data is output. The denoising expression during the inference process is as follows:

[0154] b k =STHD(b k+1 )+b k+1 .

[0155] In this embodiment, features are extracted from the face video to obtain a multi-scale spatiotemporal graph, which is specifically shown below:

[0156] Divide the face video into multiple segments based on the number of frames, each segment has M frames of video images;

[0157] Detect the video image and divide the face into n regions of interest (ROIs);

[0158] Combine n ROIs to obtain a combination of one ROI and different ROIs, totaling 2 n-1 combinations;

[0159] For each combination, the average pixel value of all ROIs in each color channel is calculated; the color channels are RGB and YUV, a total of 6;

[0160] Finally, we get the dimension 6×(2 n-1 )×M vector as a multi-scale space-time graph.

[0161] In this embodiment, the fusion of the noisy rPPG signal and the multi-scale spatiotemporal graph is specifically as follows:

[0162] The dimension of the noise rPPG signal is 1×M, and the dimension of the noise rPPG signal is expanded to 1×(2 n-1 )×M, the expanded noise rPPG signal and the multi-scale spatiotemporal map are spliced ​​in the channel dimension to obtain the fused data. The dimension of the fused data is 7×(2 n-1 )×M.

[0163] In this embodiment, the source domain is embedded in E s and target domain embedding E t Each contains different Transformer blocks, which are used to process the fused data passed in previously and output the fused data as heart rate data. The specific contents are as follows:

[0164] The embedding layer maps channel (the input data dimension, the number of features at each time step of the input sequence) to embed_dim (the embedding dimension, which is used to map the input sequence to a high-dimensional continuous space). The formula is:

[0165] R E =W e X+B bias

[0166] Where X is the input data, W e is the weight matrix of the embedding layer, with shape (channel, embed_dim), R E is the output data after embedding.

[0167] Position encoding is used in the embedding layer to record the position of the output: the position information is calculated using sine and cosine functions and added to the embedding sequence to make the model aware of the element position.

[0168] In the Transformer encoder layer, the multi-head self-attention mechanism is used to calculate the attention scores of multiple heads in parallel, focusing on different parts of the sequence. The features are further processed through a feedforward network consisting of two linear layers and an activation function. After that, the input is sent to the decoding layer, which uses the decoding layer to map the encoder output from the embed_dim dimension back to the channel dimension. The formula is:

[0169] O=W d H T +b d

[0170] Among them, O is the decoded output, W d Is the weight matrix of the decoding layer, with shape (channel, embed_dim)H T is the output of the Transformer encoder, b d is the bias vector, shape (channel).

[0171] Then, a mask mechanism is used to generate subsequent masks to prevent the model from seeing future information during prediction. Finally, in the forward propagation, the input is embedded, positionally encoded, processed through a multi-layer encoder, decoded, and finally averaged to obtain the output.

[0172] In this embodiment, taking the Teacher network as an example, LoRA is embedded in the denoiser and the internal dimension is changed as follows:

[0173] Use the search function in the code to find the module that needs to be embedded in LoRA in the denoiser model.

[0174] Embedded LoRA module:

[0175] For the module that needs to be embedded in LoRA, perform the following operations:

[0176] Linear layer processing: If it is a linear layer, replace it with the previously defined LoraInjectedLinear module. During the replacement process, the weights and biases of the original linear layer are copied to the original linear layer part of the new module.

[0177] Convolutional layer processing: If it is a convolutional layer, replace it with the LoraInjectedConv2d module, and also copy the weights and biases of the original convolutional layer to the original convolutional layer part of the new module.

[0178] The dimensions of the embeddings change as follows:

[0179] Dimension changes when the linear layer is embedded in LoRA:

[0180] Original linear layer input: B n represents the batch size, I represents the number of input features, and the input dimension is (B n ,I);

[0181] Output: O represents the number of output features. After linear transformation, the output dimension is (B n ,O);

[0182] After embedding the LoRA module:

[0183] First low rank downsampling (lora_down) input into (B n ,I), the output dimension becomes (B n ,R)), where R is the rank of the low-rank matrix; then through the selector selector, input into (B n ,R) Output dimension remains unchanged (B n ,R); then low rank upsampling (lora_up) input into (B n ,R) and then output and change the dimension back to (B n ,O);

[0184] The original linear layer output (B n ,O) and low-rank processing output (B n ,O) are added, the final output dimension is still (B n ,O);

[0185] Dimension changes when the convolutional layer is embedded in LoRA:

[0186] Original convolutional layer:

[0187] Input: B n is the batch size, C in is the number of input channels, H in 、W in are the feature map height and width respectively, and the input dimension is (B n ,C in ,Hin ,W in ); Output: C out is the number of output channels, H out 、W out The output feature map height and width, the output dimension is (B n ,C out ,H out ,W out );

[0188] After embedding the LoRA module:

[0189] In low-rank downsampling convolution (lora_down), input into (B n ,C in ,H in ,W in ) and then the output dimension becomes (B n ,R,H out ,W out ); then input into (B n ,R,H out ,W out )The output is still (B n ,R,H out ,W out ); then the low-rank up-sampling convolution (lora_up) input (B n ,R,H out ,W out ) and the output dimension changes back to (B n ,C out ,H out ,W out ); then merge the outputs and output the original convolutional layer (B n ,C out ,H out ,W out ) and low-rank processing output (B n ,C out ,H out ,W out ) are added, the final output dimension is still (B n ,C out ,H out ,W out ).

[0190] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A remote heart rate detection method based on domain adaptation, characterized in that: The following steps are involved: S1, using the source domain dataset to train the domain-adaptive Teacher network. The Teacher network is used to output the predicted rPPG signal based on the input noisy rPPG signal and face video. The source domain dataset includes the source domain face video and its corresponding source domain true label, i.e., the source domain true rPPG signal. S2, using the trained Teacher network to generate the predicted rPPG signal corresponding to the target domain face video as the target domain pseudo label, and constructing the target domain dataset; the target domain dataset includes the target domain face video and its corresponding target domain pseudo label; S3, using the target domain dataset to train the domain-adaptive Student network, which is used to output the predicted rPPG signal based on the input noisy rPPG signal and face video; S4 uses the trained Student network to perform remote heart rate detection and generate the predicted rPPG signal of the face video to be detected.

2. The method for remote heart rate detection based on domain adaptation according to claim 1, characterized in that: In step S1, the training process of the Teacher network is as follows: S11, the i-th source domain face video x i s The corresponding source domain true label y i s Denote it as the initial signal y0, add noise in time steps, and obtain the noise signal after noise addition in each time step; S12, for the i-th source domain face video x i s Perform feature extraction and process to obtain a multi-scale spatiotemporal graph; S13, fusing the noise signal with the multi-scale spatiotemporal graph to obtain fused data; S14, the fused data is input into the denoiser P conditioned on the initial signal y0 and the time step to reconstruct the noise-free rPPG signal S15, reconstruct the noise-free rPPG signal After embedding E s After processing, we get the i-th source domain face video x i s The corresponding multiple predicted rPPG signals are multiple pseudo labels First, based on the mean square error loss of power spectral density PSD, the pseudo label closest to the initial signal y0 is selected. Reconstruct the source domain embedding E s For the i-th source domain face video x i s The true source domain label y i s and predicted rPPG signals Negative Pearson correlation loss via negative Pearson correlation loss Inversely optimize the denoiser P and source domain embedding E s .

3. The method for remote heart rate detection based on domain adaptation according to claim 2, characterized in that: For any given data X, embed E through the source domain s Generate a deviation B bias , the generated deviation B bias The output data of the denoiser is used to reconstruct the noise-free rPPG signal Add together to get the source domain embedding E s Output data The mean square error loss of the power spectral density PSD is specifically calculated as follows: Among them, PSD pred [i] and PSD true [i] are respectively The i-th element in the predicted PSD signal and the true PSD signal obtained after processing y0, l represents the length of the signal, and N represents the number of elements.

4. The method for remote heart rate detection based on domain adaptation according to claim 2, characterized in that: In step S2, the reasoning process of the Teacher network, that is, using the trained Teacher network to generate the target domain pseudo label corresponding to the target domain face video, is as follows: S21, randomly generated noise signal; S22, for the j-th target domain face video x j t Perform feature extraction and process to obtain a multi-scale spatiotemporal graph; S23, fusing the randomly generated noise signal with the multi-scale spatiotemporal graph to obtain fused data; S24, the fused data is embedded into the source domain E through the denoiser P. s After processing, the jth target domain face video x is obtained j t The corresponding predicted rPPG signal is the target domain pseudo label y′ j t .

5. The method for remote heart rate detection based on domain adaptation according to claim 4, characterized in that: In step S3, the training process of the Student network is as follows: S31, the target domain face video x j t The corresponding target domain pseudo label y′ j t Noise is added in time steps to obtain the noise signal after noise addition in each time step; S32, for the j-th target domain face video x j t Perform feature extraction and process to obtain a multi-scale spatiotemporal graph; S33, fusing the noise signal with the multi-scale spatiotemporal graph to obtain fused data; S34, the fused data is input into the denoiser P, and then embedded into the target domain E t After processing, we get the j-th target domain face video x j t Corresponding predicted rPPG signal S35, build target domain embedding E t For the jth target domain face video The target domain pseudo labels and predicted rPPG signals Negative Pearson correlation loss via negative Pearson correlation loss Reverse optimization of the target domain embedding E in the Student network t .

6. The method for remote heart rate detection based on domain adaptation according to claim 5, characterized in that: The linear layer in the denoiser P is embedded in the low-rank adaptation mechanism LoRA, which is as follows: Among them, W represents the updated weight matrix, W0 represents the fixed original weight matrix, α is the scaling factor, and the low-rank matrix A∈R m×r , B∈R r×n , r is the rank of the low-rank matrix and r≤min(m,n), m represents the input dimension of the weight matrix, and n represents the output dimension of the weight matrix; In the training process, two pairs of low-rank matrix groups (A tea , B tea ) and (A stu , B stu ), two pairs of low-rank matrix groups (A tea , B tea ) and (A stu , B stu ) are introduced into the Teacher network and the Student network as trainable parameters respectively. During the training process, the weight matrix W0 corresponding to the key part of the denoiser P is decomposed and updated with low rank, while the other parts remain unchanged. In the process of forward propagation, i.e., adding noise, the updated weight matrix W is calculated according to the low rank adaptation mechanism; in the process of back propagation, i.e., removing noise, the low rank matrix group (A tea , B tea ) and (A stu , B stu ) and updates them according to the gradients, while the original weight matrix W0 remains unchanged.

7. The method for remote heart rate detection based on domain adaptation according to claim 5, characterized in that: In step S4, the inference process of the Student network, that is, using the trained Student network to perform remote heart rate detection and generate the predicted rPPG signal of the face video to be detected, is as follows: S41, sampling noise signals from Gaussian distribution to obtain H initial heart rate data assumptions y 0:H,0|T ; S42, extracting features from the face video to be detected and processing it to obtain a multi-scale spatiotemporal graph; S43, assume that H initial heart rate data are y 0:H,0|T Fuse with the multi-scale spatiotemporal graph to obtain H fused data; S44, input H fused data into the denoiser P, and predict feasible heart rate data hypotheses S45, assume that feasible heart rate data Generating noisy heart rate data via a diffusion model As the input of the denoiser P for the next time step, the process is expressed as: Where t and t′ are the current and next time steps respectively, initially t=T,∈~Ν(0,I) is y 0:H,t Independent standard Gaussian noise; Among them, ∈ t is the noise at time step t; σ t Used to control the randomness of the diffusion process; is the cumulative noise scaling factor from time step 0 to t; The process of step S45 is iterated K times, starting from T, and the time step of each iteration is expressed as:

8. The method for remote heart rate detection based on domain adaptation according to claim 7, characterized in that: During the training process of the Teacher network and the Student network, the fused data is denoised using a denoiser P, which is a spatiotemporal hybrid denoiser (STHD), as shown below: The multi-scale spatiotemporal graph and the noisy rPPG signal are concatenated into a matrix for dimension alignment; the resulting matrix is ​​then projected into a high-dimensional embedding through an embedding layer. middle; Mapping high-dimensional embedding x to query key Sum Then they are divided into spatial groups along the channel dimension and time group Where n represents the number of facial regions of interest (ROIs), J represents the number of frames of video data, C represents the number of channels, and D represents the dimension of the high-dimensional embedding after projection by the embedding layer. Spatial correlation Z between facial regions of interest ROI in the same frame S Calculated using the following formula: Z S =MSA S (Q S ,K S ,V S ) Among them, MSA S It is a spatial multi-head self-attention mechanism; Temporal correlation Z between facial regions of interest ROIs in the same frame T Calculated using the following formula: Z T =MSA T (Q T ,K T ,V T ) Among them, MSA T It is a temporal multi-head self-attention mechanism.

9. A readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed, the remote heart rate detection method based on domain adaptation according to any one of claims 1 to 8 is implemented.

10. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements a remote heart rate detection method based on domain adaptation as described in any one of claims 1 to 8.

Images