Remote multi-channel pulse wave signal linear fusion algorithm in complex scene

Abstract

The invention provides a pulse wave signal (BVP signal) extraction method based on remote photoplethysmography. The method comprises the following specific steps: step 1, acquiring a face video and detecting a face area; in the process, the system captures, recognizes and extracts all facial feature points; 2, the rPPG signal intensities of different face areas are different, and in order to maximize the signal quality, the face areas are divided according to the intensities of remote photoplethysmography signals (rPPG signals); 3, training a convolutional neural network model, calculating a weighting coefficient of each region according to the characteristics of the signals of each region, and fusing the signals of each region into a single-channel signal; 4, removing low-frequency noise and interference signals from the fused signals by using a filtering technology, wherein the superposition operation helps to enhance the effective rPPG signals; through the processing steps, the quality of the BVP signals can be improved, and high-quality input data is provided for subsequent extraction of blood oxygen, respiration, heart rate variability and other signals.

Description

Technical Field

[0001] The present invention belongs to the field of biomedical signal processing, and is specifically an algorithm for extracting pulse wave signals (BVP signals) through multi-channel remote photoplethysmography (rPPG) in complex scenarios. Background Art

[0002] Remote photoplethysmography (rPPG) is a non-contact technology based on video analysis that can detect physiological signals such as heart rate and respiratory rate. By capturing tiny changes in light reflected from the skin surface, remote photoplethysmography can be used in health monitoring. Compared with traditional contact sensors, remote photoplethysmography has the advantages of being non-invasive and comfortable. However, in complex scenarios, the extraction accuracy of remote photoplethysmography (rPPG signal) is often affected by low signal-to-noise ratio (SNR), motion artifacts and environmental interference. These problems result in low quality of the extracted rPPG signal, which in turn affects the subsequent calculation results of blood oxygen concentration, respiratory rate and heart rate.

[0003] Existing rPPG signal extraction methods can be mainly divided into two categories: nonlinear modeling methods based on neural networks and linear signal processing methods based on traditional algorithms.

[0004] 1. Neural network-based rPPG signal extraction method: In recent years, with the widespread application of deep learning technology, the use of neural networks in rPPG signal extraction has become more and more common. The neural network method relies on a large amount of training data to predict the pulse wave signal (BVP signal) by learning the complex nonlinear relationship between the input image and the physiological signal. However, there are several problems that cannot be ignored in the neural network method:

[0005] Introducing nonlinear noise: The nonlinear characteristics of the neural network model make it susceptible to the influence of data distribution. When there is a difference between the training data and the actual application scenario, the network may introduce additional nonlinear noise, which is difficult to remove through simple post-processing methods.

[0006] · False prediction results: The prediction results of neural networks depend largely on the diversity and quality of training data. In complex scenarios such as motion and lighting changes, neural networks are prone to produce false prediction results. This error will directly affect the quality of rPPG signals without strong enough prior information constraints, leading to misjudgment of subsequent physiological parameters (such as respiratory rate and heart rate).

[0007] 2. rPPG signal extraction methods based on traditional methods: Traditional rPPG signal extraction methods mainly include POS algorithm, CHROM algorithm and PCA (principal component analysis). These methods usually rely on the assumption of linear signal processing, that is, assuming that the light signals reflected from different areas of facial skin have fixed periodic characteristics and can be extracted through linear models. However, these traditional methods also face multiple challenges in practical applications:

[0008] Poor motion robustness: POS, CHROM and other algorithms are usually based on the assumption of a static environment, that is, the face remains basically still. However, in real application scenarios, the face will inevitably move, rotate and so on. These motion interferences will greatly reduce the signal extraction capabilities of these algorithms, resulting in the loss or distortion of rPPG signals.

[0009] Sensitive to lighting changes: Traditional methods usually assume that facial lighting is relatively stable and rely on small changes in reflected light to extract heart rate or breathing signals. However, in scenes with drastic changes in lighting, such as changes in outdoor light intensity, the algorithm has difficulty adapting to these changes, resulting in a decrease in the quality of the extracted signal. Light changes not only affect the amplitude of the rPPG signal, but may also introduce additional high-frequency noise, making it difficult for traditional algorithms to accurately capture low-frequency physiological signals.

[0010] Obvious environmental interference: Traditional linear models have difficulty coping with various interference sources in complex environments, such as background color changes, shadows, differences in facial skin reflectivity, etc. Under these interferences, the signal-to-noise ratio of POS, CHROM or PCA algorithms is greatly reduced, making it difficult to stably output accurate physiological signals.

[0011] To solve these problems, the present invention proposes a multi-channel rPPG signal linear fusion algorithm based on machine learning. This method combines the signal feature extraction capability of convolutional neural networks with the linear advantages of traditional signal processing methods, and effectively improves the quality and robustness of rPPG signals in complex scenes by introducing a multi-region weighted fusion mechanism. Compared with existing methods, the present invention can better cope with the challenges brought by motion and illumination changes, improve the signal extraction accuracy, and provide physiological signal extraction results with a higher signal-to-noise ratio in complex scenes. Summary of the invention

[0012] In view of the problems that the neural network method in the prior art introduces nonlinear noise and false prediction results, and the traditional algorithm has low robustness in motion and illumination changes, this paper proposes a multi-channel rPPG signal linear fusion algorithm in complex scenes. By combining the feature extraction capability of convolutional neural networks with linear weighted signal fusion technology, the algorithm improves the quality and robustness of rPPG signals, can effectively deal with motion artifacts and illumination changes in complex scenes, and extracts more accurate heart rate signals.

[0013] The specific steps of the present invention include:

[0014] 1. Facial video acquisition and regional division: First, the collected facial video is processed and the face detection algorithm is used to identify the key facial features, such as the forehead, nose tip, cheeks and other areas. According to the rPPG signal strength, the face is divided into multiple sub-areas with different signal-to-noise ratios. The division of these areas is based on the prior knowledge of the signal-to-noise ratio, and there are significant differences in the strength of the rPPG signals extracted from different areas. The overall signal quality is improved through subsequent weighted fusion.

[0015] 2. Multi-channel signal acquisition and convolutional neural network training: The rPPG signal of each sub-region is extracted by the dynamic priority convex hull method, which ensures the robustness of feature points in the case of facial movement. Subsequently, the average pixel value in the RGB space of each sub-region is calculated, and the signal stream is generated according to the timestamp interpolation of the video stream. In order to eliminate the amplitude difference of signals in different regions, the signal of each channel will be normalized to generate the processed multi-channel signal H.

[0016] According to the characteristics of different data sets, the video data is aligned with the contact pulse wave signal label. The rPPG signal in the video is extracted by traditional methods or deep learning algorithms, and the time difference between the signal and the label signal is calculated. The signal is then time-compensated to ensure the alignment of the signal timing.

[0017] After the data set is processed, it is divided into a training set and a test set according to the ratio. A convolutional neural network is used to train multi-channel signals. The network structure includes a channel combination layer, a pooling layer, and a fully connected layer to predict the weight coefficients of each sub-region. During the network training process, negative Pearson coefficient and frequency domain correlation are used as loss functions, and the network parameters are optimized through back propagation to obtain the optimal weight coefficient for each region.

[0018] 3. Multi-channel rPPG signal fusion: According to the weighted coefficients obtained through training, the rPPG signals of different sub-areas are linearly combined to generate a single-channel rPPG signal. This linear fusion method can effectively offset interference such as facial movement and lighting changes, and improve the signal-to-noise ratio of the signal. The fused rPPG signal has stronger anti-interference ability and is suitable for extracting accurate physiological signals in complex scenes.

[0019] 4. Post-processing and physiological signal extraction: The fused single-channel rPPG signal is filtered to remove low-frequency noise and interference. Specifically, a second-order cascade filter is used to filter the signal in the range of 0.5-4Hz to retain the main frequency components of the heart rate and breathing signals. The signal is analyzed in the frequency domain by Chirp Z transform (CZT), the frequency characteristics of the signal are further tracked, and the energy in the frequency domain is accumulated to estimate the heart rate at the current moment.

[0020] The present invention has the following significant advantages:

[0021] Strong ability to resist motion artifacts: The dynamic priority convex hull method is used to extract feature points and weighted fusion of multi-channel rPPG signals can effectively reduce the artifact interference caused by motion and enhance signal robustness.

[0022] Strong light adaptability: Multi-channel signal fusion can reduce the impact of light changes on the signal and enhance the stability of rPPG signals under unstable light conditions.

[0023] High signal quality: The convolutional neural network predicts the weighting coefficient based on the correlation between the rPPG signal in each region and the reference signal, and combines it with subsequent filtering processing to generate a single-channel rPPG signal with a high signal-to-noise ratio, providing high-quality input for subsequent physiological signal extraction.

[0024] Accurate timing alignment: By performing time alignment and compensation on the video signal and the contact pulse wave signal, the timing consistency of the data during training is ensured, thus improving the prediction accuracy of the network.

[0025] Through the method of the present invention, high-quality rPPG signals and physiological signals can be stably extracted in complex scenarios, which to a certain extent improves the robustness and accuracy of existing technologies in dealing with environmental interference such as motion and lighting changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention provides a framework for the anti-interference rPPG signal extraction process.

[0027] Figure 2 A schematic diagram of the process of anti-interference rPPG signal extraction provided by the present invention.

[0028] Figure 3 Schematic diagram of 17 facial skin area ROIs.

[0029] Figure 4 It is a training diagram of the remote pulse wave signal fusion network of the present invention.

[0030] Figure 5 This is the CNN classification network structure used in this article.

[0031] Figure 6 This is a schematic diagram showing the final effect of extracting the pulse wave signal energy spectrum using this method and the comparison with the traditional POS method. DETAILED DESCRIPTION

[0032] The process framework of this method is as follows Figure 1 As shown in the figure, the face is first divided into multiple ROI regions through the face detection algorithm, and the RGB signal is extracted in each region. The convex hull method is used to ensure the robustness of facial movement. Subsequently, the extracted signal is normalized and input into the convolutional neural network. The network outputs the weighted coefficients k1, k2, ..., k of each ROI region. n Then, the multi-channel signals are linearly fused through these weighting coefficients to effectively offset the artifact interference, and finally a fused high-quality rPPG signal is obtained, such as Figure 2 shown.

[0033] Step 1: Facial video collection and region division

[0034] First, a camera device is used to capture facial video. In order to ensure the accuracy of subsequent signal processing, the present invention uses a face detection algorithm to identify facial areas and extract key facial feature points, such as the forehead, nose tip, cheeks and other parts. These feature points have high stability under facial movement and lighting changes. On this basis, the face is divided into multiple sub-areas based on the difference in rPPG signal intensity in different facial areas. The signal strength of each sub-area is obtained by statistical analysis to maximize the signal-to-noise ratio. The present invention adopts a dynamic priority convex hull method to ensure the robustness of feature point extraction, that is, the feature points are grouped by the convex hull method, and the priority is dynamically adjusted to adapt to facial movement and posture changes. This method effectively reduces the impact of motion artifacts on signal extraction.

[0035] Step 2: Multi-channel signal acquisition and preprocessing

[0036] Use face detection algorithms such as Mediapipe to detect facial key points in each frame of video and identify the geometric contour information of the facial area. Construct the facial convex hull area by extracting the key points of the face. Priority convex hull partitioning method: Use a priority algorithm to determine the priority based on the estimated distance from the area to the camera (the z value returned by Mediapipe). High-priority areas are first counted and marked in the mask. If the subsequently processed area overlaps with the marked area in the mask, the system only extracts the non-overlapping part to ensure that the projection of the three-dimensional facial surface in the two-dimensional image will not be blocked or overlapped due to the ROI extraction order. Then, perform convex hull partitioning on other facial areas from high to low to form multiple levels of ROI areas, such as Figure 3 The corresponding RGB signal is extracted from each ROI area, and the average RGB value of each area is calculated.

[0037]

[0038] Since the rPPG signal is based on the change of the light signal in the video, the present invention interpolates the signal according to the timestamp of the video stream to generate a continuous signal stream. In order to reduce the amplitude difference between signals in different regions and enhance signal consistency, the present invention normalizes the signal of each channel. The normalized multi-channel signal is recorded as H. The time series of the H signal can more accurately reflect the change of the light signal in each sub-region, thereby laying the foundation for the subsequent convolutional neural network training.

[0039] Step 3: Alignment of multi-channel signals with label data

[0040] Before training the multi-channel rPPG signal with a convolutional neural network, it is necessary to ensure that the rPPG signal extracted from the video is synchronized with the contact pulse wave signal (PPG). Due to the difference between the video frame rate and the PPG signal sampling frequency, there may be a time delay between the two. Therefore, the present invention adopts a method based on convolution calculation to estimate and compensate for the time delay.

[0041] The specific steps are as follows:

[0042] 1. Calculate rPPG signal: First, use the existing method to extract the rPPG signal in the video, denoted as S rPPG (t). This signal represents the change in reflected light intensity in the facial area and is processed to generate an rPPG signal that is related to the heart rate.

[0043] 2. Invert the label PPG signal: Invert the label PPG signal S PPG (t) Perform time reversal to obtain the reversed PPG signal

[0044] 3. Convolution operation: Use convolution operation to calculate the rPPG signal S rPPG (t) and the inverted PPG signal Similarity of Δt within a preset time range:

[0045]

[0046] Among them, τ is the time delay and C(τ) is the result of the convolution calculation.

[0047] 4. Time delay estimation: In the convolution result C(τ), find the time point τ corresponding to the maximum value delay ,Right now:

[0048]

[0049] τ delayRepresents the time delay between the rPPG signal and the PPG signal. Through this time delay, the time axis of the rPPG signal can be adjusted and compensated to synchronize it with the PPG signal.

[0050] 5. Signal alignment and compensation: Based on the calculated time delay τ delay , the rPPG signal is time-compensated accordingly to ensure that the two signals are aligned on the same time axis. The aligned signal will be used to train the convolutional neural network to ensure that the input data for network training is consistent in time.

[0051] This method can accurately calculate and compensate for the time delay between the rPPG signal and the contact pulse wave signal, ensuring the timing alignment of the data. This convolution-based time alignment method improves the accuracy of signal synchronization and ensures the reliability of the training and prediction results of the convolutional neural network.

[0052] Step 4: Convolutional Neural Network Model Training

[0053] The neural network training process is as follows Figure 4 As shown, after obtaining the processed multi-channel signal H, the present invention uses a convolutional neural network to extract features and predict weighting coefficients of these signals. The structure of the convolutional neural network includes a channel combination layer, a pooling layer, and a fully connected layer. The function of the channel combination layer is to reduce the dimension of the multi-channel signal through a convolution operation and extract the rPPG signal features of each facial sub-region. The pooling layer reduces redundant information and improves computing efficiency through downsampling operations. The fully connected layer predicts the weighting coefficient of each sub-region based on the correlation between the signals of each sub-region and the contact pulse wave signal. The specific network structure is shown in Figure 5 shown.

[0054] When training a convolutional neural network to predict the weighted coefficients of a multi-channel rPPG signal, the present invention uses a combined loss function, including a negative Pearson coefficient and a frequency domain correlation loss. These two loss functions work together to ensure that the rPPG signal output by the network has a high correlation with the contact pulse wave signal (PPG) in both the time domain and the frequency domain, thereby avoiding overfitting to a certain extent.

[0055] The Pearson correlation coefficient measures the linear correlation between two signals and its value range is [-1,1]. The closer the Pearson coefficient is to 1, the stronger the linear correlation between the signals. In order to maximize the correlation between the rPPG signal and the PPG signal during training, the negative Pearson coefficient is used as the loss function.

[0056]

[0057] in:

[0058] · is the single-channel rPPG signal predicted by the model,

[0059] ·S PPG It is the contact pulse wave signal of the tag.

[0060] ·Cov(S rPPG ,S PPG )express With S PPG The covariance between

[0061] · and They are and S PPG The standard deviation of .

[0062] In order to ensure that the rPPG signal and the PPG signal have consistent spectral characteristics in the frequency domain, the frequency domain correlation loss is used. This loss is implemented by calculating the similarity between the frequency domain representations of the two signals (for example, by fast Fourier transform (FFT) or Chirp Z transform (CZT)). The frequency domain correlation loss function is defined as:

[0063]

[0064] in: N is the total number of frequency sampling points, |·| 2 Represents the Euclidean distance (squared difference). By minimizing the frequency domain correlation loss, it is possible to ensure that the energy distribution of the rPPG signal and the PPG signal in the spectrum is as consistent as possible, thereby improving the frequency accuracy of the physiological signal.

[0065] Combining the negative Pearson coefficient loss in the time domain and the correlation loss in the frequency domain, the final total loss function is:

[0066] L total =αL Pearson +βL Frequency

[0067] Among them, α and β are weight parameters used to balance the impact of the two parts of loss.

[0068] By minimizing the total loss function, the convolutional neural network can optimize the correlation between rPPG signals and PPG signals in both the time domain and the frequency domain, and avoid overfitting to a certain extent, thereby generating high-quality rPPG signals to meet the subsequent physiological signal extraction requirements.

[0069] Step 5: Multi-channel rPPG signal fusion

[0070] The weight coefficients predicted by the trained convolutional neural network are used to perform linear weighted fusion on the rPPG signals of each sub-region. This linear combination method can reduce artifacts and interference caused by facial movement, lighting changes, etc., and improve the signal-to-noise ratio.

[0071] The fused single-channel rPPG signal has stronger robustness, especially in complex scenarios, and can provide stable and high-quality signals. This signal will be used for subsequent calculations of physiological parameters such as breathing and heart rate.

[0072] Step 6: Post-processing and physiological signal extraction

[0073] In order to further improve the quality of the rPPG signal, the present invention performs filtering on the fused single-channel signal to remove low-frequency noise and interference. Specifically, a second-order cascade filter is used to filter the signal within the heart rate range (0.5-4Hz) to retain the effective frequency components in the rPPG signal.

[0074] The filtered signal is traced in the frequency domain through Chirp Z transform (CZT) to calculate the energy distribution of the signal in each frequency band. By accumulating the energy of the frequency domain signal, the heart rate at the current moment is estimated. This process not only improves the accuracy of signal extraction, but also can effectively combat various interferences in complex scenes. The result diagram is shown in the figure below. Figure 6 As shown, the first line is the reference signal, the second line is the method proposed in this paper, and the third line is the traditional method. It can be seen that the method proposed in this paper has a significant improvement in signal-to-noise ratio.

[0075] Through the above steps, the present invention can effectively extract rPPG signals from videos. The method has advantages in signal strength enhancement, motion artifact suppression and real-time performance, and is suitable for physiological state monitoring in a variety of complex scenarios. Compared with traditional methods, the method of the present invention has a certain degree of improvement in signal-to-noise ratio, anti-interference ability and real-time performance.

Claims

1. A linear fusion algorithm for multi-channel remote photoplethysmography (rPPG) signals in complex scenarios, the main features of which include: S1. Count the facial remote pulse wave signals and noise intensity distribution in the face video and the corresponding contact pulse wave signal dataset, and divide the skin into multiple different areas. S2. For each skin area, spatially average each channel to obtain a multi-channel signal as data; use the photoplethysmography signal (PPG signal) as a label, align the data signal with the label signal, preprocess, and divide the data set. S3. By training the convolutional neural network, the weighted coefficient of each channel is predicted, and a single pulse wave signal is obtained through linear combination. S4. Filter the single pulse wave signal, perform frequency domain tracking and other processing to obtain the final pulse wave signal (BVP signal) and heart rate value.

2. According to the multi-channel rPPG signal linear fusion algorithm described in claim 1, the face area is divided according to the rPPG intensity in S1, specifically including collecting complex scene data sets containing faces, counting the rPPG skin signal intensity of each small area of ​​the face of various data sets, and the divided areas include BVP high signal-to-noise ratio areas and low signal-to-noise ratio areas, strong interference signal areas, and the signal-to-noise ratio histogram of each area is obtained by analyzing various public data sets, and then the regional division method "based on prior knowledge" is obtained through statistics.

3. According to the multi-channel rPPG signal linear fusion algorithm of claim 1, preprocessing the acquired data and dividing the data set for S2 specifically comprises: a. Detect the face and the corresponding facial feature points in each frame of video, and divide it into multiple facial sub-regions according to the facial region segmentation method mentioned in S1. Each facial sub-region uses the dynamic priority convex hull method to ensure the robustness of the regional feature points to facial motion. b. Calculate the spatial average pixel value of each sub-region, and interpolate the corresponding signal stream according to the video stream timestamp, and then normalize the signal of each channel to obtain the processed multi-channel signal c. According to the characteristics of the data set, the contact pulse wave signal label of the data set is aligned with the video. The specific feature is to use traditional or machine learning algorithms to extract the rPPG signal of the face in the video, calculate the time difference between the signal and the label signal, and compensate for it. d. Divide the dataset into training set and test set according to the ratio.

4. The multi-channel rPPG signal linear fusion algorithm according to claim 1, for the convolutional neural network training and prediction method in S3, is mainly characterized by: a. Build a convolutional neural network, use channel combination and pooling to achieve dimensionality reduction of multi-channel input signals, obtain the characteristics of all facial region of interest (ROI) channels, fully connect and predict the correlation coefficients of each channel, and obtain the weighting coefficient K of each channel ROI . b. The original multi-channel signal The single-channel rPPG signal is obtained after linear weighting, and the loss between the predicted single-channel rPPG signal and the label signal is calculated using the negative Pearson coefficient and the frequency domain correlation coefficient. The network is back-propagated and iteratively trained.

5. The multi-channel rPPG signal linear fusion algorithm according to claim 1, for the single channel signal post-processing algorithm obtained in S4, is characterized in that The signal is enhanced in the time domain using the window overlap-addition method, and the signal is filtered within the heart rate range of [0.5-4 Hz] using a 3-stage second-order cascade filter. The power spectrum of the frequency range of interest is obtained using the linear frequency modulation Z transform (CZT), and the heart rate value at the current moment is estimated by accumulating the energy of the frequency domain signal in each band.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for linear fusion of remote multi-channel pulse wave signals in complex scenarios as described in any one of claims 1-5 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 5.

Images