Non-contact heart rate variability detection method and system

Abstract

The application provides a non-contact heart rate variability detection method and system, and relates to the technical field of heart rate variability detection. The application constructs a fusion space-time graph of all ROIs based on the obtained ROIs; then inputs the fusion space-time graph into a heart rate variability detection model constructed based on a conditional generative adversarial network and including a double discriminator for noise reduction processing, so as to obtain an iPPG signal after noise reduction; and finally performs non-contact heart rate variability detection based on the iPPG signal after noise reduction. The application can accurately restore the iPPG signal whose noise complexity far exceeds the expected situation, so as to obtain high-precision heart rate variability detection and estimation results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of heart rate variability detection, and particularly relates to a non-contact heart rate variability detection method and system. BACKGROUND

[0002] The non-contact heart rate variability detection technology is mainly based on image photoplethysmography (iPPG) and is performed by analyzing the iPPG signal related indicators obtained in advance to obtain the non-contact heart rate variability detection result.

[0003] The existing heart rate variability detection method (for example, ICA, CHROM, etc.) is based on a signal processing method, and such method is based on a certain hypothesis or model and focuses on a specific scene, and the iPPG signal is denoised for noise characteristics to improve the heart rate variability extraction accuracy. Meanwhile, the existing non-contact heart rate variability detection method mainly considers the fitting of the overall trend of the iPPG signal, and the existing non-contact heart rate variability detection method mainly extracts the time domain indicators such as RR interval mean, SDNN, PNN50, etc.

[0004] However, in the real scene, the complexity of the noise involved in the iPPG signal is far beyond the preset condition, the detection model of the existing non-contact heart rate variability detection method is simple and has poor generalization, and important information is easily ignored, resulting in poor heart rate variability index accuracy. In addition, the wave peak position and interval in the iPPG signal have a key influence on the calculation of the heart rate variability index, and the method of considering the overall trend fitting of the iPPG signal in the prior art will cause the heart rate variability detection result to deviate from the actual value. In addition, compared with the time domain indicators such as RR interval mean, SDNN, PNN50, etc., the frequency domain indicators such as LF, HF, LF / HF, etc. have more important value in a specific scene, and the prior art only considers the time domain indicators of the iPPG signal when extracting the heart rate variability, which will cause the non-contact heart rate variability detection accuracy to be low due to the incomplete indicators. SUMMARY

[0005] (I) Technical problems solved

[0006] In view of the deficiencies of the prior art, the present application provides a non-contact heart rate variability detection method and system, which solves the problem of low detection accuracy of the existing non-contact heart rate variability detection method.

[0007] (II) Technical solutions

[0008] To achieve the above purpose, the present application is realized by the following technical solutions:

[0009] In a first aspect, the present application first provides a non-contact heart rate variability detection method, comprising:

[0010] constructing a fusion spatio-temporal graph of all the regions of interest based on the obtained regions of interest;

[0011] inputting the fusion spatio-temporal graph into a heart rate variability detection model for noise reduction processing to obtain an iPPG signal after noise reduction; the heart rate variability detection model is constructed based on a conditional generative adversarial network, and the heart rate variability detection model further comprises a first discriminator and a second discriminator for respectively discriminating time domain signals and frequency domain signals in the fusion spatio-temporal graph;

[0012] performing time domain analysis, frequency domain analysis and nonlinear analysis on the iPPG signal after noise reduction to realize non-contact heart rate variability detection.

[0013] Preferably, the constructing a fusion spatio-temporal graph of all the regions of interest based on the obtained regions of interest comprises:

[0014] S11, detecting a face from the obtained original face video by using a face feature point positioning and face detection algorithm, and positioning and segmenting a region of interest;

[0015] S12, constructing a spatio-temporal graph based on each of the regions of interest;

[0016] S13, filtering each of the spatio-temporal graphs;

[0017] S14, fusing all the filtered spatio-temporal graphs to obtain a fusion spatio-temporal graph.

[0018] Preferably, the input of the first discriminator is a time domain signal of a gold standard PPG signal and an iPPG signal; the input of the second discriminator is a frequency domain signal corresponding to the gold standard PPG signal and a frequency domain signal corresponding to a time domain signal of the iPPG signal; and the output value of the first discriminator and the second discriminator represents a probability value that the discriminator considers the input to be real data.

[0019] Preferably, the loss function L of the generator in the conditional generative adversarial network is: G

[0020] L G =αL p +βL I

[0021]

[0022]

[0023] wherein, L p represents X g,f ​and X c,f negative Pearson correlation loss between X I peak loss function of the whole iPPG signal; X c denotes the gold standard PPG signal; X g denotes the time domain signal of the iPPG signal; X c,f , X g,f , X c , X g corresponding frequency domain signals; and α and β represent the weight coefficients of L p and L I , respectively; the PCor() function represents the Pearson correlation; L i denotes the position error between the ith peak of the iPPG signal and the corresponding peak in the PPG signal, and x represents the total number of peaks; g i denotes the position of the ith peak; c i denotes the position of the ith peak in the gold standard PPG signal corresponding to g i ; denotes the interval between the ith peak and the previous peak; c i-1 c i denotes the corresponding interval in the gold standard; denotes the interval between the ith peak and the next peak; denotes the corresponding interval in the gold standard.

[0024] Preferably, the method further comprises:

[0025] visualizing the results of the time domain analysis, the results of the frequency domain analysis, and the results of the nonlinear analysis.

[0026] In a second aspect, the present application provides a non-contact heart rate variability detection system, which comprises:

[0027] a fusion spatiotemporal graph acquisition module configured to construct a fusion spatiotemporal graph of all regions of interest based on the acquired regions of interest;

[0028] an iPPG signal denoising module configured to input the fusion spatiotemporal graph into a heart rate variability detection model for denoising to obtain a denoised iPPG signal; the heart rate variability detection model is constructed based on a conditional generative adversarial network, and the heart rate variability detection model further comprises a first discriminator and a second discriminator configured to respectively discriminate time domain signals and frequency domain signals in the fusion spatiotemporal graph;

[0029] a heart rate variability analysis module configured to perform time domain analysis, frequency domain analysis, and nonlinear analysis on the denoised iPPG signal to realize non-contact heart rate variability detection.

[0030] Preferably, the fusion spatio-temporal graph acquisition module comprises:

[0031] S11, using a face feature point positioning and face detection algorithm to detect a face from the acquired original face video, and positioning and segmenting a region of interest;

[0032] S12, constructing a spatio-temporal graph based on each of the regions of interest;

[0033] S13, filtering each of the spatio-temporal graphs;

[0034] S14, fusing all the filtered spatio-temporal graphs to obtain a fusion spatio-temporal graph.

[0035] Preferably, the input of the first discriminator is the time domain signal of the gold standard PPG signal and the iPPG signal; the input of the second discriminator is the frequency domain signal corresponding to the gold standard PPG signal and the frequency domain signal corresponding to the time domain signal of the iPPG signal; the output value of the first discriminator and the second discriminator represents the probability value that the discriminator considers the input to be real data.

[0036] Preferably, the loss function L of the generator in the conditional generative adversarial network is: G

[0037] L G =αL p +βL I

[0038] L p =1-PCor(X c ,X g )+1-PCor(X c,f ,X g,f )

[0039]

[0040]

[0041] wherein, L p represents the negative Pearson correlation loss between X g,f and X c,f ; L I represents the peak loss function of the entire iPPG signal; X c represents the gold standard PPG signal; X g represents the time domain signal of the iPPG signal; X c,f , X g,f respectively represent X c , X g ​corresponding frequency domain signal; and a and b represent weight coefficients of L p and L I i The PCor() function represents the Pearson correlation; L i represents the position of the i-th peak; c i represents the position of the i-th peak in the gold standard PPG signal; g i represents the corresponding peak position in the gold standard; represents the interval between the i-th peak and the previous peak; c i-1 c i represents the corresponding interval in the gold standard; represents the interval between the i-th peak and the next peak; represents the corresponding interval in the gold standard.

[0042] Preferably, the system further comprises:

[0043] an analysis result display module configured to visually display the results of the time domain analysis, the results of the frequency domain analysis, and the results of the nonlinear analysis.

[0044] (III) Advantages

[0045] The present application provides a non-contact heart rate variability detection method and system. Compared with the prior art, the following advantages are achieved:

[0046] 1. The present application constructs a fusion spatio-temporal graph of all ROIs based on the obtained ROIs; then inputs the fusion spatio-temporal graph into a heart rate variability detection model constructed based on a conditional generative adversarial network and including a double discriminator for noise reduction processing to obtain an iPPG signal after noise reduction; and finally performs non-contact heart rate variability detection based on the iPPG signal after noise reduction. The present application can accurately restore iPPG signals with noise complexity far exceeding the preset condition, thereby obtaining high-precision heart rate variability detection and estimation results.

[0047] 2. The heart rate variability detection model DD-cGAN in the present application discriminates the authenticity of the generated time domain signal and frequency domain signal through the design of a double discriminator, so that the generator can better learn the time-frequency features of the gold standard; at the same time, the present application also constructs a loss function that can simultaneously consider the overall trend and local features of the signal, thereby improving the accuracy of the restored (i.e., after noise reduction) iPPG signal and assisting in improving the accuracy of the non-contact heart rate variability detection result.

[0048] ​3、Compared with the existing non-contact heart rate variability detection technology, only the mean RR interval, SDNN, PNN50 and other time domain indicators are extracted, and the LF, HF, LF / HF and other frequency domain indicators are newly added, the extracted heart rate variability indicators are more comprehensive, which can be applied to subsequent cardiovascular disease monitoring, treatment, fatigue identification, cognitive function evaluation and various scenes, and the generalization and robustness of the method are better. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 The flowchart of the non-contact heart rate variability detection method of the present application is shown in the figure.

[0051] Figure 2 The network structure diagram of the generator G in the embodiment of the present application is shown in the figure.

[0052] Figure 3 The network structure diagram of the first and second discriminators in the embodiment of the present application is shown in the figure.

[0053] Figure 4 The flowchart of data processing using the heart rate variability detection model in the embodiment of the present application is shown in the figure.

[0054] Figure 5 The schematic diagram of heart rate variability index visualization display in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application is described clearly and completely. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0056] The embodiments of the present application provide a non-contact heart rate variability detection method and system, which solves the problem of low detection accuracy of the existing non-contact heart rate variability detection method, and realizes the purpose of providing accurate, fast and robust health detection service for users based on accurate heart rate variability estimation results.

[0057] The technical scheme in the embodiments of the present application is to solve the above technical problems, and the general idea is as follows:

[0058] In order to solve the problem that the existing non-contact heart rate variability detection method has poor heart rate variability index precision due to simple model, lack of attention to important information, and incomplete extraction of heart rate variability indicators, and other factors, the technical scheme of the present application proposes a non-contact heart rate variability monitoring method and system, which constructs a new heart rate variability monitoring network containing two discriminators, sets two discriminators to focus on time-frequency signals and frequency domain signals, and sets a loss function that focuses on overall trend and local features at the same time, realizes accurate restoration of iPPG signals, and the obtained heart rate variability estimation result is more accurate. The technical scheme of the present application can be widely applied to epidemic prevention, patient monitoring, stress rescue, spaceflight and navigation, etc. and provides accurate, fast and robust health detection services for users.

[0059] In order to better understand the above technical scheme, the above technical scheme will be described in detail in combination with the drawings of the specification and specific embodiments.

[0060] Heart rate variability refers to the time variation between consecutive heart beat interval sequences. Non-contact heart rate variability detection technology is mainly based on facial imaging photoplethysmography signal (iPPG). When external light irradiates the facial skin tissue, due to the contraction of the human heart, the blood volume in the facial blood vessels increases, causing more light to be absorbed by the blood, and the reflected light intensity decreases, so the pixel value of the ROI in the facial image decreases; when the blood flows back to the heart through the venous network, the blood volume in the facial blood vessels decreases, the light absorption decreases, and the pixel value in the facial image increases. Based on this phenomenon and principle, the change of the pixel value of the ROI in the facial visible light video can be used to record the heart cycle and form the iPPG signal, then the position of each wave peak in the iPPG signal is identified, the time interval between adjacent wave peaks is calculated to obtain the time length of a single heart beat cycle, and the time variation between consecutive heart beat cycles can be obtained. The time variation between consecutive heart beat cycles can be obtained. Then, the time-frequency conversion of the iPPG signal is performed using Fourier transform to calculate the frequency domain indicators of heart rate variability. Finally, by analyzing the time domain indicators, frequency domain indicators and other indicators of the iPPG signal, the non-contact heart rate variability detection result can be obtained.

[0061] Based on the above theory and concept, the following technical scheme is proposed.

[0062] Embodiment 1:

[0063] In the first aspect, the present application first proposes a non-contact heart rate variability detection method, as shown in Figure 1 The method comprises:

[0064] S1, constructing a fusion space-time graph of all regions of interest based on the obtained regions of interest;

[0065] S2, inputting the fusion spatio-temporal graph into a heart rate variability detection model for noise reduction processing to obtain a denoised iPPG signal; the heart rate variability detection model is constructed based on a conditional generative adversarial network, and the heart rate variability detection model further comprises a first discriminator and a second discriminator for respectively discriminating time domain signals and frequency domain signals in the fusion spatio-temporal graph;

[0066] S3, performing time domain analysis, frequency domain analysis and nonlinear analysis on the denoised iPPG signal to realize non-contact heart rate variability detection.

[0067] It can be seen that, in the embodiment, a fusion spatio-temporal graph of all ROIs is constructed based on the obtained ROIs; then the fusion spatio-temporal graph is input into a heart rate variability detection model constructed based on a conditional generative adversarial network and comprising double discriminators for noise reduction processing to obtain a denoised iPPG signal; finally, non-contact heart rate variability detection is performed based on the denoised iPPG signal. The embodiment can accurately restore iPPG signals with noise complexity far exceeding the preset condition, so as to obtain high-precision heart rate variability detection and estimation results.

[0068] The implementation process of one embodiment of the application will be described in detail below in combination with the accompanying drawings and the explanation of the specific steps of S1-S3. Figures 1-5 The specific implementation process of the non-contact heart rate variability detection method proposed in the embodiment comprises:

[0069] S1, constructing a fusion spatio-temporal graph of all ROIs based on the obtained ROIs.

[0070] S11, detecting a face from the obtained original face video by using a face feature point positioning and face detection algorithm, and positioning and segmenting the ROIs.

[0071] A face video of a subject is collected by using a face video collection device. Specifically, the face video collection device comprises a visible light camera, which is used to collect visible light images of the face of the subject and transmit the obtained real-time video frames to a specific position for subsequent processing.

[0072] The background environmental noise and motion noise caused by face rigid motion in the obtained original face video are denoised, and the forehead and nose regions are segmented from the original images as ROIs for extracting image photoplethysmography (iPPG) signals. Specifically:

[0073] First, facial landmark localization and face detection algorithms are used to detect faces from the original facial video sequence and locate the Region of Interest (ROI). Then, the coordinates of 81 facial landmarks are detected using a facial landmark localization model based on ensemble regression trees integrated from the Dlib library. To avoid the influence of non-rigid movements of parts such as the eyes and mouth (blinking and facial expressions) on iPPG signal extraction, and considering that the forehead and nose regions contain more iPPG information, these two regions are selected as the ROIs for iPPG signal extraction in this embodiment.

[0074] S12. Construct a spatiotemporal graph based on each of the regions of interest.

[0075] To address the characteristic that heart rate variability can undergo multiple significant changes over a long period, this embodiment aims to extract more features from each short time interval signal, thereby improving the overall performance of non-contact heart rate variability detection. For longer video sequences, firstly, they are divided into M video segments containing T frames. The value of T can be set based on a combination of camera frame rate and device performance, typically an integer between 128 and 250. Then, the ROIs corresponding to the detected forehead and nose are adjusted to H... ROI1 ×W ROI1 and H ROI2 ×W ROI2 Size, for each frame of the image, the total number of pixels extracted N = H ROI1 ×W ROI1 +H ROI2 ×W ROI2 To fully utilize temporal information, M spatiotemporal sub-blocks are constructed based on the number of segments in each video segment. Specifically, a specific ROI sequence of a T-frame is divided into n sub-regions, and then the pixel values ​​of each sub-region are averaged. For the m-th ROI, P is used... R (m,i,t) represents the average pixel value of the i-th sub-region in the red channel in frame t. Similarly, P is used... G (m,i,t) and P B (m,i,t) represent the average pixel values ​​of the green and blue channels, respectively. Therefore, the continuous signals of the red, green, and blue channels of the i-th sub-region in the RGB color space within the m-th ROI can be determined as follows:

[0076] R mi ={P B (m,i,1),…,P R (m,i,t),…,P B (m,i,T)}

[0077] G mi ={P G (m,i,1),…,PG (m,i,t),…,P G (m,i,T)}

[0078] B mi ={P B (m,i,1),…,P B (m,i,t),…,P B (m,i,T)}

[0079] Where m = 0 or 1, 0 represents the forehead ROI, and 1 corresponds to the nose ROI; i = 0, 1, ..., n.

[0080] S13. Filter each of the spatiotemporal graphs.

[0081] Since the normal human heart rate (HR) is 45-200 bpm, corresponding to a frequency band of 0.75-3.33 Hz, the Fast Fourier Transform (FFT) is first used to transform each time-domain signal R... mi G mi B mi The signal is transformed to the frequency domain, and then a bandpass filter is used to retain data with frequency values ​​between 0.75 and 3.33 Hz, while data values ​​outside this range are zeroed out. This embodiment filters the spatiotemporal maps of each forehead ROI and each nose ROI obtained above in this way, which eliminates noise frequencies and extracts frequency domain information useful for heart rate variability analysis. Next, an inverse Fast Fourier Transform (IFFT) is used to convert the frequency domain signal back to the time domain. To maximize the extraction of the iPPG signal, the R of the ROI is used... mi G mi B mi The information is used to construct a spatiotemporal graph of size n×T×3.

[0082] S14. Fuse all the filtered spatiotemporal graphs to obtain a fused spatiotemporal graph.

[0083] The spatiotemporal maps of the forehead and nose regions are fused. Considering the different scales of the forehead and nose regions, this embodiment uniformly divides the forehead and nose regions into N sub-regions when constructing the spatiotemporal map. Both the forehead and nose regions can obtain spatiotemporal maps of size N×T×3. Then, they are stitched together row by row to obtain a spatiotemporal map of size 2N×T×3. The stitched spatiotemporal map will be used as the input of the HRV (Heart Rate Variability) detection network.

[0084] S2. Input the fused spatiotemporal graph into the heart rate variability detection model for noise reduction processing to obtain the noise-reduced iPPG signal; the heart rate variability detection model is constructed based on a conditional generative adversarial network, and the heart rate variability detection model further includes: a first discriminator and a second discriminator that respectively distinguish between the time domain signal and the frequency domain signal in the fused spatiotemporal graph.

[0085] A heart rate variability (HRV) detection network is constructed based on conditional generative adversarial network (cGAN) to extract iPPG signals. The model consists of a generator G and two discriminators D1 and D2. The two discriminators are used to distinguish the authenticity of the time-domain signal and the frequency-domain signal, respectively, thereby improving the accuracy of the iPPG signal reconstructed by the generator.

[0086] Since the extraction of iPPG signals can essentially be viewed as a generation problem, and iPPG signals possess important properties in both the time and frequency domains, this embodiment constructs a heart rate variability detection model (DD-cGAN) with a dual discriminator (a first and a second discriminator) based on a conditional generative adversarial network (cGAN) to achieve accurate HRV detection using iPPG signals, thereby reducing the noise of the iPPG signals. The dual discriminators are used to distinguish the authenticity of the time-domain and frequency-domain signals, respectively, thus improving the accuracy of the iPPG signals reconstructed by the generator.

[0087] In this embodiment, DD-cGAN includes a generator G and a first discriminator D1 and a second discriminator D2. Specifically, as shown... Figure 2 As shown, the generator G consists of four "ConvBlock" blocks and four "UpBlock" blocks. Each ConvBlock contains two convolutional layers, followed by batch normalization (BN) and Rectified Linear Unit (ReLU) activation in the last layer. Each UpBlock consists of a transposed convolutional layer and two convolutional layers, followed by batch normalization (BN) and ReLU activation in the last layer. The input to the generator G is the fused spatiotemporal map obtained after preprocessing the facial video, and the output of the generator is the reconstructed (extracted) iPPG signal X. g (X g (For timing signals). Simultaneously, to improve generator efficiency, the generator learns the gold standard PPG signal (X). c The time-frequency characteristics of the iPPG signal are also considered. Furthermore, since the iPPG signal possesses important attributes in both the time and frequency domains, in this embodiment, two discriminators are used to distinguish the time-domain signal X. g and X g The corresponding frequency domain signal X g,fDiscrimination is performed. The first discriminator D1 and the second discriminator D2 have the same structure, and the specific structure is as shown in Figure 3 D1 and D2 are both composed of 5 "ConvBlock", global average pooling layer and two fully connected layers. The input of the first discriminator D1 is X c and X g , and the input of the second discriminator D2 is X c corresponding frequency domain signal X c,f and X g,f . The output value of the first discriminator D1 and the second discriminator D2 represents the probability that the discriminator considers the input to be real data.

[0088] In addition, most of the existing heart rate variability detection methods only consider the overall trend fitting effect of the waveform, and rarely pay attention to the local features such as peak position and interval. However, the local feature attributes such as peak position and interval have a key influence on the calculation of heart rate variability indicators, and the existing technology does not pay attention to these, which will cause the heart rate variability detection result to deviate from the actual value. Based on this, the heart rate variability detection model proposed in the embodiment also sets the loss function of the overall trend and local features of the signal. Specifically:

[0089] The reduction of iPPG signal only needs to focus on its periodicity, so the negative Pearson correlation loss between X g and the gold standard X c is calculated, so that the generator learns the time-frequency features of X c , and the negative Pearson correlation loss between X g,f and X c,f is also calculated:

[0090] L p =1-PCor(X x ,X g )+1-PCor(X c,f ,X g,f )

[0091] Where Pcor() represents Pearson correlation.

[0092] Considering that the peak position and interval have a very important role in HRV detection, the embodiment designs a loss function that focuses on the local features of the iPPG signal. Specifically, define each peak to consist of three attributes: the position of the i-th peak g i ; and the corresponding peak position in the gold standard PPG signal is c i ; the interval between the previous peak, that is, the interval between the i-1th peak and the i-th peak The corresponding interval in the gold standard is the interval between the latter peak and the former peak, i.e. the interval between the ith peak and the (i+1)th peak The interval corresponding to the gold standard is Based on this, the peak loss function of the entire iPPG signal can be represented by the formula:

[0093]

[0094]

[0095] wherein L I represents the peak loss function of the entire iPPG signal; L i represents the position error between the ith peak of the iPPG and the peak in the corresponding PPG signal; x represents the total number of peaks.

[0096] Therefore, the loss function of the generator can be represented by the formula:

[0097] L G = aL p + bL I

[0098] The loss function of the double discriminator is designed with reference to SEGAN, and is specifically as follows:

[0099]

[0100]

[0101] Based on the above-constructed heart rate variability detection model, the fusion spatio-temporal graph in S1 is input into the heart rate variability detection model for noise reduction processing to obtain the denoised iPPG signal. For details, please refer to Figure 4 After denoising, a high-precision iPPG signal can be obtained for subsequent analysis.

[0102] S3, performing time domain analysis, frequency domain analysis and nonlinear analysis on the denoised iPPG signal to realize non-contact heart rate variability detection.

[0103] After denoising and restoring the high-precision iPPG signal, time domain analysis, frequency domain analysis and nonlinear analysis are performed on the iPPG signal to obtain time domain, frequency domain and nonlinear indicators of heart rate variability, realizing multi-parameter detection.

[0104] 1) HRV time domain index calculation in the denoised iPPG signal. After obtaining the high-precision iPPG time domain signal using DD-cGAN, the following calculation formula is used to obtain the mean (MEAN), overall standard deviation (SDNN), difference mean square standard deviation (RMSSD) and percentage of adjacent interval difference > 50ms (PNN50). 50). At the same time, the triangular index and TINN are obtained. Specifically,

[0105]

[0106]

[0107]

[0108]

[0109] wherein P represents the position of the wave peak in the time domain signal; PP i represents the interval length between two adjacent wave peaks; N represents the number of all wave peaks; NN 50 represents the number of wave peak intervals whose time difference is greater than 50 ms; TotalNN represents the number of all wave peak intervals.

[0110] Triangular index: the total number of RR intervals divided by the height of the RR interval histogram.

[0111] TINN: the width of the approximate triangular base of the histogram of all RR intervals is obtained by using the least square method.

[0112] 2) Frequency domain analysis of the denoised iPPG signal. The high-precision iPPG time domain signal obtained is converted into a frequency domain signal using Fourier transform. From the frequency domain signal, the low-frequency index LF in the range of 0.04-0.15 Hz is selected, the high-frequency index HF in the range of 0.15-0.4 Hz is selected, and finally the ratio LF / HF between the low-frequency index and the high-frequency index is obtained.

[0113] LF = FFT(X g ) {0.04 < FFT(X g ) < 0.15}

[0114] HF = FFT(X g ) {0.15 < FFT(X g ) < 0.4}

[0115] 3) Obtain the nonlinear index in the denoised iPPG signal. For a continuous wave peak interval, the ith heartbeat interval is taken as the abscissa, and the (i+1)th heartbeat interval is taken as the ordinate. An ellipse-like graph can be drawn on the two-dimensional plane, with the center of the ellipse at the coordinate point determined by the average value of the x-axis heartbeat interval and the average value of the y-axis heartbeat interval. The semi-major axis SD1 and the semi-minor axis SD2 of the ellipse are the nonlinear index of heart rate variability.

[0116] Based on the analysis of the time domain index, the frequency domain index, and the nonlinear index of the high-precision iPPG signal after noise reduction, multi-parameter detection is realized, and the result of high-precision non-contact heart rate variability detection is obtained.

[0117] In addition, in order to more conveniently and intuitively display the result of non-contact heart rate variability detection in actual application, the non-contact heart rate variability detection method of the embodiment further comprises:

[0118] S4, visualizing and displaying the results of the time domain analysis, the frequency domain analysis, and the nonlinear analysis.

[0119] The heart rate variability indexes SDNN, TINN, PNN 50 , RMSSD, LF, HF, etc. acquired in the S3 step are transmitted to a display interface in real time for real-time display of data, as shown in detail in Figure 5 , so that users or relevant medical staff can conveniently and intuitively obtain relevant information.

[0120] At this point, the whole process of the non-contact heart rate variability detection method of the embodiment is completed.

[0121] Embodiment 2

[0122] In a second aspect, the present application further provides a non-contact heart rate variability detection system, which comprises:

[0123] The fusion spatiotemporal graph acquisition module is configured to construct a fusion spatiotemporal graph of all the regions of interest based on the acquired regions of interest;

[0124] The iPPG signal noise reduction module is configured to input the fusion spatiotemporal graph into a heart rate variability detection model for noise reduction processing to obtain an iPPG signal after noise reduction; the heart rate variability detection model is constructed based on a conditional generative adversarial network, and the heart rate variability detection model further comprises a first discriminator and a second discriminator for respectively discriminating time domain signals and frequency domain signals in the fusion spatiotemporal graph;

[0125] The heart rate variability analysis module is configured to perform time domain analysis, frequency domain analysis, and nonlinear analysis on the iPPG signal after noise reduction to realize non-contact heart rate variability detection.

[0126] Optionally, the fusion spatiotemporal graph acquisition module constructing a fusion spatiotemporal graph of all the regions of interest based on the acquired regions of interest comprises:

[0127] S11, using a facial feature point positioning and face detection algorithm to detect a face from the acquired original face video, and positioning and segmenting a region of interest;

[0128] S12, constructing a spatio-temporal graph based on each of the regions of interest;

[0129] S13, filtering each of the spatio-temporal graphs;

[0130] S14, fusing all the filtered spatio-temporal graphs to obtain a fused spatio-temporal graph.

[0131] Optionally, an input of the first discriminator is a time domain signal of a gold standard PPG signal and an iPPG signal; an input of the second discriminator is a frequency domain signal corresponding to the gold standard PPG signal and a frequency domain signal corresponding to a time domain signal of the iPPG signal; and an output value of the first discriminator and the second discriminator represents a probability value of the discriminator considering that the input is real data.

[0132] Optionally, a loss function L of the generator in the conditional generative adversarial network is: G

[0133] L G = αL p + βL I

[0134] L p = 1-PCor(X x ,X g ) + 1-PCor(X c,f ,X g,f )

[0135]

[0136]

[0137] wherein, L p represents a negative Pearson correlation loss between X g,f and X c,f ; L I represents a peak loss function of the entire iPPG signal; X c represents the gold standard PPG signal; X g represents a time domain signal of the iPPG signal; X c,f and X g,f respectively represent corresponding frequency domain signals of X c and X g ; α and β respectively represent weight coefficients of L p and L I ; PCor() function represents Pearson correlation; L i represents a position error between an iPPG ith peak and a peak in the corresponding PPG signal, x represents a total number of peaks; g i represents a position of the ith peak; c i ​denotes the interval between the i-th peak and the previous peak; c i the corresponding peak position in the gold standard PPG signal; denotes the interval between the i-th peak and the previous peak; c i-1 c i denotes the interval between the i-th peak and the previous peak; c the corresponding interval in the gold standard; denotes the interval between the i-th peak and the previous peak; c denotes the interval between the i-th peak and the previous peak; c the corresponding interval in the gold standard.

[0138] Optionally, the system further comprises:

[0139] an analysis result display module configured to visually display the results of the time domain analysis, the results of the frequency domain analysis and the results of the nonlinear analysis.

[0140] It can be understood that the non-contact heart rate variability detection system provided by the embodiments of the present application corresponds to the non-contact heart rate variability detection method described above, and the explanation, examples, beneficial effects and the like of related contents can refer to the corresponding contents in the non-contact heart rate variability detection method, which will not be described here.

[0141] In summary, compared with the prior art, the present application has the following beneficial effects:

[0142] 1. The present application constructs a fusion spatio-temporal graph of all ROIs based on the obtained ROIs, then inputs the fusion spatio-temporal graph into a heart rate variability detection model constructed based on a conditional generative adversarial network and including a double discriminator for denoising processing to obtain a denoised iPPG signal, and finally performs non-contact heart rate variability detection based on the denoised iPPG signal. The present application can accurately restore the iPPG signal with a noise complexity far exceeding the preset condition, thereby obtaining high-precision heart rate variability detection and estimation results.

[0143] 2. The heart rate variability detection model DD-cGAN in the present application discriminates the authenticity of the generated time domain signal and frequency domain signal through the design of a double discriminator, so that the generator can better learn the time-frequency features of the gold standard; meanwhile, the present application also constructs a loss function that can simultaneously consider the overall trend and local features of the signal, thereby improving the accuracy of the restored (i.e., denoised) iPPG signal and assisting in improving the accuracy of the non-contact heart rate variability detection result.

[0144] 3、Compared with the prior art non-contact heart rate variability detection technology, only the mean RR interval, SDNN, PNN50 and other time domain indexes are extracted, and the LF, HF, LF / HF and other frequency domain indexes are newly added, the extracted heart rate variability indexes are more comprehensive, and the method can be applied to subsequent cardiovascular disease monitoring, treatment, fatigue identification, cognitive function evaluation and various scenes, and the generalization and robustness of the method are better.

[0145] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual such relationship or order between these entities or operations. Moreover, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0146] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A non-contact method for detecting heart rate variability, characterized in that, The method includes: Based on the acquired regions of interest, a fused spatiotemporal graph of all regions of interest is constructed, and the fused spatiotemporal graph includes multiple spatiotemporal sub-blocks; The fused spatiotemporal graph is input into the heart rate variability detection model for noise reduction to obtain the noise-reduced iPPG signal. The heart rate variability detection model is constructed based on a conditional generative adversarial network and further includes a first discriminator and a second discriminator that distinguish between the time-domain signal and the frequency-domain signal in the fused spatiotemporal graph, respectively, and a generator that sets loss functions for the overall trend and local features of the signal. The noise-reduced iPPG signal is subjected to time-domain analysis, frequency-domain analysis, and nonlinear analysis to achieve non-contact heart rate variability detection; Wherein, the loss function of the generator for: in, express and Negative Pearson correlation loss between them; This represents the peak loss function of the entire iPPG signal; This indicates the gold standard PPG signal; The time-domain signal representing the iPPG signal; They represent , The corresponding frequency domain signal; and They represent and The weighting coefficients; the PCor() function represents the Pearson correlation; denoted by , where represents the positional error between the i-th peak of the iPPG and the corresponding peak in the PPG signal, and x represents the total number of peaks; Indicates the position of the i-th peak; This indicates that the i-th peak in the gold standard PPG signal is related to... Corresponding peak position; This represents the interval between the i-th wave crest and the previous wave crest. express The corresponding interval in the gold standard is; This represents the interval between the i-th peak and the next peak. express The corresponding interval in the gold standard.

2. The method as described in claim 1, characterized in that, The construction of a fused spatiotemporal graph of all regions of interest based on the acquired regions of interest includes: S11. Detect faces from the acquired raw facial video using facial feature point localization and face detection algorithms, and locate and segment the region of interest. S12. Construct a spatiotemporal graph based on each region of interest; S13. Filter each of the spatiotemporal graphs; S14. Fuse all the filtered spatiotemporal graphs to obtain a fused spatiotemporal graph.

3. The method as described in claim 1, characterized in that, The inputs of the first discriminator are the time-domain signals of the gold standard PPG signal and the iPPG signal; the inputs of the second discriminator are the frequency-domain signals corresponding to the gold standard PPG signal and the frequency-domain signals corresponding to the time-domain signals of the iPPG signal; the output values ​​of the first discriminator and the second discriminator represent the probability values ​​that the discriminator considers the input to be real data.

4. The method as described in claim 1, characterized in that, The method further includes: The results of the time-domain analysis, frequency-domain analysis, and nonlinear analysis are visualized.

5. A non-contact heart rate variability detection system, characterized in that, The system includes: The fusion spatiotemporal graph acquisition module is used to construct a fusion spatiotemporal graph of all regions of interest based on the acquired regions of interest, wherein the fusion spatiotemporal graph includes multiple spatiotemporal sub-blocks; The iPPG signal denoising module is used to input the fused spatiotemporal graph into the heart rate variability detection model for denoising processing to obtain the denoised iPPG signal. The heart rate variability detection model is constructed based on a conditional generative adversarial network, and the heart rate variability detection model further includes: a first discriminator and a second discriminator that distinguish between the time domain signal and the frequency domain signal in the fused spatiotemporal graph, respectively, and a generator that sets the loss function of the overall trend and local features of the signal. The heart rate variability analysis module is used to perform time-domain analysis, frequency-domain analysis, and nonlinear analysis on the noise-reduced iPPG signal to achieve non-contact heart rate variability detection. Wherein, the loss function of the generator for: in, express and Negative Pearson correlation loss between them; This represents the peak loss function of the entire iPPG signal; This indicates the gold standard PPG signal; The time-domain signal representing the iPPG signal; They represent , The corresponding frequency domain signal; and They represent and The weighting coefficients; the PCor() function represents the Pearson correlation; denoted by , where represents the positional error between the i-th peak of the iPPG and the corresponding peak in the PPG signal, and x represents the total number of peaks; Indicates the position of the i-th peak; This indicates that the i-th peak in the gold standard PPG signal is related to... Corresponding peak position; This represents the interval between the i-th wave crest and the previous wave crest. express The corresponding interval in the gold standard is; This represents the interval between the i-th peak and the next peak. express The corresponding interval in the gold standard.

6. The system as described in claim 5, characterized in that, The fusion spatiotemporal graph acquisition module constructs a fusion spatiotemporal graph of all regions of interest based on the acquired regions of interest, including: S11. Detect faces from the acquired raw facial video using facial feature point localization and face detection algorithms, and locate and segment the region of interest. S12. Construct a spatiotemporal graph based on each region of interest; S13. Filter each of the spatiotemporal graphs; S14. Fuse all the filtered spatiotemporal graphs to obtain a fused spatiotemporal graph.

7. The system as described in claim 5, characterized in that, The inputs of the first discriminator are the time-domain signals of the gold standard PPG signal and the iPPG signal; the inputs of the second discriminator are the frequency-domain signals corresponding to the gold standard PPG signal and the frequency-domain signals corresponding to the time-domain signals of the iPPG signal; the output values ​​of the first discriminator and the second discriminator represent the probability values ​​that the discriminator considers the input to be real data.

8. The system as described in claim 5, characterized in that, The system also includes: The analysis results display module is used to visualize the results of the time domain analysis, frequency domain analysis, and nonlinear analysis.

Images