Video-based far-end blood pressure measurement method

Abstract

The invention discloses a video-based far-end blood pressure measurement method, which comprises the following steps of: performing face detection on a video frame image containing a human face, extracting a face image sequence in a time dimension, and obtaining a face overall signal and a face roi signal set in the time dimension on the basis; eliminating the noise of the overall face signal and the face roi signal set by adopting but not limited to an empirical mode decomposition mode; based on the filtered overall face signal and the face roi signal set, respectively calculating spectrum features corresponding to the signals, and calculating a physiological signal distribution area; calculating TDLCC features of the signals, and learning and identifying the features by adopting a residual convolutional network model to realize rapid measurement of blood pressure. The detection method has high robustness, facial physiological signal information can be fully utilized, and the method is suitable for complex application scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of signal processing, and in particular to a remote blood pressure measurement method based on video. Background Art

[0002] Human heart rate and facial blood volume distribution are key indicators for measuring human physiological conditions. Currently, the most common way to measure human blood pressure is to use airbag devices, which have the disadvantages of large size, needing to be close to the human body, and the inflation and deflation measurement method may cause discomfort.

[0003] At present, remote detection of human physiological indicators through cameras has become a hot topic in current research. For example: the application document with publication number CN112315437A provides a system for predicting blood pressure based on video data, including: a first device and a cloud server; wherein the first device includes a first main control module, a first camera, a lighting module, a display screen and a first communication module; the cloud server includes a second main control module, a legality verification module, a parameter verification module, a data preprocessing module, an artificial intelligence blood pressure prediction module and a second communication module. Using the system for predicting blood pressure based on video data provided by an embodiment of the present invention, blood pressure can be detected and continuously monitored in real time without wearing any specific acquisition device. The application document with publication number CN106236049A describes a blood pressure measurement method based on video images, which realizes continuous measurement of blood pressure by collecting video images of human faces, and then performing spatial decomposition, time domain filtering and real-time processing on the facial video images. Specifically, the method includes: real-time tracking of the target area to be measured to determine the ROI area; performing RGB three-channel separation on each frame of the image sequence of the ROI area, and summing and averaging the obtained image to obtain a time series waveform within a period of time; removing the trend item from the above time series waveform and performing normalization processing to obtain a time series signal; using the empirical mode decomposition method to perform denoising processing on the normalized signal to obtain a three-channel time domain signal with a higher signal-to-noise ratio; finally, obtaining the peaks and troughs of the time domain signal, establishing a relationship between the time domain signal and the blood pressure, and thus obtaining the blood pressure value.

[0004] Due to the complexity of the external environment, it is easy to interfere with the remote measurement method. In order to eliminate the above interference, empirical mode decomposition, bandpass filtering and other methods are used to remove noise. At the same time, the distribution area of ​​facial physiological signals is calculated based on the signal spectrum characteristics to achieve further optimization of the signal. On this basis, the TDLCC features are calculated and combined with the deep learning model to achieve accurate measurement of human blood pressure data. Summary of the invention

[0005] In order to improve the efficiency of human blood pressure detection, the present invention proposes a new method for measuring human blood pressure by using human facial physiological signals. The method is based on the face video captured by the camera, analyzes and processes the facial data in the video, and realizes human blood pressure measurement through signal processing, feature calculation and model prediction.

[0006] A video-based remote blood pressure measurement method comprises the following steps:

[0007] (1) Detect the facial area in the video frame image, extract the facial image sequence and facial key position points in the time dimension; extract the facial overall signal and facial ROI signal set based on the facial image sequence, and pre-process the signal;

[0008] (2) Calculate the signal spectrum features based on the preprocessed facial overall signal and facial ROI signal set;

[0009] (3) calculating the facial physiological signal distribution area according to the signal spectrum characteristics;

[0010] (4) According to the distribution area of ​​facial physiological signals, the signal TDLCC features are calculated, and the residual network model is used to realize human blood pressure measurement.

[0011] In the present invention, firstly, the facial overall signal and the facial ROI signal set are extracted based on the face image sequence and preprocessed; secondly, based on the preprocessed physiological signal, the signal spectrum feature method is used to accurately calculate the distribution area of ​​the physiological signal on the face; finally, the signal TDLCC feature with good feature expression ability is calculated, and the signal TDLCC feature is learned and recognized by a robust deep learning model to achieve rapid measurement of human blood pressure.

[0012] Preferably, in step (1), a MTCNN model is used to extract facial images from the video, and facial images are extracted from each frame of the video to obtain a facial image sequence.

[0013] Furthermore, extracting the overall facial signal according to the average pixel intensity of the facial images in the facial image sequence;

[0014] The face image is divided into ROI sub-blocks of size m×n to obtain a ROI sub-block image sequence in the time dimension; the set composed of all ROI sub-block image sequences is the facial ROI signal set.

[0015] The signal preprocessing is to eliminate high and low frequency noise signals outside the range, and the preprocessing method includes an empirical mode decomposition method.

[0016] Preferably, in the step (2), the short-time Fourier spectrum and the Wigner-Williams spectrum of the facial overall signal and the facial ROI signal are calculated;

[0017] stft_face_sig=STFT(face_sig_refine)

[0018] wv_face_sig=WV(face_sig_refine)

[0019] stft_roi_sig=STFT(roi_sig_refine)

[0020] wv_roi_sig=WV(roi_sig_refine)

[0021] Among them: STFT() is the stft spectrum calculation function, WV() is the wv spectrum calculation function, stft_face_sig and wv_face_sig are the stft spectrum and wv spectrum corresponding to the overall facial signal respectively, stft_roi_sig and wv_roi_sig are the stft spectrum and wv spectrum corresponding to the facial roi signal respectively.

[0022] Preferably, the physiological signal distribution area is calculated according to the stft spectrum and wv spectrum:

[0023] dis_stft=Corr(stft_face_sig,stft_roi_sig)

[0024] dis_wv=Corr(wv_face_sig,wv_roi_sig)

[0025] Wherein: Corr() is the correlation calculation function, dis_stft is the physiological signal distribution area calculated based on the stft spectrum, and dis_wv is the physiological signal distribution area calculated based on the wv spectrum.

[0026] Calculate physiological signal distribution: dis_phy = w stft *dis_stft+w wv *dis_wv, where: dis_phy is the distribution area of ​​physiological signals, w stft 、w wv are weight values ​​respectively.

[0027] Preferably, in the step (4), the local physiological signal is calculated and denoised according to the distribution of the ROI sub-blocks containing the physiological signal;

[0028]

[0029] k_set={k0,k1,...,k i ,...,k n}

[0030] sig_local_set={sig_local0,sig_local1,...,sig_local i ,…,sig_local n}

[0031] sig_local_refine i =EMD(sig_local i )

[0032] Where: sig_local i is the i-th local physiological signal, k i is the number of roi set elements corresponding to the i-th local physiological signal, k_set is a set containing the number of roi set elements corresponding to all local physiological signals, and sig_local_set is a set containing physiological signals.

[0033] Furthermore, the TDLCC feature matrix is ​​calculated:

[0034] R ij (t w ,τ)=Corr(sig_local_refine i (t w ),sig_local_refine j (t w ,τ))

[0035] t w =[t c -w*w d / 2:t c +w*w d / 2]

[0036] Where: R ij is the TDLCC feature matrix corresponding to the i-th and j-th local physiological signals, Corr() is the cross-correlation function, t is the time variable, w is the sliding window scale variable, τ is the offset variable, t w is the signal range covered by the sliding window, w d is the minimum sliding window width, t c is the spatial position corresponding to the center of the sliding window.

[0037] According to the TDLCC features, a residual convolutional network model deep learning model is used to achieve blood pressure prediction.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1) Realize human blood pressure measurement based on remote acquisition equipment. A new method for remotely measuring human blood pressure is proposed. First, the facial overall signal and facial ROI signal set are extracted based on the face image sequence and preprocessed; secondly, based on the preprocessed physiological signal, the signal spectrum feature method is used to accurately calculate the distribution area of ​​the physiological signal on the face; finally, the signal TDLCC feature with good feature expression ability is calculated, and the signal TDLCC feature is learned and recognized by a robust deep learning model to achieve rapid measurement of human blood pressure.

[0040] 2) Calculate the time-dependent local cross-correlation (TDLCC) feature. A physiological signal feature extraction method based on time-dependent features is proposed. The morphological features of the physiological signal in the time dimension are analyzed by correlation calculation to maximize the expansion of signal features.

[0041] 3) Human blood pressure measurement based on deep learning methods. A human blood pressure measurement method based on deep learning technology and facial video data is proposed. The deep learning model based on the residual network structure is used to learn and identify the TDLCC features of human physiological signals to achieve rapid measurement of human blood pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a characteristic diagram of signal TDLCC. DETAILED DESCRIPTION

[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0044] A video-based remote blood pressure measurement method, such as Figure 1 As shown, the following steps are included:

[0045] (1) Face image extraction: The MTCNN model is used to extract face images from videos.

[0046] face i =MTCNN(frame i ) (1)

[0047] face_seq={face1,face2,...,face i ,...,face T} (2)

[0048] In Formula 1, framei represents the i-th frame image of the video, and facei represents the face image extracted from the i-th frame image of the video. face_seq is the face image sequence, as shown in Formula 2, where T is the length of the video frame sequence.

[0049] Extract the facial overall signal and facial ROI (region of interest, ROI) signal set. The facial overall signal calculation is shown in Formula 3, where: face_sig is the compressed signal, PCompress() is the function for calculating the average pixel intensity of the face image in the face image sequence, and face_seq is the face image sequence.

[0050] face_sig=PCompress(face_seq) (3)

[0051] The calculation of the facial ROI signal set is shown in formulas 4 and 5. The face image is divided into ROI sub-blocks of size m×n to obtain a ROI sub-block image sequence in the time dimension, as shown in formula 4.

[0052] face_roi_seq={face_roi1,face_roi2,...,face_roi i ,...,face_roi T} (4)

[0053] In the formula, face_roii represents the i-th roi sub-block image sequence, and face_roi_seq is the set of all roi sub-block image sequences, that is, the face roi signal set. Based on this, the sub-block image sequence shown in formula 4 is compressed, as shown in formula 5, where: face_roi_sig is the face roi signal set, and the set element calculation is shown in formula 6.

[0054] face_roi_sig={face_roi_sig1,...,face_roi_sig i ,...,face_roi_sig n×n}(5)

[0055] face_roi_sig i =PCompress(face_roi i ) (6)

[0056] (2) Signal preprocessing: The method of empirical mode decomposition (EMD) is used but not limited to eliminate high and low frequency noise signals outside the specified range, as shown in formulas 7 and 8.

[0057] face_sig_refine=EMD(face_sig) (7)

[0058] roi_sig_refine=EMD(face_roi_sig) (8)

[0059] (3) Calculate signal spectrum features: Calculate the short-time Fourier transform spectrum (STFT spectrum) and the Wigner-Ville spectrum (WV spectrum) of the overall facial signal and the facial ROI signal. As shown in formulas 9 to 12, STFT() is the STFT spectrum calculation function, WV() is the WV spectrum calculation function, stft_face_sig and wv_face_sig are the STFT spectrum and WV spectrum corresponding to the overall facial signal, and stft_roi_sig and wv_roi_sig are the STFT spectrum and WV spectrum corresponding to the facial ROI signal.

[0060] stft_face_sig=STFT(face_sig_refine) (9)

[0061] wv_face_sig=WV(face_sig_refine) (10)

[0062] stft_roi_sig=STFT(roi_sig_refine) (11)

[0063] wv_roi_sig=WV(roi_sig_refine) (12)

[0064] (4) Calculate physiological signal distribution: Based on the stft spectrum and wv spectrum obtained in step 3, calculate the physiological signal distribution area. As shown in formulas 13 and 14, Corr() is the correlation calculation function, dis_stft is the physiological signal distribution area calculated based on the stft spectrum, and dis_wv is the physiological signal distribution area calculated based on the wv spectrum.

[0065] dis_stft=Corr(stft_face_sig,stft_roi_sig) (13)

[0066] dis_wv=Corr(wv_face_sig,wv_roi_sig) (14)

[0067] Based on this, the physiological signal distribution is calculated as shown in Formula 15, where: dis_phy is the physiological signal distribution area, wstft and wwv are weight values ​​respectively.

[0068] dis_phy=w stft *dis_stft+w wv *dis_wv (15)

[0069] (5) Calculate the time dependency local cross correlation (TDLCC) feature: This feature is based on the signal correlation principle and realizes the characteristic description of the blood flow state.

[0070] (5.1) Based on step 4, the distribution of ROI sub-blocks containing physiological signals is calculated, and local physiological signals are calculated based on this, as shown in formulas 16 to 18, where: sig_locali is the i-th local physiological signal, ki is the number of ROI set elements corresponding to the i-th local physiological signal, k_set is a set containing the number of ROI set elements corresponding to all local physiological signals, and sig_local_set is a set containing physiological signals.

[0071]

[0072] k_set={k0,k1,...,k i ,...,k n} (17)

[0073] sig_local_set={sig_local0,sig_local1,...,sig_local i ,...,sig_local n}(18)

[0074] (5.2) Local physiological signal denoising processing, as shown in formula 19, where sig_local_refinei is the signal after denoising processing.

[0075] sig_local_refine i =EMD(sig_local i ) (19)

[0076] (5.3) Calculate the TDLCC feature matrix, as shown in Formula 20, where: Rij is the TDLCC feature matrix corresponding to the i-th and j-th local physiological signals, Corr() is the cross-correlation function, t is the time variable, w is the sliding window scale variable, τ is the offset variable, tw is the signal range covered by the sliding window, wd is the minimum sliding window width, and tc is the spatial position corresponding to the sliding window center.

[0077] R ij (tw ,τ)=Corr(sig_local_refine i (t w ),sig_local_refine j (t w ,τ)) (20)

[0078] t w =[t c -w*w d / 2:t c +w*w d / 2] (21)

[0079] (6) Blood pressure prediction based on deep learning model: Based on the TDLCC features calculated in step 5, the residual convolutional network model (RCNN) deep learning model is used to achieve blood pressure prediction.

[0080] (6.1) Data standardization processing, as shown in Formulas 22 to 23, where mean() is the mean function, var() is the variance function, and R_set is the TDLCC feature set.

[0081]

[0082] R_set={R 00 ,R 01 ,...,R ij ,...,R mm} (twenty three)

[0083] (6.2) Construct the sample as shown in Formula 24, where concat() is the connection function that merges the feature matrix elements in the set.

[0084] fearture=concat(R_set) (24)

[0085] (6.3) Blood pressure prediction, as shown in Formula 25, where RCNN() is a deep learning model and bloodpressure is the blood pressure prediction result.

[0086] bloodpressure=RCNN(feature) (25)

[0087] The above description is only an example of a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video-based remote blood pressure measurement method, characterized in that: The following steps are involved: (1) Detect the facial area in the video frame image, extract the facial image sequence and facial key position points in the time dimension; extract the facial overall signal and facial ROI signal set based on the facial image sequence, and pre-process the signal; (2) Calculate the signal spectrum features based on the preprocessed facial overall signal and facial ROI signal set; (3) calculating the facial physiological signal distribution area according to the signal spectrum characteristics; (4) According to the distribution area of ​​facial physiological signals, the signal TDLCC features are calculated, and the residual network model is used to realize human blood pressure measurement.

2. The video-based remote blood pressure measurement method according to claim 1, characterized in that: In the step (1), the MTCNN model is used to extract face images from the video, and the face images are extracted from each frame of the video to obtain a face image sequence.

3. The video-based remote blood pressure measurement method according to claim 2, characterized in that: Extracting the overall facial signal according to the average pixel intensity of the facial images in the facial image sequence; The face image is divided into ROI sub-blocks of size m×n to obtain a ROI sub-block image sequence in the time dimension; the set composed of all ROI sub-block image sequences is the facial ROI signal set.

4. The video-based remote blood pressure measurement method according to claim 3, characterized in that: The signal preprocessing is to eliminate high and low frequency noise signals outside the range, and the preprocessing method includes an empirical mode decomposition method.

5. The video-based remote blood pressure measurement method according to claim 1, characterized in that: In the step (2), the short-time Fourier spectrum and the Wigner-Ville spectrum of the facial overall signal and the facial ROI signal are calculated; stft_face_sig=STFT(face_sig_refine) wv_face_sig=WV(face_sig_refine) stft_roi_sig=STFT(roi_sig_refine) wv_roi_sig=WV(roi_sig_refine) Among them: STFT() is the stft spectrum calculation function, WV() is the wv spectrum calculation function, stft_face_sig and wv_face_sig are the stft spectrum and wv spectrum corresponding to the overall facial signal respectively, stft_roi_sig and wv_roi_sig are the stft spectrum and wv spectrum corresponding to the facial roi signal respectively.

6. The video-based remote blood pressure measurement method according to claim 5, characterized in that: Calculate the physiological signal distribution area according to the stft spectrum and wv spectrum: dis_stft=Corr(stft_face_sig,stft_roi_sig) dis_wv=Corr(wv_face_sig,wv_roi_sig) Wherein: Corr() is the correlation calculation function, dis_stft is the physiological signal distribution area calculated based on the stft spectrum, and dis_wv is the physiological signal distribution area calculated based on the wv spectrum.

7. The video-based remote blood pressure measurement method according to claim 6, characterized in that: Calculate physiological signal distribution: dis_phy = w stft *dis_stft+w wv *dis_wv, where: dis_phy is the distribution area of ​​physiological signals, w stft 、w wv are weight values ​​respectively.

8. The video-based remote blood pressure measurement method according to claim 1, characterized in that: In the step (4), the local physiological signal is calculated and denoised according to the distribution of the ROI sub-blocks containing the physiological signal; k_set={k0,k1,…,k i ,...,k n } sig_local_set={sig_local0,sig_local1,...,sig_local i ,...,sig_local n } sig_local_refine i =EMD(sig_local i ) Where: sig_local i is the i-th local physiological signal, k i is the number of roi set elements corresponding to the i-th local physiological signal, k_set is a set containing the number of roi set elements corresponding to all local physiological signals, and sig_local_set is a set containing physiological signals.

9. The video-based remote blood pressure measurement method according to claim 8, characterized in that: Calculate the TDLCC feature matrix: R ij (t w ,τ)=Corr(sig_local_refine i (t w ),sig_local_refine j (t w ,t)) t w =[t c -w*w d / 2:t c +w*w d / 2] Where: R ij is the TDLCC feature matrix corresponding to the i-th and j-th local physiological signals, Corr() is the cross-correlation function, t is the time variable, w is the sliding window scale variable, τ is the offset variable, t w is the signal range covered by the sliding window, w d is the minimum sliding window width, t c is the spatial position corresponding to the center of the sliding window.

10. The video-based remote blood pressure measurement method according to claim 9, characterized in that: According to the TDLCC features, a residual convolutional network model deep learning model is used to achieve blood pressure prediction.

Images