Non-contact pulse rate estimation device, non-contact pulse rate estimation method, and program

Abstract

To provide a non-contact pulse estimation device, etc. by a thermal camera that is more accurate than a conventional one.SOLUTION: A non-contact pulse estimation device for estimating a pulse from video data acquired by a thermal camera includes: a frequency analysis unit for executing frequency analysis for a plurality of pulse signals extracted from the video data; and a pulse component determination unit for determining a frequency component having a maximum amplitude in a pulse frequency range, which is a frequency range likely to be a pulse component, of frequency components output by the frequency analysis unit, as a pulse component.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a non-contact pulse estimation device, a non-contact pulse estimation method, and a program, and more particularly to a non-contact pulse estimation device that estimates pulse rate from video data acquired by a thermal camera. [Background technology]

[0002] In camera-based non-contact pulse estimation, the Photoplethysmography (PPG) technique is known, which measures changes in blood vessel volume due to pulse based on the intensity of reflected light from the blood vessels. Measurements are usually performed in a near-contact manner. While highly reliable, the sensors are expensive.

[0003] Among PPGs, rPPG, which measures non-contact using RGB cameras, has the advantage of being non-contact / non-invasive, measured from a distance of several tens of centimeters, and does not require special equipment.

[0004] On the other hand, depending on the size of the ROI (Region of Interest), the lighting conditions within the ROI may differ. This means that the results will vary depending on whether you choose the brighter side, the darker side, or a side that spans both sides, especially if half of the face is in shadow. In addition, the signal may contain lighting components, and in the case of dark skin tones, the temporal changes in hemoglobin levels are difficult to see due to the influence of melanin pigment, making analysis difficult.

[0005] The advantages of non-contact vital sensing using thermal cameras include: (1) enabling measurements at night; (2) the signal does not contain illumination components; (3) it is not affected by melanin pigment, even in people with dark skin tones; and (4) it reduces privacy concerns.

[0006] However, research into this technology is not as extensive as that into RGB camera-based technologies. It is known that non-contact vital sensing using thermal cameras has been verified using independently collected data (see Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] S. Cosar, Z. Yan, F. Zhao, T. Lambrou, S. Yue and N. Bellotto, “Thermal Camera Based Physiological Monitoring with an Assistive Robot,” 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018, pp. 5010-5013, doi: 10.1109 / EMBC.2018.8513201. [Non-Patent Document 2] Mayank Kumar, Ashok Veeraraghavan and Ashutosh Sabharwal, “DistancePPG: Robust non-contact vital signs monitoring using a camera”, Biomedical Optics Express Vol.6, Issue 5, pp.1565-1588, 2015. [Overview of the project] [Problems that the invention aims to solve]

[0008] However, common challenges with camera-based vital sign estimation technologies include low pulse wave signal intensity, superimposed noise due to movement and changes in the surrounding environment, and the presence of non-physiological noise in the same frequency band as physiological signals.

[0009] In addition, as a problem of a normal thermal camera using LWIR (Long Wave Infrared), the spatial resolution is lower than that of an RGB image, and the positions and detailed features of facial parts cannot be obtained.

[0010] In addition, regarding non-contact vital sensing by a thermal camera, since the verification has been limited to data collected independently, the performance of the methods proposed so far has not been fairly evaluated.

[0011] Therefore, an object of the present invention is to provide a non-contact pulse estimation device using a thermal camera with higher accuracy than before.

Means for Solving the Problems

[0012] A first aspect of the present invention is a non-contact pulse estimation device that estimates a pulse from video data acquired by a thermal camera, the non-contact pulse estimation device including: a frequency analysis unit that performs frequency analysis on a plurality of pulse wave signals extracted from the video data; and a pulse component determination unit that determines, as a pulse component, a frequency component having the maximum amplitude in a pulse frequency range that is a frequency range that may be the pulse frequency among the frequency components output by the frequency analysis unit.

[0013] A second aspect of the present invention is the non-contact pulse estimation device according to the first aspect, the non-contact pulse estimation device further including a pulse SNR calculation unit that calculates a pulse SNR that is a ratio of the maximum amplitude in the pulse frequency range among the frequency components to a sum or integral of amplitudes of frequency components outside the pulse frequency range, wherein the pulse component determination unit determines, as a pulse component, a frequency component having the maximum pulse SNR.

[0014] A third aspect of the present invention is the non-contact pulse estimation device according to the second aspect, wherein the pulse SNR calculation unit calculates the pulse SNR based on formula (1).

[0015]

Number

[0016] A fourth aspect of the present invention is a non-contact pulse estimation device according to any one of the first to third aspects, further comprising an interest area determination unit that determines an interest area to be used for estimating a pulse from the video data, and the interest area determination unit determines a face area as the interest area.

[0017] A fifth aspect of the present invention is a non-contact pulse estimation device according to the fourth aspect, wherein the interest area determination unit performs binarization processing on the video data and determines an area having the largest area with high luminance as the face area.

[0018] A sixth aspect of the present invention is a non-contact pulse estimation device according to the fourth or fifth aspect, comprising a jaw position determination unit that determines a jaw position among the face areas, and an image data extracted from the video data and the jaw position in the video data as teacher data, and further comprising a model generation unit that generates a determination model having an input as the image data extracted from the video data and an output as the jaw position in the video data, and the jaw position determination unit determines the jaw position using the determination model.

[0019] A seventh aspect of the present invention is a non-contact pulse estimation device according to any one of the fourth to sixth aspects, further comprising a skin area determination unit that determines a skin area among the face areas, and the skin area determination unit dynamically determines a first threshold value that is a threshold value of a luminance value representing skin.

[0020] An eighth aspect of the present invention is a non-contact pulse estimation device according to the seventh aspect, further comprising a skin ratio calculation unit that calculates a ratio of skin pixels in the face area for each frame, and the skin area determination unit updates the first threshold value so as to keep the ratio constant.

[0021] A ninth aspect of the present invention is a non-contact pulse estimation device according to the eighth aspect, further comprising a pulse area determination unit that determines an area for extracting a pulse component, and the pulse area determination unit determines an area for extracting a pulse component only from areas where the ratio is equal to or greater than a second threshold value.

[0022] A tenth aspect of the present invention is a non-contact pulse estimation device according to any of the first to ninth aspects, further comprising: a low-resolution reduction unit that reduces the resolution of image data extracted frame by frame from the video data to output a low-resolution image; and a pulse wave visualization unit that performs processing to associate the low-resolution image with the image data in the frame corresponding to the frequency component of the pulse determined by the pulse component determination unit.

[0023] An eleventh aspect of the present invention is a non-contact pulse estimation method using a non-contact pulse estimation device that estimates pulse rate from video data acquired by a thermal camera, wherein the non-contact pulse estimation device comprises a frequency analysis unit that performs frequency analysis on a plurality of pulse wave signals extracted from the video data, and a pulse component determination unit that determines the pulse frequency component from among the frequency components output by the frequency analysis unit, and the non-contact pulse estimation method includes a frequency analysis step in which the frequency analysis unit performs frequency analysis on a plurality of pulse wave signals extracted from the video data, and a pulse component determination step in which the pulse component determination unit determines as the pulse component the frequency component having the largest amplitude in the pulse frequency range, which is a frequency range that is likely to be the pulse frequency, from among the frequency components output by the frequency analysis unit.

[0024] A twelfth aspect of the present invention is a program for causing a computer to perform the non-contact pulse estimation method of the eleventh aspect. [Effects of the Invention]

[0025] According to each aspect of the present invention, it is possible to provide a non-contact pulse estimation device using a thermal camera with higher accuracy than conventional devices.

[0026] Furthermore, according to a second or third aspect of the present invention, by using the pulse SNR index invented by the present inventors, it becomes even easier to objectively estimate the pulse rate with greater accuracy than before.

[0027] Furthermore, according to a fourth or fifth aspect of the present invention, it becomes even easier to automatically extract a facial region from video data obtained from a thermal camera and to implement a non-contact pulse estimation device, etc.

[0028] Furthermore, according to a sixth aspect of the present invention, it becomes possible to detect the jaw position with high accuracy. Therefore, it becomes even easier to extract the facial region stably and accurately.

[0029] Furthermore, according to a seventh aspect of the present invention, it becomes easier to identify the skin region from the extracted facial region in which pulse wave signals should be detected.

[0030] Furthermore, according to the eighth aspect of the present invention, it becomes even easier to reliably identify skin areas even in thermal images where the brightness of the entire image tends to change gradually over time.

[0031] Furthermore, according to the ninth aspect of the present invention, it becomes even easier to determine a suitable region for extracting pulse signals.

[0032] Furthermore, according to the tenth aspect of the present invention, it becomes possible to obtain video data in which pulse wave signals extracted from thermal images are visualized. [Brief explanation of the drawing]

[0033] [Figure 1] This is a block diagram showing an overview of the non-contact pulse estimation device 1 according to this embodiment. [Figure 2] This is a flowchart illustrating the outline of the non-contact pulse rate estimation method according to this embodiment. [Figure 3] This is a flowchart illustrating the overview of Stage 1. [Figure 4] This figure illustrates the effectiveness of face detection using the method of this embodiment. [Figure 5] This is a flowchart illustrating the overview of Stage 2. [Figure 6] This is a flowchart illustrating the overview of Stage 3. [Figure 7]This figure illustrates thermal images of five subjects in this embodiment. [Figure 8] This figure illustrates the time course of reference values ​​measured by a pulse oximeter for each subject (solid line) and the period extracted to synchronize with the video used (dashed line). [Figure 9] This histogram shows the number of data points where the absolute error from the reference value of the pulse rate estimated by each method is below a certain bpm. [Figure 10] This figure shows the degree of agreement between each method and the PPG reference value. [Figure 11] This diagram shows the process for visualizing the pulse wave extracted using this embodiment. [Figure 12] This figure illustrates multiple frames of video data reconstructed by the pulse wave visualization method according to this embodiment, showing them consecutively in the time direction. [Modes for carrying out the invention]

[0034] Embodiments of the present invention will be described in detail below with reference to the drawings. However, the embodiments of the present invention are not limited to those described below. [Examples]

[0035] Figure 1 is a block diagram showing an overview of the non-contact pulse estimation device 1 according to this embodiment. The non-contact pulse estimation device 1 comprises a region of interest determination unit 3, a jaw position determination unit 5, a model generation unit 7, a skin region determination unit 9, a skin ratio calculation unit 11, a pulse region determination unit 13, a frequency analysis unit 15, a pulse component determination unit 17, a pulse SNR calculation unit 19, a low-resolution reduction unit 21, and a pulse wave visualization unit 23.

[0036] The non-contact pulse estimation device 1 estimates the pulse rate from video data acquired by a thermal camera. The region of interest determination unit 3 determines the facial region (hereinafter referred to as the "face region") included in the video data as the region of interest to be used for pulse rate estimation. The region of interest determination unit 3 also performs binarization on the image data extracted from the video data and determines the region with the highest brightness and largest area as the face region. Hereinafter, the region of interest may also be referred to as ROI.

[0037] The jaw position determination unit 5 determines the jaw position in the video data (hereinafter referred to as "jaw position"). The model generation unit 7 uses the image data extracted from the video data and the jaw position in the video data as training data, and generates a judgment model in which the image data extracted from the video data is the input and the jaw position in the video data is the output.

[0038] The skin area determination unit 9 determines the skin area within the face area. The skin area determination unit 9 also dynamically determines a first threshold value for the brightness value representing skin. The skin ratio calculation unit 11 calculates the ratio of skin pixels within the face area for each frame. The skin area determination unit 9 updates the first threshold value to maintain this skin pixel ratio constant.

[0039] The pulse region determination unit 13 determines a region from which pulse components are extracted only from regions where the ratio of skin pixels is equal to or greater than the second threshold.

[0040] The frequency analysis unit 15 performs frequency analysis on multiple pulse wave signals extracted from the video data. The pulse component determination unit 17 determines the pulse component as the frequency component that has the largest amplitude within the pulse frequency range, which is the frequency range that is likely to be a pulse component, from among the frequency components output by the frequency analysis unit 15. The pulse SNR calculation unit 19 calculates the pulse SNR, which is the ratio of the sum or integral of the amplitude of the frequency component with the largest amplitude within the pulse frequency range and the amplitude of the frequency component that is outside the pulse frequency range.

[0041] The low-resolution unit 21 reduces the resolution of the video data frame by frame and outputs a low-resolution video. The pulse wave visualization unit processes the low-resolution video output by the low-resolution unit 21 to associate it with the video data in the frame corresponding to the pulse component determined by the pulse component determination unit 17.

[0042] The following describes the outline of the non-contact pulse estimation method according to this embodiment, with reference to Figure 2. Figure 2 is a flowchart showing the outline of the non-contact pulse estimation method according to this embodiment.

[0043] The non-contact pulse estimation method according to this embodiment consists of three stages: selection of multiple ROIs (Stage 1), extraction of pulse wave signals (Stage 2), and application of BSS (Blind Source Separation) (Stage 3). The details of the processing in each stage are described below.

[0044] Refer to Figure 3 to describe Stage 1. Figure 3 is a flowchart illustrating the overview of Stage 1. Stage 1 is the selection of ROIs for pulse estimation. This stage is divided into three steps: face detection, skin detection, and ROI segmentation and selection. Figure 3 also shows the parameters required for each step.

[0045] In pulse rate estimation using a camera, selecting a stable ROI is the first crucial step. To select the ROI, the subject's face region is extracted from the thermal image.

[0046] The inventors have confirmed that BlazeFace, one of the machine learning models proposed by Google Research as a face detection technology suitable for thermal images, can also function as a face detector for thermal images. BlazeFace is a deep learning model that rapidly detects face positions and keypoints, trained using a dataset of RGB images.

[0047] However, it was also confirmed that BlazeFace performed less effectively in images where the face was tilted compared to images where the face was facing forward. Furthermore, when performing face detection on continuously captured thermal images, there was some variation in the position of the detected face region from frame to frame. Since pulse estimation using rPPG requires continuously selecting the same region as the ROI, such detection results are undesirable.

[0048] Therefore, in this embodiment, in addition to face detection using BlazeFace, a face detection method using thresholding that takes advantage of the property that human skin appears with high brightness values ​​in thermal images is used. By using brightness values ​​as a reference, the reflection of the background within the face area is reduced, and stable area selection is expected.

[0049] As a thresholding method, contrast enhancement is first applied to each frame to accurately detect only the skin portion in the video, increasing the difference in brightness between the subject's skin and the clothing and background. Then, the contrast-enhanced image is binarized, and the region with the largest detected area is designated as the face region.

[0050] However, when the subject's skin extends down to the neck in the thermal image, thresholding-based face detection makes it difficult to determine the subject's jaw position. Therefore, only the jaw position was interpolated using BlazeFace.

[0051] Figure 4 illustrates the effectiveness of face detection using the method of this embodiment. Figure 4 shows a comparison of face detection results using BlazeFace (Figure 4(a)) and face detection results using a method that combines BlazeFace and thresholding (Figure 4(b)) for each frame of the thermal image.

[0052] In the BlazeFace method shown in Figure 4(a), the detected face region changes as time progresses from left to right. On the other hand, the method using BlazeFace in combination with thresholding, shown in Figure 4(b), yields a more stable face region over time.

[0053] Next, we will describe the skin detection process in Stage 1. In this process, only skin pixels are extracted from the face region detected in the face detection process. The ROI includes pixels unrelated to the pulse, such as beards, eyebrows, and background. The presence of these regions can act as noise sources for the pulse wave signal being quantified.

[0054] Therefore, in this embodiment, non-skin pixels that do not contribute to the pulse wave signal are excluded from the ROI by thresholding similar to that used for face detection. However, unlike the process of separating the background from the face, skin and hair have relatively similar brightness values.

[0055] Because the luminance values ​​representing skin differ depending on the individual and their physical condition, a threshold is dynamically determined using k-means++ (non-hierarchical clustering). k-means is an algorithm for finding cluster centers. k-means++ is a clustering method that randomly divides data into a specified number of clusters and then optimizes the cluster assignment so that the distance between each data point and the cluster centroid is minimized. It is believed that k-means++ can automatically detect skin with different luminance values ​​for each data point.

[0056] Thermal imaging can sometimes exhibit a trend where the entire image gradually becomes brighter or darker over time. Therefore, the threshold needs to be updated frame by frame.

[0057] One possible method for updating the threshold is to run k-means++ frame by frame. However, the clustering results of k-means++ can vary depending on the initial values ​​set. In this case, the determined threshold and the extracted pulse wave signals may be affected.

[0058] Therefore, as a method for updating the threshold, we employ a method that calculates the ratio of skin pixels within the face region for each frame and updates the threshold to maintain a constant ratio.

[0059] First, in the initial frame, the threshold B for skin detection is determined using k-means++. skinThis determines the threshold. Based on this threshold, the ratio of skin to non-skin pixels in the binarized mask image is taken. This is the reference skin ratio r. base In subsequent frames, the threshold B determined in the previous frame will be used. skin After skin detection, the skin ratio r in the current frame is calculated, and the skin detection threshold B is determined according to the update rule shown in equation (2.1). skin Update.

[0060]

number

[0061] Here, z is a parameter that evaluates the consistency of the skin ratio. In the update rule shown in equation (2.1), if the change in the skin ratio for each frame exceeds z, the skin detection threshold B is set. skin This will be updated.

[0062] The method according to this embodiment assumes that the region from which pulse wave signals are extracted becomes more steady by determining an adaptive skin pixel threshold.

[0063] Next, we will describe the ROI division and selection process in Stage 1. As the final step in ROI selection, the facial region is divided into multiple ROIs by dividing it equally vertically and horizontally in order to extract multiple pulse wave signals.

[0064] Different skin regions have different epidermal thickness, capillary density, and facial features, resulting in pulse wave signals of varying quality.

[0065] In this embodiment, the number of candidate pulse wave signals is increased by setting the number of ROIs to multiple, and these are then separated into common pulse components and other non-physiological noise through subsequent processing.

[0066] The number of ROI divisions depends on the resolution of the camera used and the distance between the camera and the subject. For example, it is known that small ROIs of about 4x4 pixels or large ROIs containing multiple facial features may not be suitable. Also, some divided ROIs contain a lot of background or hair. Therefore, in this embodiment, ROIs in which the proportion of skin pixels is less than the threshold v were excluded from the region from which the pulse wave signal was extracted.

[0067] Next, we will describe the Stage 2 steps of pulse wave signal extraction and selection of valid signals. Figure 5 is a flowchart illustrating the overview of Stage 2. Stage 1 yielded multiple ROIs for pulse wave extraction. In Stage 2, signals are extracted from each ROI.

[0068] In methods for extracting pulse waves from camera footage, it is common practice to calculate the average brightness value within the ROI for each frame and use that as the pulse wave, regardless of whether the camera is an RGB or thermal camera. This is because averaging helps to reduce camera noise and improve the SNR (Single-to-Noise Ratio) of the pulse wave signal.

[0069] In this embodiment, the skin detection process in Stage 1 masks pixels other than skin within the ROI. Therefore, the average value is calculated using only skin pixels.

[0070] The signals extracted from each ROI are narrowed down to a few signals based on the following two conditions.

[0071] First, it is desirable to avoid obtaining signals containing a large amount of respiratory components from ROIs near the nostrils. Therefore, frequency spectral analysis is performed on the signals extracted from each ROI, and the respiratory frequency [b low , b high Signals with the largest spectral peak in the Hz range were excluded.

[0072] Also, the differences in values between two consecutive frames for each signal were calculated over all frames, and the value with the largest difference for each signal was calculated. If there are significant fluctuations in values between consecutive frames, it is considered that the fluctuations are due to the movement of the subject. Therefore, signals with the largest difference exceeding the threshold value d were also excluded.

[0073] Then, referring to FIG. 6, the stage of separating the pulse component by BSS and estimating the pulse rate in stage 3 will be described. FIG. 6 is a flowchart showing an overview of stage 3.

[0074] In stage 3, multiple signals are utilized to estimate the pulse rate. The pulse wave signals extracted from the video are mixed with various non - physiological noises, and the signal intensity of the pulse component is small. As a method to improve the SNR of the signal in rPPG, band - pass filtering with a fixed cut - off frequency is used. However, usually, non - physiological noises are present within the same frequency band as the pulse component, so it is considered insufficient.

[0075] Therefore, BSS is applied to multiple pulse wave signals extracted from different ROIs to separate the components contained in the signals. The method of applying BSS to a one - channel thermal video is proposed in the present invention.

[0076] In this embodiment, particular attention is focused on independent component analysis among BSS techniques. When the signals obtained from the divided ROIs are x(t)=[x1(t), x2(t),…,x n (t)] T (t = 0,1,2,…), it is considered that the observed signal x(t) contains various components such as pulse waves and other non - physiological signals. Here, when the signal sources contained in the observed signal are s(t)=[s1(t), s2(t),…,s m (t)] T a linear relationship expressed by Equation (2.2) is assumed between x(t) and s(t).

[0077]

Equation

[0078] However, T is the transpose, n and m are the number of observed signals and signal sources (n>=m), respectively, and A is an unknown n×m mixture matrix. Independent component analysis assumes that each component of s(t) is independent of the others in order to estimate the signal source s(t) using equation (2.2).

[0079] In this example, FastICA was used as the algorithm for independent component analysis. FastICA is a method that estimates independent components sequentially and is characterized by its rapid convergence. In addition, whitening and dimensionality reduction were performed using principal component analysis as preprocessing.

[0080] It is unclear which component of the signal source s(t) obtained by independent component analysis corresponds to the pulse wave signal, and it is necessary to identify the pulse wave signal using some kind of indicator.

[0081] Therefore, the inventor proposes a new index called pulse SNR. Pulse SNR is defined by equation (2.3).

[0082]

number

[0083] The selection of pulse components and estimation of pulse rate are performed by the following process. Each separated signal is subjected to trend removal processing used in biosignal analysis, and then converted to the frequency domain by Fast Fourier Transform (FFT) to obtain the frequency spectrum S(f). Here, in equation (2.3), f is the frequency in Hz, and N is the sampling frequency divided by 2.

[0084] The pulse SNR is given by [P low , P high This value is the ratio of the maximum amplitude in the pulse frequency band of Hz to the sum of the amplitudes outside the frequency band of interest. Pulse SNR is an indicator of how dominant the peak frequency is.

[0085] Assuming that a higher pulse SNR value indicates a greater certainty of the presence of a pulse component, the separated component with the highest pulse SNR is selected as the pulse wave signal.

[0086] Finally, the frequency spectrum of the selected signal is analyzed, [P low , P high We found the frequency with the largest amplitude in the Hz frequency band. Since the obtained frequency was in Hz, we multiplied it by 60 to get the number of pulses per minute.

[0087] The following describes an experiment on pulse rate estimation using thermal images and thermal video. In this experiment, pulse rate was estimated from thermal video using the method described in this embodiment.

[0088] First, let's describe the data used in this experiment. In this example, we use a dataset for physiological monitoring using thermal cameras, collected by the Lincoln Centre for Autonomous Systems Research (L-CAS) at the University of Lincoln in the UK. The dataset includes thermal images and numerical data of pulse rate measured simultaneously with the image acquisition using a pulse oximeter, as a reference value.

[0089] The camera used had a resolution of 382 x 288 pixels, a frame rate of 27 fps, and a wavelength range of 8-14 micrometers. The distance between the camera and the subject was approximately 1.5 meters.

[0090] Video and numerical data were recorded for five subjects, with each subject measured for approximately two minutes. Subjects were instructed to remain still for the first minute, and then to tilt their heads up and down and left and right for 10 seconds each during the second half.

[0091] In this experiment, each subject was assigned an ID: A, B, C, D, or E. Figure 7 shows an example of thermal images from five subjects in this dataset. Subjects were photographed facing the camera. Video data was recorded for approximately two minutes at a time, but in this experiment, only the first minute of data, during which the subject was stationary, was analyzed.

[0092] Furthermore, subject E had lower skin brightness values ​​and a pulse rate measured by the reference PPG was more than 20 beats higher on average compared to the other subjects, so the data was excluded from the study.

[0093] Furthermore, the video data captured for each subject contained interruptions, resulting in a loss of continuity in some parts. In this experiment, we decided to analyze continuous video data of 10 seconds or more, and after removing the interrupted parts of the video, we divided it into 12 videos in total for all subjects. Table 1 shows the number of frames in the divided video data used in this experiment.

[0094] [Table 1]

[0095] Reference values ​​measured by a pulse oximeter were recorded at approximately 0.1-second intervals for approximately 2 minutes. The unit is bpm (beats per minute), indicating the number of pulses per minute. To synchronize the video and the recorded numerical values, the discrete numerical data was approximated by a polynomial using cubic spline interpolation and resampled at 1 / 27-second intervals.

[0096] Figure 8 shows the time course of the reference value measured by a pulse oximeter for subject A (solid line) and the period extracted to synchronize with the video used (dashed line). The average value of the reference value in each extracted period was used as the reference value for pulse rate in this experiment.

[0097] Next, we will discuss the comparison method and parameter settings. In this experiment, we used three pulse estimation methods using thermal images as baselines. Hereafter, these three methods will be referred to as WeightedMean, WeightedMax, and FaceMean.

[0098] WeightedMean weights the signals from each divided ROI and extracts the pulse wave from its weighted average. In WeightedMean, pulse SNR is used as the weighting metric, and the pulse SNR calculated for each ROI is normalized so that the minimum value is 0 and the maximum value is 1, and then the weighted average is taken.

[0099] WeightedMax is a technique that extracts pulse waves using only one ROI (Region of Interest) that has the maximum weight assigned by the pulse SNR (Signal-to-Noise Ratio).

[0100] FaceMean is a technique that extracts pulse waves from the entire face as a single ROI without dividing it into separate ROIs.

[0101] In each method, signals were extracted only from skin pixels after skin detection. The extracted signals were post-processed with trend removal and a bandpass filter with a passband frequency of 0.90–1.65 Hz. Each signal was then converted to the frequency domain using FFT, and the frequency with the largest amplitude was multiplied by 60 to estimate the pulse rate.

[0102] The approach of this invention requires setting several parameters. Many of these parameters were determined experimentally. If you are verifying the data using data other than that used in this experiment, you may need to adjust the parameters. Table 2 shows the parameter values ​​set for each process. The Python® library used is indicated in parentheses in Table 2. In addition, for skin detection using k-means++, after clustering with the number of clusters set to 4, the average of the maximum value and the second largest value of the four obtained cluster centroids is used as the skin detection threshold B. skin That's what I decided.

[0103] [Table 2]

[0104] The experimental results of the present invention's method and comparative methods are shown below with reference to Table 3, Figure 9, and Figure 10. Table 3 shows the performance evaluation of each method using the average value of the absolute error between the reference value and the estimated value, defined by the Mean Absolute Error (MAE) in equation (3.1). The standard deviation of the absolute error is shown in parentheses in Table 3.

[0105] [Table 3]

[0106]

number

[0107] Here, i = 1, 2, ..., n. Also, when n is the number of data points (n=12), HR i This represents the estimated heart rate in each data point, GT i The values ​​shown represent the reference values ​​measured by PPG for each data point. The units of the values ​​in Table 3 are bpm, which represents the pulse rate per minute.

[0108] Figure 9 is a histogram showing the number of data points where the absolute error from the reference value to the estimated pulse rate using each method is below a specific bpm. The horizontal axis represents the absolute error in 3 bpm increments, and the vertical axis represents the number of data points in each class. Each class, from left to right, represents WeightedMean, WeightedMax, FaceMean, and Proposed (proposed method). However, Proposed was not distributed in classes with an error of 15 or more. Also, only the FaceMean and WeightedMax methods were distributed in classes with errors of 27 and 30, respectively.

[0109] Figure 10 shows the degree of agreement between each method and the PPG reference value using Bland-Altman analysis. Bland-Altman analysis is a method for checking systematic errors. Systematic errors are divided into fixed errors and proportional errors. Fixed errors are deviations that occur within a certain range in a specific direction, regardless of the true value. Proportional errors are errors that occur in a specific direction and increase or decrease in proportion to the true value. All 12 data points are plotted in each graph in Figure 10. The horizontal axis of the graphs represents the mean of the reference value and the estimated value from each method, and the vertical axis represents the error between the reference value and the estimated value from each method.

[0110] In Figure 10, the solid line represents the mean error, and the dashed line represents the 95% confidence interval. The 95% confidence interval is defined as the mean error ± 1.96 × the standard deviation of the error.

[0111] As shown in Table 3, the estimated result using the proposed method in this embodiment was an MAE of 5.65 bpm, which was the smallest error from the reference value compared to other methods.

[0112] Furthermore, looking at the estimated values ​​for each data point in Figure 9, the percentage of data points with an error of 9 bpm or less was 58% for WeightedMean, WeightedMax, and FaceMean, while the proposed method achieved 91%. This narrow error range can also be seen from the Bland-Altman analysis in Figure 10. The 95% confidence interval for the proposed method was -13.51-10.19 bpm, which was the narrowest range.

[0113] The following describes the visualization of pulse waves using image pyramids. To visualize the pulse waves extracted by the method described in this embodiment, image pyramids were applied to the video data used. An image pyramid is an image processing technique that obtains multiple images of different resolutions by gradually reducing the resolution of an image.

[0114] Figure 11 shows the process for visualizing the pulse wave extracted by this embodiment. In the visualization method according to this embodiment, the pulse wave is visualized in a visible form by replacing the time-series signal obtained by reducing the resolution of the image pyramid to the lowest possible resolution in each frame of the video with the pulse wave extracted by the proposed method of this embodiment and reconstructing the image pyramid.

[0115] The following describes the specific procedure. First, the independent component analysis of Stage 1, Stage 2, and Stage 3 in this embodiment is performed. Here, the observed signal is x(t)=[x1(t), x2(t),…,x n (t)] T (t=0,1,2,…L), mixing matrix is ​​A, independent component is s(t)=[s1(t), s2(t),…,s m (t)] T Let T be the transpose, L be the number of frames, n and m be the number of ROIs and the number of independent components (n≧m), respectively, and A be an n×m matrix.

[0116] In this case, if the component selected by the pulse SNR according to this embodiment is the u (1 ≤ u ≤ m)-th signal of the independent components, then it can be expressed as equation (3.2) using an m x m square matrix U in which there is exactly one 1 in the u row and u column, and all other elements are 0.

[0117]

number

[0118] A + This is the pseudo-inverse matrix of A, and using only the independent components selected by equation (3.2), the signal x'(t)=[x'1(t), x'2(t),…,x' n (t)] T It will be reconfigured.

[0119] The signal x'(t) is subjected to a 0.75-2.00Hz bandpass filter and post-processing that enhances its amplitude by 1.1 times, and is replaced with the time-series signal of the image pyramid corresponding to each ROI. Subsequently, the pyramid is reconstructed for each ROI and for each frame, yielding video data that visualizes the pulse wave.

[0120] Figure 12 illustrates 12 consecutive frames in the time direction of video data reconstructed by the pulse wave visualization method according to this embodiment. The ROI used in the method according to this embodiment is reconstructed, and the waveform obtained as a pulse wave is shown as blinking. [Explanation of Symbols]

[0121] 1; Non-contact pulse estimation device, 3; Region of interest determination unit, 5; Jaw position determination unit, 7; Model generation unit, 9; Skin region determination unit, 11; Skin ratio calculation unit, 13; Pulse region determination unit, 15; Frequency analysis unit, 17; Pulse component determination unit, 19; Pulse SNR calculation unit, 21; Low resolution unit, 23; Pulse wave visualization unit

Claims

1. A non-contact pulse rate estimation device that estimates pulse rate from video data acquired by a thermal camera, A frequency analysis unit performs frequency analysis on multiple pulse wave signals extracted from the aforementioned video data, A pulse component determination unit determines the pulse component as the frequency component that has the largest amplitude in the pulse frequency range, which is a frequency range that is likely to be a pulse component, from among the frequency components output by the frequency analysis unit. The system includes a pulse SNR calculation unit that calculates a pulse SNR which is the ratio of the sum or integral of the amplitude of the maximum amplitude within the pulse frequency range and the amplitude of the frequency components outside the pulse frequency range. The pulse component determination unit is a non-contact pulse estimation device that determines the frequency component that maximizes the pulse SNR as the pulse component.

2. The non-contact pulse rate estimation device according to claim 1, wherein the pulse SNR calculation unit calculates the pulse SNR based on formula (1). [Math 1]

3. A non-contact pulse rate estimation device that estimates pulse rate from video data acquired by a thermal camera, A frequency analysis unit performs frequency analysis on multiple pulse wave signals extracted from the aforementioned video data, A pulse component determination unit determines the pulse component as the frequency component that has the largest amplitude in the pulse frequency range, which is a frequency range that is likely to be a pulse component, from among the frequency components output by the frequency analysis unit. A region of interest determination unit determines the facial region as the region of interest to be used for estimating pulse rate from the aforementioned video data, Of the aforementioned facial region, a jaw position determination unit determines the jaw position, The system uses image data extracted from video data and the jaw position in the video data as training data, and includes a model generation unit that generates a judgment model that takes image data extracted from video data as input and outputs the jaw position in the video data. The jaw position determination unit is a non-contact pulse estimation device that determines the jaw position using the determination model.

4. The non-contact pulse estimation device according to claim 3, wherein the region of interest determination unit performs binarization processing on the video data and determines the region having the largest area with high brightness as the face region.

5. A non-contact pulse rate estimation device that estimates pulse rate from video data acquired by a thermal camera, A frequency analysis unit performs frequency analysis on multiple pulse wave signals extracted from the aforementioned video data, A pulse component determination unit determines the pulse component as the frequency component that has the largest amplitude in the pulse frequency range, which is a frequency range that is likely to be a pulse component, from among the frequency components output by the frequency analysis unit. A region of interest determination unit determines the facial region as the region of interest to be used for estimating pulse rate from the aforementioned video data, A skin region determination unit that determines the skin region within the aforementioned facial region, The system includes a skin ratio calculation unit that calculates the ratio of skin pixels within the face region for each frame, The skin region determination unit is, The first threshold value, which is the threshold value for the brightness representing skin, is dynamically determined. A non-contact pulse rate estimation device that updates the first threshold so as to maintain the ratio constant.

6. It further includes a pulse region determination unit that determines the region from which pulse components are extracted. The non-contact pulse estimation device according to claim 5, wherein the pulse region determination unit determines a region from which pulse components are extracted only from a region where the ratio is equal to or greater than a second threshold.

7. A low-resolution unit that reduces the resolution of image data extracted frame by frame from the aforementioned video data to output a low-resolution image, A non-contact pulse estimation device according to any one of claims 1 to 6, further comprising: a pulse wave visualization unit that performs processing to associate the low-resolution image with the image data in a frame corresponding to the pulse component determined by the pulse component determination unit.

8. A non-contact pulse estimation method using a non-contact pulse estimation device that estimates pulse rate from video data acquired by a thermal camera, The non-contact pulse rate estimation device is A frequency analysis unit performs frequency analysis on multiple pulse wave signals extracted from the aforementioned video data, A pulse rate component determination unit determines the pulse rate component from the frequency components output by the frequency analysis unit, The system includes a pulse SNR calculation unit that calculates a pulse SNR which is the ratio of the maximum amplitude in the pulse frequency range, which is a frequency range that may be a pulse component, to the sum or integral of the amplitudes of frequency components that fall outside the pulse frequency range. The frequency analysis unit performs a frequency analysis step on multiple pulse wave signals extracted from the video data, A non-contact pulse estimation method comprising: a pulse component determination step in which the pulse component determination unit determines the frequency component that maximizes the pulse SNR as the pulse component.

9. A program for causing a computer to perform the non-contact pulse estimation method described in claim 8.

Images