Device and method for obtaining heart rate

Abstract

An apparatus and method for obtaining a heart rate, the apparatus for obtaining a heart rate may include: a processor; and a memory device for obtaining a first frequency band image and a second frequency band image of different frequency bands, respectively measuring a first remote heart rate signal of the first frequency band image and a second remote heart rate signal of the second frequency band image, respectively calculating a first mass fraction of the first frequency band image and a second mass fraction of the second frequency band image, a first effective heart rate section is selected based on the first remote heart rate signal and the first mass fraction, a second effective heart rate section is selected based on the second remote heart rate signal and the second mass fraction, and a complementary heart rate is determined based on the first effective heart rate section and the second effective heart rate section.

Description

[0001] Related Applications

[0002] This application claims priority to Korean Patent Application No. 10-2023-0160496 filed on November 20, 2023, which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure relates to devices and methods for obtaining heart rate. Background Art

[0004] Conventional photoplethysmography (PPG) technology attaches sensors directly to the body and measures changes in blood flow in blood vessels in the body, while remote photoplethysmography (rPPG) technology can measure changes in blood flow in a non-contact manner, thereby recognizing the usefulness of rPPG technology in various application fields, such as continuously tracking human health status through rPPG or remotely collecting patient physiological data to provide telemedicine. In addition, rPPG technology can be used to infer heart rate and analyze heart rate changes to infer human emotional state, or monitor the state of vehicle drivers, such as stress level, excitement level, and fatigue driving. However, the accuracy of rPPG technology may be affected by changes in lighting (e.g., changes in lighting due to strong external light sources, reflected light, etc.) or movement of the object.

[0005] The information included in this background of the disclosure is only for enhancement of understanding of the general background of the disclosure and should not be taken as an acknowledgment or any form of suggestion that this information forms the prior art already known to a person skilled in the art. Summary of the invention

[0006] Various aspects of the present disclosure are directed to providing an apparatus and method for obtaining a heart rate, which are configured for determining a remote heart rate of a driver or a passenger in a vehicle with high accuracy and reliability.

[0007] An exemplary embodiment of the present disclosure provides a device for obtaining heart rate, which device may include: one or more processors; and one or more memory devices, wherein the one or more memory devices may include program codes, and the program codes may be executed by the one or more processors, and may obtain a first frequency band image and a second frequency band image of different frequency bands, respectively measure a first remote heart rate signal of the first frequency band image and a second remote heart rate signal of the second frequency band image, respectively calculate a first quality score of the first frequency band image and a second quality score of the second frequency band image, select a first effective heart rate segment based on the first remote heart rate signal and the first quality score, select a second effective heart rate segment based on the second remote heart rate signal and the second quality score, and determine a complementary heart rate based on the first effective heart rate segment and the second effective heart rate segment.

[0008] In some exemplary embodiments of the present disclosure, determining the first quality score and the second quality score may include: determining a first motion quality score and a second motion quality score, and determining the first motion quality score by measuring changes in facial feature points in frames adjacent to each other relative to the first frequency band image, and determining the second motion quality score by measuring changes in facial feature points in frames adjacent to each other relative to the second frequency band image.

[0009] In some exemplary embodiments of the present disclosure, determining the first quality score and the second quality score may include determining a first lighting quality score and a second lighting quality score, and the first lighting quality score may be determined by measuring a brightness change relative to a first frequency band image, and the second lighting quality score may be determined by measuring a brightness change relative to a second frequency band image.

[0010] In some exemplary embodiments of the present disclosure, determining the first quality score and the second quality score may include determining a first signal quality score and a second signal quality score, wherein the first signal quality score may be determined by measuring the signal quality based on the spectral characteristics of the remote heart rate signal relative to the first frequency band image, and the second signal quality score may be determined by measuring the signal quality based on the spectral characteristics of the remote heart rate signal relative to the second frequency band image.

[0011] In some exemplary embodiments of the present disclosure, selecting a first effective heart rate segment may include: predicting an error value from a model trained by a first quality score; including a segment having a prediction error value less than or equal to a predetermined first threshold in the first effective heart rate segment; and excluding a segment having a prediction error value greater than the first threshold in the first effective heart rate segment.

[0012] In some exemplary embodiments of the present disclosure, selecting a second effective heart rate segment may include: predicting an error value from a model trained by a second quality score; including a segment having a prediction error value less than or equal to a predetermined second threshold in the second effective heart rate segment; and excluding a segment having a prediction error value greater than the second threshold in the second effective heart rate segment.

[0013] In some exemplary embodiments of the present disclosure, the first threshold may be determined to correspond to an error value of the highest x% (x is a positive real number), in which the error value of the result predicted from the model trained by the first quality score is the smallest, and the second threshold may be determined to correspond to an error value of the highest y% (y is a positive real number), in which the error value of the result predicted from the model trained by the second quality score is the smallest.

[0014] In some exemplary embodiments of the present disclosure, the model may include at least one of a long short-term memory (LSTM) model, a random forest model, and a singular value decomposition (SVD) model.

[0015] In some exemplary embodiments of the present disclosure, the step of determining the complementary heart rate may include: when a portion of the first valid heart rate segment and a portion of the second valid heart rate segment overlap and the predicted heart rates in the overlapping segments are different from each other, determining the complementary heart rate by using a predetermined weight.

[0016] In some exemplary embodiments of the present disclosure, the first frequency band image may include a visible light image, and the second frequency band image may include an infrared image.

[0017] Another exemplary embodiment of the present disclosure provides a device for obtaining heart rate, which device may include: one or more processors; and one or more memory devices, wherein the one or more memory devices may include program codes, and the program codes may be executed by the one or more processors, and may obtain a first frequency band image and a second frequency band image of different frequency bands, respectively measure a first remote heart rate signal of the first frequency band image and a second remote heart rate signal of the second frequency band image, calculate a movement quality score, a lighting quality score, and a signal quality score of each of the first frequency band image and the second frequency band image, select a first effective heart rate segment based on the first remote heart rate signal and the first quality score, select a second effective heart rate segment based on the second remote heart rate signal and the second quality score, and determine a complementary heart rate based on the first effective heart rate segment and the second effective heart rate segment.

[0018] In some exemplary embodiments of the present disclosure, the movement quality score may be determined by using the following equation (1):

[0019]

[0020] Here, k represents a number indicating a region of interest (ROI), n represents an index indicating a vertex of a square ROI, b represents a current frame, and b′ represents a previous frame.

[0021] In some exemplary embodiments of the present disclosure, the lighting quality score may be determined by using the following equations (2), (3), and (4):

[0022]

[0023] I k =Max(X k )-Min(X k ) (3)

[0024] X k ={X 1,X 2 ,…,X n} (4)

[0025] Where k represents the sum of the number of divided regions and the total number of regions, n represents the number of time series data, X represents the brightness value, and Represents the average value of X.

[0026] In some exemplary embodiments of the present disclosure, the signal quality score may be determined by using the following equation (5):

[0027]

[0028] Here, U t (f) represents the template window binary function, represents the power spectral density function.

[0029] In some exemplary embodiments of the present disclosure, selecting a first effective heart rate segment may include: predicting an error value from a model trained by a first quality score; including a segment having a prediction error value less than or equal to a predetermined first threshold in the first effective heart rate segment; and excluding a segment having a prediction error value greater than the first threshold in the first effective heart rate segment.

[0030] In some exemplary embodiments of the present disclosure, selecting a second effective heart rate segment may include: predicting an error value from a model trained by a second quality score; including a segment having a prediction error value less than or equal to a predetermined second threshold in the second effective heart rate segment; and excluding a segment having a prediction error value greater than the second threshold in the second effective heart rate segment.

[0031] Various exemplary embodiments provide a method for obtaining a heart rate, which may include: obtaining a first frequency band image and a second frequency band image of different frequency bands; measuring a first remote heart rate signal of the first frequency band image and a second remote heart rate signal of the second frequency band image, respectively; determining a first quality score of the first frequency band image and a second quality score of the second frequency band image; selecting a first valid heart rate segment based on the first remote heart rate signal and the first quality score; selecting a second valid heart rate segment based on the second remote heart rate signal and the second quality score; and determining a complementary heart rate based on the first valid heart rate segment and the second valid heart rate segment.

[0032] In some exemplary embodiments of the present disclosure, determining the first quality score and the second quality score may include determining a first motion quality score and a second motion quality score. The first motion quality score may be determined by measuring changes in facial feature points in frames adjacent to each other relative to the first frequency band image, and the second motion quality score may be determined by measuring changes in facial feature points in frames adjacent to each other relative to the second frequency band image.

[0033] In some exemplary embodiments of the present disclosure, determining the first quality score and the second quality score may include determining a first lighting quality score and a second lighting quality score, and the first lighting quality score may be determined by measuring a brightness change relative to a first frequency band image, and the second lighting quality score may be determined by measuring a brightness change relative to a second frequency band image.

[0034] In some exemplary embodiments of the present disclosure, determining the first quality score and the second quality score may include determining a first signal quality score and a second signal quality score, wherein the first signal quality score may be determined by measuring the signal quality based on the spectral characteristics of the remote heart rate signal relative to the first frequency band image, and the second signal quality score may be determined by measuring the signal quality based on the spectral characteristics of the remote heart rate signal relative to the second frequency band image.

[0035] According to an exemplary embodiment of the present disclosure, a remote heart rate with high accuracy and reliability can be stably determined for a driver or passenger in a vehicle even with various lighting changes, including sunlight, traffic lights in a vehicle driving environment, or transport traffic lights, or noise such as movement of the driver or passenger's face.

[0036] The methods and apparatus of the present disclosure have other features and advantages that will be apparent from or set forth in more detail in the accompanying drawings and the following detailed description incorporated herein, which together serve to explain certain principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a block diagram for describing an apparatus for obtaining a heart rate according to an exemplary embodiment of the present invention.

[0038] Figure 2 is a flowchart for describing the operation of an apparatus for obtaining a heart rate according to an exemplary embodiment of the present disclosure.

[0039] Figures 3 to 8 is a diagram for describing an exemplary embodiment of an apparatus for obtaining a heart rate according to an exemplary embodiment of the present disclosure.

[0040] Fig. 9 is a flowchart for describing a method for determining a heart rate according to an exemplary embodiment of the present invention.

[0041] Fig.10 is a diagram for describing a computing device according to an exemplary embodiment of the present disclosure.

[0042] It will be appreciated that the accompanying drawings are not necessarily drawn to scale, and that they present somewhat simplified representations of various features illustrating the basic principles of the present disclosure. The specific design features of the present disclosure as included herein (including, for example, specific dimensions, orientations, locations, and shapes) will be determined in part by the specific intended application and use environment.

[0043] In the drawings, reference numbers refer to the same or equivalent parts of the present disclosure throughout the several figures of the drawing. DETAILED DESCRIPTION

[0044] Reference will now be made in detail to various embodiments of the present disclosure, examples of which are shown in the accompanying drawings and described below. Although the present disclosure will be described in conjunction with exemplary embodiments of the present disclosure, it should be understood that this specification is not intended to limit the present disclosure to those exemplary embodiments of the present disclosure. On the other hand, the present disclosure is intended to cover not only the exemplary embodiments of the present disclosure, but also various replacements, modifications, equivalents and other embodiments that may be included within the spirit and scope of the present disclosure as defined by the appended claims.

[0045] In the following description, various exemplary embodiments of the present disclosure will be described more fully below with reference to the accompanying drawings, in which exemplary embodiments of the present disclosure are shown. As will be appreciated by those skilled in the art, the described embodiments may be modified in various different ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and description are to be considered illustrative rather than restrictive in nature. Throughout the specification, the same reference numerals represent the same elements.

[0046] Throughout the specification and claims, unless explicitly described to the contrary, the word "comprise" and variations such as "comprises" or "comprising" will be understood to imply the inclusion of the elements stated but not the exclusion of any other elements. Terms including common numbers such as first and second are used to describe various constituent elements, but the constituent elements are not limited by the terms. These terms are only used to distinguish one component from another.

[0047] The terms "part", "unit", "module" and the like included in the specification refer to a unit that can process at least one function or operation, and this can be implemented by hardware or circuit, or software, or a combination of hardware or circuit and software. In addition, at least some components or functions of the apparatus and method for obtaining a heart rate according to the exemplary embodiments described below may be implemented as a program or software, and the program or software may be stored in a computer-readable medium.

[0048] Figure 1 is a block diagram for describing an apparatus for obtaining a heart rate according to an exemplary embodiment of the present invention.

[0049] Reference Figure 1 According to various exemplary embodiments of the present disclosure, the heart rate obtaining device 10 may include one or more processors and one or more memory devices. For example, the heart rate obtaining device 10 may be implemented as follows: Fig.10 In this case, the one or more processors may correspond to the processor 510 of the computing device 50 , and the one or more memory devices may correspond to the memory 520 of the computing device 50 .

[0050] One or more memory devices of the heart rate obtaining device 10 may include program codes executed by one or more processors. The program codes are executed to perform the function of obtaining a remote heart rate with high accuracy and reliability for a driver or a passenger in a vehicle, and for the sake of clarity and convenience of description, the functions are described in this specification by using the term "module".

[0051] The heart rate obtaining device 10 may include an image obtaining module 110 , a remote heart rate signal measuring module 120 , a quality score calculating module 130 , a first effective heart rate segment selecting module 140 , a second effective heart rate segment selecting module 150 and a complementary heart rate determining module 160 .

[0052] The image acquisition module 110 can obtain a first band image and a second band image of different bands. In some exemplary embodiments of the present disclosure, the first band image may include a visible light image, and the second band image may include an infrared image. In this specification, for the sake of clarity and convenience of description, the visible light image and the infrared image are described as images of different bands obtained by the image acquisition module 110, but the spirit of the present disclosure is not limited thereto. For example, the image acquisition module 110 can obtain an image corresponding to any first band that is not particularly limited to the visible light band and an image including any second band that is different from the first band and is not limited to the infrared band, and the heart rate acquisition device 10 can perform the heart rate acquisition described below from the image.

[0053] The image acquisition module 110 can acquire a first band image and a second band image of different frequency bands using a plurality of cameras that can acquire images of different spectral ranges. In some exemplary embodiments of the present invention, the image acquisition module 110 can acquire a first band image by using an RGB camera 20, and acquire a second band image by using a near infrared (NIR) camera 21. The RGB camera 20 can detect a visible light spectrum of about 380 to 740 nm visible to the human eye, and provide images represented by various colors by combining three basic colors of red, green and blue. The RGB camera 20 can be used in various fields such as general photography, video recording, etc., but will show limited performance under continuous sunlight changes that occur in low illumination environments or during vehicle driving. Meanwhile, the NIR camera 21 can detect infrared spectra exceeding the visible light range (specifically, in the near infrared region between about 750 nm and 1400 nm), and can generate images based on the thermal pattern or specific infrared reflection characteristics of the object. Since the NIR camera 21 does not rely on visible light, the NIR camera 21 is useful even in a low illumination environment and is not affected by the illumination caused by sunlight. The heart rate obtaining apparatus 10 may be configured to determine a complementary heart rate with improved accuracy by considering mutual characteristics of remote heart rate signals for different wavelength bands as described above.

[0054] In some exemplary embodiments of the present disclosure, the RGB camera 20 may capture images using a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The CCD sensor may operate by converting light into electric charge and then converting the electric charge into an analog signal, and may read images by moving electric charge between pixels. Meanwhile, the CMOS sensor may perform separate charge voltage conversion for each pixel, and may perform signal processing at the pixel level. Generally, CCD sensors provide high image quality, but are more expensive and consume large power, while CMOS sensors may be relatively cheap, have high power efficiency, and may provide high processing speed.

[0055] In some exemplary embodiments of the present invention, the NIR camera 21 may use a monochrome sensor. A monochrome sensor, which is an image sensor specifically for detecting light in the near infrared spectrum, does not capture color information, but may record brightness or intensity information only in a black and white image. Since the monochrome sensor does not have a color filter, the monochrome sensor provides higher sensitivity and is configured for effective image capture in low illumination environments. In addition, since the monochrome sensor does not need to process color information, the monochrome sensor may provide minute image details and higher resolution. The NIR camera 21 may use a monochrome sensor that may detect infrared wavelengths of approximately 930 nm to 950 nm, which include high sunlight absorption, particularly in the atmosphere. At the same time, the NIR camera 21 may operate in conjunction with an infrared lighting machine that is disposed in the vehicle and irradiates infrared light to the driver or passenger, and a bandpass filter that passes a predetermined range of near infrared wavelengths and blocks light of other wavelengths.

[0056] The remote heart rate signal measuring module 120 may respectively measure the first remote heart rate signal and the second remote heart rate signal of the first frequency band image and the second frequency band image obtained by the image acquisition module 110. When the first frequency band image includes a visible light image and the second frequency band image includes an infrared image, the first frequency band image may be a 3-channel image sequence obtained by using the RGB camera 20, and the second frequency band image may be a single-channel image sequence obtained by using the NIR camera 21.

[0057] In some exemplary embodiments of the present disclosure, the remote heart rate signal measurement module 120 obtains an image of the face of the vehicle driver or passenger by using the RGB camera 20, and detects the facial area and feature points in the image to define a region of interest (ROI). Next, the remote heart rate signal measurement module 120 may perform signal processing preprocessing, including obtaining the average brightness value of the skin area in the facial area, removing trend lines and bandpass filtering, and extracting the remote heart rate signal by using an algorithm for extracting the heart rate signal. In some exemplary embodiments of the present disclosure, the algorithm for extracting the heart rate signal may include at least one of a chromaticity-based method (Chrom), an optical noise injection technique (ONIT), a principal component analysis (PCA), and a plane orthogonal to the skin color (POS). Chrom can measure the blood flow changes caused by the heartbeat by detecting the skin color changes using the face whose skin color changes are more sensitive to the heartbeat than to the intensity of the light, and ONIT can accurately measure the heart rate by removing noise from the light source under various lighting conditions or in an environment where there are many movements. PCA can separate and extract heart rate related information from various color channels by identifying and extracting principal components from data collected from multiple optical channels, and POS can separate signals related to skin color changes and extract heart rate related information by analyzing components perpendicular to a plane indicating skin color changes. Thereafter, the remote heart rate signal measurement module 120 can measure the remote heart rate by performing frequency analysis on the extracted remote heart rate signal to determine the maximum frequency component.

[0058] In some exemplary embodiments of the present disclosure, the remote heart rate signal measurement module 120 obtains an image of the face of the vehicle driver or passenger by using the NIR camera 21, and detects the facial area and feature points in the image to define a region of interest (ROI). Next, the remote heart rate signal measurement module 120 may perform signal preprocessing including obtaining the average brightness value of the skin area in the facial area, removing trend lines and bandpass filtering, and extract the remote heart rate signal by using an algorithm for extracting the heart rate signal. Thereafter, the remote heart rate signal measurement module 120 may measure the remote heart rate by performing frequency analysis on the extracted remote heart rate signal to determine the maximum frequency component. In some exemplary embodiments of the present disclosure, the algorithm for extracting the heart rate signal may include at least one of a green method and a distance PPG. The green method may extract the heart rate signal by detecting a slight color change of the skin using the characteristics of the green channel, and the distance PPG may perform correction on the predetermined heart rate signal even if the distance between the camera and the target changes. Thereafter, the remote heart rate signal measurement module 120 may measure the remote heart rate by performing frequency analysis on the extracted remote heart rate signal to determine the maximum frequency component.

[0059] The quality score calculation module 130 may calculate a first quality score of the first frequency band image measured by the remote heart rate signal measurement module 120, and calculate a second quality score of the second frequency band image. The quality score may be used to select a valid heart rate segment to improve the reliability of heart rate acquisition, the valid heart rate segment being determined as a segment suitable for analysis in the remote heart rate signal measured in the acquired image.

[0060] In some exemplary embodiments of the present disclosure, the first quality score as an index for measuring noise generated due to the movement of the head of the vehicle driver or passenger or the movement of muscles in the face due to conversation or expression may include a first movement quality score. For example, the first movement quality score may be determined by measuring the change of facial feature points in frames adjacent to each other relative to the first frequency band image obtained by the image acquisition module 110, and the movement intensity may be measured by using the first movement quality score. That is, the quality score calculation module 130 may measure the change of facial feature points in adjacent frames to quantitatively measure the facial movement generated due to expressions and conversations that become noise elements when measuring the remote heart rate. In some exemplary embodiments of the present disclosure, the second quality score may include a second movement quality score determined relative to the second frequency band image in a scheme substantially the same as the first movement quality score.

[0061] In some exemplary embodiments of the present disclosure, the first quality score as an index for measuring noise due to illumination variation may include a first illumination quality score. For example, the first illumination quality score may be determined by measuring the amount of brightness variation (i.e., the illumination difference changed in a time series relative to the first frequency band image obtained by the image acquisition module 110). The quality score calculation module 130 extracts the input brightness information through the Y value in the YCbCr color space in the case of the RGB camera 20, or extracts the input brightness information from the single channel image in the case of the NIR camera 21, and uses the extracted brightness information as a quality score to quantitatively provide the validity of the extracted heart rate signal. In some exemplary embodiments of the present disclosure, the second quality score may include a second illumination quality score determined relative to the second frequency band image in a scheme substantially the same as the first illumination quality score.

[0062] In some exemplary embodiments of the present disclosure, the first quality score may include a first signal quality score for measuring the quality of the heart rate signal by using a frequency analysis and a signal-to-noise ratio (SNR) index of the remote heart rate signal. For example, the first signal quality score may be determined by measuring the signal quality based on the spectral characteristics possessed by the remote heart rate signal relative to the first frequency band image obtained by the image acquisition module 110. The heart rate cycle has a feature in which the beats per minute (BPM) change gradually changes in a time series, and for example, over a 10-second time series, the BPM change may be 6 BPM or less. This may indicate that the intensity of the frequency component corresponding to the heart rate signal is higher than that of other frequency components, and indicates that the frequency power of the heart rate band is high in the spectrum. Therefore, the remote heart rate bandwidth is regarded as a signal, and the remaining bandwidth is regarded as noise to evaluate the remote heart rate signal quality. In some exemplary embodiments of the present disclosure, the second quality score may include a second signal quality score determined relative to the second frequency band image in a scheme substantially the same as the first signal quality score.

[0063] The first valid heart rate zone selection module 140 may select the first valid heart rate zone based on the first remote heart rate signal extracted by the remote heart rate signal measurement module 120 and the first quality score determined by the quality score calculation module 130 .

[0064] In some exemplary embodiments of the present disclosure, the first effective heart rate zone selection module 140 may be configured to predict an error value from a model trained by the first quality score, and select the first effective heart rate zone according to the predicted error value.

[0065] The first effective heart rate segment selection module 140 includes a segment having a prediction error value less than or equal to a predetermined first threshold value in the first effective heart rate segment to determine the heart rate using the segment, and does not include a segment having a prediction error value greater than the predetermined first threshold value in the first effective heart rate segment and does not use the segment to determine the heart rate. Here, the first threshold value may be determined as an error value corresponding to the highest x% (x is a positive real number), in which the error value of the result predicted from the model trained by the first quality score is the smallest. For example, the first threshold value may be determined as an error value corresponding to the highest 10%, in which the error value of the result predicted from the model trained by the first quality score is the smallest, and the first effective heart rate segment selection module 140 may select the heart rate signal including the highest 10% reliability as the first effective heart rate segment.

[0066] The model trained by the first quality score can adopt a model that is easy to learn time series data, for example, at least one of a long short-term memory (LSTM) model, a random forest model, and a singular value decomposition (SVD) model. LSTM, as a recursive neural network (RNN), can learn the long-term dependencies of time series data, and the random forest model can be used for multiple decision trees to learn different parts of the data and for feature selection or for nonlinear pattern modeling of time series data. SVD can maintain important information while reducing the dimension of complex data by metric decomposition, so that SVD can be used for separate principal components of time series data. In some exemplary embodiments of the present disclosure, LSTM can be used to model the dependencies of time series data, and in some other exemplary embodiments of the present disclosure, a random forest model or an SVD model can be used to determine the main features of the time series data.

[0067] In some exemplary embodiments of the present disclosure, model training may be performed to input the above-mentioned first remote heart rate signal, the second remote heart rate signal, the movement quality score, the illumination quality score, and the signal quality score, and predict the error. Here, the error as the difference between the correct answer heart rate and the predicted heart rate may represent the average noise generated by the movement of the muscles in the face due to conversation or expression, the noise generated due to the change in illumination, and the signal noise for the heart rate signal. At the same time, the above-mentioned first threshold may represent the highest 10% error with the lowest error value in all data. When the first remote heart rate signal, the second remote heart rate signal, the movement quality score, the illumination quality score, and the signal quality score are provided as inputs to the model on which the training is completed, the error prediction of the model may be performed, and hereinafter, when the prediction error is less than or equal to the first threshold, the predicted heart rate value may be stored, and when the prediction error is greater than the first threshold, the predicted heart rate value may be discarded. The segment finally stored in the entire image by repeating such a process may eventually correspond to the first valid heart rate segment.

[0068] The second valid heart rate zone selection module 150 may select the second valid heart rate zone based on the second remote heart rate signal extracted by the remote heart rate signal measurement module 120 and the second quality score determined by the quality score calculation module 130 .

[0069] In some exemplary embodiments of the present disclosure, the second effective heart rate segment selection module 150 may be configured to predict an error value from a model trained by a second quality score, and select a second effective heart rate segment according to the predicted error value. The second effective heart rate segment selection module 150 includes a segment having a predicted error value less than or equal to a predetermined second threshold in the second effective heart rate segment to determine the heart rate using the segment, and excludes a segment having a predicted error value greater than a predetermined second threshold from the second effective heart rate segment without using the segment to determine the heart rate. Here, the second threshold may be determined as an error value corresponding to the highest y% (y is a positive real number), in which the error value of the result predicted from the model trained by the second quality score is the smallest. In addition, the matters of the first effective heart rate segment selection module 140 described above are referred to and applied to the matters of the second effective heart rate segment selection module 150, and therefore redundant descriptions will be omitted.

[0070] The complementary heart rate determination module 160 may be configured to determine a complementary heart rate based on the first effective heart rate segment and the second effective heart rate segment. Here, the complementary heart rate may refer to a heart rate whose accuracy is increased by considering the mutual characteristics of the remote heart rate signals for different wavelength bands. Therefore, compared with the case where the heart rate is determined for a single frequency band, the two frequency bands are used in a complementary manner, so that a segment in which the performance of the obtained signal is degraded due to noise in each frequency band can be replaced by another frequency band-based segment with excellent performance, and the time width of the extracted heart rate can be expanded.

[0071] In some exemplary embodiments of the present disclosure, when a portion of a first valid heart rate segment overlaps with a portion of a second valid heart rate segment, and the predicted heart rates in the overlapping segments are different from each other, the complementary heart rate determination module 160 may be configured to determine the complementary heart rate by using a predetermined weight.

[0072] Figure 2 is a flowchart for describing the operation of an apparatus for obtaining a heart rate according to an exemplary embodiment of the present disclosure.

[0073] Reference Figure 2 According to various exemplary embodiments of the present disclosure, the heart rate acquisition device may acquire a visible light image in step S201, and measure a remote heart rate signal based on visible light in step S203. Meanwhile, the heart rate acquisition device may obtain a movement quality score, obtain an illumination quality score, and obtain a signal quality score in steps S205, S207, and S209, respectively. Thereafter, the heart rate acquisition device may select a valid heart rate segment in step S211 and evaluate the reliability of the heart rate signal based on the obtained quality scores.

[0074] At the same time, the heart rate obtaining device can obtain an infrared image in step S221 and measure an infrared-based remote heart rate signal in step S223. At the same time, the heart rate obtaining device can obtain a movement quality score, obtain a lighting quality score, and obtain a signal quality score in steps S225, S227, and S229, respectively. Thereafter, the heart rate obtaining device can select a valid heart rate segment in step S231 and evaluate the reliability of the heart rate signal based on the obtained quality scores.

[0075] Thereafter, the heart rate obtaining device may be configured to determine the complementary heart rate based on the reliability of the visible light and infrared signals in step S241. The description described in this specification may refer to or be applied to more specific contents for the operation of the heart rate obtaining device, so redundant description will be omitted here.

[0076] Figures 3 to 8 is a diagram for describing an exemplary embodiment of an apparatus for determining a heart rate according to an exemplary embodiment of the present disclosure.

[0077] refer to Figure 3 In a heart rate acquisition device according to an exemplary embodiment of the present disclosure, the following can be set in a position to measure the front angle of a vehicle driver without interfering with driving: an RGB camera configured to acquire visible light images; an NIR camera configured to acquire near-infrared images; a 940nm infrared lighting machine; and a 940nm bandpass filter.

[0078] The heart rate measured by using the element as described above can be applied to the environment in the vehicle being driven. When the driver drives the vehicle, the camera can obtain images of the driver in the visible light and infrared bands. Thereafter, rPPG can be obtained in the two images, and the heart rate with high reliability can be extracted by the method according to the above exemplary embodiment. When the heart rate is less than or equal to the reference value, or when the heart rate shows an incomplete pattern, there may be a problem with the driver's condition. At this time, measures for sending a warning or an emergency help request to the outside can be taken. As an exemplary embodiment of the present disclosure, due to changes in heart rate, situations such as extreme excitement and fatigue driving of the driver can also be inferred. When the driver is driving fatigued, the driver can use the self-audio system in the vehicle or use the lighting system. In addition, even when the driver is in an extremely excited state, a lighting system or audio system for reducing the driver's tension can be similarly used. As another example, an emergency help letter can be automatically transmitted for situations where an extremely excited state is caused by a problem of dangerous invasion of the driver.

[0079] Reference Figure 4 , in some exemplary embodiments of the present disclosure, a mobile mass fraction (M) including a first mobile mass fraction and a second mobile mass fraction k) can be determined by using the following equation (1):

[0080]

[0081] Here, k may denote a number indicating a region of interest (ROI), n may denote an index indicating a vertex of a square ROI, b may denote a current frame, and b′ may denote a previous frame.

[0082] exist Figure 4 , k includes values ​​from 1 to 22.

[0083] Reference Figure 5 In some exemplary embodiments of the present disclosure, the lighting quality score including the first lighting quality score and the second lighting quality score may be determined by using the following equations (2), (3) and (4):

[0084]

[0085] I k =Max(X k )-Min(X k ) (3)

[0086] X k ={X 1 ,X 2 ,…,X n} (4)

[0087] Here, k may represent the sum of the number of divided regions and the total number of regions, n may represent the number of time series data, X may represent a brightness value, and represents the average value of X. For example, when the image acquired by the image acquisition module 110 is divided into 3×3=9 divided regions, a total of 10 ROIs may be determined including the 9 divided regions and the entire region indicating the face region in the image.

[0088] The quality score calculation module 130 may be configured to determine a standard deviation σ for the brightness variation over the time series from a total of 10 ROIs. k The quality fraction of the difference Ik between the maximum and minimum values ​​of the brightness change. Figure 6 In some exemplary embodiments of the present disclosure, the signal quality score including the first signal quality score and the second signal quality score may be determined by using the following equation (5):

[0089]

[0090] Here, U t (f) can represent a template window binary function, and The power spectral density function may be represented as a template window binary function as a function for cutting and separating, and a signal in a specific portion of the analysis window in signal processing is implemented so that the portion corresponding to the window includes a value of 1 and the other portion includes a value of 0. The power spectral density function is a function indicating how large the energy of the frequency characteristics of the signal analysis signal is in each frequency.

[0091] Reference Figure 7 , shows an implementation example of determining the reliability-based complementary heart rate of visible light and infrared signals. The final complementary heart rate determination can be based on the quality prediction value of each input, and the quality prediction can be modeled by learning the error between the remote heart rate and the actual heart rate. The final weight can indicate the reliability of each measurement and can be obtained by W RGB and W NIR To determine the complementary heart rate. The equation for the final heart rate (BPM) by summing the weights based on quality can be as follows.

[0092] Input: E RGB , E NIR

[0093] Quality = {1-MINMAX(E RGB ),1-MINMAX(E NIR )}

[0094] Quality = {Q RGB ,Q NIR}

[0095] set up Softmax(quality) = {W RGB ,W NIR}

[0096]

[0097] Here, E RGB The error prediction of the model for visible light measurements, E NIR The error prediction of the model for infrared measurements can be expressed as RGB The determination result of the visible light measurement weight can be expressed as NIRmay represent the result of determining the infrared measurement weights. MINMAX(x) may represent the minimum-maximum normalization defined by the error prediction measured from the data set, Softmax(x) may represent a function that converts the input vector x into a probability distribution by taking the exponent of each element of the input vector x and normalizing them using the exponents of all elements and thereby ensuring that the output values ​​are non-negative and sum to 1, BPM may represent the final predicted heart rate based on the supplementary rPPG measurement, and NONE may represent a measurement filtered by a quality-based selection method, i.e., an unreliable measurement.

[0098] The prediction error of the model is E RGB and E NIR It can be max-min normalized based on the data set, and since the error is inversely proportional to the quality, the quality value can be determined by subtracting the normalized error value from 1.0.

[0099] The top 10% can be selected based on the data set, and the weight W can be obtained by the selected results RGB and W NIR , and the final heart rate BPM can be determined. Here, when the final heart rate is NONE, determining the final heart rate is unreliable and the final heart rate can be excluded from the measurement.

[0100] (Table 1)

[0101]

[0102] (Table 2)

[0103]

[0104] Through Tables 1 and 2, performance enhancement can be confirmed when the supplementary rPPG measurement technology is applied. A data set for the experiment can be obtained, which includes movement in an indoor environment of 20 people. A total of 80 data can be obtained, and the experimental results measured in facial videos of approximately 2 hours and 40 minutes with a measurement time of 2 minutes for each case are organized by Table 2. When the actual supplementary rPPG measurement method is applied, higher accuracy than single sensor measurement can be confirmed. At this time, the accuracy enhancement width is greater in "motion" with a lot of noise. The current result can be a result corresponding to the purpose of the quality-based selection method, which aims to learn and infer an environment with a lot of noise through an error prediction model, and make inferences including actual high quality included in the measurement to enhance accuracy.

[0105] Reference Figure 8, R1 represents the first effective heart rate segment caused by the RGB camera, and R2 represents the second effective heart rate segment caused by the infrared camera. In addition, R3 represents the complementary heart rate segment determined by the complementary heart rate determination module 160 based on the first effective heart rate segment and the second effective heart rate segment. It can be seen that in the area represented by A, the common effective heart rate segment of R1 and R2 is included in R3, and in the area represented by B, the corresponding segment does not correspond to the effective heart rate segment in R1, but is supplemented by R2 and included in R3 as an effective heart rate segment. In addition, it can be seen that in the area represented by C, the corresponding segment does not correspond to the effective heart rate segment in R2, but is supplemented by R1 and included in R3 as an effective heart rate segment.

[0106] Fig. 9 is a flowchart for describing a method for determining a heart rate according to an exemplary embodiment of the present invention.

[0107] Reference Fig. 9 According to various exemplary embodiments of the present disclosure, the heart rate acquisition method may include: acquiring a first frequency band image and a second frequency band image of different frequency bands (S901); measuring a first remote heart rate signal of the first frequency band image and a second remote heart rate signal of the second frequency band image respectively (S902), calculating a first quality score of the first frequency band image and a second quality score of the second frequency band image (S903), selecting a first effective heart rate segment based on the first remote heart rate signal and the first quality score (S904), selecting a second effective heart rate segment based on the second remote heart rate signal and the second quality score (S905), and determining a complementary heart rate based on the first effective heart rate segment and the second effective heart rate segment (S906). The description described in this specification may refer to or be applied to more specific contents of the method, so redundant descriptions will be omitted here.

[0108] Fig.10 is a diagram for describing a computing device according to an exemplary embodiment of the present disclosure.

[0109] Reference Fig.10 , the device and method for obtaining the heart rate according to the exemplary embodiment of the present disclosure may be implemented by using the computing device 50 .

[0110] The computing device 50 may include at least one of a processor 510, a memory 530, a user interface input device 540, a user interface output device 550, and a storage device 560 that communicate with each other through a bus 520. The computing device 50 may also include a network interface 570 electrically connected to the network 40. The network interface 570 may send or receive a signal to or from another entity through the network 40.

[0111] The processor 510 may be implemented as various types including a microcontroller unit (MCU), an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and a quantum processing unit (QPU), and may be any semiconductor device that executes instructions stored in the memory 530 or the storage device 560. The processor 510 may be configured to implement the same Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 and Fig. 9 Related functions and methods.

[0112] The memory 530 and the storage device 560 may be various types of volatile or non-volatile storage media. For example, the memory may include a read-only memory (ROM) 531 and a random access memory (RAM) 532. In an exemplary embodiment of the present disclosure, the memory 530 may be located inside or outside the processor 510 and connected to the processor 510 through various well-known means.

[0113] In some exemplary embodiments of the present disclosure, at least some components or functions of the apparatus and method for obtaining heart rate according to the exemplary embodiments of the present disclosure may be implemented as a program or software executed by the computing device 50, or the program or software may be stored in a computer-readable medium. The computer-readable medium according to various exemplary embodiments of the present disclosure may record a program for executing the steps included in the apparatus and method for obtaining heart rate according to the exemplary embodiment in a computer including a processor 510 that executes a program or instruction stored in a memory 530 or a storage device 560.

[0114] In some exemplary embodiments of the present disclosure, at least some components or functions of the apparatus and method for obtaining heart rate according to exemplary embodiments of the present disclosure may be implemented by hardware or circuits of the computing device 50, or may also be implemented as separate hardware or circuits that may be electrically connected to the computing device 50.

[0115] According to an exemplary embodiment of the present disclosure, a remote heart rate with high accuracy and reliability can be determined for a driver or passenger in a vehicle even with various lighting changes in the vehicle driving environment (including sunlight, traffic lights or transport traffic lights) or noise such as movement of the driver or passenger's face.

[0116] In various exemplary embodiments of the present disclosure, each of the operations described above may be performed by a control device, and the control device may be configured by a plurality of control devices or an integrated single control device.

[0117] In various exemplary embodiments of the present disclosure, the memory and the processor may be provided as one chip, or provided as separate chips.

[0118] In different exemplary embodiments of the present disclosure, the scope of the present disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling operations of methods according to different embodiments to be executed on a device or computer, and non-transitory computer-readable media including such software or commands stored thereon and executable on a device or computer.

[0119] In various exemplary embodiments of the present disclosure, the control device may be implemented in the form of hardware or software, or may be implemented in a combination of hardware and software.

[0120] Furthermore, terms such as “unit”, “module” and the like included in the specification mean a unit for processing at least one function or operation, which can be implemented by hardware, software or a combination thereof.

[0121] In the flowcharts described with reference to the accompanying drawings, the flowcharts may be executed by a controller or a processor. The order of operations in the flowcharts may be changed, multiple operations may be combined, or any operation may be divided, and specific operations may not be performed. In addition, the operations in the flowcharts may be performed sequentially, but not necessarily sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0122] Hereinafter, the fact that a plurality of hardwares are operably coupled may include the fact that a direct and / or indirect connection between the plurality of hardwares is established by wire and / or wirelessly.

[0123] In an exemplary embodiment of the present disclosure, a vehicle may be referred to as being based on a concept including various vehicles. In some cases, a vehicle may be interpreted as being based on not only various land vehicles (such as cars, motorcycles, trucks, and buses) traveling on roads but also various vehicles such as airplanes, drones, ships, etc.

[0124] For ease of explanation and accurate definition of the appended claims, the terms "up", "down", "inside", "outside", "upward", "downward", "upward", "downward", "front", "rear", "rear", "inside", "outside", "inward", "outward", "inner", "external", "inner", "outside", "forward", and "rearward" are used to describe features of the exemplary embodiments with reference to the locations of such features as shown in the drawings. It should be further understood that the term "connected" or its derivatives refer to both direct and indirect connections.

[0125] The term "and / or" may include a combination of multiple related listed items or any one of the multiple related listed items. For example, "A and / or B" includes all three cases, such as "A", "B" and "A and B".

[0126] In an exemplary embodiment of the present invention, “at least one of A and B” may refer to “at least one of A or B” or “at least one of a combination of at least one of A and B”. Furthermore, “one or more of A and B” may refer to “one or more of A or B” or “one or more of a combination of one or more of A and B”.

[0127] In this specification, unless otherwise stated, a singular expression includes a plural expression unless the context clearly indicates otherwise.

[0128] In the exemplary embodiments of the present disclosure, it should be understood that terms such as “including” or “having” are intended to specify the presence of features, quantities, steps, operations, elements, parts, or a combination thereof described in the specification, and do not exclude the possibility of adding or existing one or more other features, quantities, steps, operations, elements, parts, or a combination thereof.

[0129] According to the exemplary embodiments of the present invention, components may be combined with each other to be implemented as one, or some components may be omitted.

[0130] For the purpose of illustration and description, the foregoing descriptions of specific exemplary embodiments of the present disclosure have been presented. They are not intended to be exhaustive or to limit the present disclosure to the precise form disclosed, and it is apparent that many modifications and variations are possible in light of the above teachings. In order to illustrate certain principles of the present invention and their practical applications, exemplary embodiments have been selected and described to enable other persons skilled in the art to carry out and utilize various exemplary embodiments of the present disclosure and various substitutions and modifications thereof. The scope of the present disclosure is intended to be limited by the appended claims and their equivalents.

Claims

1. A device for obtaining a heart rate, the device comprising: one or more processors; as well as one or more memory devices operatively connected to the one or more processors, wherein the one or more memory devices include program code, and wherein the program code is executed by the one or more processors to: Obtain a first frequency band image and a second frequency band image of different frequency bands, respectively measuring a first remote heart rate signal of the first frequency band image and a second remote heart rate signal of the second frequency band image, respectively determining a first quality score of the first frequency band image and a second quality score of the second frequency band image, selecting a first valid heart rate zone based on the first remote heart rate signal and the first quality score, selecting a second valid heart rate zone based on the second remote heart rate signal and the second quality score; and A complementary heart rate is determined based on the first effective heart rate zone and the second effective heart rate zone.

2. The device according to claim 1, in, In determining the first quality score and the second quality score, the one or more processors are configured to determine a first movement quality score and a second movement quality score, wherein the one or more processors are configured to determine the first motion quality score by measuring changes in facial feature points in adjacent frames relative to the first frequency band image, and The one or more processors are configured to determine the second motion quality score by measuring changes in facial feature points in adjacent frames relative to the second frequency band image.

3. The device according to claim 1, in, In determining the first quality score and the second quality score, the one or more processors are configured to determine a first lighting quality score and a second lighting quality score, wherein the one or more processors are configured to determine the first lighting quality score by measuring a brightness change relative to the first frequency band image, and wherein the one or more processors are configured to determine the second lighting quality score by measuring a brightness change relative to the second frequency band image.

4. The device according to claim 1, in, In determining the first quality score and the second quality score, the one or more processors are configured to determine a first signal quality score and a second signal quality score, wherein the one or more processors are configured to determine the first signal quality score by measuring signal quality based on spectral characteristics of the remote heart rate signal relative to the first frequency band image, and The one or more processors are configured to determine the second signal quality score by measuring the signal quality based on frequency spectrum characteristics of the remote heart rate signal relative to the second frequency band image.

5. The device according to claim 1, wherein: When selecting the first valid heart rate zone, the one or more processors are configured to: Predicting error values ​​from the model trained by the first quality score, including a segment having a prediction error value less than or equal to a predetermined first threshold value in the first effective heart rate segment, and A segment in which the prediction error value is greater than the first threshold is not included in the first effective heart rate segment.

6. The device according to claim 5, wherein: When selecting the second valid heart rate zone, the one or more processors are configured to: predicting error values ​​from the model trained by the second quality score, so that a segment in which the prediction error value is less than or equal to a predetermined second threshold is included in the second effective heart rate segment, and A segment for which the prediction error value is greater than the second threshold is not included in the second effective heart rate segment.

7. The device according to claim 6, in, determining the first threshold to correspond to a top x% of error values ​​in which the error values ​​of the results predicted from the model trained with the first quality score are smallest, wherein the second threshold is determined to correspond to the highest y% of error values ​​in which the error value of the result predicted from the model trained by the second quality score is the smallest, and Where x is a positive real number and y is a positive real number.

8. The device according to claim 6, wherein: The model includes at least one of a long short-term memory model, a random forest model, and a singular value decomposition model.

9. The device according to claim 1, wherein: When determining the complementary heart rate, the one or more processors are configured to determine the complementary heart rate by using a predetermined weight in response to a partial segment of the first valid heart rate segment and a partial segment of the second valid heart rate segment overlapping and the predicted heart rates in the overlapping segments being different from each other.

10. The device according to claim 1, wherein: The first frequency band image includes a visible light image, and the second frequency band image includes an infrared image.

11. A device for obtaining heart rate, the device comprising: one or more processors; as well as one or more memory devices operatively connected to the one or more processors, wherein the one or more memory devices include program code, and wherein the program code is executed by the one or more processors to: Obtain a first frequency band image and a second frequency band image of different frequency bands, respectively measuring a first remote heart rate signal of the first frequency band image and a second remote heart rate signal of the second frequency band image, calculating a motion quality score, an illumination quality score, and a signal quality score for each of the first frequency band image and the second frequency band image, selecting a first valid heart rate zone based on the first remote heart rate signal and a first quality score, selecting a second valid heart rate zone based on the second remote heart rate signal and the second quality score; and A complementary heart rate is determined based on the first effective heart rate zone and the second effective heart rate zone.

12. The device according to claim 11, in, The one or more processors are configured to determine the movement quality score by using the following equation (1): Here, k represents a number indicating an interest region, n represents an index indicating a vertex of a square interest region, b represents a current frame, and b′ represents a previous frame.

13. The device according to claim 11, in, The one or more processors are configured to determine the lighting quality score by using the following equations (2), (3) and (4): I k =Max(X k )-Min(X k ) (3) X k = {X1, X2, ..., X n } (4), and Wherein, k represents the sum of the number of divided regions and the total number of regions, and n represents the number of time series data, X represents the brightness value, and Represents the average value of X.

14. The device according to claim 11, wherein: The one or more processors are configured to determine the signal quality score by using the following equation (5): Among them, U t (f) represents the template window binary function, represents the power spectral density function.

15. The device according to claim 11, wherein When selecting the first valid heart rate zone, the one or more processors are configured to: Predicting error values ​​from the model trained by the first quality score, including a segment having a prediction error value less than or equal to a predetermined first threshold in the first effective heart rate segment, and A segment in which the prediction error value is greater than the first threshold is not included in the first effective heart rate segment.

16. The device according to claim 15, wherein: When selecting the second valid heart rate zone, the one or more processors are configured to: The prediction error value of the model trained by the second quality score is used to include a segment in which the prediction error value is less than or equal to a predetermined second threshold value in the second effective heart rate segment, and A segment in which the prediction error value is greater than the second threshold is not included in the second effective heart rate segment.

17. A method for obtaining a heart rate, the method comprising: A processor obtains a first frequency band image and a second frequency band image of different frequency bands; The processor measures, respectively, a first remote heart rate signal of the first frequency band image and a second remote heart rate signal of the second frequency band image; Determining, by the processor, a first quality score of the first frequency band image and a second quality score of the second frequency band image respectively; selecting, by the processor, a first valid heart rate zone based on the first remote heart rate signal and the first quality score; selecting, by the processor, a second valid heart rate zone based on the second remote heart rate signal and the second quality score; as well as A complementary heart rate is determined by the processor based on the first effective heart rate zone and the second effective heart rate zone.

18. The method according to claim 17, in, Determining the first mass score and the second mass score includes determining a first movement mass score and a second movement mass score, The first motion quality score is determined by the processor by measuring changes in facial feature points in adjacent frames relative to the first frequency band image, and the second motion quality score is determined by the processor by measuring changes in facial feature points in adjacent frames relative to the second frequency band image.

19. The method according to claim 17, in, Determining the first quality score and the second quality score includes determining a first lighting quality score and a second lighting quality score, wherein the processor determines the first lighting quality score by measuring a brightness change relative to the first frequency band image, and The processor determines the second lighting quality score by measuring a brightness change relative to the second frequency band image.

20. The method according to claim 17, in, Determining the first quality score and the second quality score includes determining a first signal quality score and a second signal quality score, wherein the processor determines the first signal quality score by measuring signal quality based on frequency spectrum characteristics of the remote heart rate signal relative to the first frequency band image, and The processor determines the second signal quality score by measuring the signal quality based on the frequency spectrum characteristics of the remote heart rate signal relative to the second frequency band image.

Images