Heart rate estimation method, heart rate estimation device, program, and electronic device

Abstract

The present invention relates to a method and device for estimating a heart rate, and an electronic device to which the same is applied. The heart rate estimation method includes step S100 of acquiring a face video, step S102 of performing face detection on the face video and extracting a local face region set for estimating the heart rate, step S104 of performing a first process on values ​​of pixel points in the local face region to acquire an initial heart rate signal, step S106 of performing a time domain-to-frequency domain transformation on the initial heart rate signal to acquire a frequency domain signal, and step S110 of performing a second process on the frequency domain signal within a heart rate estimation range to acquire an estimated heart rate.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to a Chinese patent application entitled "Heart Rate Estimation Method, Apparatus, and Electronic Device Applying the Same," filed with the State Intellectual Property Office of the People's Republic of China on November 25, 2019, application number 201911166001.2, the entire contents of which are incorporated herein by reference. [Technical Field]

[0002] The present invention relates to computer vision technology, in particular to a method for estimating heart rate, Heart rate estimation Device ,program, Reach Biden Regarding child devices. [Background technology]

[0003] Electrocardiography is a technique that uses an electrocardiograph to record the pattern of electrical activity that occurs during each cardiac cycle from the body surface. This method is highly accurate, but the equipment is expensive and requires specialist operation, making the device complicated and limiting its use.

[0004] Photoplethysmography (PPG) is a noninvasive detection method that uses photoelectric means to monitor changes in blood volume in biological tissue. When light of a certain wavelength is irradiated onto the skin surface, such as between the fingers, the light is transmitted or reflected by an optical receiver. The light is absorbed by the skin, muscle, blood, etc., and is attenuated throughout the process, resulting in a decrease in the intensity of the light reaching the optical receiver. While the attenuation effect of skin, muscle, etc. on light intensity is constant, the attenuation effect of blood in blood vessels on light exhibits pulsatile changes with the heartbeat. When the heart contracts, the blood volume in the blood vessels increases, increasing the absorbed light intensity and weakening the light intensity received by the optical receiver. The opposite occurs when the heart expands. This light intensity is converted into an electrical signal, and changes in blood flow can be detected through a volume pulse wave. This method requires the sensor to be fixed and in close contact with the human body, which places many limitations on how and when the user can use it. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present disclosure provide a method for estimating a heart rate that at least solves the technical problem in the prior art that a heart rate can only be estimated by a contact method. Heart rate estimation Device ,program, Reach Biden Provide child devices. [Means for solving the problem]

[0006] According to one aspect of an embodiment of the present invention, there is provided a method for estimating a heart rate, including the steps of acquiring a face video, performing face detection on the face video and extracting a local face region set for estimating a heart rate, performing a first process on values ​​of pixel points in the local face region to acquire an initial heart rate signal, performing a time domain-to-frequency domain transform on the initial heart rate signal to acquire a frequency domain signal, and performing a second process on the frequency domain signal within a heart rate estimation range to acquire an estimated value of the heart rate.

[0007] Preferably, an infrared camera is used to collect the facial video.

[0008] Preferably, the local facial region includes the region below both eyes.

[0009] Preferably, the method for estimating a heart rate further comprises the step of extracting the facial local region set for estimating a heart rate by combining face keypoint location with the face detection.

[0010] Preferably, the step of performing a first process on the values ​​of pixel points in the local facial region and obtaining an initial heart rate signal includes a step of performing a weighted average on the values ​​of pixel points in the local facial region according to weights, and setting the weighted average value as the luminance signal of the current frame, wherein the luminance signal of the current frame and the luminance signals of the previous N frames in history constitute the initial heart rate signal.

[0011] Preferably, the step of performing a time domain-to-frequency domain transform on the initial heart rate signal to obtain a frequency domain signal includes the step of transforming the initial heart rate signal from the time domain to the frequency domain by a fast Fourier transform to obtain the frequency domain signal.

[0012] Preferably, the weight is set based on the position of the pixel point within the local facial region, and the closer the pixel point is to a border position within the local facial region, the smaller the weight corresponding to the pixel point becomes.

[0013] Preferably, before performing a time domain-to-frequency domain transformation on the initial heart rate signal, the method for estimating a heart rate may further include a step of performing a first noise removal process on the initial heart rate signal in the time domain, the first noise removal process including at least one of Savitzky-Golay (SG) filtering, detrend filtering, moving average filtering, normalization processing, and bandpass filtering.

[0014] Preferably, the step of performing a second processing on the frequency domain signal within the heart rate estimation range to obtain an estimated heart rate includes the steps of: performing peak detection on the frequency domain signal within the heart rate estimation range to obtain peak values; sorting the peak values ​​to obtain sorted results; calculating a reliability based on the sorted results; and obtaining the estimated heart rate based on the reliability.

[0015] Preferably, the step of performing second processing on the frequency domain signal within the estimated range of the heart rate to obtain an estimated value of the heart rate includes the steps of: performing peak value detection on the frequency domain signal within the estimated range of the heart rate to obtain a highest peak value; calculating the energy of first and second harmonics corresponding to the dominant frequency, using the frequency of the highest peak value as the dominant frequency, to obtain a calculated energy value; dividing the calculated energy value by the energy of the remaining frequencies other than the dominant frequency to obtain a reliability; and obtaining the estimated value of the heart rate based on the reliability.

[0016] Preferably, the estimated range of the heart rate is set in advance.

[0017] Preferably, the heart rate estimation method further includes a step of determining whether the detection time exceeds a second threshold, and if the detection time exceeds the second threshold, a step of performing a third processing on the frequency domain signal to obtain an estimated range of the heart rate.

[0018] Preferably, the step of performing a third processing on the frequency domain signal to obtain an estimated range of heart rate includes the steps of: constantly selecting and buffering X consecutive frames or X seconds of frequency domain signal by a sliding window to obtain and buffer the heart rate values; then obtaining the estimated range of heart rate by a deep learning method; repeating the operation within M frames or M seconds; and averaging all the obtained estimated ranges of heart rate to obtain a final estimated range of heart rate.

[0019] Preferably, the peak values ​​are sorted in descending order using a traverse method, the highest peak value and the second peak value are selected as the sorting results, and the ratio of the highest peak value to the second peak value is used as the reliability.

[0020] Preferably, the step of obtaining an estimate of the heart rate based on the confidence level includes the steps of: comparing the confidence level with a first threshold to obtain a comparison result; if the confidence level is less than the first threshold, the comparison result indicates that the frequency domain signal is significantly disturbed by noise, and discarding the current result and detecting a next frame; if the confidence level is greater than or equal to the first threshold, the comparison result indicates that the frequency domain signal is not disturbed by noise or has little noise disturbance, and obtaining a frequency corresponding to the highest peak value as the estimate of the heart rate.

[0021] Preferably, before performing a second processing on the frequency domain signal, the heart rate estimation method further includes a step of performing a second noise removal processing on the frequency domain signal, the second noise removal processing including at least one of a discrete Fourier transform (DFT) and bandpass filtering.

[0022] According to another aspect of the present invention, there is provided a heart rate estimation device including: a capturing unit configured to capture a face video; a detection unit configured to perform face detection on the face video and extract a facial local region configured to estimate a heart rate; a first processing unit configured to perform a first processing on values ​​of pixel points in the facial local region to obtain an initial heart rate signal; a transformation unit configured to perform a time domain-to-frequency domain transformation on the initial heart rate signal to obtain a frequency domain signal; and a second processing unit configured to perform a second processing on the frequency domain signal within a heart rate estimation range to obtain an estimated heart rate. According to another aspect of the present invention, there is provided a storage medium storing a program that, when the program is executed, controls a device in which the storage medium is located to perform any one of the above heart rate estimation methods.

[0023] Preferably, the second processing unit includes: a peak detection module configured to perform peak detection on the frequency domain signal within an estimation range of a heart rate to obtain peak values; a sorting module configured to sort the peak values ​​to obtain sorting results; a confidence calculation module configured to obtain a confidence based on the sorting result; and an estimation module configured to obtain an estimation value of the heart rate based on the confidence.

[0024] Preferably, the second processing unit includes: a maximum peak value detection module configured to perform peak value detection on the frequency domain signal within an estimation range of the heart rate to obtain a maximum peak value; an energy calculation module configured to calculate energy of a first harmonic and a second harmonic corresponding to the main frequency, taking the frequency of the maximum peak value as a main frequency, to obtain an energy calculation value; a reliability calculation module configured to divide the calculated energy value by energy of remaining frequencies other than the main frequency to obtain a reliability; and an estimation module configured to obtain an estimation value of the heart rate based on the reliability.

[0025] Preferably, the heart rate estimation device further includes a determination module configured to determine whether the detection time exceeds a second threshold, and a third processing module configured to perform a third processing on the frequency domain signal to obtain an estimation range of the heart rate if the detection time exceeds the second threshold.

[0026] According to another aspect of an embodiment of the present invention, there is provided an electronic device further comprising: a processor; and a memory configured to store executable commands for the processor, the processor configured to execute the heart rate estimation method described in any one of the preceding claims by executing the executable commands.

[0027] In an embodiment of the present invention, the method includes the steps of acquiring a face video, performing face detection on the face video and extracting a local facial region set for estimating a heart rate, performing a first process on pixel points in the local facial region to acquire an initial heart rate signal, performing a time-domain-to-frequency domain transformation on the initial heart rate signal to acquire a frequency-domain signal, and performing a second process on the frequency-domain signal within a heart rate estimation range to acquire an estimated heart rate. This solves the technical problem in the prior art that heart rate can only be estimated by a contact method. [Brief explanation of the drawings]

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application, but the exemplary embodiments of the present invention and the description thereof are provided to explain the present invention and do not constitute an undue limitation on the present invention. [Figure 1] FIG. 1 is a flowchart of a first selectable heart rate estimation method according to an embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram of selectable extracted local facial regions according to an embodiment of the present invention. [Figure 3] FIG. 3 is a flowchart of a second selectable method for estimating heart rate according to an embodiment of the present invention. [Figure 4] FIG. 4 is an application scenario diagram of the heart rate estimation method provided according to an embodiment of the present invention. [Figure 5] FIG. 5 is a structural block diagram of a selectable heart rate estimation device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] In order to make those skilled in the art better understand the solutions of the present invention, the following clearly and completely describes the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention, but it should be obvious that the described embodiments are only some embodiments of the present invention, not all embodiments, and all other embodiments obtained by those skilled in the art based on the embodiments of the present invention without any creative efforts should fall within the protection scope of the present invention.

[0030] In addition, terms such as "first," "second," and the like in the specification and claims of the present invention and in the drawings are used to distinguish between similar objects and are not necessarily used to describe a particular order or chronology. It should be understood that the order used in this manner may be interchanged, under appropriate circumstances, to perform the embodiments of the present invention described herein in an order other than that illustrated or described herein. Furthermore, the terms "comprise" and "have," and any synonyms thereof, are intended to cover non-exclusive inclusions; for example, a process, method, system, product, or apparatus comprising a series of steps or units need not be limited to those explicitly recited steps or units, but may include other steps or units not explicitly recited or inherent to such process, method, product, or apparatus.

[0031] Embodiments of the present invention may be applied to electronic devices that have at least one imaging unit and may be smartphones, tablet computers, e-book readers, desktop computers, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs), medical devices, cameras or wearable devices (accessory-type devices such as watches, bracelets, glasses, headsets, etc.), electronic clothing, implantable chips, automotive electronic devices, etc.

[0032] The following is a flow chart of a selectable heart rate estimation method according to an embodiment of the present invention. It should be noted that the steps shown in the flow chart of the drawings may be performed in a computer system, such as a set of computers capable of executing commands. And, although a logical order is shown in the flow chart, in some circumstances the steps may be performed in a different order than shown or described herein.

[0033] 1 is a flowchart of a first selectable heart rate estimation method according to an embodiment of the present invention. As shown in FIG. 1, the heart rate estimation method includes steps S100, S102, S104, S106, and S110.

[0034] Step S100 is a step of acquiring a face-captured video.

[0035] In an alternative embodiment, this step involves capturing facial video using an imaging device (e.g., an RGB camera, an infrared camera, etc.), typically with a frame rate greater than 25 fps to maintain image continuity and facilitate heart rate monitoring. When using an infrared camera, infrared light is virtually invisible to the naked eye, so it does not affect human activity. Furthermore, using an infrared camera can avoid the effects of dim light and complex light beams, providing certain anti-interference and anti-obstruction capabilities. Furthermore, the infrared light source has a distinct wavelength band for transmission and reception, effectively eliminating the large amount of noise that is unavoidable when using a typical RGB camera, and further realizing active lighting. The infrared light source may be a light source built into the infrared camera or an external, independent light source. The imaging device may be an independent camera, or may be integrated with other cameras as a whole or part of one or more imaging modules, or may be installed independently, or attached to an electronic device in an embedded or external manner.

[0036] Step S102 is a step of performing face detection on the face-captured video and extracting a local face region set for estimating the heart rate.

[0037] In an alternative embodiment, this step can further extract a facial local region set for estimating the heart rate by combining facial keypoint location with face detection. By introducing facial keypoints, the facial local region set for estimating the heart rate can be extracted more accurately, and even if the face shakes, the estimated heart rate will not be deviated due to blurring caused by motion or facial deformation.

[0038] In an alternative embodiment, the local facial regions extracted in this step include the areas under the eyes, which have more venous distribution and are less likely to be occluded than other parts of the face, and thus extracting the areas under the eyes as local facial regions can improve the accuracy of the heart rate estimation method.

[0039] For example, Fig. 2 shows a schematic diagram of selectable extracted facial local regions according to an embodiment of the present invention, which combines facial keypoint location with face detection to extract facial local regions set for estimating heart rate. In Fig. 2, a rectangular frame 20 indicates a facial region determined by face detection, and an irregular mask frame 22 indicates a facial local region extracted by face keypoints.

[0040] Of course, those skilled in the art can further extract other areas of the face, such as the forehead, cheeks, etc., to be used as local facial regions, or, without requiring creative effort, those skilled in the art can extract other human body regions with many veins, such as the neck, wrist, etc., to be used as detection regions for estimating the heart rate.

[0041] Step S104 is a step of performing a first process on the values ​​of the pixel points in the local face region to obtain an initial heart rate signal.

[0042] In an alternative embodiment, this step includes a step of performing a weighted average of the values ​​of pixel points within the local facial region according to the weight, and setting the weighted average value as the luminance signal of the current frame, where the luminance signal of the current frame and the luminance signals of the previous N frames in the history constitute the initial heart rate signal, where N is an integer greater than or equal to 1. Here, the weight can be set based on the position of the pixel point within the local facial region, so that the closer the pixel point is to the edge position within the local facial region, the smaller the weight corresponding to that pixel point. The weight value may range from 0 to 1.

[0043] Step S106 is a step of performing a time domain-to-frequency domain transformation on the initial heart rate signal to obtain a frequency domain signal.

[0044] In an alternative embodiment, performing a time domain to frequency domain transform on the initial heart rate signal to obtain a frequency domain signal includes transforming the initial heart rate signal from the time domain to the frequency domain using a Fast Fourier Transform to obtain a frequency domain signal.

[0045] In an alternative embodiment, before performing a time-domain-to-frequency domain transformation on the initial heart rate signal, the heart rate estimation method may further include a step of performing a first noise reduction process on the initial heart rate signal in the time domain. Here, the first noise reduction process step includes at least one of Savitzky-Golay (SG) filtering, detrend filtering, moving average filtering, normalization, and bandpass filtering. SG (Savitzky-Golay) filtering and detrend filtering can be configured to reduce the influence of signal baseline shifts on the signal (e.g., removing the effects of ambient light). Moving average filtering can be configured to remove random noise in the signal. Normalization can facilitate signal processing and improve computational efficiency. Bandpass filtering can process signals in the normal heart rate band. The first noise reduction process can remove high-frequency noise, low-frequency noise, and noise introduced by other factors (e.g., movement), making the heart rate signal clearer and its periodicity more pronounced.

[0046] Step S110 is a step of performing a second process on the frequency domain signal within the estimated range of the heart rate to obtain an estimated value of the heart rate.

[0047] In an optional embodiment, the step of performing a second processing on the frequency domain signal to obtain an estimated heart rate includes step S1100 of performing peak detection on the frequency domain signal within an estimated range of the heart rate to obtain peak values, step S1102 of sorting the peak values ​​and obtaining a sorted result, step S1104 of obtaining a confidence level based on the sorted result, and step S1106 of obtaining an estimated heart rate based on the confidence level.

[0048] In an alternative embodiment, the peak values ​​are sorted in descending order by a traversal method, and the highest peak value and the second peak value are selected as the sorting results as necessary to obtain the sorting result. The ratio of the highest peak value to the second peak value is the reliability, i.e., reliability = highest peak value / second peak value.

[0049] In an optional embodiment, the step of performing a second processing on the frequency domain signal to obtain an estimated heart rate includes step S1110 of performing peak value detection on the frequency domain signal within the estimated heart rate range to obtain the highest peak value, step S1112 of calculating the energy of the main frequency and the first harmonic at the main frequency using the frequency of the highest peak value as the main frequency to obtain a calculated energy value, step S1114 of dividing the calculated energy value by the energy of the remaining frequencies other than the main frequency to obtain a reliability, and step S1116 of obtaining an estimated heart rate based on the reliability.

[0050] In an alternative embodiment, the estimated range of the heart rate can be preset, for example, 50 to 180 beats / minute based on the human heart rate limit.

[0051] In an alternative embodiment, obtaining an estimate of the heart rate based on the confidence level includes comparing the confidence level with a first threshold to obtain a comparison result; if the confidence level is less than the first threshold, the comparison result indicates that the frequency domain signal is significantly corrupted by noise, and discarding the current frequency domain signal and detecting the next frame; if the confidence level is greater than or equal to the first threshold, the comparison result indicates that the frequency domain signal is not corrupted by noise or has little noise corruption, and obtaining a frequency corresponding to the highest peak value as the estimate of the heart rate.

[0052] In an alternative embodiment, before performing a second processing on the frequency domain signal, the method for estimating a heart rate may further include a step of performing a second noise removal processing on the frequency domain signal. Here, the second noise removal processing method includes at least one of a discrete Fourier transform (DFT) and bandpass filtering. Bandpass filtering can process signals in a normal heart rate band. By performing the second noise removal processing on the frequency domain signal, it is possible to further emphasize required principal component signals.

[0053] In the heart rate estimation method provided in the above embodiment, if the heart rate estimation range is preset, the output value may be unstable due to an inappropriate range setting. In order to achieve a more accurate and stable heart rate estimation, the present invention further provides another heart rate estimation method with an automatic range estimation function. Please refer to Figure 3, which is a flowchart of a second selectable heart rate estimation method according to an embodiment of the present invention. As shown in FIG. 3, the heart rate estimation method includes step S300 of acquiring a face video, step S302 of performing face detection on the face video and extracting a local face region set for estimating the heart rate, step S304 of performing a first process on values ​​of pixel points in the local face region to acquire an initial heart rate signal, step S306 of performing a time domain-to-frequency domain transformation on the initial heart rate signal to acquire a frequency domain signal, step S307 of determining whether the detection time exceeds a second threshold, step S308 of performing a third process on the frequency domain signal to acquire an estimated heart rate range, and step S310 of performing a second process on the frequency domain signal within the estimated heart rate range to acquire an estimated heart rate value.

[0054] In the heart rate estimation method provided by this embodiment, steps S300, S302, S304, S306, and S310 correspond to steps S100, S102, S104, S106, and S110 in the first embodiment, respectively, and are not described here. The difference is that the heart rate estimation method provided by this embodiment further includes steps S307 and S308, which are described in detail below.

[0055] In step S307, the second threshold can be set to M frames or M seconds, where M is an integer greater than or equal to 1. The detection time is measured and compared with the second threshold to determine whether the detection time exceeds the second threshold. If the detection time exceeds the second threshold, the process jumps from step S307 to step S308, where a third processing is performed on the frequency domain signal to obtain an estimated range of the heart rate. Here, the step of performing the third processing on the frequency domain signal to obtain an estimated range of the heart rate may include: continuously selecting and buffering X consecutive frames or X seconds of the frequency domain signal using a sliding window to obtain and buffer the heart rate values, then obtaining the estimated range of the heart rate using a deep learning method, repeating this operation within M frames or M seconds, and averaging all the estimated ranges of the heart rate to obtain a final estimated range of the heart rate. If the detection time does not exceed the second threshold, the process jumps to step S310, where a second processing is performed on the frequency domain signal within the estimated range of the heart rate to obtain an estimated value of the heart rate.

[0056] The heart rate estimation method provided by the embodiments of the present invention can be used not only to detect heart rate estimation, but also to obtain cardiovascular parameters such as respiratory rate, transcutaneous arterial oxygen saturation (SpO2), and blood pressure, etc., to monitor physical conditions. This heart rate estimation method can be applied to various mobile platforms, vehicle-mounted chips, embedded chips, etc., does not require large and complex hardware devices, the detection process is simple and fast, does not contact the human body and is harmless, and at the same time has sufficient accuracy, solving the problems of traditional contact-type detection methods that rely on complex hardware devices and require contact with the human body, and significantly improving the robustness and application range of the heart rate estimation method.

[0057] As shown in Figure 4, the heart rate estimation method according to the embodiment of the present invention can be used to monitor a driver's physical condition, thereby preventing the problem of drivers wearing conventional contact-type heart rate devices affecting their driving. As can be seen from the detection results shown in Figure 4, the estimated heart rate obtained using the heart rate estimation method is close to the actual heart rate, demonstrating high accuracy. Accurate heart rate estimation can also be achieved even under low light conditions, such as at night, in a tunnel, or backlit conditions. The method can continuously track and assess the driver's physical condition, and if an abnormality in the driver's heart rate is detected, an alarm can be issued and a driving support function can be activated. Furthermore, as will be appreciated by those skilled in the art, the heart rate estimation method according to the embodiment of the present invention can also be used for monitoring heart rate during sleep, etc.

[0058] According to another aspect of the embodiment of the present invention, there is further provided a heart rate estimation device. Fig. 5 is a structural block diagram of a selectable heart rate estimation device according to the embodiment of the present invention. As shown in Fig. 5, the heart rate estimation device 50 includes an imaging unit 500, a detection unit 502, a first processing unit 504, a conversion unit 506, and a second processing unit 510.

[0059] Each unit included in the heart rate estimation device 50 will be specifically described below.

[0060] The imaging unit 500 is configured to capture facial video.

[0061] In alternative embodiments, the imaging unit 500 may be an RGB camera, an infrared camera, or the like, and typically has a frame rate greater than 25 fps to maintain image continuity and facilitate heart rate monitoring. When using an infrared camera, infrared light is barely visible to the naked eye, so it does not affect human activity. Furthermore, using an infrared camera can avoid the effects of dim light and complex light beams, providing certain anti-interference and anti-obstruction capabilities. Furthermore, the infrared light source has a distinct wavelength band for transmission and reception, effectively eliminating the large amount of noise that is unavoidable when using a typical RGB camera, and further realizing active lighting. The infrared light source may be a light source built into the infrared camera or an external, independent light source. The imaging unit 500 may be an independent camera, or may be integrated with other cameras as one or more imaging modules, or may be installed independently or attached to electronic devices in a built-in or external manner.

[0062] The detection unit 502 is configured to perform face detection on the facial video and extract a facial local region configured to estimate the heart rate.

[0063] In an alternative embodiment, the detection unit 502 not only includes a face detection module 5020 configured to detect faces, but also may include a face keypoint location module 5022 configured to extract facial local regions configured for estimating heart rate in combination with the face detection module 5020. The introduction of the face keypoint location module 5022 can more accurately extract facial local regions configured for estimating heart rate, and even if the face shakes, the estimated heart rate will not be deviated due to blurring caused by motion or facial deformation.

[0064] In an alternative embodiment, the local facial regions extracted in this step include the areas under the eyes, which have more venous distribution and are less likely to be occluded than other parts of the face, and thus extracting the areas under the eyes as local facial regions can improve the accuracy of the heart rate estimation method.

[0065] The first processing unit 504 is configured to perform a first processing on the values ​​of the pixel points in the face local region to obtain an initial heart rate signal.

[0066] In an alternative embodiment, the first processing unit 504 includes a luminance signal acquisition module 5040 configured to perform a weighted average of values ​​of pixel points in the local facial region according to a weight and set the weighted average value as the luminance signal of the current frame, and an initial heart rate signal acquisition module 5042 configured to acquire an initial heart rate signal based on the luminance signal of the current frame and the luminance signals of the previous N frames in the history, where N is an integer greater than or equal to 1. Here, the weight can be set based on the position of the pixel point in the local facial region, so that the closer the pixel point is to the edge position in the local facial region, the smaller the weight corresponding to the pixel point. The value of the weight can be in the range of 0-1.

[0067] The transform unit 506 is configured to perform a time domain to frequency domain transform on the initial heart rate signal to obtain a frequency domain signal.

[0068] In an alternative embodiment, the transform unit 506 can transform the initial heart rate signal from the time domain to the frequency domain by fast Fourier transform to obtain a frequency domain signal.

[0069] In an alternative embodiment, the heart rate estimation device 50 may further include a first noise removal module configured to perform a first noise removal process on the initial heart rate signal in the time domain before performing a time-domain-to-frequency domain transformation on the initial heart rate signal. Here, the first noise removal process method includes at least one of Savitzky-Golay (SG) filtering, detrend filtering, moving average filtering, normalization, and bandpass filtering. The Savitzky-Golay (SG) filtering and detrend filtering can be configured to reduce the effect of signal baseline shifts on the signal (e.g., to remove the effects of ambient light). The moving average filtering can be configured to remove random noise in the signal. The normalization process can facilitate signal processing and improve computational efficiency. The bandpass filtering can process the signal in a normal heart rate band. The first noise removal process can remove high frequency noise, low frequency noise and noise introduced by other factors (eg, movement), making the heart rate signal clearer and the periodicity more pronounced.

[0070] The second processing unit 510 is configured to perform a second processing on the frequency domain signal within the heart rate estimation range to obtain an estimate of the heart rate.

[0071] In an alternative embodiment, the second processing unit 510 includes a peak detection module 5100 configured to perform peak detection on the frequency domain signal within the heart rate estimation range and obtain peak values, a sorting module 5102 configured to sort the peak values ​​and obtain sorted results, a confidence calculation module 5104 configured to calculate a confidence based on the sorting result, and an estimation module 5106 configured to obtain an estimated heart rate value based on the confidence.

[0072] In an alternative embodiment, the peak values ​​are sorted in descending order by a traversal method, and the highest peak value and the second peak value are selected as the sorting results as necessary to obtain the sorting result. The ratio of the highest peak value to the second peak value is the reliability, i.e., reliability = highest peak value / second peak value.

[0073] In an alternative embodiment, the second processing unit 510 includes a maximum peak value detection module 5110 configured to perform peak value detection on the frequency domain signal within the heart rate estimation range to obtain a maximum peak value; an energy calculation module 5112 configured to calculate the energy of the first harmonic and the second harmonic corresponding to the main frequency, taking the frequency of the maximum peak value as the main frequency, to obtain an energy calculation value; a reliability calculation module 5114 configured to divide the energy calculation value by the energy of the remaining frequencies other than the main frequency to obtain a reliability; and an estimation module 5116 configured to obtain an estimated heart rate based on the reliability.

[0074] In an alternative embodiment, the estimated range of the heart rate can be preset, for example, 50 to 240 beats / minute based on the human heart rate limit.

[0075] In an alternative embodiment, the estimation module 5106 is configured to perform the steps of: comparing the reliability with a first threshold to obtain a comparison result; if the reliability is less than the first threshold, the comparison result indicates that the frequency domain signal is significantly disturbed by noise, and discarding the current frequency domain signal and detecting the next frame; if the reliability is greater than or equal to the first threshold, the comparison result indicates that the frequency domain signal is not disturbed by noise or has little noise disturbance, and obtaining the frequency corresponding to the highest peak value as an estimate of the heart rate.

[0076] In an alternative embodiment, the heart rate estimation device 50 may further include a second noise removal module configured to perform a second noise removal process on the frequency domain signal before performing the second processing on the frequency domain signal. Here, the second noise removal process method includes at least one of a discrete Fourier transform (DFT) and bandpass filtering. Bandpass filtering can process signals in a normal heart rate band. By performing the second noise removal process on the frequency domain signal, desired principal component signals can be further emphasized.

[0077] In an alternative embodiment, the heart rate estimation device 50 further includes a determination module 507 configured to determine whether the detection time exceeds a second threshold by timing the detection time and comparing the timing value with the second threshold, where the second threshold can be set to M frames or M seconds, where M is an integer greater than or equal to 1, and a third processing module 508 configured to perform a third processing on the frequency domain signal to obtain an estimated range of the heart rate if the detection time exceeds the second threshold. Here, the step of performing the third processing on the frequency domain signal to obtain an estimated range of the heart rate may include the steps of constantly selecting and buffering X consecutive frames or X seconds of the frequency domain signal by a sliding window to obtain and buffer heart rate values, then obtaining an estimated range of the heart rate by a deep learning method, repeating the operation within M frames or M seconds, and averaging all the obtained estimated ranges of the heart rate to obtain a final estimated range of the heart rate. If the detection time does not exceed the second threshold, the second processing unit 510 performs a second processing on the frequency domain signal within the estimation range of the heart rate to obtain an estimation value of the heart rate.

[0078] According to another aspect of an embodiment of the present invention, there is further provided an electronic device including a processor and a memory configured to store executable instructions for the processor, the processor being configured to perform the method for estimating a heart rate according to any one of the preceding claims by executing the executable commands.

[0079] According to another aspect of the embodiment of the present invention, there is further provided a storage medium having a program stored therein, the storage medium controlling an apparatus in which the storage medium is located to execute the heart rate estimation method described in any one of the above claims when the program is executed.

[0080] The above-mentioned embodiment numbers of the present invention are for the purpose of explanation and do not indicate the superiority or inferiority of the embodiments.

[0081] In the above embodiments of the present invention, the description of each embodiment focuses on certain aspects, and for parts that are not detailed in one embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] In some embodiments provided by this application, it should be understood that the disclosed technical contents can be realized in other ways. The device embodiments described so far are merely illustrative. For example, the division of the units may be a division of logical functions, and in actual implementation, other division methods may be used. For example, multiple units or assemblies may be combined or integrated into another system, or some features may be omitted or not implemented. In other respects, the mutual couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections via interfaces, units, or modules, and may be electrical or other forms.

[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed among multiple units, and some or all of the units may be selected according to actual needs to achieve the objectives of the solution of this embodiment.

[0084] Furthermore, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated units may be realized in the form of hardware or in the form of software functional units.

[0085] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be realized in the form of a software product, including a substantial part that contributes to the prior art, or all or part of the technical solution. The computer software product is a product stored in a storage medium and includes several commands that execute all or part of the steps of the methods described in each embodiment of the present invention by a single computer device (which may be a personal computer, a server, a network device, etc.). The storage medium includes various media capable of storing program code, such as a USB memory, a read-only memory (ROM), a random access memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.

[0086] The above description is merely a preferred embodiment of the present invention, and it is obvious that those skilled in the art can make some further improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered as within the protection scope of the present invention. [Industrial Applicability]

[0087] The means provided by the embodiments of the present application can monitor physical conditions, and the technical solution provided by the embodiments of the present application can be applied to electronic devices with at least one imaging unit, such as various mobile platforms, on-board chips, embedded chips, etc., without requiring large and complex hardware devices, the detection process is simple and fast, does not contact the human body and is harmless, and at the same time has sufficient accuracy, solving the problems of traditional contact detection methods that rely on complex hardware devices and require contact with the human body, and significantly improving the robustness and range of use of heart rate estimation methods. The driver's physical condition can be continuously tracked and judged, and if an abnormality in the driver's heart rate is detected, an alarm can be issued and a driving support function can be activated.

Claims

1. acquiring a facial video with an electronic device; a step in which the electronic device performs face detection on the face-captured video and extracts a local face region set for estimating a heart rate; the electronic device performs a first process on values ​​of pixel points in the facial local region to obtain an initial heart rate signal; the electronics performing a time domain to frequency domain transform on the initial heart rate signal to obtain a frequency domain signal; the electronic device performs a second processing on the frequency domain signal within an estimation range of a heart rate to obtain an estimate of the heart rate; Including, The step of performing a second processing on the frequency domain signal within an estimation range of a heart rate by the electronic device to obtain an estimate of the heart rate includes: The electronic device includes a step of performing peak value detection on the frequency domain signal within an estimated range of the heart rate and obtaining a maximum peak value, The method for estimating a heart rate further includes: determining whether the detection time exceeds a first threshold; When the detection time exceeds the first threshold, the electronic device performs a third process on the frequency domain signal to obtain an estimated range of a heart rate; Including, How to estimate your heart rate.

2. The method of claim 1 , wherein the electronic device acquires the facial video using an infrared camera.

3. The method for estimating a heart rate according to claim 1 , wherein the local facial region includes a region below both eyes.

4. The method of claim 1 , further comprising extracting the facial local regions configured for estimating a heart rate by combining face keypoint localization with the face detection.

5. 2. The heart rate estimation method according to claim 1, wherein the step of the electronic device performing a first process on values ​​of pixel points in the local facial region and acquiring an initial heart rate signal includes a step of the electronic device performing a weighted average on values ​​of pixel points in the local facial region according to weights, setting the weighted average value as a luminance signal of a current frame, and configuring the luminance signal of the current frame and luminance signals of N previous frames in history as the initial heart rate signal.

6. 2. The method of claim 1, wherein the step of the electronic device performing a time domain-to-frequency domain transform on the initial heart rate signal to obtain a frequency domain signal includes the step of the electronic device transforming the initial heart rate signal from the time domain to the frequency domain using a fast Fourier transform to obtain the frequency domain signal.

7. 6. The method for estimating a heart rate according to claim 5, wherein the electronic device sets the weight based on a position of the pixel point within the local facial region, and the closer the pixel point is to an edge position within the local facial region, the smaller the weight corresponding to the pixel point.

8. 2. The heart rate estimation method according to claim 1, further comprising a step in which, before the electronic device performs a time domain-to-frequency domain transformation on the initial heart rate signal, the electronic device performs a first noise removal process on the initial heart rate signal in the time domain, the first noise removal process including at least one of Savitzky-Golay (SG) filtering, detrend filtering, moving average filtering, normalization processing, and bandpass filtering.

9. The step of the electronic device performing a second processing on the frequency domain signal within an estimation range of the heart rate to obtain an estimate of the heart rate comprises: The electronic device performs peak detection on the frequency domain signal within an estimated range of heart rate to obtain a peak value; the electronic device sorting the peak values ​​and obtaining a sorting result; The electronic device acquires a reliability based on the sorting result; obtaining an estimate of the heart rate based on the confidence level; The method of claim 1 , comprising:

10. The step of performing a second processing on the frequency domain signal within an estimation range of a heart rate by the electronic device to obtain an estimate of the heart rate includes: The electronic device performs peak value detection on the frequency domain signal within an estimated range of heart rate to obtain the highest peak value; The electronic device calculates energy of first and second harmonics corresponding to the main frequency, using the frequency of the highest peak value as the main frequency, to obtain calculated energy values; The electronic device divides the calculated energy value by the energy of the remaining frequencies other than the main frequency to obtain a reliability; obtaining an estimate of the heart rate based on the confidence level; The method of claim 1 , comprising:

11. The method for estimating a heart rate according to claim 1 , wherein the estimation range of the heart rate is set in advance.

12. 2. The method for estimating a heart rate according to claim 1, wherein the step of the electronic device performing a third processing on the frequency domain signal to obtain an estimated range of the heart rate includes the steps of: the electronic device continuously selecting and buffering the frequency domain signal for X consecutive frames or X seconds by a sliding window to obtain and store the heart rate values; then obtaining the estimated range of the heart rate by a deep learning method; repeating the operation within M frames or M seconds; and averaging all the obtained estimated ranges of the heart rate to obtain a final estimated range of the heart rate.

13. The method for estimating a heart rate according to claim 9, wherein the electronic device sorts the peak values ​​in descending order using a traversal method, selects the highest peak value and the second peak value as the sorting result, and determines the reliability as a ratio between the highest peak value and the second peak value.

14. The step of the electronic device obtaining the estimate of the heart rate based on the confidence level comprises: the electronic device comparing the reliability with a second threshold to obtain a comparison result; If the confidence is less than the second threshold, the comparison result indicates that the frequency domain signal is significantly corrupted by noise, and the current result is discarded and detection of the next frame is performed; If the confidence is equal to or greater than the second threshold, the comparison result indicates that the frequency domain signal is not disturbed by noise or is less disturbed by noise, and the frequency corresponding to the highest peak value is taken as the estimated heart rate; The method of claim 1 , comprising:

15. 9. The method for estimating a heart rate according to claim 8, further comprising a step in which, before the electronic device performs a second processing on the frequency domain signal, the electronic device performs a second noise removal processing on the frequency domain signal, the second noise removal processing including at least one of a discrete Fourier transform (DFT) and bandpass filtering.

16. a capture unit configured to capture face capture video; a detection unit configured to perform face detection on the face video and extract a local face region configured to estimate a heart rate; a first processing unit configured to perform a first processing on values ​​of pixel points in the facial local region to obtain an initial heart rate signal; a transform unit configured to perform a time domain to frequency domain transform on the initial heart rate signal to obtain a frequency domain signal; a second processing unit configured to perform a second processing on the frequency domain signal within an estimation range of a heart rate to obtain an estimate of the heart rate; Including, The second processing unit includes a maximum peak value detection module configured to perform peak value detection on the frequency domain signal to obtain a maximum peak value within an estimated range of the heart rate; The heart rate estimation device further comprises: a determination module configured to determine whether the detection time exceeds a first threshold; a third processing module configured to perform a third processing on the frequency domain signal to obtain an estimated range of a heart rate when the detection time exceeds the first threshold; Including, Heart rate estimator.

17. The second processing unit a peak detection module configured to perform peak detection on the frequency domain signal to obtain a peak value within an estimated range of heart rate; a sorting module configured to sort the peak values ​​and obtain a sorting result; a confidence calculation module configured to obtain a confidence based on the sorting result; an estimation module configured to obtain an estimate of the heart rate based on the confidence level; The heart rate estimation device according to claim 16, comprising:

18. The second processing unit a maximum peak value detection module configured to perform peak value detection on the frequency domain signal to obtain the maximum peak value within an estimated range of heart rate; an energy calculation module configured to calculate the energy of a first harmonic and a second harmonic corresponding to the main frequency, the frequency of the highest peak value being a main frequency, to obtain an energy calculation value; a reliability calculation module configured to divide the calculated energy by the energy of the remaining frequencies other than the dominant frequency to obtain a reliability; an estimation module configured to obtain an estimate of the heart rate based on the confidence level; The heart rate estimation device according to claim 16, comprising:

19. A program for executing the method for estimating a heart rate according to any one of claims 1 to 15.

20. a processor; a memory configured to store executable commands for said processor; An electronic device, wherein the processor is configured to perform the method for estimating heart rate according to any one of claims 1 to 15 by executing the executable commands.

Images