Non-contact heart rate measurement method and apparatus

Abstract

Disclosed is a non-contact heart rate measurement method and device, the method comprising: acquiring a face video image of a to-be-measured object, and extracting a face skin region from each frame of the video image, and extracting a first color space vector based on the face skin region; projecting the first color space vector onto a projection plane of a predetermined color space to obtain a one-dimensional pulse signal; performing modal decomposition on the one-dimensional pulse signal to obtain a plurality of modal signals, the plurality of modal signals having at least different instantaneous frequencies and instantaneous amplitudes from each other, and performing signal recombination based on at least two of the plurality of modal signals to obtain a recombined signal; and obtaining an estimated heart rate based on the recombined signal.

Description

Technical Field

[0001] The present disclosure relates to the field of non-contact heart rate measurement. Specifically, the present disclosure provides a non-contact heart rate measurement method and device based on modal decomposition. Background Art

[0002] Heart rate is the number of heartbeats per minute in the human body and is an important physiological parameter for assessing human health. The information provided by heart rate is also widely used in medical diagnosis, and various devices and methods for detecting human heart rate have been proposed. Taking the fingertip measurement method, which is currently the most widely used method for measuring heart rate, as an example, its principle is photoplethysmography (PPG). To further improve the convenience of heart rate detection, a remote photoplethysmography (rPPG) method has also been proposed. Heart rate measurement based on rPPG can achieve non-contact measurement of heart rate. For example, methods such as independent component analysis (ICA) and principal component analysis (PCA) have been provided in the prior art, but they still have defects in accuracy and robustness. For example, the waveform recovery used to calculate heart rate is not accurate enough, making it difficult to process video signals under different lighting scenes or changing lighting scenes. Summary of the Invention

[0003] In response to the existing technical problems, the present invention provides a non-contact heart rate measurement method based on rPPG, which has high accuracy and robustness.

[0004] One aspect of the present disclosure provides a contactless heart rate measurement method, which includes: acquiring a facial video image of a subject to be measured, extracting a facial skin area from each frame of the video image, and extracting a first color space vector based on the facial skin area; projecting the first color space vector onto a projection plane of a predetermined color space to obtain a one-dimensional pulse signal; performing modal decomposition on the one-dimensional pulse signal to obtain multiple modal signals, wherein the multiple modal signals have at least different instantaneous frequencies and instantaneous amplitudes from each other; performing signal recombining based on at least two of the multiple modal signals to obtain a recombined signal; and obtaining an estimated heart rate based on the recombined signal.

[0005] Optionally, the method may further include: before performing modal decomposition on the one-dimensional pulse signal to obtain multiple modal signals, performing Fourier transform on the one-dimensional pulse signal to determine the peak frequency in the spectrum after Fourier transform, and constructing a frequency vector based on the peak frequency.

[0006] Optionally, performing modal decomposition on a one-dimensional pulse signal to obtain multiple modal signals may include: constructing orthogonal factors based on frequency vectors; performing series expansion on the one-dimensional pulse signal through the orthogonal factors to obtain multiple modal signals; and performing constrained optimization on the multiple modal signals to determine the instantaneous frequency and instantaneous amplitude of each modal signal in the multiple modal signals.

[0007] Optionally, the one-dimensional pulse signal can be expressed as:

[0008]

[0009] Among them, X(t) is a one-dimensional pulse signal, X i (t) is the i-th modal signal among multiple modal signals, is the frequency vector, and the orthogonality factor is and a i (t) and b i (t) is the amplitude function of the i-th mode signal, and represents the i-th mode signal in the orthogonal factor and The weight on the i (τ) is the instantaneous frequency of the i-th mode signal, A i (t) is the instantaneous amplitude of the i-th modal signal, θ i is the initial phase of the i-th modal signal.

[0010] Optionally, the constraint conditions of the constraint optimization can be

[0011]

[0012] in, Indicates a i The square norm of the nth derivative of (t), Indicates b i The square norm of the nth derivative of (t), represents the square norm of the redundant energy values ​​of the multiple modal signals, and α is the weight.

[0013] Optionally, performing signal recombination based on at least two of the multiple modal signals to obtain a recombined signal may include: calculating the power spectral density of each modal signal in the multiple modal signals to determine the energy amplitude of each frequency, and selecting a modal signal with a maximum energy amplitude and a frequency within a predetermined frequency range for signal recombination; and calculating the correlation between each modal signal in the multiple modal signals and the one-dimensional pulse signal, and selecting the modal signal with the largest correlation for signal recombination.

[0014] Optionally, the predetermined frequency range is greater than or equal to 0.7 Hz and less than or equal to 4 Hz.

[0015] Optionally, the projection plane of the predetermined color space is perpendicular to the second color space vector of the human face skin area, wherein the second color space vector is a vector used to represent skin color information in the human face skin area.

[0016] Optionally, the projection plane of the predetermined color space includes a first projection axis and a second projection axis orthogonal to the first projection axis, wherein projecting the first color space vector onto the projection plane of the predetermined color space to obtain a one-dimensional pulse signal may include: projecting the first color space vector onto the first projection axis to obtain a first component, projecting the first color space vector onto the second projection axis to obtain a second component, the first projection axis and the second projection axis being set so that the value of the first component and the value of the second component are positive; and weightedly fusing the first component and the second component to obtain a one-dimensional pulse signal.

[0017] Optionally, the color space is an RGB color space, and the first color space vector is a pixel mean of a facial skin area corresponding to an RGB three-channel image in each frame of the video image.

[0018] Another aspect of the present disclosure provides a device for contactless heart rate measurement, which includes: an image acquisition unit, configured to acquire a facial video image of a subject to be measured, and extract a facial skin area from each frame of the video image, and extract a first color space vector based on the facial skin area; a projection unit, configured to project the first color space vector onto a projection plane of a predetermined color space to obtain a one-dimensional pulse signal; a modal decomposition unit, configured to perform modal decomposition on the one-dimensional pulse signal to obtain a plurality of modal signals, wherein the plurality of modal signals have at least different instantaneous frequencies and instantaneous amplitudes from each other; a signal recombination unit, configured to perform signal recombination based on at least two of the plurality of modal signals to obtain a recombined signal; and a heart rate estimation unit, configured to obtain an estimated heart rate based on the recombined signal.

[0019] Optionally, the device may further include an initialization unit, configured to: perform Fourier transform on the one-dimensional pulse signal output by the projection unit to determine the peak frequency in the spectrum after Fourier transform, and construct a frequency vector based on the peak frequency.

[0020] Optionally, the modal decomposition unit can also be configured to: construct an orthogonal factor based on the frequency vector output by the initialization unit; perform series expansion on the one-dimensional pulse signal through the orthogonal factor to obtain multiple modal signals; and perform constrained optimization on the multiple modal signals to determine the instantaneous frequency and instantaneous amplitude of each modal signal in the multiple modal signals.

[0021] Optionally, the signal recombination unit can be further configured to: calculate a power spectral density of each of the plurality of modal signals to determine an energy magnitude at each frequency, select a modal signal having a frequency with a largest energy magnitude within a predetermined frequency range for signal recombination; and calculate a correlation of each of the plurality of modal signals with a one-dimensional pulse signal, select a modal signal having a largest correlation for signal recombination.

[0022] Optionally, the projection plane of the predetermined color space comprises a first projection axis and a second projection axis orthogonal to the first projection axis, wherein the projection unit is further configured to: project the first color space vector onto the first projection axis to obtain a first component, and project the first color space vector onto the second projection axis to obtain a second component, the first projection axis and the second projection axis being set such that the value of the first component and the value of the second component are positive; and weight and fuse the first component and the second component to obtain the one-dimensional pulse signal.

[0023] Another aspect of the present disclosure provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the non-contact heart rate measurement method as described above.

[0024] Another aspect of the present disclosure provides a computer device, comprising: a processor; a memory storing a computer program, which, when executed by the processor, implements the non-contact heart rate measurement method as described above.

[0025] According to one or more aspects of the present disclosure, the present disclosure provides a non-contact heart rate measurement method and device, which projects a first color space vector onto a projection plane of a predetermined color space, thereby eliminating noise caused by a light scene (e.g., variation in light); decomposes a signal into a plurality of modal signals, and recombines the signal based on at least two of the plurality of modal signals to obtain a recombined signal, which helps to more accurately and robustly estimate a heart rate because at least two of the plurality of modal signals are selected for signal recombination to retain an effective signal related to the heart rate. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and other aspects, features, and advantages of the present disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1 is a schematic diagram of an application scenario of the present disclosure;

[0028] Figure 2 is a flowchart of a non-contact heart rate measurement method according to an embodiment of the present disclosure;

[0029] Figure 3is a schematic diagram of a human face skin area according to an embodiment of the present disclosure;

[0030] Figure 4 Schematic diagram of consistency analysis between the embodiment and the comparative example according to the present disclosure;

[0031] Figure 5 is a schematic diagram of a device for non-contact heart rate measurement according to an embodiment of the present disclosure; and

[0032] Figure 6 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following detailed description is provided to help the reader gain a comprehensive understanding of the methods, apparatus and / or systems described herein. However, various changes, modifications and equivalents of the methods, apparatus and / or systems described herein will be apparent to those of ordinary skill in the art. For example, the order of operations described herein is merely an example and is not limited to the order set forth herein, but changes may be made that will be apparent to those of ordinary skill in the art, except for operations that must be performed in a particular order. In addition, descriptions of features and structures that will be known to those of ordinary skill in the art may be omitted for clarity and brevity. The features described herein may be implemented in different forms and are not to be construed as being limited to the examples described herein. More specifically, the examples described herein have been provided so that this disclosure will be thorough and complete and will fully convey the scope of this disclosure to those of ordinary skill in the art.

[0034] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the accompanying drawings to explain the present disclosure.

[0035] Figure 1 It is a schematic diagram of an application scenario of the present disclosure.

[0036] The application scenarios of this disclosure mainly involve non-contact heart rate measurement technology, which is suitable for situations where remote, non-invasive or continuous heart rate monitoring is required. Figure 1 As shown, the facial image of the subject to be measured can be obtained by only using the camera of the non-contact heart rate measurement device without contacting the subject to be measured, and then the heart rate of the subject to be measured can be obtained based on the facial image.

[0037] Specific application scenarios may include but are not limited to:

[0038] Remote medical monitoring: In remote medical services, cameras can be used to capture images of patients' faces. By analyzing images of the facial skin area, patients' heart rates can be monitored in real time and contactlessly, providing doctors with real-time physiological data for remote diagnosis and treatment recommendations.

[0039] Health management and fitness monitoring: In home health management and fitness scenarios, the camera can capture the user's facial image, analyze the image of the facial skin area, and monitor the user's heart rate in real time, helping the user understand their physical condition and providing a basis for health management and exercise.

[0040] Driver fatigue monitoring: During driving, cameras can be installed inside a car to capture the driver's face in real time. By analyzing the facial skin area, the driver's heart rate can be monitored in real time to determine whether the driver is fatigued, thereby improving road safety.

[0041] Emotional Analysis: Cameras can be used to capture images of respondents' faces. By analyzing images of the facial skin area and monitoring the respondent's heart rate in real time, physiological data can be provided for the emotional analysis process, assisting in determining the respondent's emotions and / or intentions.

[0042] Figure 2 is a flowchart of a non-contact heart rate measurement method according to an embodiment of the present disclosure. Figure 3 Schematic diagram of a human face skin area according to an embodiment of the present disclosure.

[0043] like Figure 2 As shown in , in step S10, a face video image of the object to be tested is obtained, and a face skin area is extracted from each frame of the video image, and a first color space vector is extracted based on the face skin area.

[0044] In the embodiment, a common camera can be used to capture a video image of the face of the subject to be tested to obtain a video image. The facial skin area (face region of interest) is extracted for each frame based on the obtained video image. Specifically: a face recognition network is used to calibrate the facial feature points, such as Figure 3 As shown in (a) in the figure. Based on the calibrated feature points, the selected area is framed and converted from the RGB color space to the YCbCr space. The skin color of ordinary people is concentrated in the elliptical area of ​​the CbCr plane. Each pixel value of the selected area is projected onto the CbCr plane, where the pixels within the elliptical range projected on the CbCr plane are skin pixels, thereby obtaining the human face skin area in the selected area (as shown in Figure 1). Figure 3 (as shown in (b) in the figure).

[0045] In an embodiment of the present disclosure, the color space is an RGB color space, and the first color space vector is the pixel mean of the facial skin area corresponding to the RGB three-channel image in each frame of the video image.

[0046] For example, the first color space vector C can be expressed as:

[0047]

[0048] Among them, R(n), G(n), and B(n) are the red, green, and blue channel signals in the RGB three-channel signal respectively; M(x, y, n) is the pixel intensity of the pixel point with coordinates (x, y) in the n-th frame video image, K is the number of pixels detected in the n-th frame video image, and x, y∈skin indicates that the pixel with coordinates (x, y) is located in the facial skin area.

[0049] In an embodiment, if a video image with a sampling duration of 30 seconds and a frame rate of 60 frames per second is sampled, the value of n will range from 1 to 1800. In this case, the first color space vector will be a one-dimensional vector with 1800 elements.

[0050] In step S20, the first color space vector is projected onto a projection plane of a predetermined color space to obtain a one-dimensional pulse signal.

[0051] Specifically, according to the skin reflection model, for the pixel value C of the kth pixel in the video image k (t), can be expressed as:

[0052] C k (t)=1·(1+i(t))+A·u p ·I0·h(t)+A·u s ·I0·p(t)

[0053] Where i(t) is the time-varying signal of the light intensity at the pixel; h(t) is the specular reflection component at the pixel; p(t) is the pulse wave signal at the pixel; u p is the spectral unit color vector; u s is the pulse intensity vector; I0 is the light intensity constant. is a diagonal matrix used to achieve normalization.

[0054] In an exemplary embodiment of the present disclosure, the three-dimensional signal is reduced to a two-dimensional signal by projecting the first color space vector onto a predetermined projection plane D, thereby eliminating the time-varying light intensity signal i(t) of the original rPPG signal and removing additional noise.

[0055] For example, the projection plane D of the predetermined color space is perpendicular to the second color space vector of the facial skin region, where the second color space vector is a vector used to represent skin color information in the facial skin region. For example, the second color space vector can be the color average, color median, or truncated mean of the facial skin region, and can be adjusted and changed according to the skin color of the subject to be measured.

[0056] In the projecting step, the projection plane of the predetermined color space includes a first projection axis and a second projection axis orthogonal to the first projection axis, the first color space vector is projected onto the first projection axis to obtain a first component, and the first color space vector is projected onto the second projection axis to obtain a second component.

[0057]

[0058] Where P1(t) and P2(t) are the first and second components, respectively. Since the second color space vector represents the skin color information in the facial skin region and is unrelated to the time-varying light intensity signal i(t), the time-varying light intensity signal i(t) is eliminated from the projection result, and only the color variation value is included.

[0059] In an embodiment, the first projection axis and the second projection axis are set so that the value of the first component and the value of the second component are positive, thereby making the first component and the second component in-phase pulse signals.

[0060] Subsequently, the first component and the second component are weightedly fused to obtain a one-dimensional pulse signal. On the projection surface, the mirror component h(t) and the pulse component p(t) have almost opposite phases. For example, when the pulsation change dominates P(t), P1(t) and P2(t) appear in phase, and the addition of the signals will enhance the signal strength generated; when the mirror reflection change dominates P(t), P1(t) and P2(t) appear in anti-phase, and the addition of the signals will weaken the signal strength generated. However, there is often a large difference in the values ​​of P1(t) and P2(t). Therefore, the formula for weighted fusion of the first component and the second component can be expressed as

[0061]

[0062] Where X(t) is the fused one-dimensional pulse signal, and std(·) represents the standard deviation. Adding two anti-phase signals with the same amplitude cancels out the mirror reflection distortion.

[0063] In an embodiment, the time t used here may correspond to the frame number n used in step S10. For example, the one-dimensional pulse signal X(1s) at 1 second may be obtained by calculating the first color space vector at the frame number n. In this case, X(t)(t∈[0s,30s]) is also a one-dimensional vector having 1800 elements.

[0064] The non-contact heart rate measurement method according to the present disclosure helps reduce data complexity, reduce calculation amount and processing time by projecting multi-dimensional color space vectors into a low-dimensional space (such as a one-dimensional pulse signal), making the heart rate measurement process more efficient. By projecting onto a specific projection plane, signal components related to heart rate changes can be better extracted, thereby improving the accuracy of non-contact heart rate measurement. The projection process helps to filter out noise and other irrelevant signal components, making the obtained one-dimensional pulse signal more stable and reliable, which helps to achieve non-contact heart rate measurement under different environmental conditions.

[0065] In step S30 , modal decomposition is performed on the one-dimensional pulse signal to obtain a plurality of modal signals, where the plurality of modal signals at least have different instantaneous frequencies and instantaneous amplitudes.

[0066] The contactless heart rate measurement method according to an embodiment of the present disclosure also includes a frequency vector initialization step performed before step S30. For example, a Fourier transform is performed on the one-dimensional pulse signal X(t) to determine the peak frequency in the Fourier transformed spectrum, and a frequency vector is constructed based on the peak frequency. A frequency vector is a vector with the same length as the one-dimensional pulse signal X(t), and the values ​​in the frequency vector can be identical. Performing a Fourier transform on the one-dimensional pulse signal X(t) allows the signal to be analyzed in the frequency domain, thereby determining the main frequency components in the signal. This helps to more accurately extract signal components related to heart rate changes. Constructing a frequency vector based on the peak frequency provides a reasonable initial condition for the subsequent modal decomposition. This helps to improve the convergence speed and accuracy of the modal decomposition, thereby more accurately extracting each modal signal. Constructing a frequency vector based on the peak frequency facilitates signal reconstructing based on multiple modal signals, thereby improving the accuracy of signal reconstruction. This will help to more accurately estimate the heart rate in subsequent steps.

[0067] Subsequently, an orthogonal factor is constructed based on the frequency vector; the one-dimensional pulse signal X(t) is expanded in series by the orthogonal factor to obtain multiple modal signals; and the multiple modal signals are constrainedly optimized to determine the instantaneous frequency and instantaneous amplitude of each modal signal in the multiple modal signals.

[0068] For example, the one-dimensional pulse signal X(t) contains H components. In this case, the one-dimensional pulse signal X(t) can be expressed as

[0069]

[0070] Where, X(t) is the one-dimensional pulse signal, X i (t) is the i-th modal signal among the multiple modal signals, A i (t)>0,g i (t)>0,A i (t), g i (t), θ i are the instantaneous amplitude, instantaneous frequency and initial phase of the i-th mode signal respectively. The value of H can be set to 4 or 5. Represents high-order redundancy.

[0071] Based on frequency vector Constructing orthogonal factors and And based on the above orthogonal factors, the one-dimensional pulse signal X(t) is expanded in series:

[0072]

[0073] Among them, a i (t) and b i (t) is the amplitude function of the i-th modal signal, and represents the i-th modal signal in the orthogonal factor and The weight on the i (τ) is the instantaneous frequency of the i-th modal signal, A i (t) is the instantaneous amplitude of the i-th modal signal, θ i is the initial phase of the i-th modal signal. And the constraint condition of the constrained optimization is

[0074]

[0075] in, Indicates a i The square norm of the nth derivative of (t), Indicates b i The square norm of the nth derivative of (t). Here n can be 2, which has an amplifying effect on the high-frequency part and can act as a high-pass filter. represents the square norm of the redundant energy values ​​of the multiple modal signals, and α is the weight. Because the orthogonal factor is set to and Therefore, the instantaneous amplitude A of the i-th modal signal is i (t) can also be passed To express.

[0076] In an embodiment, the time t used here may correspond to the number of frames n used in step S10. In the case of sampling for 30 seconds and 60 frames per second, the instantaneous amplitude A of the i-th modal signal is i (t) and instantaneous frequency g i (t)(t∈[0s,30s]) are all one-dimensional vectors with 1800 elements.

[0077] In step S40, signal recombination is performed based on at least two of the multiple modal signals to obtain a recombined signal. Specifically, the power spectral density of each modal signal in the multiple modal signals is calculated to determine the energy amplitude at each frequency, and the modal signal with the maximum energy amplitude within a predetermined frequency range is selected for signal recombination. The correlation between each modal signal in the multiple modal signals and the one-dimensional pulse signal is calculated, and the modal signal with the maximum correlation is selected for signal recombination.

[0078] In an embodiment, a modal signal whose frequency of maximum energy amplitude is within a predetermined frequency range can be considered to best represent a pulse signal. Under normal circumstances, a person's heart rate range is between 42 and 240 bpm, and the corresponding frequency range is [0.7, 4]. A modal signal whose frequency of maximum energy amplitude is not within this range can be considered as noise, which does not contain heart rate information. In another embodiment, if a person's heart rate range is set between 45 and 240 bpm, the corresponding frequency range is [0.75, 4]. When calculating the power spectral density, the instantaneous amplitude A of the i-th modal signal can be used. i (t) and instantaneous frequency g i The power spectral density is calculated by Fourier transforming the time domain signal to obtain the frequency signal. The power spectral density represents the power per frequency unit.

[0079] In an embodiment, the correlation between the modal signal and the one-dimensional pulse signal may be calculated using the Pearson correlation coefficient.

[0080] By calculating the power spectral density of the modal signal and its Pearson correlation with the one-dimensional pulse signal, modal signals closely related to heart rate measurement can be specifically selected for reconstruction. This helps extract the most representative and accurate heart rate information from multiple modal signals. During the signal reconstruction process, the modal signals with the maximum energy amplitude within a specific range and the highest correlation are selected for fusion, which helps improve the quality of the reconstructed signal and the accuracy of heart rate measurement, while also eliminating noise interference.

[0081] In step S50, an estimated heart rate is obtained based on the recombined signal.

[0082] Specifically, for the recombination signal, a 5-point moving average filter is used to smooth the pulse signal by using the time average of adjacent frames to eliminate random noise; a Hamming window-based band-pass filter with a cutoff frequency of 0.7-4 Hz is used to eliminate components unrelated to heart rate; the denoised pulse signal is subjected to fast Fourier transform, and the frequency corresponding to the highest amplitude in the power spectrum is multiplied by 60 to estimate the heart rate (bpm).

[0083] The present disclosure provides a non-contact heart rate measurement method, which eliminates noise caused by illumination scenarios (e.g., changes in illumination) by projecting a first color space vector onto a projection plane of a predetermined color space; by modal decomposition of the signal, and based on at least two of the plurality of modal signals, the signal is recombined to obtain a recombined signal, because the recombined signal selects the modal signal with the maximum energy amplitude in a certain range and the maximum correlation for fusion, it helps to improve the quality of the recombined signal and the accuracy of heart rate measurement, thereby retaining the effective signal related to heart rate, and helps to more accurately and robustly estimate the heart rate.

[0084] Figure 4 An analysis diagram is shown for consistency with the embodiments and comparative examples according to the present disclosure.

[0085] The present disclosure selects 12 participants to wear a transmission type finger pulse oximeter on the index finger to test heart rate as a reference heart rate. During the experiment, the participants are required to be in a stationary state, remove their glasses and not to wear makeup, sit in front of the camera, and conduct the experiment under natural light. The experimental scene is shown in Figure 1 .

[0086] The Bland-Altman plot is used to analyze the consistency of the heart rate estimated by the method proposed in the embodiments of the present disclosure and the reference heart rate obtained by the transmission type finger pulse oximeter in the comparative examples. The results show that the average deviation of the heart rate estimated by the method proposed in the present disclosure is -0.32 bpm, and the consistency limit range is -4.36-3.28 bpm at 95%.

[0087] Figure 5 is a schematic diagram of a non-contact heart rate measurement device according to an embodiment of the present disclosure.

[0088] Referring to Figure 5 , the non-contact heart rate measurement device includes an image acquisition unit 110, a projection unit 120, a modal decomposition unit 130, a signal recombination unit 140, and a heart rate estimation unit 150.

[0089] The image acquisition unit 110 is configured to acquire a face video image of the subject to be tested, extract a face skin area from each frame of the video image, and extract a first color space vector based on the face skin area. Figure 2 The step S10 described above is omitted here for redundant description.

[0090] The projection unit 120 is configured to project the first color space vector onto a projection plane of a predetermined color space to obtain a one-dimensional pulse signal. The projection unit 120 is configured to perform a reference Figure 2 The step S20 described above is omitted here for redundant description.

[0091] The modal decomposition unit 130 is configured to perform modal decomposition on the one-dimensional pulse signal to obtain a plurality of modal signals, wherein the plurality of modal signals have at least different instantaneous frequencies and instantaneous amplitudes. Figure 2 The step S30 described above is omitted here for redundant description.

[0092] The signal recombining unit 140 is configured to perform signal recombining based on at least two of the plurality of modal signals to obtain a recombined signal. Figure 2 The step S40 described above is omitted here for redundant description.

[0093] The heart rate estimation unit 150 is configured to obtain an estimated heart rate based on the recombined signal. Figure 2 The step S50 described above is omitted here for redundant description.

[0094] Regarding the apparatus in the above embodiment, the specific manner in which each module / unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0095] Figure 6 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0096] Figure 6 The electronic device 600 shown includes at least one memory 601 and at least one processor 602 . The memory 601 stores a computer program. When the computer program is executed by the processor 602 , the processor is prompted to implement the non-contact heart rate measurement method as described above.

[0097] Figure 5Each unit in the illustrated non-contact heart rate measurement device can be configured as software, hardware, firmware, or any combination thereof to perform a specific function. For example, each unit may correspond to a dedicated integrated circuit, pure software code, or a module combining software and hardware. Furthermore, one or more functions implemented by each unit may also be collectively performed by components in a physical device (e.g., a processor, client, or server).

[0098] In addition, refer to Figures 1 to 3 The described non-contact heart rate measurement method can be implemented by a program (or instructions) recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may be provided. When the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the non-contact heart rate measurement method according to the present disclosure.

[0099] The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in computer devices such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above-mentioned steps or to perform more specific processing when executing the above-mentioned steps. The contents of these additional steps and further processing have been mentioned in the description of the relevant method with reference to the accompanying drawings, so they will not be repeated here to avoid repetition.

[0100] It should be noted that the various units in the device for non-contact heart rate measurement according to the exemplary embodiment of the present disclosure can completely rely on the operation of the computer program to realize the corresponding functions, that is, the various units correspond to the steps in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding functions.

[0101] on the other hand, Figure 5 The various units shown may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments for performing the corresponding operations may be stored in a computer-readable medium such as a storage medium, so that the processor can perform the corresponding operations by reading and running the corresponding program code or code segments.

[0102] For example, an exemplary embodiment of the present disclosure may also be implemented as a computing device, which includes a storage component and a processor, wherein a set of computer-executable instructions is stored in the storage component. When the set of computer-executable instructions is executed by the processor, the non-contact heart rate measurement method according to the exemplary embodiment of the present disclosure is executed.

[0103] Specifically, the computing device can be deployed in a server or client, or deployed on a node device in a distributed network environment. In addition, the computing device can be a PC, tablet device, personal digital assistant, smart phone, web application, or other device capable of executing the above instruction set.

[0104] Here, the computing device is not necessarily a single computing device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0105] In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, a processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0106] Some operations described in the non-contact heart rate measurement method according to an exemplary embodiment of the present disclosure may be implemented by software, some operations may be implemented by hardware, and furthermore, these operations may be implemented by a combination of software and hardware.

[0107] The processor may execute instructions or codes stored in one of the memory components, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0108] The storage component can be integrated with the processor, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the storage component can include a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The storage component and the processor can be operatively coupled or can communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor can access files stored in the storage component.

[0109] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.) All components of the computing device may be connected to each other via a bus and / or a network.

[0110] The contactless heart rate measurement method according to the exemplary embodiment of the present disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logical device or operate according to non-precise boundaries.

[0111] Therefore, refer to Figures 1 to 3 The described non-contact heart rate measurement method can be implemented by a system including at least one computing device and at least one storage device storing instructions.

[0112] According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device according to the non-contact heart rate measurement method of the exemplary embodiment of the present disclosure, and a computer executable instruction set is stored in the storage device. When the computer executable instruction set is executed by the at least one computing device, the reference Figures 1 to 3 A contactless heart rate measurement method is described.

[0113] While various exemplary embodiments of the present disclosure have been described above, it should be understood that the foregoing description is merely illustrative and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present disclosure. Therefore, the scope of protection of the present disclosure should be determined by the scope of the claims.

Claims

1. A non-contact heart rate measurement method, characterized in that: The method comprises: Acquire a facial video image of a subject to be tested, extract a facial skin region from each frame of the video image, and extract a first color space vector based on the facial skin region; Projecting the first color space vector onto a projection plane of a predetermined color space to obtain a one-dimensional pulse signal; Performing modal decomposition on the one-dimensional pulse signal to obtain a plurality of modal signals, wherein the plurality of modal signals have at least mutually different instantaneous frequencies and instantaneous amplitudes; performing signal recombining based on at least two of the plurality of modal signals to obtain a recombined signal; obtaining an estimated heart rate based on the recombined signal, The step of performing signal recombination based on at least two of the plurality of modal signals to obtain a recombined signal includes: calculating the power spectral density of each modal signal in the plurality of modal signals to determine the energy amplitude of each frequency, and selecting a modal signal having a maximum energy amplitude and a frequency within a predetermined frequency range for the signal recombination; The correlation between each modal signal in the plurality of modal signals and the one-dimensional pulse signal is calculated, and the modal signal with the largest correlation is selected for the signal recombination.

2. The method according to claim 1, characterized in that The method further includes: before performing modal decomposition on the one-dimensional pulse signal to obtain multiple modal signals, performing Fourier transform on the one-dimensional pulse signal to determine the peak frequency in the spectrum after Fourier transform, and constructing a frequency vector based on the peak frequency.

3. The method according to claim 2, characterized in that Performing modal decomposition on the one-dimensional pulse signal to obtain multiple modal signals includes: constructing an orthogonal factor based on the frequency vector; Performing series expansion on the one-dimensional pulse signal using the orthogonal factors to obtain multiple modal signals; and A constrained optimization is performed on the plurality of modal signals to determine an instantaneous frequency and an instantaneous amplitude of each of the plurality of modal signals.

4. The method according to claim 3, characterized in that The one-dimensional pulse signal is expressed as: in, is the one-dimensional pulse signal, is the i-th modal signal among the multiple modal signals, is the frequency vector, and is the orthogonality factor, and is the amplitude function of the i-th modal signal, respectively represents the i-th modal signal in the orthogonal factor and The weight on is the instantaneous frequency of the i-th modal signal, is the instantaneous amplitude of the i-th modal signal, is the initial phase of the i-th modal signal.

5. The method according to claim 4, characterized in that The constraints of the constrained optimization are in, express The square norm of the n-th derivative of , express The square norm of the n-th derivative of , represents the square norm of the redundant energy values ​​of the multiple modal signals, is the weight.

6. The method according to claim 1, characterized in that The predetermined frequency range is greater than or equal to 0.7 Hz and less than or equal to 4 Hz.

7. The method according to claim 1, characterized in that The projection plane of the predetermined color space is perpendicular to the second color space vector of the human face skin area, wherein the second color space vector is a vector used to represent skin color information in the human face skin area.

8. The method according to claim 7, characterized in that The projection plane of the predetermined color space includes a first projection axis and a second projection axis orthogonal to the first projection axis, wherein projecting the first color space vector onto the projection plane of the predetermined color space to obtain a one-dimensional pulse signal includes: Projecting the first color space vector onto the first projection axis to obtain a first component, and projecting the first color space vector onto the second projection axis to obtain a second component, wherein the first projection axis and the second projection axis are set so that the values ​​of the first component and the second component are positive; The first component and the second component are weightedly fused to obtain the one-dimensional pulse signal.

9. The method according to claim 1, characterized in that The color space is an RGB color space, and the first color space vector is a pixel mean of a facial skin area corresponding to an RGB three-channel image in each frame of the video image.

10. A non-contact heart rate measurement device, characterized in that: The device comprises: an image acquisition unit configured to acquire a face video image of a subject to be tested, extract a face skin region from each frame of the video image, and extract a first color space vector based on the face skin region; a projection unit configured to project the first color space vector onto a projection plane of a predetermined color space to obtain a one-dimensional pulse signal; a modal decomposition unit configured to perform modal decomposition on the one-dimensional pulse signal to obtain a plurality of modal signals, wherein the plurality of modal signals have at least different instantaneous frequencies and instantaneous amplitudes; a signal recombination unit configured to perform signal recombination based on at least two of the multiple modal signals to obtain a recombined signal, wherein the power spectral density of each modal signal in the multiple modal signals is calculated to determine the energy amplitude of each frequency, and a modal signal having a maximum energy amplitude and a frequency within a predetermined frequency range is selected for the signal recombination; a correlation between each modal signal in the multiple modal signals and the one-dimensional pulse signal is calculated, and the modal signal with the maximum correlation is selected for the signal recombination; The heart rate estimation unit is configured to obtain an estimated heart rate based on the recombined signal.

11. The device according to claim 10, characterized in that The apparatus further includes an initialization unit, which is configured to: Performing Fourier transform on the one-dimensional pulse signal output by the projection unit to determine a peak frequency in the spectrum after the Fourier transform, and constructing a frequency vector based on the peak frequency.

12. The device according to claim 11, characterized in that The modal decomposition unit is further configured to: constructing an orthogonal factor based on the frequency vector output by the initialization unit; Performing series expansion on the one-dimensional pulse signal using the orthogonal factors to obtain multiple modal signals; as well as A constrained optimization is performed on the plurality of modal signals to determine an instantaneous frequency and an instantaneous amplitude of each of the plurality of modal signals.

13. The device according to claim 12, characterized in that The one-dimensional pulse signal is expressed as: in, is the one-dimensional pulse signal, is the i-th modal signal among the multiple modal signals, is the frequency vector, and is the orthogonality factor, and is the amplitude function of the i-th modal signal after modal decomposition, and represents the i-th modal signal in the orthogonal factor and The weight on is the instantaneous frequency of the i-th mode signal, is the instantaneous amplitude of the i-th modal signal, is the initial phase of the i-th modal signal.

14. The device according to claim 13, characterized in that The constraints of the constrained optimization are in, express The square norm of the n-th derivative of , express The square norm of the n-th derivative of , represents the square norm of the redundant energy values ​​of the multiple modal signals, is the weight.

15. The device according to claim 10, characterized in that The predetermined frequency range is greater than or equal to 0.7 Hz and less than or equal to 4 Hz.

16. The device according to claim 10, characterized in that The projection plane of the predetermined color space is perpendicular to the second color space vector of the human face skin area, wherein the second color space vector is a vector used to represent skin color information in the human face skin area.

17. The device according to claim 16, characterized in that The projection plane of the predetermined color space includes a first projection axis and a second projection axis orthogonal to the first projection axis, wherein the projection unit is further configured to: Projecting the first color space vector onto the first projection axis to obtain a first component, and projecting the first color space vector onto the second projection axis to obtain a second component, wherein the first projection axis and the second projection axis are set so that the values ​​of the first component and the second component are positive; The first component and the second component are weightedly fused to obtain the one-dimensional pulse signal.

18. The device according to claim 10, characterized in that The color space is an RGB color space, and the first color space vector is a pixel mean of a facial skin area corresponding to an RGB three-channel image in each frame of the video image.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the non-contact heart rate measurement method according to any one of claims 1 to 9 is implemented.

20. A computer device, characterized in that: The computer device comprises: processor; The memory stores a computer program, and when the computer program is executed by the processor, the non-contact heart rate measurement method according to any one of claims 1 to 9 is implemented.

Images