Heart rate measuring method based on mobile phone camera

Abstract

The invention discloses a heart rate measuring method based on a mobile phone camera. The heart rate measuring method comprises the steps that camera parameters are set; continuously collecting image data of the finger through a mobile phone camera; calculating the sum value of three color values of the image; carrying out band-pass filtering processing and correction; and heart rate calculation and accuracy evaluation are performed and then output is performed, so that the effect of stably and accurately calculating the heart rate through the mobile phone camera is realized.

Description

[Technical field]

[0001] The invention relates to a fan lamp power supply with high power factor and low direct current voltage. [Background Technology]

[0002] At present, heart rate is generally measured on the market through electrocardiogram measuring instruments, finger heart rate testers and other equipment. Although these professional equipment are stable and accurate, they are expensive and inconvenient to carry. Nowadays, people's lives are inseparable from mobile phones. Through the heart rate measurement method of mobile phone cameras, heart rate can be measured anytime and anywhere without the need to purchase additional equipment. It is simple and fast.

[0003] At present, the heart rate measurement method based on mobile phone camera has the problem that the finger image signal collected by mobile phone camera is easily affected by factors such as ambient light changes, human hand jitter, smartphone multi-tasking system sampling time jitter, camera focus jitter, AWB automatic white balance jitter, AE automatic exposure jitter, etc., and can only be simply filtered, resulting in unstable and inaccurate measurement. [Summary of the invention]

[0004] The present invention overcomes the deficiencies of the prior art and provides a heart rate measurement method based on a mobile phone camera.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A heart rate measurement method based on a mobile phone camera, characterized in that:

[0007] S1. Turn on the mobile phone camera and detect the ambient brightness through the image;

[0008] S2, setting the mobile phone camera to the shortest focal length, setting the automatic white balance function of the mobile phone camera to one of the fixed ambient light modes, and fixing the camera target frame rate to N fps, and allowing the mobile phone camera to continuously acquire preview images;

[0009] S3, after obtaining a frame of preview image, respectively calculate the sum of the three color values ​​in the preview image, namely redSum, greenSum, blueSum, and record the current time, and save the data into the FIFO queue rawList;

[0010] S4, determine redSum>(greenSum+blueSum)*threshold F0; the difference of redSum between the previous and next two preview frames divided by (255*number of image pixels)<=threshold F1; within a period of time, the difference of redSum between the previous and next two preview frames>(difference of greenSum+difference of blueSum)*threshold F2, the number of times this condition is satisfied>the number of times it is not satisfied; the number of times the peak value is found>threshold F3;

[0011] S5, when the finger is placed behind the camera and the four conditions in S4 are met at the same time, the automatic exposure lock of the mobile phone camera in this state is set, otherwise the cache is cleared and the process returns to step S2;

[0012] S6, use filters to filter redSum, greenSum, and blueSum respectively, and save the filtered data redSumFilter, greenSumFilter, and blueSumFilter into the FIFO queue filterlist;

[0013] S7, get the data of point N1 after rawList of FIFO queue, determine whether point N1 / 2 is a peak value, if it is a peak value, save it in rawPeakList, and correct the peak point time by the size of the points before and after the peak value; get the data of point N1 after filterlist of FIFO queue, determine whether point N1 / 2 is a peak value, if it is a peak value, save it in filterPeakList, and correct the peak point time by the size of the points before and after the peak value;

[0014] S8. When the number of filterPeakList points>N2, calculate the heart rate. The calculation method is to take the last N3 points of filterPeakList, heart rate = 60*(N3-1) / (time of the last point of filterPeakList-time of the last N3 points of filterPeakList);

[0015] S9, determine whether the maximum redSum-minimum redSum>threshold F4, and calculate the time interval between two points in all rawPeakLists, and calculate the average value and sampling error, and determine whether the sampling error < average value * percentage F5; calculate the time interval between two points in all filterPeakLists, and determine whether the calculated time interval > minimum time interval minT and the time interval < maximum time interval maxT;

[0016] S10. Normalize the data in rawList and perform the FFT (Fast Fourier Transform). The frequency resolution = image frame rate / number of data points in the FFT. Among them, fft_amp1 is the amplitude value of the point before the maximum amplitude value point, fft_amp2 is the maximum amplitude value, and fft_amp3 is the amplitude value of the point after the maximum amplitude value point;

[0017] When fft_amp1 > fft_amp3, then

[0018] fft heart rate minimum value = (sequence number of the maximum amplitude value - 0.5) * frequency resolution * 60;

[0019] fft heart rate maximum value = sequence number of the maximum amplitude value * frequency resolution * 60;

[0020] When fft_amp1 < fft_amp3, then

[0021] fft heart rate minimum value = sequence number of the maximum amplitude value * frequency resolution * 60;

[0022] fft heart rate maximum value = (sequence number of the maximum amplitude value + 0.5) * frequency resolution * 60;

[0023] When fft_amp1 = fft_amp3, then

[0024] fft heart rate minimum value = sequence number of the maximum amplitude value * frequency resolution * 60;

[0025] fft heart rate maximum value = sequence number of the maximum amplitude value * frequency resolution * 60;

[0026] S11. Normalize the data in rawList and perform the FFT. Take all amplitude values where the frequency >= minimum frequency minFrequency and the frequency <= maximum frequency maxFrequency. Judge whether the maximum amplitude value / average amplitude value > threshold F6, whether the maximum amplitude value > threshold F7, and judge whether the heart rate result is between the minimum and maximum values of the fft heart rate;

[0027] S12. When S9 - S11 are all satisfied, output the heart rate result.

[0028] A heart rate measurement method based on a mobile phone camera as described above, characterized in that: in S1, if the light is too dark, turn on the flash.

[0029] A heart rate measurement method based on a mobile phone camera as described above, characterized in that: in S6, the filter is a FIR filter.

[0030] The heart rate measurement method based on a mobile phone camera as described above is characterized in that the minimum filtering value of the filtering range of the FIR filter is less than 60 and the maximum filtering value is greater than 100.

[0031] The heart rate measurement method based on a mobile phone camera as described above is characterized in that the FIR filter uses a Hamming window function for filtering.

[0032] The above-mentioned method for measuring heart rate based on a mobile phone camera is characterized in that: the corrected peak point time in S7 includes

[0033] Take three points (Vf, Tf), (Vp, Tp), (Vr, Tr), where Vp is the peak value, Tp is the time of the peak, Vf is the value before the peak, Tf is the time before the peak, Vr is the value after the peak, Tr is the time after the peak;

[0034] When (Vp-Vf)>(Vp-Vr), the corrected peak time = Tp+((Tp-Tf) / 2)*((Vp-Vf)-(Vp-Vr)) / (Vp-Vf)*(1-(Vp-Vr) / (Vp-Vf));

[0035] When (Vp-Vf) < (Vp-Vr), the corrected peak time = Tp + ((Tr-Tp) / 2)*((Vp-Vr)-(Vp-Vf)) / (Vp-Vr)*(1-(Vp-Vf) / (Vp-Vr))

[0036] When (Vp-Vf)==(Vp-Vr), the corrected peak time=Tp.

[0037] The beneficial effects of the present invention are:

[0038] After setting the camera parameters, the present invention continuously collects image data of the finger through the mobile phone camera, and can calculate the heart rate more stably and accurately after calculating the sum of the three color values ​​of the image, bandpass filtering, heart rate calculation and accuracy evaluation. [Drawings]

[0039] Figure 1 It is a flow chart of the present invention;

[0040] Figure 2 This is a schematic diagram of the waveform before bandpass filtering;

[0041] Figure 3 This is a schematic diagram of the waveform after bandpass filtering. [Specific implementation method]

[0042] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.

[0043] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the descriptions of "preferred", "sub-preferred", etc. in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "preferred" or "sub-preferred" may explicitly or implicitly include at least one such feature.

[0044] like Figure 1 As shown, a heart rate measurement method based on a mobile phone camera includes:

[0045] S1. Turn on the phone camera and detect the ambient brightness through the image. When the light is too dark, turn on the flash, but turn off the flash when measuring the heart rate.

[0046] S2. The focal length of the mobile phone camera is set to the shortest, the AWB automatic white balance function is set to a certain fixed ambient light mode (non-AWB), the target frame rate of the fixed camera is N fps, and the mobile phone camera continuously obtains preview images.

[0047] Among them, the finger is close to the camera when the mobile phone camera obtains the finger image, so fixing the shortest focal length can obtain the clearest image and prevent the mobile phone's automatic focus from causing image blur and color value jitter; AWB is set to a certain fixed ambient light mode to prevent color value jitter.

[0048] S3. After obtaining a frame of preview image, calculate the sum of the three color values ​​in the image, namely redSum, greenSum, and blueSum, respectively, and record the current time, and save these data in the FIFO queue rawList.

[0049] S4, determine redSum> (greenSum + blueSum) * threshold F0; the difference in redSum between the previous and next two frames divided by (255 * number of image points) is less than or equal to threshold F1; within a period of time, the difference in redSum between the previous and next two frames is greater than (difference in greenSum + difference in blueSum) * threshold F2, and the number of times this condition is met is greater than the number of times it is not met; the number of times the peak value is found is greater than threshold F3.

[0050] Among them, the advantage of judging redSum>(greenSum+blueSum)*threshold F0 is that the light appears red when passing through the finger. This method can be used to judge whether the finger has been placed on the camera. Since the background color of the environment may be red and cause misjudgment, within a period of time, the difference in redSum between the previous and next frames of the picture is greater than (difference in greenSum+difference in blueSum)*threshold F2. The method in which the number of times this condition is met is greater than the number of times it is not met can exclude the static red background of the environment; the difference in redSum between the previous and next frames of the picture divided by (255*number of image points) is less than or equal to threshold F1, and the number of times the peak value is found is greater than threshold F3. Meeting these two conditions can prevent the finger from shaking over a large range, causing large fluctuations in the total color value of the image, and affecting the calculation of the heart rate.

[0051] S5. When the finger is placed on the camera and the four conditions in S4 are met at the same time, the finger has been stably placed on the mobile phone camera. At this time, the mobile phone camera in this state is set to AE lock (non-AE). If the four conditions in S4 cannot be met at the same time, clear the cache and return to step S2.

[0052] Among them, when the camera is initialized, the AE mode is not locked immediately. Instead, the camera target frame rate is fixed to Nfps. This is because the exposure time and sensitivity parameters of different mobile phones are different. In this way, a reasonable exposure time and sensitivity can be obtained. The AE mode is locked only after waiting for the hand to be stable and the image to be stable. This can prevent the AE algorithm from causing image fluctuations.

[0053] S6. Use a bandpass FIR filter to filter redSum, greenSum, and blueSum. The filtered data redSumFilter, greenSumFilter, and blueSumFilter are saved in the FIFO queue filterlist. The filter parameters can be adjusted dynamically to adjust the filter range.

[0054] Among them, the bandpass FIR filter uses the Hamming window function, and the waveform before filtering is as follows Figure 2 , the waveform after filtering is as follows Figure 3 At the same time, the normal heart rate range of an adult is 60 to 100. Setting a FIR bandpass filter with a range larger than the heart rate range can filter the fluctuation of the total color value of the image caused by factors such as environmental changes, hand shaking, and camera noise. Therefore, the minimum filter value of the filter range of the FIR filter is set to be less than 60 and the maximum filter value is greater than 100.

[0055] S7, get the N1 point data after rawList, determine whether the N1 / 2th point is the highest peak, if it is the peak, save it to rawPeakList, and correct the peak point time by the size of the points before and after the peak; get the N1 point data after filterlist, determine whether the N1 / 2th point is the highest peak, if it is the peak, save it to filterPeakList, and correct the peak point time by the size of the points before and after the peak.

[0056] (Vf, Tf)(Vp, Tp)(Vr, Tr), three points, Vp is the value of the peak, Tp is the time of the peak, Vf is the value of the point before the peak, Tf is the time of the point before the peak, Vr is the value of the point after the peak, Tr is the time of the point after the peak.

[0057] If (Vp-Vf)>(Vp-Vr), then corrected peak time = Tp+((Tp-Tf) / 2)*((Vp-Vf)-(Vp-Vr)) / (Vp-Vf)*(1-(Vp-Vr) / (Vp-Vf));

[0058] If (Vp-Vf)<(Vp-Vr), then corrected peak time = Tp+((Tr-Tp) / 2)*((Vp-Vr)-(Vp-Vf)) / (Vp-Vr)*(1-(Vp-Vf) / (Vp-Vr));

[0059] If (Vp-Vf)==(Vp-Vr), then the corrected peak time=Tp.

[0060] Among them, since the maximum frame rate of many mobile phone cameras is only 30fps, the total color value sampling rate is relatively low. After finding the peak, this method compares the sizes of the two points before and after. The actual peak time should be closer to the larger point between the front and rear points. The peak point time is corrected at a certain ratio rate1, so that the peak time is more accurate.

[0061] S8. When the number of filterPeakList points is greater than N2, calculate the heart rate. Calculation method: take the last N3 points of filterPeakList, heart rate = 60*(N3-1) / (time of the last point of filterPeakList - time of the last N3 points of filterPeakList).

[0062] Among them, the smartphone is a multi-tasking system, and the time interval between each image is not fixed. The fluctuation of the intermediate time does not affect the result. Therefore, the deviation of the time of the first and last points of the above heart rate calculation method is averaged by multiple points, and the error is smaller.

[0063] S9, determine the number of rawPeakList points>N3, filterPeakList points>N3; determine the time of the last point of filterPeakList-the time of the first point of filterPeakList>T1; determine the maximum redSum-minimum redSum>threshold F4; calculate the time interval between all two points in rawPeakList, and calculate the average value and sampling error, and determine the sampling error <average value*percentage F5; calculate the time interval between all two points in filterPeakList, and determine the time interval>minT and the time interval <maxT。

[0064] Among them, the time interval between two points of all rawPeakList is calculated, and the average value and sampling error are calculated. The sampling error < average value * percentage F5. It can be judged whether the heartbeat fluctuates greatly or the noise is too large.

[0065] S10, normalize the data of rawList and perform fft fast Fourier transform, frequency resolution = image frame rate / number of fft data points, fft_amp1 is the amplitude value of the point before the maximum amplitude value, fft_amp2 is the maximum amplitude value, and fft_amp3 is the amplitude value of the point after the maximum amplitude value;

[0066] If fft_amp1 is greater than fft_amp3, then

[0067] FFT heart rate minimum value = (serial number of maximum amplitude value - 0.5) * frequency resolution * 60;

[0068] FFT heart rate maximum value = maximum amplitude value number * frequency resolution * 60;

[0069] If fft_amp1 is less than fft_amp3, then

[0070] FFT heart rate minimum value = maximum amplitude value sequence number * frequency resolution * 60;

[0071] fft heart rate maximum value = (the number of the maximum amplitude value + 0.5) * frequency resolution * 60; if fft_amp1 is equal to fft_amp3, then

[0072] FFT heart rate minimum value = maximum amplitude value sequence number * frequency resolution * 60;

[0073] FFT heart rate maximum value = maximum amplitude value number * frequency resolution * 60.

[0074] Among them, because the sampling rate is low, the heart rate error calculated by FFT is relatively large, so the S10 method is used to estimate the range of heart rate values.

[0075] S11, normalize the data of rawList and perform fft fast Fourier transform, take all amplitude values ​​with frequency>=minFrequency and frequency<=maxFrequency, maximum amplitude value / average amplitude value>threshold F6. Maximum amplitude value>threshold F7. The heart rate result should be between the minimum and maximum values ​​of fft heart rate.

[0076] Among them, through the S11 method, the noise level of the original signal can be evaluated. If the noise is too large, the heartbeat measurement will continue to avoid errors in the results.

[0077] S12. When steps 9 to 11 are all satisfied, the signal is considered stable, the heart rate is stable, the final heart rate is output, and the measurement ends.

[0078] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A heart rate measurement method based on a mobile phone camera, characterized in that: Included S1. Turn on the mobile phone camera and detect the ambient brightness through the image; S2, setting the mobile phone camera to the shortest focal length, setting the automatic white balance function of the mobile phone camera to one of the fixed ambient light modes, and fixing the camera target frame rate to N fps, and allowing the mobile phone camera to continuously acquire preview images; S3, after obtaining a frame of preview image, respectively calculate the sum of the three color values ​​in the preview image, namely redSum, greenSum, blueSum, and record the current time, and save the data into the FIFO queue rawList; S4, determine redSum>(greenSum+blueSum)*threshold F0; the difference of redSum between the previous and next two preview frames divided by (255*number of image pixels)<=threshold F1; within a period of time, the difference of redSum between the previous and next two preview frames>(difference of greenSum+difference of blueSum)*threshold F2, the number of times this condition is satisfied>the number of times it is not satisfied; the number of times the peak value is found>threshold F3; S5, when the finger is placed behind the camera and the four conditions in S4 are met at the same time, the automatic exposure lock of the mobile phone camera in this state is set, otherwise the cache is cleared and the process returns to step S2; S6, use filters to filter redSum, greenSum, and blueSum respectively, and save the filtered data redSumFilter, greenSumFilter, and blueSumFilter into the FIFO queue filterlist; S7, get the data of point N1 after rawList of FIFO queue, determine whether point N1 / 2 is a peak value, if it is a peak value, save it in rawPeakList, and correct the peak point time by the size of the points before and after the peak value; get the data of point N1 after filterlist of FIFO queue, determine whether point N1 / 2 is a peak value, if it is a peak value, save it in filterPeakList, and correct the peak point time by the size of the points before and after the peak value; S8. When the number of filterPeakList points>N2, calculate the heart rate. The calculation method is to take the last N3 points of filterPeakList, heart rate = 60*(N3-1) / (time of the last point of filterPeakList-time of the last N3 points of filterPeakList); S9, determine that maximum redSum-minimum redSum>threshold F4, calculate the time interval between two points in all rawPeakLists, calculate the average value and sampling error, and determine that sampling error<average value*percentage F5; Calculate the time interval between two points in all filterPeakLists, and determine whether the calculated time interval > the minimum time interval minT and the time interval < the maximum time interval maxT; S10. Normalize the data in rawList and perform the FFT (Fast Fourier Transform). The frequency resolution = image frame rate / number of data points in the FFT. Among them, fft_amp1 is the amplitude value of the point before the point with the maximum amplitude value, fft_amp2 is the maximum amplitude value, and fft_amp3 is the amplitude value of the point after the point with the maximum amplitude value; When fft_amp1 > fft_amp3, then fft heart rate minimum value = (sequence number of the maximum amplitude value - 0.5) * frequency resolution * 60; fft heart rate maximum value = sequence number of the maximum amplitude value * frequency resolution * 60; When fft_amp1 < fft_amp3, then fft heart rate minimum value = sequence number of the maximum amplitude value * frequency resolution * 60; fft heart rate maximum value = (sequence number of the maximum amplitude value + 0.5) * frequency resolution * 60; When fft_amp1 = fft_amp3, then fft heart rate minimum value = sequence number of the maximum amplitude value * frequency resolution * 60; fft heart rate maximum value = sequence number of the maximum amplitude value * frequency resolution * 60; S11. Normalize the data in rawList and perform the FFT. Take all amplitude values where the frequency >= minimum frequency minFrequency and the frequency <= maximum frequency maxFrequency. Judge whether the maximum amplitude value / average amplitude value > threshold F6, and whether the maximum amplitude value > threshold F7. Judge that the heart rate result is between the minimum and maximum values of the fft heart rate; S12. When S9 - S11 are all satisfied, output the heart rate result.

2. A heart rate measurement method based on a mobile phone camera according to claim 1, characterized in that: In S1, if the light is too dim, turn on the flash.

3. The heart rate measurement method based on a mobile phone camera according to claim 1, characterized in that: In S6, the filter is a FIR filter.

4. The heart rate measurement method based on a mobile phone camera according to claim 3, characterized in that: The filtering minimum value of the filtering range of the FIR filter is less than 60 and the filtering maximum value is greater than 100.

5. The heart rate measurement method based on a mobile phone camera according to claim 3, characterized in that: The FIR filter uses the Hamming window function for filtering processing.

6. The heart rate measurement method based on a mobile phone camera according to claim 1, characterized in that: The corrected peak point time in S7 includes Take three points (Vf, Tf), (Vp, Tp), (Vr, Tr), where Vp is the value of the wave peak, Tp is the time of the wave peak, Vf is the value of the point before the wave peak, Tf is the time of the point before the wave peak, Vr is the value of the point after the wave peak, and Tr is the time of the point after the wave peak; When (Vp - Vf) > (Vp - Vr), then the corrected peak time = Tp + ((Tp - Tf) / 2) * ((Vp - Vf) - (Vp - Vr)) / (Vp - Vf) * (1 - (Vp - Vr) / (Vp - Vf)); When (Vp - Vf) < (Vp - Vr), then the corrected peak time = Tp + ((Tr - Tp) / 2) * ((Vp - Vr) - (Vp - Vf)) / (Vp - Vr) * (1 - (Vp - Vf) / (Vp - Vr)). When (Vp - Vf) == (Vp - Vr), then the corrected peak time = Tp.

Images