Heart rate detection method realized by utilizing composite remote photoelectric volume pulse wave

Abstract

The invention discloses a heart rate detection method implemented by using a composite remote photoplethysmography (rPPG), comprising the following steps: continuously capturing an input frame, the input frame having a face image; calculating a plurality of feature points in the face image to obtain a skin color mean value and storing the skin color mean value in a first queue; executing a POS algorithm and a CHROM algorithm based on the skin color mean value of the plurality of input frames in the first queue to generate a first rPPG waveform signal and a second rPPG waveform signal respectively; performing reverse merging processing on the first rPPG waveform signal and the second rPPG waveform signal to generate a merged waveform signal; performing a fast Fourier transform process on the merged waveform signal to generate a merged spectrum; and performing spectrum analysis processing on the combined spectrum to extract heart rate output.

Description

Technical Field

[0001] The present invention relates to heart rate detection, and in particular to a heart rate detection method utilizing a non-contact method for performing heart rate detection. Background Art

[0002] Currently, there are many non-contact heart rate monitors on the market that can directly measure the user's heart rate without touching the user's body.

[0003] For example, some detectors use the Robust Pulse Rate from Chrominance-based rPPG (CHROM) algorithm. The CHROM algorithm obtains a user's facial image through a non-contact method and performs image analysis on the image to obtain the RGB color change ratio. This is used to detect changes in blood volume in the user's subcutaneous blood vessels, and the frequency of these changes is used to estimate the user's pulse.

[0004] For example, some detectors use the Plane-Orthogonal-to-Skin (POS) algorithm. The POS algorithm is a descendant of the CHROM algorithm, differing in that it detects changes in the color dimension projection matrix.

[0005] However, the CHROM and POS algorithms are suitable for calculating different skin colors and image brightness. Therefore, the detectors currently on the market that use a single algorithm are not suitable for detecting the heart rate of all users. This is because different users, different usage environments, and different image capture units may cause differences in skin color and image brightness, resulting in inaccurate detection results. Summary of the Invention

[0006] The present invention provides a heart rate detection method using a composite remote photoplethysmography algorithm, which improves the accuracy of heart rate detection by simultaneously referring to the rPPG signals generated by the CHROM algorithm and the POS algorithm.

[0007] In one embodiment, the present invention provides a heart rate detection method using hybrid remote photoplethysmography (rPPG) for application to an electronic device comprising at least one image capture unit and a processing unit, and includes the following steps: the image capture unit continuously captures input frames over time, wherein the input frames include facial images; the processing unit calculates a plurality of feature points in the facial images to obtain a skin color average, and stores the skin color average and a corresponding timestamp in a first queue; the processing unit executes a plane-orthogonal-to-skin (POS) algorithm and a robust pulse rate from chrominance-based rPPG algorithm based on the skin color averages of the plurality of input frames in the first queue. The invention relates to a method for detecting a heart rate in a heartbeat signal, wherein the heart rate is detected by the heart rate monitoring device and the heart rate monitoring device is connected to the heart rate monitoring device. The method comprises the following steps: applying an rPPG, CHROMA) algorithm to generate a first rPPG waveform signal and a second rPPG waveform signal respectively; performing reverse merging processing on the first rPPG waveform signal and the second rPPG waveform signal by the processing unit to generate a merged waveform signal; performing a fast Fourier transform (FFT) processing on the merged waveform signal to generate a merged spectrum; and performing spectrum analysis processing on the merged spectrum to extract a heart rate output.

[0008] Compared with related technologies, the detection method of the present invention can adapt to a wider range of skin color samples and can increase the intensity of the main frequency of the generated rPPG signal, thereby improving the accuracy of the heart rate detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an embodiment of an electronic device of the present invention;

[0010] Figure 2 is an embodiment of a flow chart of a detection method of the present invention;

[0011] Figure 3 An embodiment of a schematic diagram of a facial image;

[0012] Figure 4 An embodiment of the image capture unit adjustment flow chart of the present invention;

[0013] Figure 5 An embodiment of a flow chart for establishing skin color mean values ​​of the present invention;

[0014] Figure 6A is an embodiment of a first schematic diagram of an rPPG waveform signal of the present invention;

[0015] Figure 6B is an embodiment of a second schematic diagram of an rPPG waveform signal of the present invention;

[0016] Figure 7 is an embodiment of a signal merging flow chart of the present invention;

[0017] Figure 8 is an embodiment of a schematic diagram of a first queue of the present invention;

[0018] Figure 9 is an embodiment of a schematic diagram of a spectrum of the present invention;

[0019] Figure 10 This is an embodiment of the spectrum analysis flow chart of the present invention;

[0020] Wherein, the reference numerals:

[0021] 1: electronic device;

[0022] 11: processing unit;

[0023] 12: Image capture unit;

[0024] 13: storage unit;

[0025] 131: first queue;

[0026] 1311: Overlay window;

[0027] 132: second queue;

[0028] 2: user;

[0029] 3: facial image;

[0030] 31: Feature points;

[0031] 61, 611: first rPPG waveform signal;

[0032] 62, 621: second rPPG waveform signal;

[0033] 63: Merge waveform signals;

[0034] 91: Merge spectrum;

[0035] 92: skin color pixel variation spectrum;

[0036] 93: Final spectrum;

[0037] D: first length;

[0038] S21-S27: detection step;

[0039] S41-S43: adjustment steps;

[0040] S51-S55: establishment steps;

[0041] S71-S74: merging step;

[0042] S101~S107: Analysis steps. DETAILED DESCRIPTION

[0043] This invention discloses a heart rate detection method (hereinafter referred to as the detection method) using hybrid remote photoplethysmography (rPPG). This detection method can be applied to any electronic device that can capture a user's facial image. By analyzing the user's facial image, the detection method can directly detect the user's heart rate without requiring physical contact.

[0044] See also Figure 1 , which is an embodiment of a block diagram of an electronic device of the present invention. Figure 1 The disclosed electronic device 1 comprises a processing unit 11 , an image capturing unit 12 and a storage unit 13 , wherein the processing unit 11 is electrically connected to the image capturing unit and the storage unit 13 .

[0045] In one embodiment, the electronic device 1 is, for example, a smart phone, a tablet computer, a laptop computer, or a personal computer. The processing unit 11 is, for example, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a central processing unit (CPU), a system on chip (SoC), a field programmable gate array (FPGA), or a combination of the above components. The image capture unit 12 is, for example, an RGB sensor unit or a camera. The storage unit 13 is, for example, a flash memory (Flash Memory), a read-only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), or a combination of the above components. However, the above are only some embodiments of the present invention and are not limited thereto.

[0046] The detection method of the present invention involves the electronic device 1 controlling the image capture unit 12 to continuously capture the image of the user 2 whose heart rate is to be detected. The resulting facial image is then processed using at least two algorithms to generate an rPPG signal. Finally, the user's heart rate is calculated based on the rPPG signal. Because heart rate is calculated using facial images, direct contact with the user 2 is not required during the detection process. Furthermore, because at least two algorithms are used simultaneously to perform the calculations, optimal detection results are achieved regardless of the user's skin color or image brightness.

[0047] See also Figure 2 , which is an embodiment of a flow chart of the detection method of the present invention. Figure 2 The specific steps of the detection method of the present invention are disclosed, and these specific steps can be Figure 1 The electronic device 1 shown is used to implement the present invention, but the present invention is not limited thereto.

[0048] like Figure 2 As shown, to perform the detection method of the present invention, first, user 2 can use an electronic device with an image capture unit (eg Figure 1 The electronic device 1 is used to take a picture to continuously capture input frames (step S21). Specifically, the electronic device takes a picture toward the user 2 to continuously obtain the input frames, wherein each input frame has a complete facial image of the user 2.

[0049] It's worth noting that in the detection method of the present invention, electronic device 1 continuously captures user 2's facial image to continuously obtain multiple input frames, and processing unit 11 of electronic device 1 continuously executes the various steps described below based on these multiple input frames. Thus, as long as electronic device 1 continuously obtains input frames containing user 2's facial image, processing unit 11 can continuously calculate user 2's heart rate.

[0050] After step S21, the processing unit 11 uses an image analysis algorithm to analyze the facial image (eg Figure 3 The facial image 3 shown in FIG. 3 is analyzed to identify multiple feature points in the facial image (eg Figure 3 The plurality of feature points 31 shown in FIG. 3 are calculated to obtain the skin color mean of the facial image (step S22). The processing unit 11 stores the skin color mean and the corresponding timestamp into the first queue 131 (step S23).

[0051] Please also see Figure 3, which is an embodiment of a schematic diagram of a facial image. In the present invention, processing unit 11 analyzes the facial image in the input frame using an image analysis algorithm to identify multiple feature points preset by the algorithm. These feature points are primarily located in flat areas that are not affected by head rotation, such as the sides of the nose, cheeks, and forehead. Because these feature points can still be captured even if user 2 turns their head while the image capture unit 12 is recording, using these feature points as a basis can improve the accuracy of the calculated skin color mean.

[0052] In the present invention, the electronic device 1 continuously captures images via the image capture unit 12. The brightness of the input frames obtained can also affect the accuracy of the resulting heart rate. More specifically, the detection method of the present invention is adapted to a specific brightness range. Therefore, the detection method of the present invention can also selectively adjust the image capture unit 12 used for recording based on the calculated skin color mean.

[0053] Please also see Figure 4 , is an embodiment of the image capture unit adjustment flow chart of the present invention. Figure 4 As shown, in one embodiment, the processing unit 11 executes Figure 2 In step S22, the processing unit 11 obtains the skin color mean based on multiple feature points in the facial image in the input frame (step S41). Then, the processing unit 11 can perform brightness detection processing on the skin color mean and generate a brightness detection result (step S42). The brightness detection processing can be an algorithm that determines whether the current skin color mean is sufficient for image analysis or sufficient to generate an rPPG signal that meets quality requirements, but is not limited to this. After step S42, the processing unit 11 can feed back the brightness detection result to the image capture unit 12, thereby directly adjusting the exposure used by the image capture unit 12 when shooting the next input frame (step S43). Specifically, the processing unit 11 adjusts one of the physical parameters of the image capture unit 12 based on the brightness detection result, thereby changing the brightness of the facial image in the next input frame.

[0054] The brightness detection result may include skin color that is too dark, skin color that is appropriate, and skin color that is too light. In step S43, the processing unit 11 may control the image capture unit 12 to increase the exposure if the brightness detection result indicates that the skin color is too dark; control the image capture unit 12 to decrease the exposure if the brightness detection result indicates that the skin color is too light; and control the image capture unit 12 to maintain the current exposure if the brightness detection result indicates that the skin color is appropriate.

[0055] Different from the related art, the present invention does not perform image processing on the input frames, but directly adjusts the physical parameters used by the image capture unit 12 when shooting, which can optimize the obtained images more directly, quickly and effectively.

[0056] As previously described, the present invention calculates the skin color mean based on multiple preset feature points on a facial image. However, the algorithm has a limited number of preset feature points (e.g., 400 feature points). Calculating the skin color mean based on the skin color of a relatively small number of feature points may result in poor accuracy. Therefore, in one embodiment, processing unit 11 may sample skin color pixels from multiple preset feature points and multiple sampling points surrounding these feature points (e.g., eight sampling points surrounding a single feature point) to jointly calculate the skin color mean.

[0057] Please also see Figure 5 , which is an embodiment of a flowchart for establishing the skin color mean value of the present invention. Figure 5 Yes Figure 2 Step S22 is described in more detail. Figure 5 As shown, after obtaining an input frame, the processing unit 11 first identifies multiple feature points in the facial image through an algorithm (step S51), then respectively captures the skin color values ​​of multiple sampling points around the multiple feature points, and calculates a skin color benchmark based on the multiple feature points and the skin color values ​​of the multiple sampling points (step S52).

[0058] After step S52, the processing unit 11 removes the eye area from the facial image to obtain a facial skin region (step S53). The processing unit 11 then filters out multiple skin color pixels within the facial skin region that are within a certain standard deviation from the skin color reference (step S54). In this embodiment, the processing unit 11 establishes the skin color mean based on the filtered multiple skin color pixels (step S55).

[0059] Specifically, the detection method of the present invention first calculates skin color baselines for multiple features in the facial image. Pixels that exceed the skin color baseline by a certain standard deviation are likely not part of user 2's skin. Therefore, the detection method of the present invention removes these pixels through the aforementioned steps. By using only the filtered skin color pixels to establish a skin color average, the skin color average can be more accurate.

[0060] At Figure 5 In the embodiment of the present invention, the processing unit 11 first processes the facial image to remove the image of the eye area, and then obtains a plurality of skin color pixels that meet the requirements from the processed facial image. However, in other embodiments, the processing unit 11 may also directly obtain a plurality of skin color pixels from the facial image, and then remove the plurality of skin color pixels belonging to the eye area from the obtained plurality of skin color pixels. In other words, Figure 5 Steps S53 and S54 do not have a necessary order in execution.

[0061] In one embodiment, the standard deviation is between plus or minus 1.5 and plus or minus 3. That is, the processing unit 11 filters out multiple skin color pixels within the facial skin range that are within a range of plus or minus 1.5 and plus or minus 3 from the skin color reference, and establishes a skin color mean based on these skin color pixels. However, the above is only one embodiment of the present invention and is not intended to be limiting.

[0062] Back to Figure 2 . After the processing unit 11 receives and processes multiple input frames, multiple skin color averages and corresponding timestamps are stored in the first queue 131. At this time, after step S23, the processing unit 11 executes the POS algorithm and the CHROM algorithm based on the skin color averages of the multiple input frames in the first queue 131 to generate a first rPPG waveform signal and a second rPPG waveform signal, respectively (step S24). For ease of understanding, the result of the processing unit 11 executing the POS algorithm based on the skin color averages of the multiple input frames in the first queue 131 is referred to as the first rPPG waveform signal, and the result of the processing unit 11 executing the CHROM algorithm based on the skin color averages of the multiple input frames in the first queue 131 is referred to as the second rPPG waveform signal.

[0063] It is worth mentioning that the POS algorithm and CHROM algorithm used in the present invention are obtained by modifying the output of the core algorithm and the superposition window (described in detail later).

[0064] After step S24, the processing unit 11 further performs reverse merging processing on the first and second rPPG waveform signals to generate a merged waveform signal (step S25). The detection method of the present invention does not solely use the first rPPG waveform signal generated by the POS algorithm to calculate user 2's heart rate, nor does it solely use the second rPPG waveform signal generated by the CHROM algorithm to calculate user 2's heart rate. Instead, it first merges the first and second rPPG waveform signals before calculating user 2's heart rate based on the merged waveform signal. This method leverages the advantages of both the POS and CHROM algorithms, resulting in more accurate heart rate detection results.

[0065] Please also see Figure 6A , which is an embodiment of the first schematic diagram of the rPPG waveform signal of the present invention. Figure 6A The first waveform signal is the first rPPG waveform signal 61 generated after the processing unit 11 executes the POS algorithm, and the second waveform signal is the second rPPG waveform signal 62 generated after the processing unit 11 executes the CHROM algorithm.

[0066] Through experiments, it can be seen that Figure 6AAs shown, although the POS and CHROM algorithms used in the present invention project different vector values ​​into the three-dimensional color space, their output waveforms exhibit similar inverse directions and frequency consistency. Within the same time period (based on the number of frames), the output signal of the POS and CHROM algorithms is not necessarily greater. To enhance the strength of the final rPPG effective signal, the present invention takes the signal with the larger absolute value at each node of the rPPG output from the two algorithms and combines these signal intensities to generate the combined waveform signal. By performing the reverse merging of the first rPPG waveform signal 61 and the second rPPG waveform signal 62, the combined waveform signal can be adapted to a wider variety of skin color samples and the strength of the rPPG dominant frequency (i.e., the dominant frequency of the combined waveform signal) can be increased.

[0067] Please also see Figure 6B , which is an embodiment of the second schematic diagram of the rPPG waveform signal of the present invention. Figure 6B The first waveform signal in the figure is the first rPPG waveform signal 611 after amplitude limiting processing has been performed on the first rPPG waveform signal 61. The second waveform signal is the second rPPG waveform signal 621 after amplitude limiting processing has been performed on the second rPPG waveform signal 62. The third waveform signal is the merged waveform signal 63 generated after the processing unit 11 performs reverse merging processing on the first rPPG waveform signal 611 and the second rPPG waveform signal 621. In one embodiment, the detection method of the present invention performs offset correction and amplitude limiting processing on the first rPPG waveform signal 61 and the second rPPG waveform signal 62 before performing reverse merging processing. The processed first rPPG waveform signal 611 and the second rPPG waveform signal 621 are then reverse merged to generate the merged waveform signal 63 (described in detail later).

[0068] Please also see Figure 7 , which is an embodiment of the signal merging flow chart of the present invention. Figure 7 Yes Figure 2 Step S25 will be described in more detail. For ease of explanation, the following example uses the reverse merging of the first rPPG waveform signal 61 and the second rPPG waveform signal 62 to generate the merged waveform signal 63. However, the detection method of the present invention may also first generate the first rPPG waveform signal 611 and the second rPPG waveform signal 621, and then reversely merge the first rPPG waveform signal 611 and the second rPPG waveform signal 621 to generate the merged waveform signal 63.

[0069] like Figure 7As shown, after the POS algorithm and the CHROM algorithm are used to obtain the first rPPG waveform signal 61 and the second rPPG waveform signal 62, for each frame node (i.e., each input frame), the processing unit 11 extracts the maximum intensity signal in the first rPPG waveform signal 61 (step S71), and also extracts the maximum intensity signal in the second rPPG waveform signal 62 (step S72). Next, for each frame node, the processing unit 11 compares the absolute value of the maximum intensity signal of the first rPPG waveform signal 61 with the absolute value of the maximum intensity signal of the second rPPG waveform signal 62 (step S73), and uses the larger absolute value of the two as the main signal intensity of the corresponding frame node in the merged waveform signal 63 (step S74). After comparing the absolute values ​​of the maximum intensity signals of the first rPPG waveform signal 61 and the second rPPG waveform signal 62 in each frame node, the merged waveform signal 63 can be generated, as shown in FIG. Figure 6B shown.

[0070] As mentioned above, the algorithm used in the present invention is a modified POS algorithm and a modified CHROM algorithm. In one embodiment, the detection method of the present invention needs to be Figure 6A The first rPPG waveform signal 61 and the second rPPG waveform signal 62 are further processed to generate Figure 6B The first rPPG waveform signal 611 and the second rPPG waveform signal 621 are shown.

[0071] Please also see Figure 8 , which is an embodiment of a schematic diagram of the first queue of the present invention. Figure 8 A specific embodiment of the first queue 131 of the present invention is disclosed. In one embodiment, the first queue 131 has a queue length M and records M skin color averages in a first-in, first-out (FIFO) manner. When the number of skin color average data entries exceeds the queue length M, the processing unit 11 deletes the oldest skin color average data entry from the leftmost field of the first queue 131 and inserts the latest skin color average data entry into the rightmost field of the first queue 131. In other words, the first queue 131, with a queue length M, can record the latest M skin color averages.

[0072] The first queue 131 further includes an overlapping window 1311 having an overlapping length N, which is smaller than the queue length M. In one embodiment, the queue length M of the first queue 131 may be, for example, 250 frames, and the overlapping length N of the overlapping window 1311 may be, for example, 50 frames. Figure 8 In the example, the queue length M is 6 and the stacking length M is 3 for simplified description, but the present invention does not use Figure 8 Quantities shown are limited.

[0073] The superposition window 1311 is used to record the latest N skin color average values ​​in the first queue 131. More specifically, when the POS algorithm and CHROM algorithm of the present invention generate the first rPPG waveform signal 61 and the second rPPG waveform signal 62 based on the skin color average values ​​in the first queue 131, after the latest skin color average value is stored in the first queue 131 and the superposition window 1311, the updated N skin color average values ​​in the superposition window 1311 are superimposed with the latest N skin color average values ​​in the first queue 131.

[0074] like Figure 8 As shown, after the processing unit 11 calculates and generates the first skin color average value (e.g., 0.2) (i.e., the 0th round), the processing unit 11 fills the latest skin color average value into the latest field of the first queue 131 and the superposition window 1311 ( Figure 8 Here, before executing the POS algorithm and the CHROM algorithm based on the data in the first queue 131, the processing unit 11 superimposes the data of the first field in the superposition window 1311 (i.e., 0.2) with the data of the first field in the first queue 131 (i.e., 0.2), superimposes the data of the second field in the superposition window 1311 (i.e., 0) with the data of the second field in the first queue 131 (i.e., 0), and superimposes the data of the third field in the superposition window 1311 (i.e., 0) with the data of the third field in the first queue 131 (i.e., 0), and so on.

[0075] It is worth mentioning that, taking the queue length M as 250 frames and the superposition length N as 50 frames as an example, since all fields in the superposition window 1311 cannot have values ​​before the electronic device 1 obtains the first 50 frames, the calculation of the data in the first queue 131 is incomplete. Therefore, in one embodiment, the processing unit 11 Figure 2 In the process, steps S21 to S23 are continuously executed until the number of data frames stored in the first queue 131 is greater than the superposition length N of the superposition window 1311 (e.g., 50 frames). Then, step S24 is executed to calculate the first rPPG waveform signal 61 and the second rPPG waveform signal 62.

[0076] In addition, since the processing unit 11 will still superimpose the data in the superimposition window 1311 to the corresponding field of the first queue 131 before the superimposition window 1311 is filled, the data of the first N frames (taking 50 frames as an example) will be inaccurate. Figure 2In the process, steps S21 to S23 are continuously executed until the number of data frames stored in the first queue 131 exceeds the sum of the superposition length N of the superposition window 1311 and the queue length M (e.g., 300 frames). At this point, step S24 is executed to calculate the first rPPG waveform signal 61 and the second rPPG waveform signal 62. In other words, the processing unit 11 waits until the oldest 50 frames of data in the first queue 131 have been removed before beginning to calculate the first rPPG waveform signal 61 and the second rPPG waveform signal 62.

[0077] Back to Figure 8 After the processing unit 11 calculates and generates the second average skin color value (e.g., 0.7) (i.e., the first time), the processing unit 11 shifts the data in the first queue 131 and the superposition window 1311 to the left by one unit and fills the latest average skin color value into the latest field of the first queue 131 and the superposition window 1311. At this time, when executing the POS algorithm and the CHROM algorithm based on the data in the first queue 131, the processing unit 11 superimposes the data in the first field of the superposition window 1311 (i.e., 0.7) with the data in the first field of the first queue 131 (i.e., 0.7), superimposes the data in the second field of the superposition window 1311 (i.e., 0.1) with the data in the second field of the first queue 131 (i.e., 0.3), and superimposes the data in the third field of the superposition window 1311 (i.e., 0) with the data in the third field of the first queue 131 (i.e., 0).

[0078] It is worth noting that the present invention also includes an alpha-tuning process during the step of adding the latest skin color average value to the superposition window 1311. Therefore, even though there is only one new data in the superposition window 1311, the conversion result of the old data in the superposition window 1311 is different each time.

[0079] Since signals need to be superimposed when performing the POS and CHROM algorithms, if the signals are offset, the offset will be further amplified after superposition. Therefore, the present invention performs offset correction on the signals before each signal superposition.

[0080] Specifically, before executing the POS and CHROM algorithms based on the data in the first queue 131, the processing unit 11 also corrects the most recent multiple frames of a first length D in the superposition window 1311. The correction amount for each frame is 1 / D of the mean difference between the multiple frames, where the first length D is less than the superposition length N. In one embodiment, the first length D is one-fifth of the superposition length N. For example, if the superposition length N is 50 frames, the first length D can be 10 frames. Before superposition, the detection method of the present invention first performs a small segment offset correction on the data in the superposition window 1311 of 1 / 5 of the superposition length N. For example, if the superposition length N is 50 frames, the length of the small segment offset correction is 10 frames. Due to the accumulation, the correction amount for each frame is 1 / 10 of the mean difference between the 10 frames, thereby gradually aligning the signal toward the mean.

[0081] Back to Figure 8 After processing unit 11 calculates and generates the third average skin color value (e.g., 1.1) (i.e., the second time), processing unit 11 shifts the data in first queue 131 and superposition window 1311 one unit to the left and fills the latest average skin color value into the latest field of first queue 131 and superposition window 1311. During superposition, processing unit 11 superimposes the data in the first field of superposition window 1311 (i.e., 1.1) with the data in the first field of first queue 131 (i.e., 1.1), superimposes the data in the second field of superposition window 1311 (i.e., 0.6) with the data in the second field of first queue 131 (i.e., 1.3), and superimposes the data in the third field of superposition window 1311 (i.e., 0.2) with the data in the third field of first queue 131 (i.e., 0.5).

[0082] Similarly, after processing unit 11 calculates and generates the fourth average skin color value (e.g., 0.5) (i.e., the third time), processing unit 11 shifts the data in first queue 131 and superposition window 1311 one unit to the left and fills the latest average skin color value into the latest field of first queue 131 and superposition window 1311. During superposition, processing unit 11 superimposes the data in the first field of superposition window 1311 (i.e., 0.5) with the data in the first field of first queue 131 (i.e., 0.5), superimposes the data in the second field of superposition window 1311 (i.e., 1.0) with the data in the second field of first queue 131 (i.e., 2.1), and superimposes the data in the third field of superposition window 1311 (i.e., 0.6) with the data in the third field of first queue 131 (i.e., 1.9). It is worth mentioning that at this time, the position of the fourth field in the first queue 131 does not have a corresponding superposition window 1311 , so the data of the fourth field in the first queue 131 (ie, 0.5) will not be further superimposed and corrected.

[0083] After processing unit 11 calculates and generates the fifth average skin color value (e.g., 0.1) (i.e., the fourth time), processing unit 11 shifts the data in first queue 131 and superposition window 1311 one unit to the left and populates the latest average skin color value into the latest field of first queue 131 and superposition window 1311. During superposition, processing unit 11 superimposes the data in the first field of superposition window 1311 (i.e., 0.1) with the data in the first field of first queue 131 (i.e., 0.1), superimposes the data in the second field of superposition window 1311 (i.e., 0.6) with the data in the second field of first queue 131 (i.e., 1.1), and superimposes the data in the third field of superposition window 1311 (i.e., 0.9) with the data in the third field of first queue 131 (i.e., 3.0). After processing unit 11 calculates and generates the sixth average skin color value (e.g., -0.3) (i.e., the fifth time), processing unit 11 shifts the data in first queue 131 and superposition window 1311 one unit to the left and fills the latest average skin color value into the latest field of first queue 131 and superposition window 1311. During superposition, processing unit 11 superimposes the data in the first field of superposition window 1311 (i.e., -0.3) with the data in the first field of first queue 131 (i.e., -0.3), superimposes the data in the second field of superposition window 1311 (i.e., 0.0) with the data in the second field of first queue 131 (i.e., 0.1), and superimposes the data in the third field of superposition window 1311 (i.e., 0.5) with the data in the third field of first queue 131 (i.e., 1.6).

[0084] While the electronic device 1 continuously captures images and acquires input frames, the processing unit 11 continuously performs superposition processing on the data in the first queue 131 to update the data in the first queue 131. The detection method of the present invention continuously updates the first queue 131 and continuously performs the POS algorithm and the CHROM algorithm based on the updated data in the first queue 131, thereby generating a first rPPG waveform signal 61 and a second rPPG waveform signal 62 based on a continuous time series.

[0085] It is worth mentioning that before outputting the first rPPG waveform signal 61 and the second rPPG waveform signal 62, the processing unit 11 can also perform amplitude limiting processing on the first rPPG waveform signal 61 and the second rPPG waveform signal 62. By limiting the positive and negative signal strengths of the first rPPG waveform signal 61 and the second rPPG waveform signal 62 to half of the original signal strength, high-frequency noise and low-frequency noise are filtered out. Figure 6A and Figure 6BAccording to the embodiment, the upper limit of the original signal strength of the first rPPG waveform signal 61 and the second rPPG waveform signal 62 is +50 to -50, and after the amplitude limiting processing, the signal strength of the first rPPG waveform signal 611 and the second rPPG waveform signal 621 is limited to between +25 and -25.

[0086] Specifically, after the offset correction, the signal moves closer to the centerline. Amplitude limiting is then performed to limit the upper and lower limits of the intensity. This eliminates small oscillations at the tail end of the signal without affecting signal accuracy, thereby outputting a clear signal that is closer to a square wave. The present invention performs offset correction and amplitude limiting before outputting the first and second rPPG waveform signals 611 and 621. This allows for a relatively clear heart rate rhythm to be obtained from the combined waveform signal 63 generated based on the first and second rPPG waveform signals 611 and 621. Therefore, the processing unit 11 can directly calculate a clear frequency based on the combined waveform signal 63, eliminating the need for further noise elimination in subsequent processing.

[0087] Back to Figure 2 After step S25, the processing unit 11 performs a Fast Fourier Transform (FFT) on the combined waveform signal 63 to convert the combined waveform signal 63 in the time domain into a combined spectrum in the frequency domain (step S26). Finally, the processing unit 11 performs a Fast Fourier Transform (FFT) on the combined waveform signal 63 (for example, Figure 9 The combined spectrum 91 shown is subjected to spectrum analysis processing to extract the heart rate output of the user 2 (step S27).

[0088] Please also see Figure 9 , is an embodiment of a schematic diagram of a spectrum of the present invention. Figure 2 In step S26, the processing unit 11 performs fast Fourier transform processing on the combined waveform signal 63 to generate frequency domain information, and takes the absolute value of the frequency domain information and normalizes it to obtain the combined spectrum 91. Figure 9 In the embodiment of the present invention, the X-axis of the spectrum is the frequency converted based on the number of frames stored in the first queue 131 and the time. Figure 9 0 to 4 is used as an example; the Y axis of the spectrum is the normalized frequency intensity. Figure 9 Take 0 to 1 as an example.

[0089] Taking the 250 frames of data stored in the first queue 131 as an example, after converting the combined waveform signal 63 in the time domain into the combined spectrum 91 in the frequency domain, the processing unit 11 can determine how long the current 250 frames occurred and how many amplitudes are contained therein. In this way, the processing unit 11 can calculate the number of frequencies contained therein and thereby analyze the heart rate of user 2. Specifically, after obtaining a clear spectrum, the processing unit 11 can calculate the actual frequency represented by the spectrum by combining the number of sampled frames with the timestamps of the first and last frames. Furthermore, the processing unit 11 can extract the spectrum corresponding to the human heart rate range of approximately 0.66 to 4 Hz (corresponding to 40 to 240 bpm) from the spectrum, and then analyze the current heart rate of user 2 based on the intensity distribution of each frequency.

[0090] It's worth noting that when the electronic device 1 continuously records images, the total number of pixels in each input frame may vary due to the movement or rotation of the user 2's head. This difference in total pixel count may also affect the resulting spectrum. To optimize the spectrum used as the basis for heart rate analysis, the detection method of the present invention further deducts this difference in total pixel count from the spectrum to optimize it. This optimized spectrum allows the processing unit 11 to perform more accurate spectrum analysis.

[0091] Specifically, in Figure 2 In steps S22 and S23, the processing unit 11 analyzes the facial image to obtain a skin color reference and a plurality of skin color pixels that meet the standard deviation, and then establishes a skin color mean based on the plurality of skin color pixels and stores the skin color mean in the first queue 131. In one embodiment, the processing unit 11 also records the total number of skin color pixels in each input frame when analyzing the facial image, and then calculates the skin color pixel variation between the preceding and following frames in time, and stores the skin color pixel variation in the second queue (e.g., Figure 1 The second queue 132 is shown).

[0092] exist Figure 2 In step S26, the processing unit 11 performs a fast Fourier transform on the combined waveform signal to generate a combined spectrum 91. Furthermore, the processing unit 11 also performs a fast Fourier transform on the skin color pixel variations of the plurality of input frames in the second queue 132 to generate a skin color pixel variation spectrum 92.

[0093] In one embodiment, the processing unit 11 can directly perform spectral analysis on the combined spectrum 91 to extract the heart rate output of user 2. In another embodiment, the processing unit 11 subtracts the skin color pixel variation spectrum 92 from the combined spectrum 91 to generate a final spectrum 93. The processing unit 11 then performs spectral analysis on the final spectrum 93 to extract the heart rate output of user 2. By directly subtracting the skin color pixel variation spectrum 92 from the main spectrum generated based on the rPPG signal (i.e., the combined spectrum 91), a clearer main frequency can be obtained, making it easier for the processing unit 11 to extract the correct heart rate.

[0094] from Figure 9 It can be seen from the figure that in the final spectrum 93, the strongest frequency is concentrated around 1.2-1.3 Hz. Therefore, the detection method of the present invention uses the following method to extract the heart rate output of user 2 from the final spectrum 93. Figure 10 , which is an embodiment of the spectrum analysis flow chart of the present invention. As previously described, processing unit 11 can perform spectrum analysis on combined spectrum 91 or final spectrum 93 to extract the heart rate output of user 2. For ease of explanation, the following example uses processing unit 11 performing spectrum analysis on combined spectrum 91 as an example, but the present invention is not limited to this.

[0095] At Figure 10 In the embodiment, processing unit 11 first determines whether there are no frequencies with frequency intensities greater than 0.5 in combined spectrum 91 (step S101). If it is determined that there are no frequencies with frequency intensities greater than 0.5 in combined spectrum 91, calculation is difficult due to the lack of a clear dominant frequency, and thus processing unit 11 temporarily refrains from calculating the heart rate output (step S102).

[0096] If it is determined that there is at least one frequency with a frequency intensity greater than 0.5 in the combined spectrum 91 (i.e., the determination in step S101 is no), the processing unit obtains a frequency with a maximum intensity in the combined spectrum 91 (step S103). Figure 9 Taking the combined spectrum 91 as an example, because the frequency intensity of the multiple frequencies in the combined spectrum 91 is the highest at 1, in step S103, the processing unit 11 obtains the frequency with an intensity of 1 as the frequency with the highest intensity. At the same time, the processing unit 11 obtains one or more frequencies in the combined spectrum 91 whose frequency intensity is greater than 50% of the frequency with the highest intensity (step S104). For example, if the frequency intensity of the frequency with the highest intensity is 0.8, then in step S104, the processing unit 11 only obtains one or more frequencies in the combined spectrum 91 whose frequency intensity is greater than 0.4, and ignores one or more frequencies whose frequency intensity is less than or equal to 0.4.

[0097] Next, the processing unit 11 calculates the average frequency of the merged spectrum 91 based on the one or more frequencies obtained in step S104. It also calculates the difference between the average frequency and the maximum intensity frequency, converts the difference to beats per minute (BPM), and then determines whether the difference between the average frequency and the maximum intensity frequency is greater than 5 bpm (step S105). If the difference between the average frequency and the maximum intensity frequency is greater than 5 bpm in step S105, it indicates that the current signal is unstable (e.g., there are multiple main frequencies), and the processing unit 11 temporarily suspends the calculation of the heart rate output (step S102). Meanwhile, the electronic device 1 continues to capture images via the image capture unit 12, and the processing unit 11 continuously generates the first rPPG waveform signal 61, the second rPPG waveform signal 62, the first rPPG waveform signal 611, the second rPPG waveform signal 621, the merged waveform signal 63, the merged spectrum 91, the skin color pixel variation spectrum 92 (optional), and the final spectrum 93 (optional).

[0098] If it is determined in step S105 that the difference between the average frequency and the maximum intensity frequency is no more than 5 bpm, it means that the current signal is stable. At this time, the processing unit 11 calculates the weighted average of the maximum intensity frequency and one or more frequencies with frequency intensities greater than 50% of the maximum intensity frequency, and converts the weighted average into the number of heart contraction beats as the heart rate output (step S106).

[0099] The above is only one of the spectrum analysis methods used in the detection method of the present invention, but the present invention is not limited to the above method.

[0100] The detection method of the present invention uses at least two modified algorithms to generate a unique rPPG signal and then performs an inverse merging process on the two rPPG signals. This allows for a clearer spectrum to be obtained after Fast Fourier Transform processing, thereby improving the accuracy of heart rate detection results.

Claims

1. A heart rate detection method using a composite remote photoplethysmography system, applied to an electronic device comprising at least one image capture unit and a processing unit, characterized in that: include: Step a) the image capture unit continuously captures input frames over time, wherein the input frames have facial images; Step b) the processing unit calculates a plurality of feature points in the facial image to obtain a skin color average, and stores the skin color average and a corresponding timestamp in a first queue; Step c) the processing unit executes a Plane-Orthogonal-to-Skin (POS) algorithm and a Robust Pulse Rate from Chrominance-based rPPG (CHROM) algorithm based on the skin color mean values ​​of the plurality of input frames in the first queue to generate a first rPPG waveform signal and a second rPPG waveform signal, respectively; Step d) the processing unit performs reverse merging processing on the first rPPG waveform signal and the second rPPG waveform signal to generate a merged waveform signal; Step e) performing fast Fourier transform processing on the combined waveform signal by the processing unit to generate a combined spectrum; and Step f) The processing unit performs spectrum analysis on the combined spectrum to extract the heart rate output.

2. The heart rate detection method using composite remote photoplethysmography according to claim 1, characterized in that: The step b) then comprises: Step b11) performing a brightness detection process on the skin color mean and generating a brightness detection result; and Step b12) Feedback the brightness detection result to the image capture unit to adjust the exposure used by the image capture unit when capturing the next input frame.

3. The heart rate detection method using composite remote photoplethysmography according to claim 1, wherein: The first queue has a queue length M, and the first queue has an overlapping window, wherein the overlapping length N of the overlapping window is less than the queue length M, and the step b) further comprises: Step b21) When the number of data frames in the first queue is greater than the superposition length N, execute step c).

4. The heart rate detection method using composite remote photoplethysmography according to claim 1, wherein: The step b) comprises: Step b01) identifying the plurality of feature points in the facial image; Step b02) capturing skin color values ​​of the plurality of feature points and a plurality of sampling points surrounding the plurality of feature points, and calculating an average of the plurality of skin color values ​​to obtain a skin color reference; Step b03) filtering out a plurality of skin color pixels within a certain range of standard deviation from the skin color reference within a facial skin area, wherein the facial skin area is the facial image after removing the eyes; and Step b04) establishing the skin color average based on the plurality of skin color pixels, and storing the skin color average and the corresponding timestamp in the first queue.

5. The heart rate detection method using composite remote photoplethysmography according to claim 4, characterized in that: The step b03) is to filter out multiple skin color pixels within a standard deviation of plus or minus 1.5 times to plus or minus 3 times of the skin color reference.

6. The heart rate detection method using composite remote photoplethysmography according to claim 1, wherein: The first queue has a queue length M and an overlapping window, wherein the overlapping window has an overlapping length N less than the queue length M and is used to record the latest N skin color average values ​​in the first queue. When the POS algorithm and the CHROM algorithm generate the first rPPG waveform signal and the second rPPG waveform signal, the N skin color average values ​​in the overlapping window updated in step b) are respectively superimposed with the latest N skin color average values ​​in the first queue.

7. The heart rate detection method using composite remote photoplethysmography according to claim 6, characterized in that: Before outputting the first rPPG waveform signal and the second rPPG waveform signal, the POS algorithm and the CHROM algorithm correct the latest multiple frames of a first length D in the superposition window, where the correction amount for each frame is 1 / D of the mean difference of the multiple frames, and the first length D is less than the superposition length N.

8. The heart rate detection method using composite remote photoplethysmography according to claim 7, characterized in that: Before outputting the first rPPG waveform signal and the second rPPG waveform signal, the POS algorithm and the CHROM algorithm perform amplitude limiting processing on the first rPPG waveform signal and the second rPPG waveform signal, so that the positive and negative signal strengths of the first rPPG waveform signal and the second rPPG waveform signal are limited to half of the original signal strength, thereby filtering out high-frequency noise and low-frequency noise.

9. The heart rate detection method using composite remote photoplethysmography according to claim 1, wherein: The reverse merging process is to extract the maximum intensity at each frame node of the first rPPG waveform signal and the second rPPG waveform signal, and take the larger absolute value of the two as the signal intensity of each frame node in the merged waveform signal.

10. The heart rate detection method using composite remote photoplethysmography according to claim 1, wherein: include: Step g) After step b), the processing unit records the total number of skin color pixels, calculates the skin color pixel variation between the previous and next frames, and stores the variation in the calculation in a second queue; Step h) performing the fast Fourier transform process on the skin color pixel variation of the plurality of input frames in the second queue to generate a skin color pixel variation spectrum; and Step i) after step e), subtracting the skin color pixel variation spectrum from the combined spectrum to generate a final spectrum; Wherein, the step f) is to perform spectrum analysis processing on the final spectrum by the processing unit to extract the heart rate output.

11. The heart rate detection method using composite remote photoplethysmography according to claim 1, wherein: The step e) includes the processing unit performing the fast Fourier transform processing on the combined waveform signal to generate frequency domain information, taking the absolute value of the frequency domain information and normalizing it to obtain the combined spectrum, wherein the signal strengths of multiple frequencies in the combined spectrum are between 0 and 1.

12. The heart rate detection method using composite remote photoplethysmography according to claim 11, characterized in that: The spectrum analysis process includes calculating the real frequency represented by the combined spectrum based on the number of frames included in the first queue and the timestamp, extracting the spectrum within the human heart rate range, and obtaining the heart rate output based on the intensity distribution of each frequency.

13. The heart rate detection method using composite remote photoplethysmography according to claim 11, characterized in that: The step f) comprises: In step f1), when there is no frequency with a frequency intensity greater than 0.5 in the combined spectrum, the heart rate output is not calculated.

14. The heart rate detection method using composite remote photoplethysmography according to claim 13, wherein: The step f) comprises: Step f2) obtaining the frequency with the maximum intensity in the combined spectrum; Step f3) obtaining one or more frequencies in the combined spectrum whose frequency intensity is greater than 50% of the frequency with the maximum intensity; and Step f4) Calculate the weighted average of the maximum intensity frequency and the one or more frequencies whose frequency intensity is greater than 50% of the maximum intensity frequency, and convert the weighted average into beats per minute (BPM) to output as the heart rate.

15. The heart rate detection method using composite remote photoplethysmography according to claim 14, characterized in that: The step f) comprises: Step f5) calculating the average frequency of the combined spectrum; and Step f6) When the difference between the average frequency and the maximum frequency of the intensity is greater than 5 bpm, the heart rate output is not calculated.

Images