Region of interest based heart rate determination method, apparatus and electronic device

Abstract

The application is suitable for the field of remote physiological parameter monitoring, and provides a heart rate determination method and device based on a region of interest and an electronic device, which comprises the following steps: acquiring a video stream comprising a face of a user to be measured; performing region of interest detection on the video stream by using at least two region of interest templates to obtain corresponding regions of interest; for each region of interest, determining a time sequence signal corresponding to the region of interest, performing blind source separation on the time sequence signal by using an independent component algorithm to obtain an independent component signal related to a heart rate, and calculating the heart rate according to the independent component signal to obtain a heart rate corresponding to the region of interest; determining a weight value of each region of interest; and determining a heart rate of the user to be measured according to the weight value and the heart rate corresponding to each region of interest. By using the above method, local abnormal disturbance can be inhibited, and the accuracy of the heart rate can be improved.

Description

Technical Field

[0001] This application belongs to the field of remote physiological parameter monitoring technology, and in particular relates to methods, devices, electronic devices, computer-readable storage media, and computer program products for determining heart rate based on regions of interest. Background Technology

[0002] Heart rate refers to the number of times the heart beats per minute, and it is a basic physiological parameter for assessing cardiovascular function. Heart rate can be detected by methods such as palpation and auscultation, or it can be detected using the Independent Component Analysis (ICA) algorithm.

[0003] When using the ICA algorithm to detect heart rate, the process typically involves the following: A continuous video image sequence of the subject's face is acquired non-contactly using a camera device. A region of interest (ROI) is determined from this sequence. Based on this ROI, time-varying red (R), green (G), and blue (B) temporal signals are identified. These RGB temporal signals contain subtle changes in light intensity caused by blood flow variations. The ICA algorithm is then applied to perform blind source separation on the RGB temporal signals to obtain independent components related to heart rate. These independent components are analyzed, and the corresponding heart rate (or heart rate value) is determined based on the analysis results.

[0004] However, the heart rate determined by the above method has low accuracy. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for determining heart rate based on a region of interest, which can solve the problem of low accuracy in determining heart rate using existing methods.

[0006] In a first aspect, embodiments of this application provide a method for determining heart rate based on a region of interest, including:

[0007] Acquire a video stream including the face of the user under test; At least two region of interest (ROI) templates are used to perform ROI detection on the video stream to obtain the corresponding ROI. For each region of interest, the time series signal corresponding to the region of interest is determined, and the independent component algorithm is used to perform blind source separation on the time series signal to obtain the independent component signal related to heart rate. The heart rate is calculated based on the independent component signal to obtain the heart rate corresponding to the region of interest. Determine the weight value of each region of interest; The heart rate of the user to be tested is determined based on the weight value corresponding to each region of interest and the heart rate.

[0008] The beneficial effects of the embodiments in this application compared with the prior art are: In this embodiment, after obtaining the heart rate corresponding to at least two regions of interest (ROIs) of the user under test, the weight value of each ROI is determined. Then, based on the weight value of each ROI and the heart rate, the heart rate of the user under test is determined. Since local motion or uneven illumination in a single ROI is unlikely to affect all ROIs simultaneously, while different ROIs can provide spatially diverse physiological signal samples, and the fusion process can automatically suppress local abnormal disturbances, when the heart rate of the user under test is obtained by fusing the heart rates of different ROIs on the user's face, the heart rate estimation of the user under test becomes more resistant to local interference, and the accuracy is systematically improved.

[0009] Secondly, embodiments of this application provide a heart rate determination device based on a region of interest, comprising: The video stream acquisition module is used to acquire a video stream including the face of the user under test; The region of interest (ROI) determination module is used to perform ROI detection on the video stream using at least two ROI templates to obtain the corresponding ROI. The heart rate calculation module for the region of interest is used to determine the time series signal corresponding to each region of interest, perform blind source separation on the time series signal using the independent component algorithm to obtain independent component signals related to heart rate, and calculate the heart rate corresponding to the region of interest based on the independent component signals. The weight value determination module for regions of interest is used to determine the weight value of each region of interest; The heart rate determination module for the user under test is used to determine the heart rate of the user under test based on the weight value corresponding to each region of interest and the heart rate.

[0010] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0011] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0012] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect.

[0013] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0015] Figure 1 This is a flowchart illustrating a method for determining heart rate based on a region of interest, as provided in an embodiment of this application. Figure 2 This is a schematic diagram illustrating the division of a facial region of interest according to an embodiment of this application; Figure 3 This is a schematic diagram of a heart rate determination device based on a region of interest according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0017] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0018] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0019] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0020] When using the ICA algorithm to detect heart rate, detecting only a single region of interest (such as the face as a whole or a specific area of ​​the face as a region of interest) may result in low accuracy. This is because a single region of interest is easily affected by local factors (such as beards, wrinkles, and pigmentation). Therefore, when detecting heart rate in a single region of interest, the measurement accuracy will significantly decrease under conditions of localized movement or uneven lighting.

[0021] To improve the accuracy of heart rate detection using the ICA algorithm, this application provides a heart rate determination method based on a region of interest.

[0022] In this heart rate determination method, the face of the user to be tested is divided into at least two regions of interest (ROIs), and heart rate is detected in each ROI separately. After obtaining the heart rate corresponding to each ROI, the heart rates determined by fusing the weight values ​​of each ROI are used to obtain the heart rate of the user to be tested.

[0023] Since the heart rate of the user being tested is obtained by fusing different regions of interest on the user's face, the fused heart rate can effectively suppress the influence of local motion or uneven illumination on a single region of interest, thereby improving the accuracy of the determined heart rate of the user being tested.

[0024] The method for determining heart rate based on region of interest provided in this application is described below with reference to the accompanying drawings.

[0025] Figure 1 A flowchart illustrating a heart rate determination method based on a region of interest provided in this application is shown below, in detail: S11, acquire a video stream including the face of the user to be tested.

[0026] In this embodiment, considering the significant channel differences and high data fidelity of the Red (R), Green (G), and Blue (B) color spaces, the color space of the aforementioned video stream can be the RGB color space. Specifically, an RGB camera can be used to capture the user's face to obtain a video stream with a color space including the RGB color space. An RGB camera is a device that generates a color image by capturing the brightness values ​​of the R, G, and B color channels.

[0027] Optionally, the frame rate of the aforementioned video stream shall not be less than 30 frames per second (30 fps) to meet the requirements of heart rate measurement range (30 fps can fully capture the detailed changes in heart rate signals), motion artifact suppression (30 fps can better capture subtle head movements and facial expression changes), and so on.

[0028] S12, use at least two region of interest templates to perform region of interest detection on the above video stream to obtain the corresponding region of interest.

[0029] The aforementioned region of interest (ROI) template is a pre-defined template for the face. Optionally, considering that regions with different blood vessel densities typically reflect varying levels of heart rate signal richness, corresponding ROIs can be defined according to different blood vessel densities. For example, since the blood vessel densities corresponding to areas such as the forehead, left cheek, right cheek, nose, and chin are usually different, the face can be divided into five ROIs. Figure 2 As shown, the face can be divided into the forehead region 201, left cheek region 202, right cheek region 203, nose region 204, and chin region 205. Among them, the forehead region 201 has the highest blood vessel density. Figure 2 The blood vessels in the left cheek region 202 and right cheek region 203 are also highly dense and suitable for signal acquisition. The blood vessel density in the nasal region 204 is moderate. Figure 2 The vascular density in the middle region (represented by a medium-density lattice) is easily affected by respiratory movements; the vascular density in the chin region 205 is lower ( Figure 2 (represented by sparse dots), and is easily affected by facial movements such as speaking.

[0030] In this embodiment, region of interest (ROI) detection is performed on the image frame sequence in the video stream to detect the ROI corresponding to the ROI template in each image frame. For example, assuming the ROI template includes the left cheek and the right cheek, when performing ROI detection on the image frame based on the ROI template, it will detect whether the image frame has a region matching the left cheek, and whether the image frame has a region matching the right cheek. If both exist, the obtained ROI are: the left cheek region and the right cheek region.

[0031] S13. For each region of interest, determine the time series signal corresponding to the region of interest, use the independent component algorithm to perform blind source separation on the time series signal to obtain the independent component signal related to heart rate, and calculate the heart rate corresponding to the region of interest based on the independent component signal.

[0032] In this embodiment, for each region of interest (ROI) in the aforementioned ROI sequence, the spatial average value of the ROI across each channel is calculated. Considering that different channels have varying sensitivities to heart rate signals and different channels typically exhibit varying robustness to different types of noise, and that while using only a single channel (e.g., only the G channel) can extract heart rate signals under ideal conditions, it suffers from the following problems in complex environments: a single channel is easily affected by specific types of noise (e.g., changes in illumination primarily affect the V and Y channels, while changes in skin color primarily affect the Cr and Cb channels); a single channel lacks redundant information and cannot effectively separate heart rate signals from noise using blind source separation techniques such as ICA; and in low signal-to-noise ratio scenarios, the measurement accuracy and robustness of a single channel significantly decrease. Therefore, in this embodiment, the spatial average value of the ROI across multiple channels is calculated. For example, if the color space of the video frame is RGB, when calculating the spatial average value of a ROI in that video frame, the spatial average values ​​of the ROI in the R, G, and B channels can be calculated separately.

[0033] After obtaining the spatial average value of each channel, the time series signal corresponding to each channel can be determined based on the spatial average value of each region of interest in each channel. A mixed signal matrix is ​​then constructed based on the time series signals corresponding to each channel. Specifically, after obtaining the time series signal, a second-order polynomial can be used to fit the time series signal to remove the low-frequency baseline drift. The mixed signal matrix is ​​then constructed based on the time series signal after removing the low-frequency baseline drift. Assume the constructed mixed signal matrix is... , where x i Let x be the complete time series signal after filtering of the i-th channel. i It is not a numerical value, but a time series vector.

[0034] in, …………Formula (1); In the above formula (1), T is the length of the time series signal (i.e., the total number of video frames in the video stream), x i (t) represents the spatial average value of the i-th channel in the t-th video frame (i.e., time t) (this spatial average value can be the spatial average value after baseline shift and bandpass filtering).

[0035] For a single region of interest, the dimension of its corresponding mixed signal matrix X is n×T, where: n is the number of channels (when there are 9 channels R, G, B, Y, Cr, Cb, H, S, V, n is 9), and T is the length of the time series signal (total number of frames).

[0036] Each row of the mixed signal matrix X is a complete time series signal of one channel, and each column is the spatial average of all channels at a certain time.

[0037] In this embodiment of the application, when there are multiple regions of interest, an independent hybrid signal matrix X is constructed for each region of interest. For example, if there are 5 regions of interest, 5 independent hybrid signal matrices are constructed: X1, X2, X3, X4, X5.

[0038] Based on the aforementioned mixed signal matrix and the preset objective function, the separation matrix to be iterated is iterated until one of the following iteration stopping conditions is met to obtain the target separation matrix. The objective function includes the negative entropy of the separated signal and a heart rate signal constraint term. The heart rate signal constraint term is used to constrain the separated signal obtained to conform to the physiological law of heart rate. The iteration stopping conditions include: the objective function reaching a preset convergence threshold and the number of iterations reaching a preset upper limit.

[0039] After obtaining the target separation matrix, the above mixed signal matrix is ​​converted into an independent component matrix based on the target separation matrix, and then the heart rate is calculated based on the independent components related to heart rate in the independent component matrix.

[0040] Assuming the target separation matrix is ​​W and the independent component matrix is ​​S, the separation matrix W is used to transform the mixed signal matrix X into the independent component matrix S. W has dimensions m×n, where m is the number of independent components to be separated (usually m=n), and n is the number of input channels. When n=9, the above W can be a 9×9 matrix, with each row w... i is a separation vector used to extract the i-th independent component. Since typically only one of the multiple separated signals contained in the independent component matrix S is the heart rate signal, while the others are usually signals corresponding to interference components such as noise, motion artifacts, and illumination changes, the score of the heart rate signal constraint term corresponding to each separated signal in the independent component matrix can be calculated. Based on the scores of the heart rate signal constraint terms corresponding to each separated signal, the separated signal that best matches the physiological characteristics of heart rate is selected as the independent component signal for calculating heart rate. The physical meaning of W is: each row of W represents a "projection direction," and each row of W... i This is used to project the mixed signal from the nine channels onto this direction to obtain a single component signal.

[0041] Since Gaussian distributions have the highest signal entropy, while non-Gaussian signals have lower entropy, negative entropy (i.e., negative signal entropy) is used as part of the objective function. The iteration stopping conditions include the objective function reaching a preset convergence threshold and the number of iterations reaching a preset upper limit. Therefore, when iterating the target separation matrix based on the preset objective function, the probability of obtaining the separated signal being the least Gaussian signal is high. Furthermore, if the calculated objective function does not reach the preset convergence threshold but the number of iterations reaches the preset upper limit, the iteration can be stopped promptly. In complex scenarios (such as motion or changes in lighting), the independent components separated solely by negative entropy may not be heart rate signals but motion artifacts or other noise. Therefore, a heart rate signal constraint term is introduced into the objective function. This heart rate signal constraint term ensures that the separated signal conforms to the physiological laws of heart rate, thus making the heart rate calculation based on this independent component signal more accurate.

[0042] S14, determine the weight values ​​for each of the above-mentioned regions of interest.

[0043] In this embodiment, the weight value of each region of interest can be a fixed value, for example, the weight value of the region of interest can be determined based on the vascular density of each region of interest; the weight value of each region of interest can also be a variable value, for example, the weight value of the region of interest can be determined based on the quality score corresponding to each region of interest at the current time (or based on vascular density and quality score).

[0044] Optionally, determining the weight values ​​for each of the aforementioned regions of interest includes: A1. Obtain the blood vessel density corresponding to each of the above-mentioned regions of interest, and / or determine the quality score corresponding to each of the above-mentioned regions of interest at the current time. The quality score reflects the signal quality of the above-mentioned regions of interest.

[0045] The blood vessel density of the aforementioned regions of interest is determined according to the proportion of blood vessels per unit area or unit volume. In this embodiment, the blood vessel density of each region of the face is arranged in descending order as follows: forehead region, cheek region (left cheek region, right cheek region), nose region, and chin region.

[0046] The quality score mentioned above can be determined based on at least one of the following: the degree of interference of head movement on the video stream signal, the signal-to-noise ratio of the independent component signal related to heart rate, the stability of heart rate in the time domain, the sufficiency and uniformity of illumination, etc.

[0047] A2. Based on the above vascular density and / or quality scores, determine the corresponding weight values ​​for the above-mentioned regions of interest.

[0048] In this embodiment, the greater the blood vessel density of the region of interest, the greater the weight value corresponding to the region of interest; the higher the quality score of the region of interest, the greater the weight value corresponding to the region of interest.

[0049] Optionally, when determining weight values ​​based on vascular density and quality score, the weight value of the region of interest can be determined based on the product of the vascular density and quality score.

[0050] Because changes in blood volume cause variations in reflection / absorption, i.e., vascular density affects the photoplethysmography (PPG) signal, and thus the signal-to-noise ratio of the region of interest (ROI), determining the weight value of the ROI based on vascular density can adjust the weight of the heart rate determined from that ROI, thereby improving the accuracy of the heart rate obtained through subsequent fusion. Furthermore, since the quality score reflects the signal quality of the ROI, i.e., the reliability of the signal used to calculate the heart rate, determining the weight value of the ROI based on the quality score can adjust the weight of the heart rate determined from that ROI, thereby improving the accuracy of the heart rate obtained through subsequent fusion.

[0051] In this embodiment of the application, considering that the weight values ​​of the determined regions of interest may be too small, in order to avoid introducing noise, the excessively small weight values ​​are adjusted to 0. That is, in S14 above, determining the weight values ​​of each of the aforementioned regions of interest includes: B1. Determine the candidate weight values ​​for each of the above-mentioned regions of interest.

[0052] B2. For any candidate weight value of a region of interest, if the candidate weight value is less than the preset minimum weight threshold, the candidate weight value is adjusted to 0 to obtain the weight value of the region of interest.

[0053] In this embodiment, a candidate weight value that may need adjustment is determined (this candidate weight value can be determined based on vascular density and / or quality score). This candidate weight value is compared with a preset minimum weight threshold (e.g., set to 0.05). If it is determined that the candidate weight value is less than the preset minimum weight threshold, the candidate weight value is set to 0. Since setting excessively small weight values ​​to 0 is equivalent to removing the heart rate contribution determined based on the corresponding region of interest in the subsequent weighted fusion stage, thereby avoiding the bias of the final heart rate by low-confidence or high-noise regions of interest, and thus improving the accuracy of the heart rate obtained by subsequent weighted fusion.

[0054] Optionally, the aforementioned minimum weight threshold is determined in the following way: Determine the current scenario of the user under test, including resting scenarios and moving scenarios; determine the minimum weight threshold based on the current scenario of the user under test.

[0055] In this embodiment of the application, considering that the quality score determined based on the video stream of the user under test is usually different when the user is in a resting scene and an active scene, the minimum weight threshold for filtering the minimum weight value can be determined according to the scene in which the user under test is currently located, so as to improve the accuracy of the filtering results.

[0056] Optionally, the minimum weight threshold for a user in a resting state can be set to be lower than the minimum weight threshold for a user in an active state. For example, the minimum weight threshold can be set to 0.05 when the user is in a resting state, and to 0.1 when the user is in an active state. This setting helps to more rigorously filter out higher-quality regions of interest during exercise.

[0057] In this embodiment, considering that the left and right cheek regions are symmetrical, meaning that under normal circumstances, the heart rate detected from the left cheek region should be the same as or have a small difference from the heart rate detected from the right cheek region, a large difference in heart rate detected from these two regions of interest indicates interference. In this case, the weight values ​​of these two regions of interest can be reduced. That is, in S14 above, determining the weight value of each of the aforementioned regions of interest includes: C1. Determine the candidate weight value of the left cheek region and the candidate weight value of the right cheek region.

[0058] C2. If the absolute value of the difference between the heart rate corresponding to the left cheek region and the heart rate corresponding to the right cheek region is greater than a preset heart rate difference threshold, then the left cheek candidate weight value is reduced to obtain the left cheek weight value of the left cheek region, and the right cheek candidate weight value is reduced to obtain the right cheek weight value of the right cheek region.

[0059] The heart rate difference threshold mentioned above can be set according to the actual situation. For example, the heart rate difference threshold can be set to 10 bpm. Of course, it can also be set to other values, which are not limited here.

[0060] In this embodiment, after determining the candidate weight value (or candidate weight value) of the left cheek region (or right cheek region), the candidate weight value of the left cheek is reduced. The reduction value can be set according to the actual situation. For example, the reduction value can be set to half of the candidate weight value of the left cheek, or set to a fixed reduction value; this is not limited here. Optionally, the aforementioned candidate weight value (or candidate weight value) of the left cheek region (or right cheek region) can be determined based on the vascular density and / or quality score of the left cheek region (or right cheek region); this is not limited here.

[0061] For example, assuming the heart rate in the left cheek region is represented by HR2, the heart rate in the right cheek region by HR3, the candidate weight value for the left cheek region by w2', and the candidate weight value for the right cheek region by w3', and the heart rate difference threshold is 10 bpm, with the reduction value being half of the candidate weight value of the region of interest, then the left cheek weight value w2 and the right cheek weight value w3 can be determined according to the following formula: If |HR2-HR3|>10bpm…………Formula (2), then Formula (3) and Formula (4) are used to reduce the weights of the two respectively: w2=w2'×0.5……………………Formula (3); w3=w3'×0.5……………………Formula (4); Optionally, after determining w2 and w3 as described above, all weight values ​​can be renormalized.

[0062] In this embodiment of the application, to avoid drastic fluctuations in weight values, time-domain smoothing of the weight values ​​can be performed. That is, in S14 above, determining the weight values ​​of each of the aforementioned regions of interest includes: D1. Determine the candidate weight values ​​for each of the above regions of interest at the current time.

[0063] D2. For any candidate weight value of a region of interest at the current time, determine the weight value of the region of interest at the current time based on the historical weight values ​​corresponding to the region of interest and the candidate weight values ​​of the region of interest at the current time.

[0064] The aforementioned historical weight values ​​can be the weight values ​​of the previous time point of the current time point. Of course, in actual practice, the weight values ​​of the two time points above the current time point can also be selected, which is not limited here.

[0065] In this embodiment of the application, corresponding weights can be assigned to the historical weight values ​​and the candidate weight values ​​at the current moment, and the weight corresponding to the candidate weight value at the current moment is greater than the weight corresponding to the historical weight value. Then, the weight value of the region of interest at the current moment is determined based on the historical weight value, the candidate weight value at the current moment, and the assigned corresponding weights.

[0066] Taking the weight value of the previous time step (i.e., (t-1) in the following formula) as the historical weight value as an example, if we assign a weight of 0.3 to the weight value of the previous time step (i.e., (t) in the following formula) and assign a weight of 0.7 to the candidate weight value of the current time step (i.e., (t) in the following formula), then the weight value w_i(t) of the region of interest at the current time step can be: w_i(t)= 0.7×w_i^(current)(t)+0.3×w_i(t-1)…………Formula (5); In the above formula (5), w_i^(current)(t) is the candidate weight value of the region of interest at the current time t, and w_i(t-1) is the weight value of the region of interest at time (t-1).

[0067] It should be noted that the 0.3 and 0.7 mentioned above can be set to other values ​​according to the actual situation, as long as the weight corresponding to the candidate weight value at the current moment is greater than the weight corresponding to the historical weight value, and the sum of the weight corresponding to the candidate weight value at the current moment and the weight corresponding to the historical weight value is 1.

[0068] The above describes several scenarios for adjusting weight values. In practice, these adjustments can be integrated. For example, after determining candidate weight values ​​for the region of interest (ROI), it can be first determined whether the candidate weight value is less than a preset minimum weight threshold. If so, the candidate weight value is filtered out. If not, the candidate weight value for the ROI at the current moment is adjusted based on the historical weight values ​​of the ROI to obtain the current weight value for the ROI. Alternatively, if the ROI is the left or right cheek region, the absolute value of the difference between the heart rate of the left and right cheek regions can be calculated. It can then be determined whether this absolute value is greater than a preset heart rate difference threshold. If so, the weight values ​​of both the left and right cheek regions are reduced.

[0069] S15, determine the heart rate of the user to be tested based on the weight value corresponding to each of the above regions of interest and the heart rate.

[0070] The weight values ​​corresponding to each region of interest are normalized weight values. In this embodiment, the heart rate of the user under test can be determined by accumulating the weight values ​​of each region of interest and the heart rate. Optionally, the heart rate of the user under test can also be determined based on the weight values ​​of each region of interest, the quality score, and the heart rate. In this case, the heart rate HR_final(t) of the user under test can be: HR_final(t)=Σ i [w i ×HR i (t)×Q i (t)] / Σ i [w i ×Q i (t)] ……Formula (6); In the above formula (6): w i This represents the weight value of the i-th region of interest, w.i Based on the vascular density and quality score of the region of interest, such as w i It can be equal to the product of the vascular density and quality score of the region of interest; HR i (t) represents the heart rate estimate of the i-th region of interest at time t; Q i (t) represents the quality score of the i-th region of interest at time t.

[0071] When HR_final(t) is determined using the above formula, it is equivalent to using a double weighting mechanism: w i (Static weights, based on the characteristics of the region of interest (vessel density)) × Q i (t) (dynamic weights, based on real-time quality), i.e., w i It primarily reflects the prior quality and vessel density of the region of interest, Q. i (t) reflects the real-time signal quality of the region of interest at the current moment. Therefore, the above dual weighting mechanism realizes the combination of static prior and dynamic evaluation, thereby improving the accuracy of the determined HR_final(t).

[0072] In this embodiment, after obtaining the heart rate corresponding to at least two regions of interest (ROIs) of the user under test, the weight value of each ROI is determined. Then, based on the weight value of each ROI and the heart rate, the heart rate of the user under test is determined. Since local motion or uneven illumination in a single ROI is unlikely to affect all ROIs simultaneously, while different ROIs can provide spatially diverse physiological signal samples, and the fusion process can automatically suppress local abnormal disturbances, when the heart rate of the user under test is obtained by fusing the heart rates of different ROIs on the user's face, the heart rate estimation of the user under test becomes more resistant to local interference, and the accuracy is systematically improved.

[0073] In some embodiments, to improve the accuracy of the determined heart rate of the user to be tested, outlier filtering can be performed on the fused heart rate. That is, in S15 above, determining the heart rate of the user to be tested based on the weight values ​​corresponding to each of the aforementioned regions of interest and the heart rate includes: E1. Based on the weight values ​​and heart rates of each of the aforementioned regions of interest at the current moment, determine the suspected heart rate of the user to be tested at the current moment.

[0074] E2. Based on the heart rate of the user under test at the previous moment and the preset abnormal heart rate threshold, determine whether the suspected heart rate of the user under test at the current moment is abnormal.

[0075] E3. If the suspected heart rate of the aforementioned user is normal at the current moment, then the suspected heart rate of the aforementioned user at the current moment is determined to be the actual heart rate of the aforementioned user at the current moment.

[0076] In this embodiment, the heart rate of the user at the current moment, calculated based on the weight values ​​of each region of interest at the current moment and the heart rate, is taken as the suspected heart rate. The suspected heart rate is compared with the heart rate calculated for the user at the previous moment. If the difference between the two is large, such as the absolute value of the difference between the suspected heart rate and the heart rate calculated for the user at the previous moment being greater than a preset abnormal heart rate threshold, then the suspected heart rate is determined to be an abnormal heart rate. Otherwise, the suspected heart rate can be determined to be a normal heart rate. In this case, the suspected heart rate is taken as the heart rate of the user at the current moment.

[0077] Since the difference in heart rate between adjacent moments is usually small, combining the heart rate of the user at the previous moment with the current heart rate to determine abnormalities helps improve the accuracy of the final determined heart rate.

[0078] Optionally, the above-mentioned abnormal heart rate threshold can be set according to actual conditions, or it can be determined in the following ways: Calculate the standard deviation of the heart rate of the user under test within the most recent preset time period; determine the abnormal heart rate threshold based on the standard deviation of the heart rate.

[0079] The preset duration can be set according to the actual situation; for example, it can be set to the 10 seconds closest to the current time.

[0080] In this embodiment of the application, a preset multiple of the standard deviation can be used as the abnormal heart rate threshold mentioned above.

[0081] Optionally, considering that the heart rate of the user under test is close to 5 bpm in a resting scenario, but increases during an active scenario, the above abnormal heart rate threshold can be set as follows: threshold=max(5bpm,3×σ_HR), where σ_HR is the standard deviation of the heart rate of the user being tested within the most recent preset time period.

[0082] When the heart rate is stable, σ_HR is small, and the heart rate is close to 5 bpm. Therefore, setting the abnormal heart rate threshold to 5 bpm is equivalent to strictly detecting abnormalities in the heart rate of the identified users, which helps improve the accuracy of the judgment. Conversely, when the heart rate fluctuates greatly, σ_HR is larger, and the heart rate also increases. Therefore, setting the abnormal heart rate threshold to 3 × σ_HR is equivalent to applying a more lenient abnormality detection method to the heart rate of the identified users, which can reduce the probability of false positives.

[0083] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0084] Corresponding to the heart rate determination method based on region of interest described in the above embodiments, Figure 3 The diagram shows a structural block diagram of a heart rate determination device based on a region of interest according to an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0085] Reference Figure 3 The heart rate determination device 3 based on the region of interest (ROI) is applied to an electronic device and includes: a video stream acquisition module 31, a ROI determination module 32, a ROI heart rate calculation module 33, a ROI weight value determination module 34, and a heart rate determination module 35 for the user being tested. Wherein: The video stream acquisition module 31 is used to acquire a video stream including the face of the user under test.

[0086] The region of interest determination module 32 is used to perform region of interest detection on the above video stream using at least two region of interest templates to obtain the corresponding region of interest.

[0087] The heart rate calculation module 33 for each region of interest is used to determine the time series signal corresponding to the region of interest, perform blind source separation on the time series signal using the independent component algorithm to obtain the independent component signal related to heart rate, and calculate the heart rate corresponding to the region of interest based on the independent component signal.

[0088] The weight value determination module 34 for regions of interest is used to determine the weight value of each of the aforementioned regions of interest.

[0089] The heart rate determination module 35 for the user under test is used to determine the heart rate of the user under test based on the weight values ​​corresponding to each of the above-mentioned regions of interest and the heart rate.

[0090] In this embodiment, since local motion or uneven illumination of a single region of interest is unlikely to affect all regions of interest simultaneously, while different regions of interest can provide spatially diverse physiological signal samples, and the fusion process can automatically suppress local abnormal disturbances, when the heart rate of the user to be tested is obtained by fusing the heart rates of different regions of interest on the user's face, the heart rate estimation of the user to be tested can be more resistant to local interference, and the accuracy can be systematically improved.

[0091] Optionally, the weight value determination module 34 for the region of interest includes: The weight parameter acquisition unit is used to acquire the blood vessel density corresponding to each of the above-mentioned regions of interest, and / or to determine the quality score corresponding to each of the above-mentioned regions of interest at the current time, wherein the quality score reflects the signal quality of the above-mentioned regions of interest. The weight value determination unit is used to determine the weight value of the corresponding region of interest based on the above-mentioned vascular density and / or quality score.

[0092] Optionally, the aforementioned weight value determination unit is specifically used for: Determine the candidate weight values ​​for each of the aforementioned regions of interest; For any candidate weight value of a region of interest, if the candidate weight value is less than a preset minimum weight threshold, the candidate weight value is adjusted to 0 to obtain the weight value of the region of interest.

[0093] Optionally, the aforementioned minimum weight threshold is determined in the following way: Determine the current scenario of the user being tested, which includes: resting scenario and moving scenario; The minimum weight threshold is determined based on the current scenario of the user being tested.

[0094] Optionally, the aforementioned region of interest includes the left cheek region and the right cheek region, and the aforementioned weight value determination unit is specifically used for: Determine the candidate weight value for the left cheek region and the candidate weight value for the right cheek region. If the absolute value of the difference between the heart rate corresponding to the left cheek region and the heart rate corresponding to the right cheek region is greater than a preset heart rate difference threshold, then the left cheek candidate weight value is reduced to obtain the left cheek weight value of the left cheek region, and the right cheek candidate weight value is reduced to obtain the right cheek weight value of the right cheek region.

[0095] Optionally, the aforementioned weight value determination unit is specifically used for: Determine the candidate weight values ​​for each of the aforementioned regions of interest at the current time; For any candidate weight value of a region of interest at the current time, the weight value of the region of interest at the current time is determined based on the historical weight values ​​corresponding to the region of interest and the candidate weight values ​​of the region of interest at the current time.

[0096] Optionally, the heart rate determination module 35 for the user to be tested includes: The suspected heart rate determination unit is used to determine the suspected heart rate of the user under test at the current moment based on the weight value and heart rate of each of the above regions of interest at the current moment. The suspected heart rate abnormality judgment unit is used to determine whether the suspected heart rate of the user under test is abnormal at the current moment based on the user's heart rate at the previous moment and the preset abnormal heart rate threshold. The heart rate determination unit for the user under test is used to determine the suspected heart rate of the user under test at the current moment as the actual heart rate of the user under test if the suspected heart rate of the user under test at the current moment is normal.

[0097] Optionally, the above-mentioned abnormal heart rate thresholds are determined according to the following method: Calculate the standard deviation of the heart rate of the above-mentioned test users within the most recent preset time period; The abnormal heart rate thresholds are determined based on the standard deviation of the heart rate.

[0098] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0099] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4 The diagram shows only one processor, memory 41, and computer program 42 stored in the memory 41 and executable on at least one processor 40. When the processor 40 executes the computer program 42, it implements the steps in any of the above method embodiments.

[0100] The aforementioned electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0101] The processor 40 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0102] In some embodiments, the memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. In other embodiments, the memory 41 may be an external storage device of the electronic device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 4. Furthermore, the memory 41 may include both internal and external storage units of the electronic device 4. The memory 41 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0104] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0105] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0106] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0108] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0109] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] It should be noted that the information collection process (such as the facial image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is carried out with the user's knowledge and permission. That is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

Claims

1. A method for determining heart rate based on a region of interest, characterized in that, include: Acquire a video stream including the face of the user under test; At least two region of interest (ROI) templates are used to perform ROI detection on the video stream to obtain the corresponding ROI. For each region of interest, the time series signal corresponding to the region of interest is determined, and the independent component algorithm is used to perform blind source separation on the time series signal to obtain the independent component signal related to heart rate. The heart rate is calculated based on the independent component signal to obtain the heart rate corresponding to the region of interest. Determine the weight value of each region of interest; The heart rate of the user to be tested is determined based on the weight value corresponding to each region of interest and the heart rate.

2. The heart rate determination method based on region of interest as described in claim 1, characterized in that, Determining the weight values ​​of each of the regions of interest includes: Obtain the blood vessel density corresponding to each of the regions of interest, and / or determine the quality score corresponding to each of the regions of interest at the current time, wherein the quality score reflects the signal quality of the region of interest; The weight value of the corresponding region of interest is determined based on the blood vessel density and / or quality score.

3. The heart rate determination method based on region of interest as described in claim 1 or 2, characterized in that, Determining the weight values ​​of each of the regions of interest includes: Determine the candidate weight values ​​for each of the regions of interest; For any candidate weight value of the region of interest, if the candidate weight value is less than a preset minimum weight threshold, the candidate weight value is adjusted to 0 to obtain the weight value of the region of interest.

4. The heart rate determination method based on region of interest as described in claim 3, characterized in that, The minimum weight threshold is determined in the following way: Determine the current scenario of the user under test, including: a resting scenario and a moving scenario; The minimum weight threshold is determined based on the current scenario of the user being tested.

5. The heart rate determination method based on region of interest as described in claim 1 or 2, characterized in that, The regions of interest include the left cheek region and the right cheek region. Determining the weight value of each region of interest includes: Determine the left face candidate weight value of the left cheek region, and determine the right face candidate weight value of the right cheek region; If the absolute value of the difference between the heart rate corresponding to the left cheek region and the heart rate corresponding to the right cheek region is greater than a preset heart rate difference threshold, then the left cheek candidate weight value is reduced to obtain the left cheek weight value of the left cheek region, and the right cheek candidate weight value is reduced to obtain the right cheek weight value of the right cheek region.

6. The heart rate determination method based on region of interest as described in claim 1 or 2, characterized in that, Determining the weight values ​​of each of the regions of interest includes: Determine the candidate weight value for each region of interest at the current time; For any candidate weight value of the region of interest at the current time, the weight value of the region of interest at the current time is determined based on the historical weight value corresponding to the region of interest and the candidate weight value of the region of interest at the current time.

7. The heart rate determination method based on region of interest as described in claim 1 or 2, characterized in that, Determining the heart rate of the user to be tested based on the weight values ​​corresponding to each region of interest and the heart rate includes: Based on the weight value of each region of interest at the current time and the heart rate, determine the suspected heart rate of the user to be tested at the current time; Based on the heart rate of the user under test at the previous moment and the preset abnormal heart rate threshold, determine whether the suspected heart rate of the user under test at the current moment is abnormal; If the suspected heart rate of the user under test is normal at the current moment, then the suspected heart rate of the user under test at the current moment is determined to be the actual heart rate of the user under test at the current moment.

8. The heart rate determination method based on region of interest as described in claim 7, characterized in that, The abnormal heart rate threshold is determined according to the following method: Calculate the standard deviation of the heart rate of the user under test within the most recent preset time period; The abnormal heart rate threshold is determined based on the standard deviation of the heart rate.

9. A heart rate determination device based on a region of interest, characterized in that, include: The video stream acquisition module is used to acquire a video stream including the face of the user under test; The region of interest (ROI) determination module is used to perform ROI detection on the video stream using at least two ROI templates to obtain the corresponding ROI. The heart rate calculation module for the region of interest is used to determine the time series signal corresponding to each region of interest, perform blind source separation on the time series signal using the independent component algorithm to obtain independent component signals related to heart rate, and calculate the heart rate corresponding to the region of interest based on the independent component signals. The weight value determination module for regions of interest is used to determine the weight value of each region of interest; The heart rate determination module for the user under test is used to determine the heart rate of the user under test based on the weight value corresponding to each region of interest and the heart rate.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program, which, when run, causes the electronic device to perform the method according to any one of claims 1 to 8.

Images