A remote and large-scale heart rate estimation method based on deep learning and face segmentation

Abstract

The invention discloses a remote large-scale heart rate estimation method based on face segmentation, comprising: acquiring near-infrared thermal imaging video, extracting 68 key points for each frame of the image; dividing the face into 25 regions according to the key points; calculating the average grayscale for the forehead, left face, right face and chin in the region to obtain four-way local face heart rate signals, and calculating the frequency domain confidence; weighted averaging the signals according to the frequency domain confidence to obtain the facial heart rate signal; using a recurrent neural network model to perform binary classification on the facial heart rate signal to obtain a classification result of high heart rate or low heart rate; performing wavelet decomposition on the facial heart rate signal, and eliminating the subband signal that does not match the classification result in the wavelet decomposition result, and then recombining the facial heart rate signal; based on a dynamic window, intercepting the facial heart rate signal for multiple times in succession, and using the intercepted signal for heart rate estimation each time; and determining the final heart rate estimation result according to the multiple consecutive estimated heart rate values.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a remote large-scale heart rate estimation method based on deep learning and face segmentation. Background Art

[0002] Heart rate is one of the important vital signs of the human body, and it is also the most important and basic physiological indicator of the human cardiovascular system. Generally, the heart rate of normal people is relatively stable at rest, but the heart rates of people of different genders and ages are often different, and strenuous exercise or changes in emotional state can often cause significant changes in heart rate. For example, people who are asleep will have a low heart rate, while their heart rate will increase significantly after strenuous exercise or when they are affected by emotions such as anger and excitement. People with cardiovascular diseases often have large fluctuations in heart rate when the symptoms occur. If the patient's heart rate information is not understood in time, the best time for diagnosis and treatment may be missed. Therefore, it is imperative to conduct effective heart rate monitoring for people diagnosed with cardiovascular diseases, and even for people with potential health risks.

[0003] Compared with contact heart rate detection, non-contact video heart rate detection can detect heart rate remotely without wearing special detection equipment, and is more convenient to use. In the related technology, the patent application with publication number CN114912487A discloses an end-to-end remote heart rate detection method based on a channel-enhanced spatiotemporal attention network. This method uses an end-to-end deep learning network to directly map the original face video and the specific heart rate value. However, this method has very high requirements for the data set required for training the network, and requires the quality and quantity of samples to reach a certain level, which limits its practicality. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a remote large-scale heart rate estimation method based on deep learning and face segmentation.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] A remote large-range heart rate estimation method based on deep learning and face segmentation, comprising:

[0007] Acquire near infrared thermal imaging video containing human face images;

[0008] For each frame of the near-infrared thermal imaging video, 68 key points of the face in the image are extracted; according to the 68 key points, the face in the image is divided into 25 areas; for each area of ​​the 25 areas belonging to the forehead, left face, right face and chin, the average grayscale of the area is calculated;

[0009] The average grayscale of the forehead, the average grayscale of the left face, the average grayscale of the right face, and the average grayscale of the chin calculated from each frame of the image are integrated in time sequence to obtain four local face heart rate signals;

[0010] Calculate the frequency domain confidence of each local face heart rate signal;

[0011] Performing weighted averaging on the four local face heart rate signals according to the calculated frequency domain confidence to obtain a facial heart rate signal;

[0012] Using a recurrent neural network model to perform binary classification on the facial heart rate signal to obtain a classification result of whether the facial heart rate signal corresponds to a high heart rate or a low heart rate;

[0013] Performing wavelet decomposition on the facial heart rate signal, and removing sub-band signals in the wavelet decomposition result that do not match the classification result, and then reconstructing the facial heart rate signal;

[0014] The reconstructed facial heart rate signal is intercepted multiple times continuously based on a dynamic window, and the heart rate is estimated using the signal intercepted each time;

[0015] The final heart rate estimation result is determined based on multiple consecutive estimated heart rate values.

[0016] In one embodiment, the step of calculating the frequency domain confidence of each local face heart rate signal includes:

[0017] Detrending processing is performed on each channel of partial face heart rate signal, and band-pass filtering is performed on each channel of partial face heart rate signal according to the heart rate range;

[0018] For each local face heart rate signal after detrending and bandpass filtering, the local face heart rate signal is converted to the frequency domain, and the ratio of the main frequency signal value in the frequency domain conversion result to the frequency domain average signal value is calculated as the frequency domain confidence of the local face heart rate signal.

[0019] In one embodiment, the method of intercepting the reconstructed facial heart rate signal multiple times continuously based on the dynamic window and using each intercepted signal to estimate the heart rate includes:

[0020] A. Set the initial window size of the Hamming window;

[0021] B. intercepting the reconstructed facial heart rate signal using the current Hamming window, and padding the intercepted data with zeros to obtain a windowed facial heart rate signal;

[0022] C. performing Hilbert transform on the windowed facial heart rate signal in step B, and obtaining an envelope of the windowed facial heart rate signal according to the transform result;

[0023] D. Convert the envelope in step C to the frequency domain, and multiply the main frequency in the frequency domain conversion result by 60 to obtain the estimated heart rate value of this time;

[0024] E. Determine whether the window size of the current Hamming window has reached the upper limit; if not, increase the window size of the Hamming window according to the preset step and return to step B; if it has reached, continue to execute the step of determining the final heart rate estimation result based on the heart rate values ​​estimated for multiple consecutive times.

[0025] In one embodiment, determining a final heart rate estimation result according to a plurality of consecutive estimated heart rate values ​​includes:

[0026] When the multiple estimated heart rate values ​​are all maintained at a stable value, the stable value is output as the final heart rate estimation result;

[0027] When the multiple estimated heart rate values ​​show the same upward trend or the same downward trend, the multiple estimated heart rate values ​​are output as the final heart rate estimation result.

[0028] In one embodiment, the method further includes: when performing facial region segmentation on a human face in the near-infrared thermal imaging video, performing convolution sharpening processing on each frame image in the near-infrared thermal imaging video, so as to perform facial region segmentation using the image after convolution sharpening processing.

[0029] In one embodiment, the recurrent neural network model is trained based on multiple sample heart rate signals and the classifications corresponding to the sample heart rate signals; wherein, the classification corresponding to the sample heart rate signal with a heart rate lower than 90 is a low heart rate; and the classification corresponding to the sample heart rate signal with a heart rate not lower than 90 is a high heart rate.

[0030] The remote large-range heart rate estimation method based on deep learning and face segmentation provided by the present invention can realize remote non-contact heart rate estimation without relying on a powerful data set, and has high practicality.

[0031] In addition, the remote large-scale heart rate estimation method based on deep learning and face segmentation provided by the present invention has the following more beneficial effects:

[0032] (1) The facial region with physical meaning is accurately segmented. The local face heart rate signal extracted by the partition is more valuable for heart rate estimation, making the estimation result more accurate.

[0033] (2) Perform frequency domain confidence evaluation on the local face heart rate signal, so as to highlight the contribution of the local face heart rate signal with high confidence to the heart rate estimation result, and suppress the influence of the local face heart rate signal with low confidence on the heart rate estimation result.

[0034] (3) A dynamic window is used to perform multiple consecutive heart rate estimates, which ensures the accuracy of the heart rate estimation results while taking into account the rapid output of the value. This solves the problem of being difficult to achieve both rapid output and accuracy when using remote video signals for heart rate detection.

[0035] (4) The facial heart rate signal is classified into two categories using a recurrent neural network model. The harmonic signals (sub-band signals) that do not meet the classification requirements are removed based on the classification results, thereby improving the signal-to-noise ratio of the facial heart rate signal and thus improving the accuracy of the heart rate estimation results.

[0036] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of a remote large-scale heart rate estimation method based on deep learning and face segmentation provided by an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of performing face region segmentation in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0040] In order to solve the technical problems raised in the background technology, the embodiment of the present invention provides a remote large-scale heart rate estimation method based on deep learning and face segmentation, such as Figure 1 As shown, the method comprises the following steps:

[0041] S10: Acquire a near-infrared thermal imaging video containing a face image.

[0042] The near-infrared thermal imaging video may be directly obtained by shooting with a camera, or may be obtained by importing, which is not limited in the embodiment of the present invention.

[0043] It is worth mentioning that most non-contact video heart rate detection solutions so far use visible light videos, and most of the existing public data sets are also RGB image data sets. This is not conducive to heart rate detection in dark light scenes. Therefore, in the embodiment of the present invention, near-infrared thermal imaging video is used for heart rate estimation, so that effective heart rate estimation can be achieved in both bright light scenes and dark light scenes.

[0044] S20: For each frame of the near-infrared thermal imaging video, extract 68 key points of the face in the image; divide the face in the image into 25 areas according to the 68 key points; for each area of ​​the 25 areas belonging to the forehead, left face, right face and chin, calculate the average grayscale of the area.

[0045] from Figure 2 It can be seen that areas 1 to 4 belong to the forehead, areas 11 and 12 belong to the left face, areas 13 and 14 belong to the right face, and areas 21 to 25 belong to the chin. The average grayscale in these areas is calculated respectively.

[0046] In practical applications, the extraction of 68 key points and the partitioning of the face into 25 regions can be achieved using the Dlib facial boundary detector (Dlib library).

[0047] It is understandable that by extracting 68 key points and performing 25-region partitioning operations based on these key points, accurate face segmentation with physical meaning can be achieved, so that a more accurate local face heart rate signal can be calculated. In the prior art, a simple rectangular frame is usually used to frame different facial regions. This segmentation method is relatively rough and is not conducive to the accurate calculation of the heart rate signal.

[0048] In addition, when performing facial region segmentation on faces in near-infrared thermal imaging videos, each frame of the near-infrared thermal imaging video can also be subjected to convolution sharpening processing, so that the images after convolution sharpening processing can be used for facial region segmentation, which is more conducive to the detection of facial key points.

[0049] S30: Integrate the average grayscale of the forehead, the average grayscale of the left face, the average grayscale of the right face, and the average grayscale of the chin calculated according to each frame image in chronological order to obtain four local face heart rate signals.

[0050] It can be understood that the four local face heart rate signals mentioned here include forehead heart rate signal, left face heart rate signal, right face heart rate signal and chin heart rate signal. The data length of these four heart rate signals is equal to the frame number of the above-mentioned near-infrared thermal imaging video.

[0051] S40: Calculate the frequency domain confidence of each local face heart rate signal.

[0052] Specifically, each local face heart rate signal is detrended and band-pass filtered according to the heart rate range; then, for each local face heart rate signal after detrending and band-pass filtering, the local face heart rate signal is converted to the frequency domain, and the ratio of the main frequency signal value to the frequency domain average signal value in the frequency domain conversion result is calculated as the frequency domain confidence of the local face heart rate signal. It can be understood that the frequency domain confidence reflects the proportion of the main signal and its harmonics contained in the local face heart rate signal, and it measures the prominence of the main frequency in the frequency domain.

[0053] Among them, the detrending processing can remove the abnormal deviation of the local face heart rate signal caused by shooting the video, so as to better present the original characteristics of the local face heart rate signal. For the specific implementation method of the detrending processing, please refer to the relevant prior art, and the embodiments of the present invention will not be repeated.

[0054] The heart rate interval used in bandpass filtering, i.e., the passband, can be set to 45 to 150 bpm, which is a wide enough range, but is not necessarily limited to this.

[0055] By calculating the frequency domain confidence of each local face heart rate signal, the embodiment of the present invention achieves a significant beneficial effect different from the prior art: the embodiment of the present invention uses another set of evaluation criteria different from the heart rate standard value to evaluate the quality of the heart rate signal. On the one hand, this will not treat the heart rate signal that is different from the heart rate standard value but actually exists as interference and eliminate it. On the other hand, it can eliminate the pseudo heart rate signal that is similar to the heart rate standard value but is an interference signal, thereby improving the accuracy of the final heart rate detection result, especially improving the accuracy of the heart rate detection estimation result for unconventional people. It can be understood that the unconventional people mentioned here include non-static people who are exercising, sub-healthy people whose heart rate is high or low, etc.

[0056] S50: performing weighted averaging on the four local face heart rate signals according to the calculated frequency domain confidence to obtain a facial heart rate signal.

[0057] In addition, the frequency domain confidences of the four local face heart rate signals can be added together as the confidence of the face heart rate signal. If the confidence value is low, the subsequent steps can be ignored and the method provided by the embodiment of the present invention can be re-executed starting from step S10.

[0058] S60: Using the recurrent neural network model to perform binary classification on the facial heart rate signal, and obtain a classification result of whether the facial heart rate signal corresponds to a high heart rate or a low heart rate.

[0059] In this step, the facial heart rate signal is classified into two categories using a recurrent neural network model, and the harmonic signals (subband signals) that do not meet the classification are removed according to the classification results, thereby improving the signal-to-noise ratio of the facial heart rate signal and thus improving the accuracy of the heart rate estimation results.

[0060] Among them, the above-mentioned recurrent neural network model is trained based on multiple sample heart rate signals and the classifications corresponding to the sample heart rate signals; among them, the classification corresponding to the sample heart rate signal with a heart rate lower than 90 is a low heart rate; the classification corresponding to the sample heart rate signal with a heart rate not lower than 90 is a high heart rate.

[0061] In practical applications, there may be a variety of recurrent neural network models that can be used in the embodiments of the present invention. For example, recurrent neural network models that can process long sequence data, such as long short-term memory networks (LSTMs), can be applied in the embodiments of the present invention.

[0062] Exemplarily, the recurrent neural network model may include an input layer, a hidden layer, and an output layer, wherein the hidden layer includes an LSTM unit, which uses a tanh function as an activation function, and the output layer uses a sigmoid function as an activation function.

[0063] When training the recurrent neural network model, the cross entropy function can be used to calculate the loss of the model, so that when the cross entropy is less than a predetermined threshold, a trained recurrent neural network model is obtained. When the cross entropy does not meet the requirement of being less than the threshold, the optimizer provided by pytorch (such as Adam) can be used to update the network parameters so that the loss function converges to the minimum as soon as possible.

[0064] In addition, after the recurrent neural network model is trained, its classification performance can be tested using test samples that have not participated in training, so that it can be put into practical use when its classification accuracy and generalization ability meet the evaluation standards.

[0065] S70: performing wavelet decomposition on the facial heart rate signal, and removing the sub-band signals in the wavelet decomposition result that do not match the above classification result, and then reconstructing the facial heart rate signal.

[0066] In this step, the wavelet decomposition used is a method for decomposing a signal into different frequency sub-bands. It can use wavelet basis functions to perform multi-layer decomposition on the approximate components and detail components of the signal to obtain a multi-scale representation of the signal. The specific method is to perform wavelet packet transform (WPT) on the facial heart rate signal to obtain sub-band signals of different frequency bands and form a wavelet packet tree. Since high heart rate and low heart rate correspond to different sub-band signals, if the classification result in step S60 is a high heart rate, the sub-band signal that does not correspond to the high heart rate in the wavelet packet tree is removed, and then the remaining sub-band signals are reorganized into a facial heart rate signal; and if the classification result in step S60 is a low heart rate, the sub-band signal that does not correspond to the low heart rate in the wavelet packet tree is removed, and then the remaining sub-band signals are reorganized into a facial heart rate signal.

[0067] In practical applications, the wavelet function dmey can be used to perform wavelet transform, and the maximum number of analysis layers can be set to 3, that is, at most 8 sub-signals in different frequency bands can be obtained, but it is certainly not limited to this.

[0068] It is worth mentioning that most non-contact video heart rate detection solutions to date usually limit the heart rate range to 55-99bpm. However, this range is the ideal heart rate range for people in a quiet state. Therefore, these existing solutions are difficult to accurately estimate the heart rate of people after exercise or for people with special high heart rates.

[0069] In the embodiment of the present invention, the facial heart rate signals of a wide range of people can be classified into two categories using a recurrent neural network model, so that the facial heart rate signals of different individuals can be purified according to their characteristics, thereby estimating the heart rate based on the purified facial heart rate to obtain a more accurate estimation result. This is because if the heart rate is estimated directly using a spectrum with a heart rate range that is too large, the heart rate estimation will be inaccurate due to the frequency doubling phenomenon that is easy to appear in the spectrum. At the same time, this is also the main reason why the conventional method of directly estimating the heart rate using deep learning in the prior art is not effective.

[0070] In addition, in comparison, the remote heart rate detection method using an end-to-end deep learning network mentioned in the background technology requires mapping the original near-infrared thermal imaging video of the face and the specific heart rate value. Therefore, in addition to the high requirements for the training data set, the network structure itself is more complex, and the robustness of the algorithm used in the model is relatively low. In the embodiment of the present invention, only two classifications are required for the facial heart rate signal, so the training data set is easy to obtain, and the network model structure for classification is relatively simple, which is much easier than the prior art method of directly mapping data to a specific heart rate. Therefore, the algorithm used in the model is more robust and more suitable for promotion and application.

[0071] S80: intercepting the reconstructed facial heart rate signal multiple times continuously based on the dynamic window, and using each intercepted signal to estimate the heart rate.

[0072] Specifically, step S80 includes:

[0073] A. Set the initial window size of the Hamming window;

[0074] B. intercepting the facial heart rate signal using the current Hamming window, and padding the intercepted data with zeros to obtain a windowed facial heart rate signal;

[0075] C. performing Hilbert transform on the windowed facial heart rate signal in step B, and obtaining an envelope of the windowed facial heart rate signal according to the transform result;

[0076] D. Convert the envelope in step C to the frequency domain, and multiply the main frequency in the frequency domain conversion result by 60 to obtain the estimated heart rate value of this time;

[0077] E. Determine whether the window size of the current Hamming window has reached the upper limit; if not, increase the window size of the Hamming window according to the preset step and return to step B; if it has reached, continue to execute step S90.

[0078] For example, the initial window size of the Hamming window is set to 10s, that is, the facial heart rate signal of 10s is intercepted for the first time to perform Hilbert transform to obtain its envelope, and then the envelope is converted to the frequency domain to obtain its main frequency, and the main frequency is multiplied by 60 to obtain the first estimated heart rate value in bpm. Then, because the window size of the Hamming window does not reach the set upper limit of 60s, the window size of the Hamming window is increased to 20 according to a step of 10, that is, the facial heart rate signal of 20s is intercepted for Hilbert transform to obtain its envelope, and then the envelope is converted to the frequency domain to obtain its main frequency, and the main frequency is multiplied by 60 to obtain the second estimated heart rate value; and so on, until the window size of the Hamming window reaches 60s, the estimation is stopped, and a total of 5 consecutive estimated heart rate values ​​are obtained.

[0079] It is understandable that the role of the Hamming window is to reduce the impact of frequency leakage when performing frequency domain analysis on the signal. Frequency leakage refers to the spectrum distortion caused by signal truncation, which makes it difficult to accurately measure the true frequency and amplitude of the signal. The Hamming window is a non-rectangular window, which is characterized by large sidelobe attenuation, and the main lobe peak and the first sidelobe peak attenuation can reach 40dB. This can effectively suppress the interference of adjacent frequencies and improve the resolution of the signal. In addition, the signal is padded with zeros after windowing, which adds some zero values ​​after the time domain signal, so that the length of the signal becomes an integer power of 2, which is convenient for the use of the fast Fourier transform (FFT) algorithm. In addition, zero padding can also increase the sampling points of the frequency domain data and improve the resolution details of the frequency domain.

[0080] The beneficial effect of performing Hilbert transform on the windowed facial heart rate signal in step C is: expanding the signal into an analytical signal on the complex plane, thereby obtaining the envelope of the signal, which can improve the signal resolution and reduce the clutter components other than the non-main frequency components.

[0081] S90: Determine a final heart rate estimation result according to the multiple consecutive estimated heart rate values.

[0082] Specifically, when multiple estimated heart rate values ​​are maintained at a stable value, the stable value is output as the final heart rate estimation result; and when multiple estimated heart rate values ​​show the same upward trend or the same downward trend, the estimated heart rate values ​​of these times are output as the final heart rate estimation result.

[0083] In the remote large-scale heart rate estimation method based on deep learning and face segmentation provided by the embodiment of the present invention, remote non-contact heart rate estimation can be achieved without relying on a powerful data set, and has high practicality. The method performs precise segmentation of facial areas with physical meaning, so the local face heart rate signal extracted by the partition is more valuable for reference to heart rate estimation, making the estimation result more accurate. The method performs frequency domain confidence evaluation on the local face heart rate signal, so that the contribution of the local face heart rate signal with high confidence to the heart rate estimation result can be highlighted, and the influence of the local face heart rate signal with low confidence on the heart rate estimation result can be suppressed. The method uses a dynamic window to perform multiple consecutive heart rate estimates, while taking into account the rapid output of the value and ensuring the accuracy of the heart rate estimation result, solving the problem that it is difficult to take into account both rapid output and accuracy when using remote video signals for heart rate detection.

[0084] In summary, the embodiments of the present invention solve the problem of excessive errors in heart rate estimation results caused by harmonics, individual differences among people, etc. when using remote video signals for large-scale heart rate estimation, and at the same time solve the problem of difficulty in balancing fast value output and accuracy. The embodiment of the present invention is very suitable for remote medical monitoring in practice.

[0085] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0086] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0087] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0088] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and the meaning of "multiple" is two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0089] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A remote large-scale heart rate estimation method based on deep learning and face segmentation, characterized in that: include: Acquire near infrared thermal imaging video containing human face images; For each frame of the near-infrared thermal imaging video, extract 68 key points of the face in the image; According to the 68 key points, the face in the image is divided into 25 regions; for each region of the 25 regions belonging to the forehead, left face, right face and chin, the average grayscale of the region is calculated; The average grayscale of the forehead, the average grayscale of the left face, the average grayscale of the right face, and the average grayscale of the chin calculated from each frame of the image are integrated in time sequence to obtain four local face heart rate signals; Calculate the frequency domain confidence of each local face heart rate signal; Performing weighted averaging on the four local face heart rate signals according to the calculated frequency domain confidence to obtain a facial heart rate signal; Using a recurrent neural network model to perform binary classification on the facial heart rate signal to obtain a classification result of whether the facial heart rate signal corresponds to a high heart rate or a low heart rate; Performing wavelet decomposition on the facial heart rate signal, and removing sub-band signals in the wavelet decomposition result that do not match the classification result, and then reconstructing the facial heart rate signal; The reconstructed facial heart rate signal is intercepted multiple times continuously based on a dynamic window, and the heart rate is estimated using the signal intercepted each time; The final heart rate estimation result is determined based on multiple consecutive estimated heart rate values.

2. The remote large-scale heart rate estimation method based on deep learning and face segmentation according to claim 1, characterized in that: The calculating of the frequency domain confidence of each local face heart rate signal includes: Detrending processing is performed on each channel of partial face heart rate signal, and band-pass filtering is performed on each channel of partial face heart rate signal according to the heart rate range; For each local face heart rate signal after detrending and bandpass filtering, the local face heart rate signal is converted to the frequency domain, and the ratio of the main frequency signal value in the frequency domain conversion result to the frequency domain average signal value is calculated as the frequency domain confidence of the local face heart rate signal.

3. The remote large-scale heart rate estimation method based on deep learning and face segmentation according to claim 1, characterized in that: The method of intercepting the reconstructed facial heart rate signal for multiple times continuously based on the dynamic window and using each intercepted signal to estimate the heart rate includes: A. Set the initial window size of the Hamming window; B. intercepting the reconstructed facial heart rate signal using the current Hamming window, and padding the intercepted data with zeros to obtain a windowed facial heart rate signal; C. performing Hilbert transform on the windowed facial heart rate signal in step B, and obtaining an envelope of the windowed facial heart rate signal according to the transform result; D. Convert the envelope in step C to the frequency domain, and multiply the main frequency in the frequency domain conversion result by 60 to obtain the estimated heart rate value of this time; E. Determine whether the window size of the current Hamming window has reached the upper limit; if not, increase the window size of the Hamming window according to the preset step and return to step B; if it has reached, continue to execute the step of determining the final heart rate estimation result based on the heart rate values ​​estimated for multiple consecutive times.

4. The remote large-scale heart rate estimation method based on deep learning and face segmentation according to claim 1, characterized in that: The step of determining a final heart rate estimation result based on a plurality of consecutive estimated heart rate values ​​includes: When the multiple estimated heart rate values ​​are all maintained at a stable value, the stable value is output as the final heart rate estimation result; When the multiple estimated heart rate values ​​show the same upward trend or the same downward trend, the multiple estimated heart rate values ​​are output as the final heart rate estimation result.

5. The remote large-scale heart rate estimation method based on deep learning and face segmentation according to claim 1, characterized in that: The method also includes: when performing facial region segmentation on the human face in the near-infrared thermal imaging video, each frame image in the near-infrared thermal imaging video is subjected to convolution sharpening processing, so as to perform facial region segmentation using the image after convolution sharpening processing.

6. The remote large-scale heart rate estimation method based on deep learning and face segmentation according to claim 1, characterized in that: The recurrent neural network model is trained based on multiple sample heart rate signals and classifications corresponding to the sample heart rate signals; wherein, the classification corresponding to the sample heart rate signal with a heart rate lower than 90 is a low heart rate; and the classification corresponding to the sample heart rate signal with a heart rate not lower than 90 is a high heart rate.

Images