Non-contact heart rate detection method, system and medium based on facial feature point detection

Abstract

The present invention discloses a contactless heart rate detection method, system, and medium based on facial feature point detection. The method comprises: obtaining a facial video image and determining a facial region based on the facial video image; inputting the facial region into a facial feature point detection network model to obtain facial feature points; the facial feature point detection network model comprises a feature extraction portion and a multi-scale fusion prediction portion, wherein the feature extraction portion comprises a standard convolutional layer and multiple block layers, wherein the block layers are depthwise separable convolution modules proposed by MobileNetV3; the multi-scale fusion prediction portion enhances the expressiveness of the model by fusing multiple images of different scales; determining a region of interest based on the facial feature points, thereby obtaining an observation signal; and using a pulse wave extraction algorithm to separate a blood volume pulse wave signal from the observation signal, thereby calculating the heart rate value. The present invention utilizes facial feature point detection to determine the region of interest, thereby improving detection accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and signal processing, and in particular to a non-contact heart rate detection method, system, computer equipment and storage medium based on facial feature point detection. Background Art

[0002] Heart rate is an important vital sign, and monitoring it can help prevent heart disease. Traditional heart rate monitoring methods, such as electrocardiograms (ECGs), rely on contact methods. These require multiple surface electrodes to make direct contact with the subject's body surface, which can be inconvenient for the examiner. Furthermore, ECG equipment is expensive and complex to operate. Non-contact heart rate monitoring methods like Doppler radar also require specialized equipment and are relatively expensive. Therefore, these methods are not suitable for daily heart rate monitoring.

[0003] In recent years, research in imaging photoplethysmography (iPPG) has rapidly advanced. This method, due to its non-contact, low-cost, and portable nature, has attracted considerable attention from researchers. iPPG utilizes a camera sensor to capture video signals from the human skin surface. Using signal processing techniques, it detects subtle changes in skin color intensity caused by differences in ambient light absorption, thereby estimating heart rate. This method requires no direct contact, poses no inconvenience to the subject, and features low hardware cost, portability, and ease of use. This method will significantly advance the development of daily heart rate monitoring and telemedicine. However, its drawback is that it is susceptible to background light and facial motion artifacts, resulting in low detection accuracy. Summary of the Invention

[0004] In order to address the deficiencies of the above-mentioned prior art, the present invention provides a non-contact heart rate detection method, system, computer device and storage medium based on facial feature point detection. This method can maintain high-precision detection results even when the eyes and lips move obviously, and can avoid noise interference caused by the background. The detection speed is faster and is very suitable for actual scenarios.

[0005] The first object of the present invention is to provide a non-contact heart rate detection method based on facial feature point detection.

[0006] The second object of the present invention is to provide a non-contact heart rate detection system based on facial feature point detection.

[0007] A third object of the present invention is to provide a computer device.

[0008] A fourth object of the present invention is to provide a storage medium.

[0009] The first object of the present invention can be achieved by adopting the following technical solutions:

[0010] A non-contact heart rate detection method based on facial feature point detection, the method comprising:

[0011] Acquire a face video image, and determine a face area based on the face video image;

[0012] The face region is input into a facial feature point detection network model to obtain facial feature points; the facial feature point detection network model includes a feature extraction part and a multi-scale fusion prediction part, wherein the feature extraction part includes a standard convolution layer and multiple Block layers, and the Block layer is a depth-separable convolution module proposed by MobileNetV3; the multi-scale fusion prediction part enhances the expressiveness of the model by fusing multiple images of different scales;

[0013] Determine a region of interest based on the facial feature points, and separate channel signals in the region of interest as observation signals;

[0014] Separating a blood volume pulse wave signal from the observation signal using a pulse wave extraction algorithm;

[0015] A heart rate value is calculated according to the blood volume pulse wave signal.

[0016] Furthermore, there are 11 Block layers in total;

[0017] The multi-scale fusion prediction part includes multiple average pooling layers, convolutional layers and fully connected layers;

[0018] Inputting the face region into a face feature point detection network model to obtain face feature points includes:

[0019] Inputting the face area into the standard convolution layer, and the output image is processed by 11 Block layers;

[0020] The outputs of the first, third, sixth, and ninth Block layers are taken as S1, S2, S3, and S4 respectively;

[0021] The output of the last Block layer is processed by the convolution layer and output as S5;

[0022] After S1, S2, S3 and S4 are processed by the average pooling layer respectively, the pooled S1, S2, S3, S4 and S5 are spliced, and the spliced ​​results are input into the fully connected layer to output the coordinates of the facial feature points; and the kernel size and step size of each pooling layer are the length of the input image size.

[0023] Furthermore, the Block layer includes an expansion layer, a depth-wise separable convolution, a projection layer, a residual connection and an SE attention module, wherein the expansion layer is used to map the low-dimensional space to the high-dimensional space; the depth-wise separable convolution is used to complete the convolution function and reduce the amount of calculation and parameters; the projection layer is used to map the high-dimensional features to the low-dimensional space; the residual connection is used to reduce the complexity of the model to reduce overfitting and prevent gradient disappearance; the SE attention module is used to enhance the channel features of the input feature map; in the Block layer, the residual connection is only used when the step size is equal to 1 and the number of input channels is equal to the number of output channels.

[0024] Furthermore, separating the channel signals in the region of interest as observation signals includes:

[0025] Separate the red, green, and blue channel signals from each region of interest (ROI) and take the average values ​​to obtain r(t), g(t), and b(t).

[0026] For a single video, three sets of original signals with a length equal to the number of video frames are obtained: R(t), G(t), and B(t) as observation signals.

[0027] Furthermore, the method of separating the blood volume pulse wave signal from the observation signal using a pulse wave extraction algorithm includes:

[0028] The pulse wave extraction algorithm is defined as follows:

[0029]

[0030] Where σ(·) represents the standard deviation, X x and Y s It is obtained by combining the RGB three-channel signals. In the case of a single light source, they are usually expressed as:

[0031] X s =G n -B n

[0032] Y s =G n +B n -2R n

[0033] Among them, R n ,G n ,B n Respectively represent the normalized results of R(t), G(t), and B(t);

[0034] For a video sequence with N frames, the pulse wave extraction algorithm uses a sliding window of size T to obtain the window output value and uses the overlap-add method to obtain the final pulse signal output as the blood volume pulse wave signal; where T = 1.6*fps, and fps is the camera sampling frequency.

[0035] Furthermore, the calculating of the heart rate value based on the blood volume pulse wave signal includes:

[0036] Bandpass filtering is performed on the blood volume pulse wave signal, fast Fourier transform is performed on the filtered signal to obtain a power spectrum density map, and the heart rate value is calculated according to the power spectrum density map, specifically including:

[0037] Applying a bandpass filter to the blood volume pulse wave signal to eliminate frequencies outside a specified frequency range, and then performing a fast Fourier transform on the filtered blood volume pulse wave signal to obtain a power spectrum density map;

[0038] Detect the peak of the power spectrum density graph and find the frequency f corresponding to the peak hr , calculate the heart rate HR:

[0039] HR=60*f hr .

[0040] Furthermore, a face detector BlaceFace is used to detect the face video image and determine the face area.

[0041] Furthermore, before inputting the face region into the face feature point detection network model, the face feature point detection network model is trained, and the face region is input into the trained face feature point detection network model;

[0042] The training of the facial feature point detection network model includes:

[0043] The public dataset WFLW is used to train the face feature point detection network model, where the loss function is L2Loss:

[0044]

[0045] Among them, y gt Indicates the true feature point coordinate value, y pred Represents the coordinate value of the feature point predicted by the model, Represents the coordinate value of the i-th real feature point, Represents the coordinate value of the i-th predicted feature point;

[0046] The normalized mean error (NME) of the facial feature point evaluation index is defined as:

[0047]

[0048] Among them, x i To predict the coordinates of the i-th feature point, is the true value of the i-th coordinate, and d is the distance between the pupils of the two eyes.

[0049] The second object of the present invention can be achieved by adopting the following technical solutions:

[0050] A non-contact heart rate detection system based on facial feature point detection, the system comprising:

[0051] An image acquisition module is used to acquire a face video image and determine a face area based on the face video image;

[0052] A facial feature point detection module is configured to input the facial region into a facial feature point detection network model to obtain facial feature points. The facial feature point detection network model includes a feature extraction part and a multi-scale fusion prediction part. The feature extraction part includes a standard convolution layer and multiple Block layers, and the Block layer is a depth-separable convolution module proposed by MobileNetV3. The multi-scale fusion prediction part enhances the expressiveness of the model by fusing multiple images of different scales.

[0053] a channel signal separation module, configured to determine a region of interest based on the facial feature points, and separate the channel signal in the region of interest as an observation signal;

[0054] a blood volume pulse wave signal separation module, configured to separate the blood volume pulse wave signal from the observation signal using a pulse wave extraction algorithm;

[0055] A heart rate value calculation module is used to calculate the heart rate value according to the blood volume pulse wave signal.

[0056] The third object of the present invention can be achieved by adopting the following technical solutions:

[0057] A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned non-contact heart rate detection method is implemented.

[0058] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0059] A storage medium stores a program, which, when executed by a processor, implements the above-mentioned non-contact heart rate detection method.

[0060] The present invention has the following beneficial effects compared to the prior art:

[0061] 1. This paper constructs a facial feature point detection network model, comprising a feature extraction component and a multi-scale fusion prediction component. The feature extraction component includes standard convolutional layers and multiple block layers, using the depthwise separable convolutional module proposed in MobileNetV3. The multi-scale fusion prediction component enhances the model's expressiveness by fusing multiple images of different scales. This constructed facial feature point detection network model effectively reduces the model's computational complexity and parameter requirements, improving detection speed while also meeting the accuracy and inference time requirements for contactless heart rate detection.

[0062] 2. This invention utilizes facial feature point detection to identify regions of interest (ROIs), overcoming the effects of facial expressions, background noise, and other factors on detection accuracy while retaining a larger area of ​​skin for BVP signal extraction. BlazeFace, used in face detection, is a very lightweight face detector that can achieve super-real-time efficiency in many embedded devices. Furthermore, the pulse wave extraction algorithm (POS) employed is a high-performance, computationally efficient, simplified algorithm. By employing these methods, the readiness and reliability of non-contact heart rate detection methods are improved while ensuring faster computational speeds, enabling porting to embedded devices for daily heart rate monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0064] Figure 1 This is a flow chart of a non-contact heart rate detection method based on facial feature point detection according to Example 1 of the present invention.

[0065] Figure 2 The Block module of Example 1 of the present invention is a structural diagram of the depth-separable convolution module proposed by MobileNetV3.

[0066] Figure 3 This is a schematic diagram of the effect of extracting the region of interest in Example 1 of the present invention.

[0067] Figure 4 This is a structural block diagram of a non-contact heart rate detection system based on facial feature point detection according to embodiment 2 of the present invention.

[0068] Figure 5 This is a structural block diagram of a computer device according to embodiment 3 of the present invention. DETAILED DESCRIPTION

[0069] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. It should be understood that the specific embodiments described are only used to explain the present application and are not used to limit the present application.

[0070] Example 1:

[0071] The intensity of light reflected from a person's face changes constantly due to heart activity. These changes are so subtle that they are difficult for the human eye to detect, but cameras can capture these tiny changes. Therefore, pulse information can be obtained by analyzing changes in skin color.

[0072] like Figure 1 As shown, the non-contact heart rate detection method based on facial feature point detection provided by this embodiment includes the following steps:

[0073] S101: Acquire a face video image, and determine a face area based on the face video image.

[0074] Use the camera to obtain face video images, perform face detection on the obtained images, and extract the face area.

[0075] Under natural lighting conditions, an ordinary color camera is used to acquire facial video images of the subject at a sampling frequency of 30 frames per second. The camera is required to be 0.5-1 meter away from the subject, and the subject is required to remain still. A total of 600 frames of facial video images are collected over 20 seconds, and the face detector BlaceFace is used to detect the face and determine the facial area.

[0076] S102: Determine a region of interest based on the face region using facial feature points, and separate red, green, and blue channel signals of the region of interest as observation signals.

[0077] The facial feature points are determined according to the facial area, the region of interest is determined according to the facial feature points, and the red, green and blue channel signals of the region of interest are separated as observation signals.

[0078] Furthermore, step S102 includes:

[0079] (1) Input the face area into the face feature point detection network model to obtain the face feature points.

[0080] (1-1) Build a facial feature point detection network model.

[0081] The facial feature point detection network model based on depthwise separable convolution has an output of 98 feature point coordinates. The model structure is shown in Table 1.

[0082] Table 1 Structure of face feature point detection network model

[0083]

[0084]

[0085] The entire network structure consists of two parts. The first is the feature extraction part based on the depth-separable convolution Block module, including the first standard convolution layer and all 11 Block layers in Table 1. Among them, the input size of the standard convolution layer is 112ⅹ112ⅹ3, the number of output channels is 16, the convolution kernel is 3ⅹ3, the step size is 2, and the feature map size output after convolution is 56ⅹ56ⅹ16. It is followed by a batch normalization BN layer, and then the activation function hardswich is used for activation. The parameters used correspond to the parameters in Table 1. Other parameters not provided are default values. The output is then used as the input of the next Block layer. The Block layer is a depth-separable convolution module proposed by MobileNetV3, which is used to replace the standard convolution. Its structure is as follows Figure 2 As shown. The Block layer consists of five parts: (1) The expansion layer uses 1×1 convolution to map the low-dimensional space to the high-dimensional space; (2) Depthwise Convolution, which is used to complete the convolution function and reduce the amount of calculation and parameters; (3) The projection layer uses 1×1 convolution to map the high-dimensional features to the low-dimensional space; (4) The residual connection is used to reduce the complexity of the model to reduce overfitting and prevent the gradient from disappearing; (5) The SE attention module is used to enhance the channel features of the input feature map. The residual connection is only used when the step size is equal to 1 and the number of input channels is equal to the number of output channels. Among them, the dimension increase size in Table 1 represents the dimension increase of the expansion layer in the Block module, is the number of output channels of the expansion layer convolution, the channel output represents the number of channels output by the Block layer, and the activation function represents the activation function used in the Block layer ( Figure 2 The stride in Table 1 represents the stride of the depthwise separable convolution in the Block module.

[0086] The parameters used in each Block layer correspond to those in Table 1. Taking the second Block layer as an example, the input feature map size is 56ⅹ56ⅹ16. The expansion layer in the Block increases the dimension to 32. The stride used in the depthwise separable convolution is 2, the convolution kernel is 3ⅹ3, the number of groups defaults to the increased dimension size of 32, the padding defaults to the convolution kernel size / / 2, the projection layer output is 24, and the feature map size obtained by this Block layer is 28ⅹ28ⅹ24, which is the input size of the third Block layer. Only the first Block layer in Table 1 does not use the expansion layer, so the increased dimension size of the first Block layer is consistent with the channel output. The feature map size obtained by the feature extraction backbone network is 7ⅹ7ⅹ192. Each Block layer uses the SE attention module. The input of the SE module is the increased dimension size of the layer. The input of the SE module of the second Block layer is the increased dimension size of 32.

[0087] The second part of the network structure is multi-scale fusion prediction, which is used to extract features from the input image and fuse them at five scales to increase the expressive power of the model. They are the output S1 (56ⅹ56ⅹ16) after the first Block layer, the output S2 (28ⅹ28ⅹ24) after the third Block layer, the output S3 (14ⅹ14ⅹ48) after the sixth Block layer, the output S4 (7ⅹ7ⅹ192) after the ninth Block layer, and the output of the feature extraction backbone network. After a 7×7 convolution, the output S5 (1×1×192) is obtained. S1, S2, S3, and S4 are each connected to an average pooling layer. The kernel size and step size of the pooling layer are the length of the input size. Taking S1 as an example, the input size is 56×56×16, the average pooling layer kernel size is 56, the step size is 56, and the output size is 1×1×16. After that, the five inputs are expanded to 1 dimension and spliced. A fully connected layer with an input of 376 and an output of 196 is used for prediction, and 98 feature point coordinates are output.

[0088] Among them, (1-2) uses the public dataset WFLW to train the facial feature point detection network model. WFLW contains 10,000 faces, of which 7,500 are used for training and 2,500 are used for testing.

[0089] Among them, the loss function used is L2Loss:

[0090]

[0091] Among them, y gt Indicates the true feature point coordinate value, y pred Represents the coordinate value of the feature point predicted by the model, Represents the coordinate value of the i-th real feature point, Represents the coordinate value of the i-th predicted feature point;

[0092] The optimizer used in training is Adam, the batch size is 256, the learning rate is 0.0001, and the number of iterations is 120. After training, the test results of the model are shown in Table 2.

[0093] Table 2 Facial feature point detection network model test structure

[0094] NME Model size Inference time 0.0727 1.5M 3.6ms

[0095] Among them, the normalized mean error (NME) of the facial feature point evaluation index is defined as:

[0096]

[0097] Among them, x i To predict the coordinates of the i-th feature point, is the true value of the i-th coordinate, and d is the distance between the pupils of the two eyes.

[0098] According to the test results of the facial feature point detection network model, the model structure adopted in the present invention can effectively reduce the model calculation amount and parameter amount. The entire model size is only 1.5M, and the inference time on the CPU (i5-12400) is 3.3ms. It can also ensure the detection accuracy (NME is 0.0727). The accuracy and inference time meet the requirements of non-contact heart rate detection tasks.

[0099] (1-3) Input the face area into the face feature point detection network model to obtain the face feature points.

[0100] The face area is input into the face feature point detection network model, and the coordinates of 98 feature points are output.

[0101] (2) Determine the region of interest based on facial feature points.

[0102] According to the obtained feature point coordinates, the pixels of the background area, eyes and lips are removed, and the remaining skin area is used as the region of interest (ROI), such as Figure 3 shown.

[0103] (3) Separate the red, green, and blue channel signals in the region of interest as observation signals.

[0104] For each ROI, separate the red, green, and blue channel signals and take the average value to obtain: For a single video, three sets of original signals with a length equal to the number of video frames can be obtained: R(t), G(t), and B(t).

[0105] S103 : Separate the blood volume pulse wave signal from the observation signal using the pulse wave extraction algorithm POS.

[0106] The definition of the plane orthogonal tiling method POS algorithm is as follows:

[0107]

[0108] Where σ(·) represents the standard deviation, X x and Y s It is obtained by combining the RGB three-channel signals. In the case of a single light source, they are usually expressed as:

[0109]

[0110] Y s =G n +B n -2R n

[0111] Among them, R n ,G n ,B n Represent the normalized results of R(t), G(t), and B(t) respectively.

[0112] For a video sequence with N frames, the POS algorithm uses a sliding window of size T, applies the above algorithm to obtain the window output value, and uses the overlap-add method to obtain the final pulse signal output; where T is usually:

[0113] T=1.6*fps

[0114] Where fps is the camera sampling frequency.

[0115] S104 , performing band-pass filtering on the blood volume pulse wave signal, performing fast Fourier transform on the filtered signal to obtain a power spectrum density map, and calculating the heart rate value based on the power spectrum density map.

[0116] Furthermore, step S104 includes:

[0117] (1) Bandpass filtering is used on the blood volume pulse wave signal (BVP signal) to eliminate frequencies outside [0.75, 4] Hz, and then the filtered BVP signal is fast Fourier transformed to obtain a power spectrum density map;

[0118] (2) Detect the peak of the power spectrum density graph and find the frequency f corresponding to the peak hr , calculate the heart rate value.

[0119] The heart rate calculation formula is:

[0120] HR=60*f hr.

[0121] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0122] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0123] Example 2:

[0124] like Figure 4 As shown, this embodiment provides a non-contact heart rate detection system based on facial feature point detection. The system includes an image acquisition module 401, a facial feature point detection module 402, a channel signal separation module 403, a blood volume pulse wave signal separation module 404, and a heart rate value calculation module 405, wherein:

[0125] An image acquisition module 401 is configured to acquire a face video image and determine a face region based on the face video image;

[0126] The facial feature point detection module 402 is configured to input the facial region into a facial feature point detection network model to obtain facial feature points. The facial feature point detection network model includes a feature extraction portion and a multi-scale fusion prediction portion. The feature extraction portion includes a standard convolution layer and multiple Block layers, wherein the Block layer is a depthwise separable convolution module proposed by MobileNetV3. The multi-scale fusion prediction portion enhances the expressiveness of the model by fusing multiple images of different scales.

[0127] a channel signal separation module 403, configured to determine a region of interest based on the facial feature points, and separate the channel signals in the region of interest as observation signals;

[0128] a blood volume pulse wave signal separation module 404 for separating the blood volume pulse wave signal from the observation signal using a pulse wave extraction algorithm;

[0129] The heart rate calculation module 405 is configured to calculate the heart rate value based on the blood volume pulse wave signal.

[0130] The specific implementation of each module in this embodiment can be found in the above-mentioned embodiment 1, and will not be described one by one here; it should be noted that the system provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0131] Example 3:

[0132] This embodiment provides a computer device, which can be a computer, such as Figure 5 As shown, a processor 502, a memory, an input device 503, a display 504, and a network interface 505 are connected via a system bus 501. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 506 and an internal memory 507. The non-volatile storage medium 506 stores an operating system, a computer program, and a database. The internal memory 507 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 502 executes the computer program stored in the memory, the non-contact heart rate detection method of the above-mentioned embodiment 1 is implemented as follows:

[0133] Acquire a face video image, and determine a face area based on the face video image;

[0134] The face region is input into a facial feature point detection network model to obtain facial feature points; the facial feature point detection network model includes a feature extraction part and a multi-scale fusion prediction part, wherein the feature extraction part includes a standard convolution layer and multiple Block layers, and the Block layer is a depth-separable convolution module proposed by MobileNetV3; the multi-scale fusion prediction part enhances the expressiveness of the model by fusing multiple images of different scales;

[0135] Determine a region of interest based on the facial feature points, and separate channel signals in the region of interest as observation signals;

[0136] Separating a blood volume pulse wave signal from the observation signal using a pulse wave extraction algorithm;

[0137] A heart rate value is calculated according to the blood volume pulse wave signal.

[0138] Example 4:

[0139] This embodiment provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the non-contact heart rate detection method of the above embodiment 1 is implemented as follows:

[0140] Acquire a face video image, and determine a face area based on the face video image;

[0141] The face region is input into a facial feature point detection network model to obtain facial feature points; the facial feature point detection network model includes a feature extraction part and a multi-scale fusion prediction part, wherein the feature extraction part includes a standard convolution layer and multiple Block layers, and the Block layer is a depth-separable convolution module proposed by MobileNetV3; the multi-scale fusion prediction part enhances the expressiveness of the model by fusing multiple images of different scales;

[0142] Determine a region of interest based on the facial feature points, and separate channel signals in the region of interest as observation signals;

[0143] Separating a blood volume pulse wave signal from the observation signal using a pulse wave extraction algorithm;

[0144] A heart rate value is calculated according to the blood volume pulse wave signal.

[0145] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0146] The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which falls within the scope of protection of the present invention.

Claims

1. A non-contact heart rate detection method based on facial feature point detection, characterized in that: The method comprises: Acquire a face video image, and determine a face area based on the face video image; The face area is input into a facial feature point detection network model to obtain facial feature points; the facial feature point detection network model includes a feature extraction part and a multi-scale fusion prediction part, wherein the feature extraction part includes a standard convolution layer and 11 Block layers, and the Block layer is a depth-separable convolution module proposed by MobileNetV3; the multi-scale fusion prediction part includes multiple average pooling layers, convolution layers and fully connected layers, and enhances the expressiveness of the model by fusing multiple images of different scales; Based on the facial feature points, pixels of the background area, eyes and lips are removed, and the remaining skin area is used as the region of interest; the channel signal in the region of interest is separated as the observation signal; Separating a blood volume pulse wave signal from the observation signal using a POS algorithm; Calculating a heart rate value according to the blood volume pulse wave signal; The step of inputting the face region into a face feature point detection network model to obtain face feature points includes: The face area is input into the standard convolution layer, and the output image is processed by 11 Block layers; The outputs of the first, third, sixth, and ninth Block layers are taken as S1, S2, S3, and S4 respectively; The output of the last Block layer is processed by the convolution layer and output as S5; After S1, S2, S3 and S4 are processed by the average pooling layer respectively, the pooled S1, S2, S3, S4 and S5 are spliced, the spliced ​​result is input into the fully connected layer, and the coordinates of the facial feature points are output; and the kernel size and step size of each pooling layer are the length of the input image size.

2. The non-contact heart rate detection method according to claim 1, characterized in that: The Block layer includes an expansion layer, a depth-wise separable convolution, a projection layer, a residual connection and an SE attention module, wherein the expansion layer is used to map the low-dimensional space to the high-dimensional space; the depth-wise separable convolution is used to complete the convolution function and reduce the amount of calculation and parameters; the projection layer is used to map the high-dimensional features to the low-dimensional space; the residual connection is used to reduce the complexity of the model to reduce overfitting and prevent gradient disappearance; the SE attention module is used to enhance the channel features of the input feature map; in the Block layer, the residual connection is only used when the step size is equal to 1 and the number of input channels is equal to the number of output channels.

3. The non-contact heart rate detection method according to claim 1, wherein: The separating the channel signals in the region of interest as observation signals includes: For each region of interest (ROI), separate the red, green, and blue channel signals and take the average value to obtain: b(t); For a single video, three sets of original signals with a length equal to the number of video frames are obtained: R(t), G(t), and B(t) as observation signals.

4. The non-contact heart rate detection method according to claim 3, characterized in that: The method of separating the blood volume pulse wave signal from the observation signal by using the POS algorithm includes: The pulse wave extraction algorithm is defined as follows: Where σ(·) represents the standard deviation, X x and Y s It is obtained by combining the RGB three-channel signals. In the case of a single light source, they are usually expressed as: X s =G n -B n Y s =G n +B n -2R n Among them, R n ,G n ,B n Respectively represent the normalized results of R(t), G(t), and B(t); For a video sequence with N frames, the pulse wave extraction algorithm uses a sliding window of size T to obtain the window output value and uses the overlap-add method to obtain the final pulse signal output as the blood volume pulse wave signal; where T = 1.6*fps, and fps is the camera sampling frequency.

5. The non-contact heart rate detection method according to claim 1, wherein: Calculating the heart rate value according to the blood volume pulse wave signal includes: Bandpass filtering is performed on the blood volume pulse wave signal, fast Fourier transform is performed on the filtered signal to obtain a power spectrum density map, and the heart rate value is calculated according to the power spectrum density map, specifically including: Applying a bandpass filter to the blood volume pulse wave signal to eliminate frequencies outside a specified frequency range, and then performing a fast Fourier transform on the filtered blood volume pulse wave signal to obtain a power spectrum density map; Detect the peak of the power spectrum density graph and find the frequency f corresponding to the peak hr , calculate the heart rate HR: HR=60*f hr 。 6. The non-contact heart rate detection method according to claim 1, characterized in that: The face detector BlaceFace is used to detect the face video image and determine the face area.

7. The non-contact heart rate detection method according to any one of claims 1 to 6, characterized in that: Before inputting the face area into the face feature point detection network model, training the face feature point detection network model, and inputting the face area into the trained face feature point detection network model; The training of the facial feature point detection network model includes: The public dataset WFLW is used to train the face feature point detection network model, where the loss function is L2Loss: Among them, y gt Indicates the true feature point coordinate value, y pred Represents the coordinate value of the feature point predicted by the model, Represents the coordinate value of the i-th real feature point, Represents the coordinate value of the i-th predicted feature point; The normalized mean error (NME) of the facial feature point evaluation index is defined as: Among them, x i To predict the coordinates of the i-th feature point, is the true value of the i-th coordinate, and d is the distance between the pupils of the two eyes.

8. A non-contact heart rate detection system based on facial feature point detection, characterized in that: The system comprises: An image acquisition module is used to acquire a face video image and determine a face area based on the face video image; A facial feature point detection module is used to input the facial region into a facial feature point detection network model to obtain facial feature points; the facial feature point detection network model includes a feature extraction part and a multi-scale fusion prediction part, wherein the feature extraction part includes a standard convolution layer and 11 Block layers, and the Block layer is a depth-separable convolution module proposed by MobileNetV3; the multi-scale fusion prediction part includes multiple average pooling layers, convolution layers, and fully connected layers, and enhances the expressiveness of the model by fusing multiple images of different scales; The channel signal separation module is used to remove pixels of the background area, eyes and lips based on facial feature points, and use the remaining skin area as the region of interest; and separate the channel signal in the region of interest as the observation signal; a blood volume pulse wave signal separation module, configured to separate the blood volume pulse wave signal from the observation signal using a POS algorithm; a heart rate value calculation module, configured to calculate the heart rate value based on the blood volume pulse wave signal; The facial feature point detection module is specifically used to: The face area is input into the standard convolution layer, and the output image is processed by 11 Block layers; The outputs of the first, third, sixth, and ninth Block layers are taken as S1, S2, S3, and S4 respectively; The output of the last Block layer is processed by the convolution layer and output as S5; After S1, S2, S3 and S4 are processed by the average pooling layer respectively, the pooled S1, S2, S3, S4 and S5 are spliced, the spliced ​​result is input into the fully connected layer, and the coordinates of the facial feature points are output; and the kernel size and step size of each pooling layer are the length of the input image size.

9. A storage medium storing a program, characterized in that: When the program is executed by a processor, the non-contact heart rate detection method according to any one of claims 1 to 7 is implemented.

Images