A heart rate detection system and method based on CardiA2Net

Abstract

The present invention relates to computer vision technology and RPG heart rate detection technology, and specifically to a heart rate detection method based on CardiA2Net. The present invention utilizes CardiA2Net to combine a self-attention convolutional hybrid network (ACmix) with an attention-based long short-term memory (ALSTM) network to perform enhanced feature extraction from raw data and improve time series learning with an integrated attention mechanism. The present invention also proposes a comprehensive set of preprocessing methods for facial video datasets, effectively reducing noise interference in facial videos and improving the accuracy of heart rate detection. The present invention calculates heart rate through facial video analysis and simultaneously processes noise in the video, effectively reducing the difficulty of contactless heart rate detection and improving the accuracy of contactless heart rate detection.

Description

Technical Field

[0001] The present invention relates to the field of RPG heart rate detection technology, computer vision, deep learning and other technologies, and more specifically to a heart rate detection system and method based on CardiA2Net. Background Art

[0002] The heart is one of the most important organs in the human body. It supports the activities of almost all cells in the body. Therefore, having a healthy heart is a prerequisite for a healthy person. The latest cardiovascular disease report released by the National Center for Cardiovascular Diseases counted the incidence of cardiovascular disease between 1990 and 2017. The results show that the mortality rate of cardiovascular disease is much higher than that of other types of diseases, and the mortality rate is showing a slowly increasing trend.

[0003] Heart rate is one of the most important vital signs that reflects a person's physiological and mental health. Closely monitoring heart rate changes can help detect and prevent common cardiovascular diseases early. Traditional heart rate monitoring methods are primarily contact-based, which poses challenges in terms of comfort, cost, and suitability for users with sensitive skin.

[0004] Video-based remote heart rate monitoring provides a new approach to solving this problem. In recent years, non-contact heart rate detection technologies, such as rPPG-based methods, have attracted increasing attention. This technology remotely monitors an individual's heart rate by analyzing changes in weak blood pulse signals in facial videos, providing a convenient and non-invasive new approach for heart health monitoring. People only need to place an ordinary RGB camera at home to detect their heart rate at any time. Once an abnormal heart rate occurs, they can seek medical attention immediately. At the same time, this also poses a huge challenge to the accuracy and stability of remote heart rate monitoring results. How to improve the accuracy and stability of remote heart rate monitoring has become an urgent problem that needs to be solved. To this end, we have developed a heart rate detection method based on CardiA2Net. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention provides a heart rate detection system and method based on CardiA2Net, which solves the problems raised in the background technology.

[0006] This paper combines several existing network models and addresses the shortcomings of existing technologies to propose a heart rate detection method based on CardiA2Net. This method can effectively process face videos to detect heart rate, thereby improving the accuracy and stability of remote heart rate monitoring. The specific steps are as follows:

[0007] (1) Collect videos containing faces and preprocess them:

[0008] (1.1) Use the open source model to detect the 68 key points of the face frame by frame in the face video in the dataset;

[0009] (1.2) Based on the key points detected in step 1.1, the face area is cropped from each frame of the video. Then, the Skin Segmentation Model is used to extract the truly meaningful skin area in the face and convert it to the YUV color space;

[0010] (1.3) Cut out the ROI area from the skin area obtained in step 1.2;

[0011] (1.4) Divide the face image into n grids according to the predefined ROI obtained in step (1.3), and then calculate and merge the pixel signals in the grids to obtain a PixelMap;

[0012] (1.5) Apply a series of operations to all time series of PixelMap to obtain STMap;

[0013] (2) Construct the CardiA2Net neural network, which is composed of a self-attention convolutional hybrid network ACmix and an attention-based long short-term memory ALSTM network, where:

[0014] (2.1) Using the STMap calculated in step (1) as input, feature extraction is performed through the ACMix network, and then the predicted heart rate of each frame is obtained;

[0015] (2.2) After the predicted heart rate is calculated in step (2.1), a fully connected layer converts the output of ACMix into a form that meets the input requirements of the subsequent ALSTM network;

[0016] (2.3) The input format that meets the requirements obtained in step (2.2) is passed to the ALSTM network. Finally, the ALSTM network learns and models the dependencies between adjacent heart rate measurements from the input, thereby improving the accuracy of heart rate estimation.

[0017] 1. Compared with existing remote heart rate monitoring methods, the present invention simultaneously considers a series of preprocessing of facial video datasets, including face detection, cropping, skin detection, YUV color space conversion, ROI acquisition, and finally generation of PixelMap and STMap, to obtain the separation of heart rate-related features and noise features in a more specific way, thereby obtaining cleaner heart rate-related features, thereby greatly improving the accuracy of remote heart rate monitoring. At the same time, combined with the ACMix network and ALSTM network to enhance feature extraction and heart rate analysis of the data, effectively improving the accuracy of non-contact heart rate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the overall flow chart of face video preprocessing;

[0019] Figure 2 This is the result of face detection and key point identification using an open source model;

[0020] Figure 3 It is a schematic diagram of the Skin Segmentation Model;

[0021] Figure 4 This is a schematic diagram of the ACMix network structure;

[0022] Figure 5 This is a schematic diagram of the ALSTM network structure. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] A heart rate detection system and method based on CardiA2Net:

[0025] (1) Collect videos containing human faces and preprocess them. Figure 1 This is the overall flow chart of face video preprocessing, and the specific process is as follows:

[0026] (1.1) Use the open source model to detect the 68 key points of the face frame by frame in the face video in the dataset;

[0027] (1.1.1) Use the open source model RetinaFace to perform face detection on face videos. Mark the position of the rectangular box containing the face in each frame of the video. (x1, y1) represents the horizontal and vertical coordinates of the upper left corner of the rectangular box, and (x2, y2) represents the horizontal and vertical coordinates of the lower right corner of the rectangular box. That is, the coordinates of the rectangular box where the face appears can be expressed as (x1, y1, x2, y2). Figure 2 (a) is the result image of face detection;

[0028] (1.1.2) Use the open source model MobileNet to further identify the faces within the calibrated rectangular frame detected in step (1.1.1) and obtain 68 key points on each face. The specific distribution of key points is as follows: Figure 2 As shown in (b);

[0029] (1.2) Based on the key points detected in step 1.1, the face area is cropped from each frame of the video. Then, the Skin Segmentation Model is used to extract the truly meaningful skin area in the face and convert it to the YUV color space;

[0030] (1.2.1) Based on the calibration of a series of facial regions obtained in step (1.1), only the facial region is obtained from the image and the background is removed. The skin region is then extracted using the Skin Segmentation Model to remove the excessive noise in the skin region that is not detected due to coarse facial hair, excessive brightness, and specular reflection. The Skin Segmentation Model is a fully convolutional neural network (FCN) that uses an encoder-decoder structure. Its overall architecture is as follows: Figure 3 As shown in Figure 2, the model was trained on the ECU dataset using a binary cross entropy loss function and a stochastic gradient descent (SGD) optimizer, where the loss function can be expressed as follows:

[0031] (11)

[0032] where N represents the number of categories (here N = 2 for skin pixels and non-skin pixels), represents a label, and represents the predicted probability,

[0033] This model gives a skin probability mask as output, which is used in our main architecture to obtain output skin frames. To retrieve skin frames, a thresholding operation is performed on the mask generated from the FCN model so that 0 represents non-skin pixels and 1 represents skin pixels. This calculated mask is then multiplied with the RGB channels of the original face frame, thus providing the desired skin region from the corresponding input video frame. Standard data augmentation techniques are used during training, along with image color variation, to accommodate different skin tones.

[0034] The obtained skin area image is then converted to the YUV color space. The YUV color space conversion formula is:

[0035] (12)

[0036] (1.3) Cut out the ROI region from the skin area obtained in step 1.2;

[0037] (1.3.1) Based on the location information of the calibrated facial key points, remove non-skin facial areas, such as the eyes and mouth, and appropriately reduce the ROI area to reduce the impact of noise during heart rate detection. At the same time, using a smaller ROI area can reduce the amount of data required for model calculation;

[0038] (1.4) Divide the face image into n grids according to the predefined ROI obtained in step (1.3), and then calculate and merge the pixel signals in the grids to obtain the PixelMap;

[0039] (1.4.1) Divide the face image into n grids according to the predefined ROI, calculate the average pixel value of each grid, and then connect it to the T sequence of C channels. The n grids are placed directly in a row. We represent the combined signal sequence as PixelMap. The dimension of PixelMap is , where n represents the number of ROIs, T represents the number of frames in the video clip, and C = 3 represents the three channels of R, G, and B;

[0040] (1.5) Apply a series of operations to all time series of PixelMap to obtain STMap;

[0041] (1.5.1) Apply the zoom operation and max-min normalization to all time series of PixelMap, scale the time series to [0, 255], and get STMap. PixelMap and STMap have the same dimensions ;

[0042] The calculated PixelMap and STMap are represented as pm and m respectively. Then the goal is to build a map:

[0043] [1] (13)

[0044] in, Indicates the mapping relationship from PixelMap to STMap, pm represents PixelMap, and m represents STMap.

[0045] (2) Construct the CardiA2Net neural network, which is composed of a self-attention convolutional hybrid network (ACmix) network and an attention-based long short-term memory (ALSTM) network, such as Figure 5 In this work, the final goal is to predict heart rate from videos using ACMix. The role of ALSTM is to learn and model the dependencies between adjacent heart rate measurements. The loss function then applies to normalize the ACMix predictions, where:

[0046] (2.1) Using the STMap calculated in step (1) as input, we perform feature extraction through the ACMix network and then obtain the predicted heart rate for each frame.

[0047] (2.1.1) ACMix is ​​specifically designed to extract complex feature representations from STMaps. ACMix can independently predict the heart rate of each frame based on these features. ACMix uses multiple parallel convolution paths with different kernel sizes to capture the extensive contextual information contained in STMaps, such as Figure 4 The ACMix attention module operates in parallel on the outputs of the convolution branch and the self-attention branch, and the strengths in the branches are respectively Scalar control, the formula is as follows:

[0048] (14)

[0049] in, 、 They are the output results of the self-attention branch and the convolution branch respectively. is the output result of the final branch,

[0050] The relevant formulas for the frame-by-frame heart rate prediction stage are:

[0051] ACMix output results Expressed as:

[0052] (15)

[0053] in, The output of STMap after ACmix network processing.

[0054] (2.2) After the predicted heart rate is calculated in step (2.1), a fully connected layer converts the output of ACMix into a form that meets the input requirements of the subsequent ALSTM network;

[0055] The processing of the fully connected layer can be expressed as:

[0056] (16)

[0057] Where W and b represent the weight and bias of the fully connected layer respectively, and x is the output of the fully connected layer;

[0058] (2.3) The input format obtained in step (2.2) is passed to the ALSTM network. Finally, the ALSTM network learns and models the dependencies between adjacent heart rate measurements from the input, thereby improving the accuracy of heart rate estimation.

[0059] (2.3.1) The ALSTM model has an integrated attention mechanism built in. Figure 5 ,The architecture of ALSTM consists of a sequence of layers, first through a fully connected layer and activation function to align the input dimensions, followed by an LSTM layer that captures the dependencies in the data, the penultimate stage involves an attention-driven layer that assigns weights to the outputs at different time steps, followed by a final linear output layer;

[0060] ALSTM related formula:

[0061] Output of ALSTM Expressed as:

[0062] (17)

[0063] in, After being processed by ALSTM, x is output from the fully connected layer.

[0064] The key attention mechanism, its calculation process can be expressed as the following formula:

[0065] (18)

[0066] in, is the hidden state of LSTM, and are the projection parameters to be learned during training, is the weighted hidden vector of the attention mechanism, where and yes The parameters of the layer. And It ensures that the sum of the attention scores is 1. Represents the weight calculated by the attention mechanism;

[0067] Then, the LSTM output adjusted by attention weights can be expressed as:

[0068] (19)

[0069] in is the LSTM output after attention weight adjustment

[0070] The final output is a combination of the last state of the LSTM and the attention output using a fully connected layer:

[0071] (20)

[0072] in It is the final output of ALSTM;

[0073] (2.3.2) The specific formula of the loss function used in the final model training process is as follows:

[0074] (twenty one)

[0075] in represents the L1 loss function, which is calculated based on the results predicted by the ACMix network. is a parameter used to balance the two terms;

[0076] According to the calculation results of the ALSTM network, considering the dependency between adjacent heart rate measurements, the average HR within T consecutive measurements is calculated as:

[0077] (twenty two)

[0078] Smoothing loss Defined as:

[0079] (twenty three)

[0080] In summary, the heart rate detection method based on CardiA2Net proposed in the present invention provides a new dataset processing method for the RPPG heart rate detection task. At the same time, the method of combining the attention mechanism model for detection multiple times makes the heart rate detection more accurate.

[0081] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

Claims

1. A heart rate detection method based on CardiA2Net, characterized in that: Enhanced feature extraction and heart rate analysis of the data are performed using the ACMix network and the ALSTM network. The specific steps are as follows: (1) Collect videos containing faces and preprocess them: (1.1) Use the open source model to detect the 68 key points of the face frame by frame in the face video in the dataset; (1.2) Based on the key points detected in step (1.1), the face area is cropped from each frame of the video. Then, the Skin Segmentation Model is used to extract the truly meaningful skin area in the face and convert it to the YUV color space; (1.3) Cut out the ROI area from the skin area obtained in step (1.2); (1.4) Divide the face image into n grids according to the predefined ROI obtained in step (1.3), and then calculate and merge the pixel signals in the grids to obtain a PixelMap; (1.5) Apply a series of operations to all time series of PixelMap to obtain STMap; (2) Construct the CardiA2Net neural network, which is composed of a self-attention convolutional hybrid network ACmix network and an attention-based long short-term memory ALSTM network, where: (2.1) Using the STMap calculated in step (1) as input, feature extraction is performed through the ACMix network, and then the heart rate prediction result is obtained; (2.2) After the predicted heart rate is calculated in step (2.1), a fully connected layer converts the output of ACMix into a form that meets the input requirements of the subsequent ALSTM network; (2.3) The input format obtained in step (2.2) is passed to the ALSTM network. Finally, the ALSTM network learns and models the dependencies between adjacent heart rate measurements from the input, thereby improving the accuracy of heart rate estimation. In step (2.1), the STMap calculated in step (1) is used as input and firstly subjected to feature processing through the ACMix network, specifically including: (2.1.1) ACMix is ​​specifically designed to extract complex feature representations from STMaps. ACMix can independently predict the heart rate of each frame based on these features. ACMix uses multiple parallel convolution paths with different kernel sizes to capture the extensive contextual information contained in the STMap. The ACMix attention module operates in parallel on the outputs of the convolution branch and the self-attention branch. The strength of the branches is respectively determined by Scalar control, the formula is as follows: （4） in, 、 They are the output results of the self-attention branch and the convolution branch respectively. is the output result of the final branch, The relevant formulas for the frame-by-frame heart rate prediction stage are: ACMix output results Expressed as: （5） in, The output after processing STMap for ACmix network; In step (2.2), after the feature extraction in step (2.1), a fully connected layer converts the output of ACMix into a form that meets the input requirements of the subsequent ALSTM network, specifically including: The processing of the fully connected layer is expressed as: （6） Where W and b represent the weight and bias of the fully connected layer respectively, and x is the output of the fully connected layer; In the step (2.3), the input format that meets the requirements obtained according to the step (2.2) is transmitted to the ALSTM network. Finally, the ALSTM network learns and models the dependency between adjacent heart rate measurements from the input, thereby improving the accuracy of heart rate estimation, specifically including: (2.3.1) The ALSTM model has an integrated attention mechanism built into it. The ALSTM architecture consists of a sequence of layers, first through a fully connected layer and activation function to align the input dimensions, followed by an LSTM layer that captures the dependencies in the data. The penultimate stage involves an attention-driven layer that assigns weights to the outputs at different time steps, followed by a final linear output layer. ALSTM related formula: Output of ALSTM Expressed as: （7） in, After ALSTM processing, x is output from the fully connected layer. The key attention mechanism is expressed in the following formula: （8） in, is the hidden state of LSTM, and are the projection parameters to be learned during training, is the weighted hidden vector of the attention mechanism, where and yes The parameters of the layer, and It ensures that the sum of the attention scores is 1. Then, the LSTM output after attention weight adjustment is expressed as: （9） in is the LSTM output after attention weight adjustment, Represents the weight calculated by the attention mechanism; The final output is a fully connected layer that combines the last state of the LSTM and the attention output: （10） in It is the final output of ALSTM.

2. A heart rate detection method based on CardiA2Net according to claim 1, characterized in that: In step (1.1), the open source model is used to detect the 68 key points of the face frame by frame in the face video in the dataset, specifically including: (1.1.1) Use the open source model RetinaFace to perform face detection on face videos and mark the location of the face in each frame of the video, that is, the coordinates of the rectangular box where the face appears; (1.1.2) Use the open source model MobileNet to further calibrate the faces within the calibration frame detected in step (1.1.1) to obtain 68 key points on each face.

3. A heart rate detection method based on CardiA2Net according to claim 1, characterized in that: In the step (1.2), the face area is cropped from each frame of the video, and then the skin area is extracted using the Skin Segmentation Model and converted to the YUV color space, specifically including: (1.2.1) Based on the calibration of a series of facial regions obtained in step (1.2), only the facial region is obtained from the image, and the background is removed. Then, the skin region is extracted using the Skin Segmentation Model to remove the excessive noise in the skin region that is not detected due to coarse facial hair, excessive brightness, and specular reflection. The Skin Segmentation Model is a fully convolutional neural network (FCN) with an encoder-decoder structure. The Skin Segmentation Model is trained on the ECU dataset using a binary cross entropy loss function and a stochastic gradient descent (SGD) optimizer. The loss function is expressed as follows: （1） Where N represents the number of categories (here N=2 for skin pixels and non-skin pixels), represents a label, and represents the predicted probability, The obtained skin area image is then converted to the YUV color space. The YUV color space conversion formula is: （2）。 4. A heart rate detection method based on CardiA2Net according to claim 1, characterized in that: In the step (1.3), the ROI region cut out from the skin region obtained in the step (1.2) specifically includes: (1.3.1) Based on the location information of the calibrated facial key points, remove non-skin facial areas, such as the eyes and mouth, and appropriately reduce the ROI area to reduce the impact of noise during heart rate detection. At the same time, using a smaller ROI area can reduce the amount of data used for model calculation.

5. A heart rate detection method based on CardiA2Net according to claim 1, characterized in that: In the step (1.4), the face image is divided into n grids according to the predefined ROI obtained in step (1.3), and then the pixel signals in the grids are calculated and merged to obtain a PixelMap, which specifically includes: (1.4.1) Divide the face image into n grids according to the predefined ROI, calculate the average value of the pixel value of each grid, and then connect it to the T sequence of C channels. The n grids are placed directly in a row. We represent the combined signal sequence as PixelMap. The dimension of PixelMap is , where n represents the number of ROIs, T represents the number of frames in the video clip, and C=3 represents the three channels of R, G, and B.

6. A heart rate detection method based on CardiA2Net according to claim 1, characterized in that: In the step (1.5), a series of operations are applied to all time series of PixelMap to obtain STMap, specifically including: (1.5.1) applying a zoom operation and maximum-minimum normalization to all time series of PixelMap, scaling the time series to [0, 255], thereby obtaining STMap, PixelMap and STMap have the same dimensions , Denote the calculated PixelMap and STMap as pm and m respectively, and then the goal is to build a map: （3） in, Indicates the mapping relationship from PixelMap to STMap, pm represents PixelMap, and m represents STMap.

Images