A mobile terminal platform-oriented remote visual heart rate measurement deep learning method

Abstract

The present application relates to the technical field of heart rate measurement, in particular to a remote visual heart rate measurement deep learning method for mobile terminal platform, comprising: acquiring a face video image and collecting human body heart rate data corresponding to each frame of the face video image; acquiring a grid area and key points in the grid area to obtain an important target area corresponding to each key point; constructing a lightweight neural network model; and acquiring human body heart rate data corresponding to the current frame of the face video image. The present application can efficiently and accurately measure remote human body heart rate data on a mobile device.

Description

Technical Field

[0001] This invention relates to the field of heart rate measurement technology, and more specifically to a deep learning method for remote visual heart rate measurement for mobile platforms. Background Technology

[0002] Among the various indicators used in popular health monitoring, heart rate is the most widely used and most important physiological indicator. Real-time monitoring of a user's heart rate and analysis of heart rate variability can identify potential health problems and provide early warnings, reminding users with chronic cardiovascular and cerebrovascular diseases to take timely intervention and treatment measures to avoid tragedies.

[0003] Existing heart rate measurement technologies traditionally utilize electrocardiograms (ECGs) or wearable heart rate sensors. However, these methods are inconvenient, limiting the real-time performance and portability of heart rate detection, and can also cause discomfort by pressing on the user's skin and restricting daily activities. Unlike more invasive contact monitoring, remote physiological signal monitoring captures images of the human body via a camera. When light shines on the skin, the amount absorbed is influenced by the current peripheral blood volume. Therefore, the pulse waveforms generated by the subtle movements caused by blood pumping around the body can be captured by deep learning models to deduce heart rate and heart rate variability. Thus, developing a remote visual heart rate measurement method for mobile platforms is of great significance.

[0004] In the field of remote visual heart rate measurement, several studies and technologies have proposed deep learning-based methods. These methods use convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to extract heart rate-related features from image sequences and perform heart rate estimation. However, most of these methods are designed for desktop or dedicated devices, and their adaptability and performance on mobile platforms still need improvement. Furthermore, applying deep learning to mobile platforms requires consideration of computational resource and power consumption limitations. Mobile devices typically have limited computing power and battery life. In addition, current vision-based remote heart rate measurement methods often require complex and computationally intensive image preprocessing methods, making them difficult to port to mobile platforms.

[0005] Therefore, it is necessary to design lightweight deep learning models and optimization algorithms to efficiently perform remote heart rate measurement on mobile devices to solve the above problems. Summary of the Invention

[0006] This invention provides a deep learning method for remote visual heart rate measurement on mobile platforms. By designing a lightweight deep learning model and optimizing the algorithm, it enables efficient remote heart rate measurement on mobile devices.

[0007] The present invention provides a deep learning method for remote visual heart rate measurement on mobile platforms, which adopts the following technical solution: including:

[0008] Acquire multiple consecutive frames of facial video images and collect the human heart rate data corresponding to each frame of facial video image;

[0009] The face video image is divided into multiple grid regions of the same size; the target region corresponding to each key point is obtained based on the key points in the grid regions;

[0010] Obtain the attention entropy of each target region, and obtain the importance value of the target region by using a linear classifier and based on the attention entropy of each target region and the preset probability vector of the target region as an important target region. Determine the most important target region based on the importance value of the target region.

[0011] A lightweight neural network model is established based on the time translation module. The important target region corresponding to each frame of face video image is used as input, and the human heart rate data corresponding to each frame of face video image is used as output to obtain the trained lightweight neural network model.

[0012] The important target region corresponding to the current frame of the face video image is input into the trained lightweight neural network model to obtain the human heart rate data corresponding to the current frame of the face video image.

[0013] Preferably, the lightweight neural network model includes:

[0014] The input layer is used to input important target regions corresponding to multiple consecutive frames of face video images. It divides the image sequence composed of multiple consecutive frames of face video images into multiple first image segments, and extracts one frame from each image segment to form a second image segment.

[0015] The temporal translation convolutional layer is used to perform single-branch temporal translation and two-dimensional convolution operations on the second image segment and each first image segment respectively, and outputs the graph matrix corresponding to the second image segment and each first image segment;

[0016] The first connection layer is used to connect the second image segment and the graph matrix corresponding to each first image segment, and output the target graph matrix corresponding to the second image segment and each first image segment.

[0017] The second connection layer is used to connect the second image segment with the target image matrix corresponding to each first image segment, and output the final image matrix corresponding to the image sequence.

[0018] The attention-based face masking layer is used to obtain the attention weight matrix composed of the attention weights of all frames of face video in the image sequence, and multiply the attention weight matrix with the final image matrix to output the feature matrix corresponding to the image sequence.

[0019] The average pooling layer is used to perform average pooling operations on the feature matrices corresponding to the image sequence to output the human heart rate data corresponding to each frame of face video image.

[0020] And an output layer, used to output the human heart rate data corresponding to each frame of face video image.

[0021] Preferably, the formula for calculating the attention weights corresponding to all frames of face video in the image sequence is:

[0022]

[0023] In the formula, Y k This represents the attention weight corresponding to the k-th frame of the face video image in the image sequence;

[0024] X k This represents the feature map matrix corresponding to the k-th frame of the face video image in the image sequence;

[0025] H k This represents the 1x1 convolution weight matrix used for the k-th frame of the face video image in the image sequence, used to generate attention.

[0026] Mask;

[0027] W k This indicates that scalar weights are used to scale the attention mask of the k-th frame face video image;

[0028] σ represents the sigmoid activation function, which is used to normalize the attention mask to the range [0,1].

[0029] ω k Represents a 1×1 convolution;

[0030] b k This represents the bias term of a 1x1 convolutional layer;

[0031] L1 represents the L1 norm, which is used to normalize the generated attention mask.

[0032] Preferably, the expression for the attention entropy of the target region is:

[0033]

[0034] In the formula, e h This represents the attention entropy of the h-th target region;

[0035] h represents the index of the attention head in the multi-attention mechanism, and h represents the h-th target region;

[0036] This represents the attention score calculated for the i-th grid cell in the h-th target region;

[0037] N i This indicates the total number of grid cells.

[0038] Preferably, the step of determining the probability vector of the preset target region as an important target region is as follows:

[0039] When the target area is the forehead or cheek area, the probability vector of the target area being an important target area is (1.0);

[0040] When the target region is the eye, nose, or mouth region, the probability vector of the target region being an important target region is (0.5.0.5);

[0041] When the target region is the ear region, the probability vector of the target region being an important target region is (0.1).

[0042] Preferably, the expression for the importance value in the target region is:

[0043]

[0044] In the formula, c represents the importance value of the target region;

[0045] e represents the attention entropy vector of all target regions in each frame of a face video image;

[0046] This represents the first weight vector of the neural network model corresponding to the linear classifier;

[0047] This represents the second weight vector of the neural network model corresponding to the linear classifier;

[0048] b represents the bias vector of the neural network model corresponding to the linear classifier;

[0049] p represents the probability vector that the preset target region is an important target region.

[0050] Preferably, the step of obtaining the target region corresponding to each key point based on the key points in the grid region is as follows:

[0051] The facial landmark recognition algorithm is used to identify key points in a grid area;

[0052] Based on the key points of each identified raster region, differential normalization is performed on all raster regions with the same key points to obtain the target region corresponding to each key point.

[0053] Preferably, the video frame image is rasterized to obtain a raster region.

[0054] Preferably, the step of acquiring human heart rate data is as follows: while acquiring each frame of facial video image, the human heart rate data is simultaneously measured using a heart rate measuring instrument.

[0055] The beneficial effects of this invention are:

[0056] By designing a lightweight neural network model, and using the important target regions corresponding to each frame of a face video image, along with the human heart rate data corresponding to each frame of a face video image as a dataset, the lightweight neural network model is trained to obtain a well-trained model. Finally, based on the important target regions corresponding to the current frame of the face video image, heart rate measurement can be efficiently performed on mobile devices. This invention utilizes the time-shifting convolutional layer of the lightweight neural network model to achieve the processing speed of two-dimensional convolution and the effect of three-dimensional convolution. Simultaneously, it segments and processes the image sequence composed of consecutive frames of face video images in parallel, extracting short-term information between adjacent video frames and periodic information between multiple video frames to achieve accurate feature extraction and thus obtain accurate human heart rate data. Furthermore, rasterization technology is used to extract the raster regions that need to be processed from the video frames, reducing the computational complexity of preprocessing and achieving end-to-end detection. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a deep learning method for remote visual heart rate measurement on a mobile platform according to the present invention.

[0059] Figure 2 This is a flowchart of the process of obtaining the target area in this embodiment;

[0060] Figure 3 This is a schematic diagram of the lightweight neural network model in this embodiment. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] This invention provides an embodiment of a deep learning method for remote visual heart rate measurement on mobile platforms. This embodiment primarily targets mobile platforms (phones, Raspberry Pi)—a crucial future platform for deploying intelligent applications. It designs a lightweight deep learning model and visual preprocessing algorithm to efficiently measure human heart rate data remotely on mobile devices. Specifically, as follows... Figure 1 As shown, this embodiment includes:

[0063] S1. Acquire facial video images and collect the human heart rate data corresponding to each frame of facial video images;

[0064] Specifically, multiple consecutive frames of facial video images are acquired, and heart rate data is measured using a heart rate measuring instrument while acquiring each frame of facial video image.

[0065] S2. Obtain the grid region and key points within the grid region to obtain the important target region corresponding to each key point;

[0066] The face video image is divided into multiple grid regions of the same size. The target region corresponding to each key point is obtained based on the key points in the grid region. The attention entropy of each target region is obtained. Based on the attention entropy of each target region and the preset probability vector of the target region being an important target region, the important target region in the target region is obtained.

[0067] Step 21: Based on the prior knowledge that the most significant regions of remote physiological signals are the forehead and cheeks, the input video frame image is rasterized, that is, it is divided into different regions on an average basis and converted into a token sequence, that is, a raster region sequence with serial number information.

[0068] Step 22, as follows Figure 2 As shown, the steps for obtaining the target region corresponding to each key point based on key points in the grid region are as follows: Key points in the grid region are identified using a facial landmark recognition algorithm; based on the identified key points in each grid region, all grid regions with the same key points are differentially normalized to obtain the target region corresponding to each key point. In this embodiment, key points in the grid region are identified using a facial landmark recognition algorithm. These key points identify the contours of the face and organs, and the positions of different regions in the image, such as the eyes and nose; using these key points, the grid regions containing the key points can be combined to divide the face into different target regions. Each target region may include one or more grid regions; then, the importance order of human facial features is sorted to obtain a target region sequence.

[0069] Step 23: Obtain the attention entropy of each target region. Based on the attention entropy of each target region and the preset probability vector of the target region being an important target region, obtain the importance value of the target region. Determine the most important target region based on the importance value of the target region.

[0070] In this embodiment, an attention mechanism is introduced. A linear classifier distinguishes target regions into important target regions and unimportant target regions based on the attention entropy of different target regions. The formula for calculating the attention entropy of a target region is as follows:

[0071]

[0072] In the formula, e h The attention entropy of the h-th target region is represented; h represents the index of the attention head in the multi-attention mechanism, and represents the h-th target region. N represents the attention score calculated for the i-th grid cell in the h-th target region; i This indicates the total number of grid cells.

[0073] In this embodiment, the step of setting the probability vector of the preset target area as an important target area is as follows: when the target area is the forehead or cheek area, the probability vector of the target area as an important target area is (1.0); when the target area is the eye, nose or mouth area, the probability vector of the target area as an important target area is (0.5.0.5); when the target area is the ear area, the probability vector of the target area as an important target area is (0.1).

[0074] The expression for the importance value of the target region is as follows:

[0075]

[0076] In the formula, c represents the importance value of the target region; e represents the attention entropy vector of all target regions in each frame of face video image. The length of the attention entropy vector is equal to the number of target regions, and each element in the attention entropy vector is the entropy value corresponding to each target region. This represents the first weight vector of the neural network model corresponding to the linear classifier; denoted as b, it represents the second weight vector of the neural network model corresponding to the linear classifier; b represents the bias vector of the neural network model corresponding to the linear classifier; p represents the probability vector that the preset target region is an important target region. It should be noted that the first weight vector is used during neural network training. Second weight vector These are the parameter vectors that will be updated. Specifically, during the training phase, the element values ​​of these two weight vectors are updated using the backpropagation algorithm based on the training data and the loss function. Their lengths are equal to the number of target regions and the dimension of the predicted probability, respectively. This design allows the lightweight neural network model to adaptively learn the relative importance of the attention entropy and predicted probability of the target region to the final result. By training the lightweight neural network model, the optimal weight combination can be found, allowing the attention entropy and predicted probability to jointly contribute to the final classification result. It is the first trainable weight vector, with a length equal to the number of target regions. This involves transposing the weight vector w1 into a row vector, then performing a dot product operation between this row vector and the attention entropy vector e to obtain a scalar value. This scalar value represents the contribution of the attention entropy vector e to the importance value c of the target region. Similarly, Another approach involves transposing the weight vector into a row vector. The dot product of this row vector and the probability vector p indicating that the target region is an important target region represents the contribution of this probability vector to the importance value c of the target region. The bias vector b represents the bias term of the neural network model corresponding to the linear classifier. It is a scalar value and a trainable parameter. Its function is to provide a learnable constant term to adjust the output of the linear classifier, allowing it to depend not only on the input feature attention entropy vector e and the probability vector p indicating that the target region is an important target region, but also on an additional adjustable bias. By introducing this bias term, the expressive power of the linear classifier is further enhanced, allowing it to better fit the training data and thus improve the final prediction performance.

[0077] S3. Construct a lightweight neural network model;

[0078] A lightweight neural network model is established based on the time translation module. The important target region corresponding to each frame of face video image is used as input, and the human heart rate data corresponding to each frame of face video image is used as output to obtain the trained lightweight neural network model.

[0079] S31. For the task of detecting human heart rate data, high frame rate and low latency detection on mobile devices are necessary for long-term monitoring and disease prediction of heart rate variability. Human heart rate data detection captures subtle changes in reflected light from the body caused by physiological or physiological factors, measuring changes in light absorption caused by changes in blood volume on the skin surface (the higher the blood volume, the higher the light absorption). This, in turn, affects the amount of visible light reflected by the skin. In addition, since human heart rate data exhibits temporal and periodic characteristics, and the input to the neural network model is a spatial image, spatiotemporal modeling is required to capture human heart rate data in the image. However, due to limitations in the computing resources of mobile devices and the problem that current deep learning models focus on extracting explicit features, an accurate and efficient neural network model is needed for processing. Therefore, this embodiment establishes a lightweight neural network model based on a time translation module.

[0080] In this embodiment, the lightweight neural network model includes: an input layer, a temporal translation convolutional layer, a first connection layer, a second connection layer, an attention-based face mask layer, an average pooling layer, and an output layer. The input layer is used to input important target regions corresponding to multiple consecutive frames of face video images, divides the image sequence composed of multiple consecutive frames of face video images into multiple first image segments, and extracts one frame from each image segment to form a second image segment. The temporal translation convolutional layer is used to perform single-branch temporal translation and two-dimensional convolution operations on the second image segment and each first image segment respectively, outputting the graph matrix corresponding to the second image segment and each first image segment. The first connection layer is used to connect the second image segment and each first image segment... The first image segment is connected to the corresponding graph matrix to output the second image segment and the target graph matrix corresponding to each first image segment. The second connection layer is used to connect the second image segment and the target graph matrix corresponding to each first image segment to output the final graph matrix corresponding to the image sequence. The attention-based face mask layer is used to obtain the attention weights composed of the attention weights corresponding to all frames of face video in the image sequence, and multiply the attention weight matrix with the final graph matrix to output the feature matrix corresponding to the image sequence. The average pooling layer is used to perform average pooling operation on the feature matrix corresponding to the image sequence to output the human heart rate data corresponding to each frame of face video image. The output layer is used to output the human heart rate data corresponding to each frame of face video image.

[0081] It should be noted that the lightweight neural network model in this embodiment is built based on a time-shifting module: the time-shifting module is used to facilitate information exchange between adjacent frames by shifting part of the channel along the time dimension of the feature map. It can be inserted into a two-dimensional convolution to achieve zero-computation, zero-parameter time modeling. Addressing the issue that a single time-shifting module can only perform spatiotemporal modeling on adjacent frames and cannot cover long sequence data, this embodiment divides the image sequence consisting of multiple consecutive frames of face video images into multiple first image segments at the input layer of the lightweight neural network model, and extracts one frame from each image segment to form a second image segment. That is, in this embodiment, the image sequence consisting of multiple consecutive frames of face video images is divided into multiple first image segments, with each first image segment containing 3 frames of face video images. Then, feature extraction is performed in the time-shifting convolutional layer.

[0082] In this embodiment, the purpose of designing a face mask layer based on attention in the lightweight neural network model is as follows: Spatiotemporal modeling solely through a time-shifting module introduces additional temporal information and noise. Therefore, it is crucial to pay attention to pixels containing human heart rate data; otherwise, there is a risk of amplifying noise. Therefore, this invention inserts a face mask layer between the second connection layer and the average pooling layer of the lightweight neural network model to minimize the negative impact of tensor shifting and to supplement the information discarded during preprocessing, enabling the face mask layer to focus on the target signal. The face mask layer can assign higher weights to certain shifted pixels, which have stronger signals in the intermediate representation of the convolution operation. Specifically, this embodiment uses a Softmax attention mask, which is generated by a 1×1 convolution before the average pooling layer. The specific calculation formula for the Softmax attention mask is:

[0083]

[0084] In the formula, Y k This represents the attention weight corresponding to the k-th frame of the face video image in the image sequence;

[0085] X k This represents the feature map matrix corresponding to the k-th frame of the face video image in the image sequence;

[0086] H k This represents the 1x1 convolution weight matrix used for the k-th frame of the face video image in the image sequence, used to generate attention.

[0087] Mask;

[0088] W k This indicates that scalar weights are used to scale the attention mask of the k-th frame face video image;

[0089] σ represents the sigmoid activation function, which is used to normalize the attention mask to the range [0,1].

[0090] ω k Represents a 1×1 convolution;

[0091] b k This represents the bias term of a 1x1 convolutional layer;

[0092] L1 represents the L1 norm, which is used to normalize the generated attention mask.

[0093] S4. Obtain the human heart rate data corresponding to the current frame of the face video image;

[0094] The important target region corresponding to the current frame of the face video image is input into the trained lightweight neural network model to obtain the human heart rate data corresponding to the current frame of the face video image.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A mobile terminal platform oriented deep learning method for remote visual heart rate measurement, characterized in that, The method comprises the following steps: Obtain a plurality of continuous face video images and collect the human heart rate data corresponding to each frame of face video image; Divide the face video image into a plurality of grid regions of the same size; obtain the target region corresponding to each key point according to the key points in the grid region; Obtain the attention entropy of each target region, obtain the importance value of the target region through a linear classifier and according to the attention entropy of each target region and a preset probability vector of the target region being an important target region, and determine the most important important target region according to the importance value of the target region; Establish a lightweight neural network model based on a time translation module, input the important target region corresponding to each frame of face video image, input the human heart rate data corresponding to each frame of face video image as output, and obtain a trained lightweight neural network model; Input the important target region corresponding to the current frame of face video image into the trained lightweight neural network model to obtain the human heart rate data corresponding to the current frame of face video image. 2.The mobile platform oriented remote visual heart rate measurement deep learning method according to claim 1, wherein, The lightweight neural network model comprises: An input layer for inputting the important target region corresponding to a plurality of continuous face video images, dividing an image sequence formed by the plurality of continuous face video images into a plurality of first image segments, and extracting one frame from each image segment to form a second image segment; A time translation convolution layer for performing single-branch time translation and two-dimensional convolution operations on the second image segment and each first image segment respectively, and outputting a graph matrix corresponding to the second image segment and each first image segment; A first connection layer for connecting the graph matrix corresponding to the second image segment and each first image segment, and outputting a target graph matrix corresponding to the second image segment and each first image segment; A second connection layer for connecting the target graph matrix corresponding to the second image segment and each first image segment, and outputting a final graph matrix corresponding to the image sequence; An attention-based face mask layer for obtaining an attention weight matrix formed by the attention weights corresponding to all frames of face video in the image sequence, and multiplying the attention weight matrix and the final graph matrix to output a feature matrix corresponding to the image sequence; An average pooling layer for performing average pooling operation on the feature matrix corresponding to the image sequence to output the human heart rate data corresponding to each frame of face video image; And an output layer for outputting the human heart rate data corresponding to each frame of face video image. 3.The mobile platform oriented remote visual heart rate measurement deep learning method of claim 2, wherein, The calculation formula of the attention weights corresponding to all frames of face video in the image sequence is: In the formula, Y k indicates the attention weight corresponding to the kth frame of the face video image in the image sequence; X k Xk represents the feature map matrix corresponding to the kth frame of face video image in the image sequence; H k denotes a 1x1 convolution weight matrix used for generating the attention mask for the k-th face video image in the image sequence; W k denotes that the scalar weight is used to scale the attention mask for the k-th frame of the face video image; σ represents an s-type activation function, i.e., a sigmod activation function, for normalizing the attention mask to the range of [0, 1]; ω k denotes a 1 x 1 convolution; b k bias term for a 1x1 convolutional layer; L1 represents an L1 norm for normalizing the generated attention mask. 4.The mobile platform oriented remote visual heart rate measurement deep learning method of claim 1, wherein, The expression of the attention entropy of the target region is: In the formula, e h represents the attention entropy of the hth target region; h represents the index of the attention head in the multi-attention mechanism, and represents the hth target region; represents the attention score computed for the i-th grid in the h-th target region; N i represents the total number of grids. 5.The mobile platform oriented remote visual heart rate measurement deep learning method of claim 1, wherein, The step of the preset probability vector of the target region being an important target region is: When the target region is the forehead or cheek region, the probability vector of the target region being an important target region is (1.0); when the target region is the eye, nose or mouth region, the probability vector of the target region being an important target region is (0.5.0.5). When the target region is an ear region, the probability vector of the target region being an important target region is (0.1).

6. The mobile platform oriented remote visual heart rate measurement deep learning method according to claim 4, characterized in that, An expression of the importance value in the target region is: In the formula, c represents the importance value of the target region; e represents an attention entropy vector of all target regions in each frame of the face video image; a first weight vector representing a neural network model corresponding to the linear classifier; a second weight vector representing a neural network model corresponding to the linear classifier; b represents a bias vector of the neural network model corresponding to the linear classifier; p represents a preset probability vector of the target region being an important target region.

7. The mobile platform oriented remote visual heart rate measurement deep learning method according to claim 1, wherein, The step of obtaining the target region corresponding to each key point according to the key points in the grid region is: The key points in the grid region are identified by using a face key point identification algorithm; According to the key points of each identified grid region, the grid regions with the same key points are differentiated and normalized to obtain the target region corresponding to each key point. 8.The mobile platform oriented remote visual heart rate measurement deep learning method of claim 1, wherein, The video frame image is rasterized to obtain a grid region. 9.The mobile platform oriented remote visual heart rate measurement deep learning method of claim 1, wherein, The step of obtaining the human heart rate data is that when each frame of the face video image is collected, a heart rate measuring instrument is used to measure the human heart rate data.

Images