A remote heart rate monitoring algorithm based on neural network model

Abstract

The present invention belongs to the field of computer vision technology, and specifically is a remote heart rate monitoring algorithm based on a multi-stream neural network model. The present invention adopts a dual-stream Transformer neural network, which can process facial information and background information at the same time, and use the background information to guide the separation of heart rate-related signals and noise signals in the facial information, which can significantly improve the stability of remote heart rate monitoring; wherein, the multi-stream neural network is composed of two or more feature extraction networks connected in series with a result regression network; the feature extraction network is composed of two or more networks with the same structure and independent of each other; the result regression network is composed of a neural network, and the output is the predicted heart rate; the present invention calculates the heart rate through facial video analysis, while taking into account facial and background information, and uses background information to guide the suppression of non-physiological signals in facial information, thereby significantly improving the accuracy of the remote heart rate monitoring algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a remote heart rate monitoring algorithm. Background Art

[0002] The heart is one of the most vital organs in the human body, supporting the activities of almost every cell in the body. Therefore, a healthy heart is essential for good health. The latest cardiovascular disease report released by the National Center for Cardiovascular Diseases, which examined the incidence of cardiovascular disease between 1990 and 2017, shows that the mortality rate from cardiovascular disease is significantly higher than that of other diseases, and the mortality rate is showing a slowly increasing trend. In 2017, cardiovascular disease accounted for nearly half of all deaths from illness, far exceeding other causes such as cancer and respiratory diseases. Given the high rates of disability and mortality associated with cardiovascular disease, early detection, prevention, and treatment are crucial to combating it. However, currently, the prevention and detection of cardiovascular disease relies primarily on scheduled physical examinations, which not only consumes significant time and money, but also, due to the long checkup period (typically every six months or annually), cannot detect abnormal cardiovascular activity in a timely manner, limiting its effectiveness in detecting and preventing cardiovascular disease. Video-based remote heart rate monitoring offers a new solution to this problem. Simply placing an ordinary RGB camera at home allows people to monitor their heartbeat at all times, allowing them to immediately identify and seek medical attention if any abnormalities occur. However, this also poses significant challenges to the accuracy and stability of remote heart rate monitoring results. Improving the accuracy and stability of remote heart rate monitoring has become a pressing issue.

[0003] Existing remote heart rate monitoring methods mainly consist of two sub-steps: first, feature extraction is performed on the region of interest (ROI) in the facial video to obtain relatively rough features; then, a deep neural network is used to extract heart rate-related features from the rough features and regress them to obtain the heart rate. However, research has shown that existing remote heart rate monitoring still has many problems, such as only considering facial information while ignoring background information, or assuming that the relationship between background information and facial information is only a simple linear relationship. This makes it difficult for existing methods to separate heart rate-related information from facial information, which greatly affects the accuracy and stability of the algorithm. Summary of the Invention

[0004] The present invention aims to solve the above-mentioned deficiencies of the existing remote heart rate monitoring algorithms and proposes a remote heart rate monitoring algorithm with high accuracy and good stability.

[0005] The remote heart rate monitoring algorithm proposed in this paper adopts a dual-stream Transformer[8] neural network, which can process facial information and background information simultaneously, and use background information to guide the separation of heart rate-related signals and noise signals in facial information, which can significantly improve the accuracy and stability of remote heart rate monitoring. The specific steps are as follows:

[0006] (1) Obtain a video containing a face and preprocess the video:

[0007] (1.1) Use the open source face 81 key point detection algorithm

[11] to detect facial key points in the video frame by frame;

[0008] (1.2) The facial region of each frame in the video is cropped based on one of various combinations of facial key points. There are four representative cropping methods:

[0009] (1.2.1) The facial region formed by the convex hull of the 81 key points of the face or its subset;

[0010] (1.2.2) The facial area contained in the polygon formed by connecting 81 key points of the face or its subset in a specific order;

[0011] (1.2.3) The facial area formed by the largest inscribed rectangle of the 81 key points of the face or its subset;

[0012] (1.2.4) The facial area formed by the minimum bounding rectangle of the 81 key points of the face or its subset;

[0013] (1.3) Based on the facial region obtained in (1.2), a background region with a certain width and height is cropped around it. Specifically, there are four representative methods for cropping the background region:

[0014] (1.3.1) Take the background regions with a width of 0.2w and a height of h on the left and right sides of the face and concatenate them together to obtain the background region, where w and h represent the width and height of the facial region extracted in step (1.2), respectively.

[0015] (1.3.2) Take the background regions with a width of w and a height of 0.2h on the upper and lower sides of the face and concatenate them together to obtain the background region, where w and h represent the width and height of the facial region extracted in step (1.2), respectively.

[0016] (1.3.3) Select any area outside the convex hull of the 81 key points in the entire face area to form the background area;

[0017] (1.3.4) Take the background regions with a width of 0.2w and a height of h on the left and right sides of the face, and combine the background regions with the facial region with a width of w and a height of h obtained in step (1.2) into one region;

[0018] (1.4) Specify k sliding windows of different scales, and perform sliding window averaging on the pixel values ​​of the face and background areas extracted in steps (1.2) and (1.3) using the settings of each sliding window, so that the number of sliding windows of the i-th scale is m i *n i , here m i and n i The number of grids for dividing the window horizontally and vertically; the three RGB color channels are averaged by sliding windows respectively, so the number of windows of this scale is 3*m i *n i , where there needs to be an overlap between adjacent sliding windows; here, the types of windows with different resolutions k, and the number of grids m for horizontal and vertical window divisions corresponding to each resolution i and n i are all positive integers, and the grid division is adapted to the size of the face. The empirical value k ranges from 1 to 10, and m i and n i Both values ​​are between 1 and 50. There are two sliding window calculation methods:

[0019] (1.4.1) The sliding window can be defined as a single size, i.e., k = 1, and the overlap between two adjacent sliding windows is no more than 100% of the window size;

[0020] (1.4.2) Sliding windows can be defined in various sizes, i.e., k > 1, and the overlap between two adjacent sliding windows is no more than 100% of the window size;

[0021] (1.5) The dimension of all sliding averages is 3*m i *n i The facial region is expanded into a column vector, and the column vectors are spliced ​​according to the time sequence of the corresponding video frames to obtain 3m i n i dimensional time series as facial region features Feat f ; Perform the same operation on the background area to obtain the background area feature Feat b .

[0022] (2) Construct a multi-stream neural network, which consists of two or more feature extraction networks connected in series with a result regression network, where:

[0023] (2.1) The feature extraction network consists of two or more networks with the same structure and independent of each other, and at least one feature extraction network has a Feat input. f The foreground flow network, at least one feature extraction network is input as Feat b The background flow network, the foreground flow network and the background flow network are both stacked by several encoding modules, and the two are respectively based on the facial area feature Feat in step (1.5) f And the feature Feat containing the background area b As input, the output of the feature extraction network is used as the input of the result regression network; there are two specific feature extraction network structures:

[0024] (2.1.1) The encoding module of the feature extraction network is Transformer[8] and adopts sine-cosine position embedding, and the facial feature Feat in step (1.5) f Or the feature Feat containing the background area b As input;

[0025] (2.1.2) The encoding module of the feature extraction network is ResNet[9], and the facial feature Feat in step (1.5) f Or the feature Feat containing the background area b As input;

[0026] (2.2) The result regression network consists of a neural network that takes the combination of the outputs of the foreground flow network and the background flow network in step (2.1) as input, and outputs the predicted heart rate. Specifically, there are the following result regression network structures:

[0027] (2.2.1) The result regression network is composed of a multi-layer perceptron neural network [7], which takes the difference between the output of the foreground flow network and the background flow network in (2.1) as input and outputs the predicted heart rate;

[0028] (2.2.2) The result regression network is composed of a multi-layer perceptron neural network [7], which takes the concatenation of the outputs of the foreground stream network and the background stream network in step (2.1) as input and outputs the predicted heart rate;

[0029] (2.2.3) The resulting regression network consists of a radial basis function (RBF) neural network

[10] , which takes the difference between the outputs of the foreground flow network and the background flow network in step (2.1) as input and outputs the predicted heart rate;

[0030] (2.2.4) The resulting regression network consists of a radial basis function (RBF) neural network

[10] , which takes the concatenation of the outputs of the foreground flow network and the background flow network in step (2.1) as input and outputs the predicted heart rate;

[0031] (2.2.5) The background flow network uses the features of the region obtained in (1.3.4) as input. Its output is subtracted from the output of the foreground flow network to replace the output of the original background flow network and serve as one input of the result regression network. The rest is the same as (2.2.1).

[0032] Compared with existing remote heart rate monitoring methods, the present invention takes into account both facial information and background information, and uses background information to guide the separation of heart rate-related features and noise features in facial information. Therefore, cleaner heart rate-related features can be obtained, thereby greatly improving the accuracy of remote heart rate monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the overall flow chart of video preprocessing.

[0034] Figure 2 This is the distribution map of 81 key points on the face.

[0035] Figure 3 Schematic diagram of the face and background area.

[0036] Figure 4 Schematic diagram of sliding window averaging.

[0037] Figure 5 Schematic diagram of the two-stream Transformer[8]. DETAILED DESCRIPTION

[0038] Example 1

[0039] (1) Obtain a video containing a face and preprocess the video. Figure 1 The overall process of video preprocessing is to first extract the facial and background regions based on the 81 facial feature points, and then embed the facial and background regions to obtain corresponding features. Embedding refers to mapping the data from the original image sequence to a high-dimensional time series. The preprocessing algorithm steps are as follows:

[0040] (1.1) Use the existing open source algorithm to detect 81 key points of the face in each frame of the video; here, the key point detection uses the DLIB

[11] open source software library, and the distribution of the 81 key points detected is as follows Figure 2 As shown;

[0041] (1.2) The facial region is represented by a rectangle formed by the following key points of the 81 facial key points according to the following rules: (x1, y1) and (x2, y2) are used to represent the coordinates of the upper left corner and lower right corner of the facial region, where x1 = LMK(1,0), x2 = LMK(17,0), y1 = [LMK(71,1)+LMK(72,1)] / 2, y2 = LMK(9,1), LMK(k,0) and LMK(k,1) are the horizontal and vertical coordinates of the kth key point, respectively. Figure 3 The red box shown is a schematic diagram of the facial area;

[0042] (1.3) Based on the facial region obtained in (1.2), a background region with a certain width and height is cut out around it; here, background regions with a width of 0.2w and a height of h on the left and right sides of the face are cut out and spliced ​​together to obtain the background region, where w and h represent the width and height of the facial region extracted in step (1.2), respectively, as shown in Figure 3 The green box shown is a schematic diagram of the background area;

[0043] (1.4) Specify k sliding windows of different scales, and perform sliding window averaging on the pixel values ​​in the face and background areas extracted by (1.2) and (1.3) using the settings of each sliding window, so that the number of sliding windows of the i-th scale is m i *n i , the three color channels of RGB are averaged by sliding windows respectively, so the number of windows of this scale is 3*m i *n i , where there needs to be an overlap between adjacent sliding windows; here, the number of windows is set to {25,81,169}, and the overlap of adjacent sliding windows is 1 / 2 of the window size Figure 4 Shown is a schematic diagram of sliding window averaging;

[0044] (1.5) The dimension of all sliding averages is 3*m i *n i The facial region is expanded into a column vector, and the column vectors are spliced ​​according to the time sequence of the corresponding video frames to obtain 3m i n i dimensional time series as facial region features Feat f , which is the embedding process described in step (1); perform the same operation on the background area to obtain the background area feature Feat b .

[0045] (2) Construct a two-stream neural network, which consists of two feature extraction networks connected in series with a result regression network, where:

[0046] (2.1) The 1-way feature extraction network is input as Feat f The foreground flow network, the other feature extraction network is input as Feat b The background flow network, the foreground flow network and the background flow network are both stacked by several encoding modules, and the two are respectively based on the facial area features Feat obtained by preprocessing in (1.5) f and background area features Feat b As input, the output of the network is used as the input of the result regression network; Here, the feature extraction network consists of a series of M units (cells), each cell consists of two parallel Transformer[8] encoding modules, namely the face encoding module and the background encoding module. The face encoding modules in all cells together constitute the foreground flow network, and all background encoding modules together constitute the background flow network, and both use sine-cosine position encoding;

[0047] (2.2) The result regression network consists of a three-layer perceptron neural network [7], which takes the difference between the output of the foreground flow network and the background flow network in (2.1) as input and outputs the predicted heart rate; Figure 5 Schematic diagram of the two-stream neural network corresponding to (2.1) and (2.2).

[0048] The experimental comparison between this embodiment and the existing method is shown in Table 1 below, where MAE is the mean absolute error, RMSE is the root mean square error, and Std is the error variance.

[0049] On the MAHNOB-HCI dataset

[12] , the value of k in (1.5) is 3, corresponding to 3*m i *n i They are {(3*5*5), (3*9*9), (3*13*13)}, (2.1) the number of cells M is 3, (2.1) the number of neurons in the three-layer perceptron [7] are 77, 256, and 1 respectively. During the offline training process, 80% of the data (421 videos) in the randomly sampled data set are used as the training set, the loss function adopts L1 loss, the optimizer adopts Adam [1], the learning rate is set to 0.0001, and the training rounds are 80; in the test phase, the remaining 20% ​​of the data (106 videos) are selected as the test set. The root mean square error of the final test result is 4.83, which is reduced by (5.10-4.83) / 5.10=5.3% compared with the existing best method AutoHR [6].

[0050] On the VIPL-HR dataset

[13] , the value of k in (1.5) is 3, corresponding to 3*m i *n iThey are {(3*5*5), (3*9*9), (3*13*13)}, (2.1) the number of cells M is 4, (2.2) the number of neurons in the three-layer perceptron [7] are 77, 256, and 1 respectively. During the offline training process, 80% of the data (1902 videos) in the randomly sampled data set are used as the training set, the loss function adopts L1 loss, the optimizer adopts Adam [1], the learning rate is set to 0.00005, and the training rounds are 120; in the test phase, the remaining 20% ​​of the data (476 videos) are selected as the test set. The root mean square error of the final test result is 7.31, which is (7.97-7.31) / 7.97=8.3% lower than the existing best method CVDNet [5].

[0051] Example 2

[0052] (1) Same as Example 1;

[0053] (1.1) Same as Example 1;

[0054] (1.2) Same as Example 1;

[0055] (1.3) Based on the facial region obtained in (1.2), a background region with a certain width and height is cut out around it. Here, the background regions with a width of 0.2w and a height of h on the left and right sides of the face are cut out and concatenated with the facial region with a width of w and a height of h obtained in (1.2) to obtain the region containing the background, as shown in the following example: Figure 3 The rectangular area formed by the union of the green box and the red box shown;

[0056] (1.4) Same as Example 1;

[0057] (1.5)Feat f The calculation of Feat is the same as in Example 1. b The calculation is performed on the background area obtained in (1.3) and the Feat f The same operation is performed.

[0058] (2) Same as Example 1;

[0059] (2.1) Same as Example 1;

[0060] (2.2) The features of the region obtained in (1.3) are used as input to the background flow network. The output of the background flow network is subtracted from the output of the foreground flow network to replace the output of the original background flow network and serve as one input of the result regression network. The rest is the same as in Example 1.

[0061] The experimental comparison between this embodiment and the existing method is shown in Table 1 below, where MAE is the mean absolute error, RMSE is the root mean square error, and Std is the error variance.

[0062] On the MAHNOB-HCI dataset

[12] , the value of k in (1.5) is 3, corresponding to 3*m i *n i They are {(3*5*5), (3*9*9), (3*13*13)}, (2.1) the number of cells M is 3, (2.1) the number of neurons in the three-layer perceptron [7] are 77, 256, and 1 respectively; during the offline training process, 80% of the data (421 videos) in the randomly sampled data set are used as the training set, the loss function adopts L1 loss, the optimizer adopts Adam [1], the learning rate is set to 0.0001, and the training rounds are 80; in the test phase, the remaining 20% ​​of the data (106 videos) are selected as the test set, and the root mean square error of the final test result is 4.86, which is reduced by (5.10-4.86) / 5.10=4.7% compared with the existing best method AutoHR [6].

[0063] On the VIPL-HR dataset

[13] , the value of k in (1.5) is 3, corresponding to 3*m i *n i They are {(3*5*5), (3*9*9), (3*13*13)}, (2.1) the number of cells M is 4, (2.2) the number of neurons in the three-layer perceptron [7] are 77, 256, and 1 respectively; during the offline training process, 80% of the data (1902 videos) in the randomly sampled data set are used as the training set, the loss function adopts L1 loss, the optimizer adopts Adam [1], the learning rate is set to 0.00005, and the training rounds are 120; in the test phase, the remaining 20% ​​of the data (476 videos) are selected as the test set, and the root mean square error of the final test result is 7.31, which is (7.97-7.24) / 7.97=9.1% lower than the existing best method CVDNet [5].

[0064] Table 1: Experimental results

[0065]

[0066]

[0067] References

[0068] [1]KINGMA D P,BA J.Adam:A method for stochastic optimization[C] / / Proceedings of the International Conference for Learning Representations.[S.l.:s.n.],2015:245-252.

[0069] [2]DE HAAN G,JEANNE V.Robust pulse rate from chrominance-based rppg[J].IEEE Transactions on Biomedical Engineering,2013,60(10):2878-2886.

[0070] [3]LYAKOV S,ALAMEDA-PINEDA X,RICCI E,et al.Self-adaptive matrixcompletion for heart rate estimation from face videos under realisticconditions[C] / / Proceedings of the IEEE conference on computer vision andpattern recognition.[S.l.:s.n.],2016:2396-2404.

[0071] [4]CHEN W,MCDUFF D.Deepphys:Video-based physiological measurementusing convolutional attention networks[C] / / Proceedings of the EuropeanConference on Computer Vision.[S.l.:s.n.],2018:349-365.

[0072] [5]NIU X,YU Z,HAN H,et al.Video-based remote physiologicalmeasurement via cross-verified feature disentangling[C] / / Proceedings of theEuropean Conference on Computer Vision.[S.l.:s.n.],2020:295-310.

[0073] [6]YU Z,LI X,NIU X,et al.Autohr:A strong end-to-end baseline forremote heart rate measurement with neural searching[J].IEEE Signal ProcessingLetters,2020,27:1245-1249.

[0074] [7]Rosenblatt F.Principles of neurodynamics.perceptrons and thetheory of brain mechanisms[R].Cornell Aeronautical Lab Inc Buffalo NY,1961.

[0075] [8]VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C] / / Proceedings of the Advances in neural information processing systems.[S.l.:s.n.],2017:5998-6008.

[0076] [9]HE K,ZHANG X,REN S,et al.Deep residual learning for imagerecognition[C] / / Proceedings of the IEEE conference on computer vision andpattern recognition.[S.l.:s.n.],2016:770-778

[0077]

[10] Vt S E,Shin Y C.Radial basis function neural network forapproximation and estimation of nonlinear stochastic dynamic systems[J].IEEEtransactions on neural networks,1994,5(4):594-603.

[0078]

[11] King D E.Dlib-ml:A machine learning toolkit[J].The Journal ofMachine Learning Research,2009,10:1755-1758.

[0079]

[12] Soleymani M,Lichtenauer J,Pun T,et al.A multimodal database foraffect recognition and implicit tagging[J].IEEE transactions on affectivecomputing,2011,3(1):42-55.

[0080]

[13] Niu X,Han H,Shan S,et al.VIPL-HR:A multi-modal database for pulseestimation from less-constrained face video[C] / / Asian Conference on ComputerVision.Springer,Cham,2018:562-576。

Claims

1. A remote heart rate monitoring method based on a multi-stream neural network model, characterized in that: A dual-stream Transformer neural network is used to simultaneously process facial and background information. The background information is used to guide the separation of heart rate-related signals and noise signals in facial information, significantly improving the accuracy and stability of remote heart rate monitoring. The specific steps are as follows: (1) Obtain a video containing a face and preprocess the video: (1.1) Use the open source face 81 key point detection algorithm to detect facial key points in the video frame by frame; (1.2) cropping the facial region of each frame in the video based on one of various combinations of facial key points; (1.3) Based on the facial region obtained in step (1.2), a region containing the background with a certain width and height is cut out around it; (1.4) Specify k sliding windows of different scales, and perform sliding window averaging on the pixel values ​​of the face and background areas extracted in steps (1.2) and (1.3) using the settings of each sliding window, so that the number of sliding windows of the i-th scale is m i *n i , here m i and n i Defined as the number of grids used to divide the window horizontally and vertically. The three color channels of RGB are averaged by sliding windows respectively. The number of windows of this scale is 3*m i *n i , where there needs to be an overlap between adjacent sliding windows; (1.5) The dimension of all sliding averages is 3*m i *n i The facial region is expanded into a column vector, and the column vectors are spliced ​​according to the time sequence of the corresponding video frames to obtain 3m i n i dimensional time series as facial region features Feat f ; Perform the same operation on the background area to obtain the feature Feat of the background area b ; (2) Construct a multi-stream neural network, which consists of two or more feature extraction networks connected in series with a result regression network; (2.1) The feature extraction network consists of two or more networks with the same structure and independent of each other, and at least one feature extraction network has a Feat input. f The foreground flow network, at least one feature extraction network is input as Feat b The background flow network, the foreground flow network and the background flow network are both stacked by several encoding modules, and the two are respectively based on the facial area feature Feat in step (1.5) f And the feature Feat containing the background area b As input, the output of the feature extraction network is used as the input of the result regression network; (2.2) The resulting regression network consists of a neural network that takes the combination of the outputs of the foreground flow network and the background flow network in step (2.1) as input, and the output is the predicted heart rate.

2. The remote heart rate monitoring method based on the multi-stream neural network model according to claim 1 is characterized in that: In step (1.2), the facial region of each frame in the video is cropped according to one of the various combinations of facial key points. The specific cropping method is one of the following: (1.2.1) The facial region formed by the convex hull of the 81 key points of the face or its subset; (1.2.2) The facial area contained in the polygon formed by connecting 81 key points of the face or its subset in a specific order; (1.2.3) The facial area formed by the largest inscribed rectangle of the 81 key points of the face or its subset; (1.2.4) The facial area formed by the minimum circumscribed rectangle of the 81 key points of the face or its subset.

3. The remote heart rate monitoring method based on the multi-stream neural network model according to claim 2 is characterized in that: In step (1.3), the facial region obtained in step (1.2) is clipped to obtain a background region with a certain width and height around it. The specific method for clipping the background region is one of the following: (1.3.1) Take the background regions with a width of 0.2w and a height of h on the left and right sides of the face and concatenate them together to obtain the background region, where w and h represent the width and height of the facial region extracted in step (1.2), respectively. (1.3.2) Take the background regions with a width of w and a height of 0.2h on the upper and lower sides of the face and concatenate them together to obtain the background region, where w and h represent the width and height of the facial region extracted in step (1.2), respectively. (1.3.3) Select any area outside the convex hull of the 81 key points in the entire face area to form the background area; (1.3.4) Take the background areas with a width of 0.2w and a height of h on the left and right sides of the face, and combine the background areas on the left and right sides of the face with the facial area with a width of w and a height of h obtained in step (1.2) into one area.

4. The remote heart rate monitoring method based on the multi-stream neural network model according to claim 3 is characterized in that: The sliding window calculation method described in step (1.4) is one of the following: (1.4.1) The sliding window is defined as a single size, i.e., k = 1, and the overlap between two adjacent sliding windows is no more than 100% of the window size; (1.4.2) Sliding windows are defined as multiple sizes, i.e., k>1, and the overlap between two adjacent sliding windows is no more than 100% of the window size.

5. The remote heart rate monitoring method based on the multi-stream neural network model according to claim 4 is characterized in that: The feature extraction network described in step (2.1) is one of the following: (2.1.1) The encoding module of the feature extraction network is Transformer and uses sine-cosine position embedding, and the facial feature Feat in step (1.5) f Or the feature Feat containing the background area b As input; (2.1.2) The encoding module of the feature extraction network is ResNet, and the facial feature Feat in step (1.5) f Or the feature Feat containing the background area b as input.

6. The remote heart rate monitoring method based on the multi-stream neural network model according to claim 5 is characterized in that: The resulting regression network structure described in step (2.2) is one of the following: (2.2.1) The result regression network consists of a multi-layer perceptron neural network, which takes the difference between the output of the foreground flow network and the background flow network in step (2.1) as input and outputs the predicted heart rate; (2.2.2) The result regression network consists of a multi-layer perceptron neural network, which takes the concatenation of the outputs of the foreground stream network and the background stream network in step (2.1) as input and outputs the predicted heart rate; (2.2.3) The resulting regression network consists of a radial basis function (RBF) neural network, which takes the difference between the outputs of the foreground flow network and the background flow network in step (2.1) as input and outputs the predicted heart rate; (2.2.4) The resulting regression network consists of a radial basis function (RBF) neural network, which takes the concatenation of the outputs of the foreground flow network and the background flow network in step (2.1) as input and outputs the predicted heart rate; (2.2.5) The background flow network uses the features of the region obtained in (1.3.4) as input. Its output is subtracted from the output of the foreground flow network to replace the output of the original background flow network and serve as one input of the result regression network. The rest is the same as (2.2.1).

7. The remote heart rate monitoring method based on a multi-stream neural network model according to any one of claims 3 to 6, characterized in that: The background flow network uses the features of the region obtained in (1.3.4) as input, and its output minus the output of the foreground flow network is used to replace the output of the original background flow network and serve as one input of the result regression network.

Images