Training method of physiological feature extraction model and remote heart rate measurement method

Abstract

Disclosed are a training method for a physiological feature extraction model and a remote heart rate measurement method, the training method comprising: obtaining a video containing a facial region and a label as heart rate waveform data; extracting the facial region from the video; separating non-physiological features through a first encoder; separating physiological features through a second encoder; encoding the label through a third encoder to obtain a third feature for characterizing the heart rate waveform data; inputting a first concatenated feature obtained by concatenating the non-physiological feature with the physiological feature and a second concatenated feature obtained by concatenating the non-physiological feature with the third feature into a generator of a generative adversarial network to generate a pseudo-facial region, and inputting the facial region and the pseudo-facial region into a discriminator of the generative adversarial network to discriminate the pseudo-facial region; and jointly training the first encoder, the second encoder, the third encoder, the generator, and the discriminator until the discriminator cannot distinguish them.

Description

Technical Field

[0001] The present disclosure relates to the field of remote heart rate measurement. Specifically, the present disclosure provides a training method for a physiological feature extraction model, a remote heart rate method, a computer-readable storage medium and a system. Background Art

[0002] Remote photoplethysmography (rPPG) is a technology that uses a camera to capture changes in light absorption by skin tissue to indirectly measure heart rate. The specific principle of rPPG is that each contraction and relaxation of the heart causes changes in the hemoglobin concentration in the blood, which changes the amount of light absorbed, and thus causes changes in facial skin color. This subtle change in skin color can be captured by high-resolution imaging devices such as cameras, computers, or mobile phone cameras, and the human heartbeat cycle can be intuitively detected through signal processing and other methods.

[0003] In recent years, deep learning methods based on rPPG have been widely studied in heart rate measurement tasks. Deep learning methods can automatically learn heart rate-related feature representations from a large amount of data without being restricted by various assumptions. rPPG signal extraction usually requires a large number of labeled training data sets (i.e., gold standard physiological signals), in which each sample needs to be manually annotated with heart rate information. Previous deep learning-based rPPG heart rate estimation methods heavily rely on data sets including gold standard (GroundTruth, GT) physiological signals, and obtaining gold standard physiological signals usually requires measurement using contact sensors and synchronization with facial videos. During the data collection process, problems such as gold standard signal loss or misalignment with facial videos often occur. In addition, for existing unsupervised methods, due to the lack of learning models to address the impact of data changes in real environments, it is difficult to obtain robust results in practical applications. Summary of the invention

[0004] This disclosure is provided to introduce selected concepts in a simplified form, and these concepts are further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In view of the problems existing in the prior art, the present invention discloses a rPPG heart rate measurement method based on weakly supervised learning. The weakly supervised learning method can use a large amount of unlabeled data for learning when the labeled data is limited. That is, on the basis of making full use of the existing high-quality labeled data, pseudo-labeled data is generated or directly combined with a large amount of unlabeled data that is easy to collect. On the one hand, the data volume is enriched and the data scale is expanded; on the other hand, the model performance is improved. This can significantly reduce the demand for labeled data, reduce the burden and cost of data labeling, and at the same time improve the generalization ability of the model.

[0006] One aspect of the present disclosure provides a training method for a physiological feature extraction model, the training method comprising: step A1, obtaining a video containing a facial region and a label corresponding to the video as heart rate waveform data; step A2, extracting the facial region from the video; step A3, separating a first feature that is unrelated to heart rate information from the facial region through a first encoder; step A4, separating a second feature that only includes characterizing heart rate information from the facial region through a second encoder; step A5, encoding the label through a third encoder to obtain a third feature, the third feature being a feature used to characterize heart rate waveform data; step A6, concatenating the first feature with the second feature to obtain a first concatenated feature, and concatenating the first feature with the third feature to obtain a second concatenated feature; step A7, inputting the first concatenated feature and the second concatenated feature into a generator of a generative adversarial network to generate a pseudo-facial region, and using the facial region and the pseudo-facial region as inputs of a discriminator of the generative adversarial network to discriminate the pseudo-facial region through the discriminator; and step A8, jointly training the first encoder, the second encoder, the third encoder, the generator, and the discriminator until the discriminator cannot distinguish between the facial region and the pseudo-facial region.

[0007] Optionally, the step of jointly training the first encoder, the second encoder, the third encoder, the generator and the discriminator may include: step B1, fixing the first encoder, the second encoder, the third encoder and the generator, and iteratively executing steps A2 to A7 to generate the pseudo-facial region required for training to train the discriminator, so that the probability that the discriminator determines the facial region as true and the probability that the pseudo-facial region is determined as false is maximized; step B2, fixing the discriminator, iteratively executing steps A2 to A7 to generate the pseudo-facial region required for training, and training the first encoder, the second encoder, the third encoder and the generator through back propagation, so that the probability that the discriminator determines the pseudo-facial region as true is maximized; iteratively executing steps B1 and B2 until the discriminator cannot distinguish between the facial region and the pseudo-facial region.

[0008] Optionally, the training method may also include: step A2-1, performing channel separation on the facial area to obtain a red channel, a blue channel and a green channel, wherein the first feature in step A3 is separated from the red channel and the blue channel by the first encoder, and the second feature in step A4 is separated from the green channel by the second encoder.

[0009] Optionally, at least a portion of the video is an unlabeled video without a corresponding label, and step A1 may also include: using a chromaticity-based method or a principal component analysis method to perform heart rate estimation on the unlabeled video to obtain an estimated rPPG signal, and using the estimated remote photoplethysmography signal as a label for the unlabeled video.

[0010] Another aspect of the present disclosure provides a remote heart rate measurement method, which includes: collecting a video to be measured including a facial area; extracting the facial area from the video to be measured; separating physiological features that only include reaction heart rate information from the facial area through a second encoder of a physiological feature extraction model; and obtaining estimated heart rate waveform data based on the physiological features through an estimator composed of a convolutional neural network, wherein the physiological feature extraction model is trained by the training method as described above.

[0011] Optionally, the estimator may include an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and an output layer.

[0012] Optionally, the remote heart rate measurement method may also include training the estimator through the following steps: obtaining a second video including a facial area and a second label corresponding to the second video as heart rate waveform data; extracting the facial area from the second video; inputting the facial area into a second encoder of a trained physiological feature extraction model to obtain physiological features; providing the physiological features to an input layer, and minimizing the difference between the output of the output layer and the second label to train the estimator.

[0013] Optionally, at least a portion of the second video is an unlabeled video without a corresponding second label, and the remote heart rate measurement method may further include: using a chromaticity-based method or a principal component analysis method to perform heart rate estimation on the unlabeled video to obtain an estimated rPPG signal, and using the estimated remote photoplethysmography signal as a label for the unlabeled video.

[0014] Another aspect of the present disclosure provides a computer-readable storage medium storing instructions, which, when executed by at least one computing device, prompts the at least one computing device to perform the remote heart rate measurement method as described above or the remote heart rate measurement method as described above.

[0015] Another aspect of the present disclosure provides a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the remote heart rate measurement method as described above or the remote heart rate measurement method as described above.

[0016] In response to the problems existing in the prior art, the present disclosure provides an rPPG heart rate measurement method based on weakly supervised learning, which avoids the dependence on gold standard physiological signals by separating and extracting physiological features and non-physiological features, while improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an application scenario diagram for implementing the remote heart rate measurement method according to an embodiment of the present disclosure;

[0018] Figure 2 is a schematic diagram of a training method for a physiological feature extraction model according to an embodiment of the present disclosure;

[0019] Figure 3 is a flow chart of a remote heart rate measurement method according to an embodiment of the present disclosure;

[0020] Figure 4 is a schematic diagram of training an rPPG estimator according to an embodiment of the present disclosure; and

[0021] Figure 5 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent to those of ordinary skill in the art. For example, the order of operations described herein is merely an example and is not limited to the order set forth herein, but changes that will be apparent to those of ordinary skill in the art may be made in addition to the operations that must be performed in a particular order. In addition, in order to improve clarity and brevity, descriptions of features and structures that will be known to those of ordinary skill in the art may be omitted. The features described herein may be implemented in different forms and will not be construed as being limited to the examples described herein. More specifically, the examples described herein have been provided so that the present invention will be thorough and complete, and will fully convey the scope of the present invention to those of ordinary skill in the art.

[0023] Figure 1 is an application scenario diagram for implementing the remote heart rate measurement method according to an embodiment of the present disclosure.

[0024] Remote photoplethysmography (rPPG) signals, which are heart rate waveform data, usually appear as a periodic waveform, which is similar to the pulse waveform in an electrocardiogram, except that it is extracted from the color changes on the skin surface through a non-contact video monitoring method. Specifically: The waveform of the rPPG signal usually appears as a periodically changing curve (i.e., a one-dimensional signal), and the interval between the peak and the trough corresponds to the rhythm of the heartbeat. Each waveform cycle includes a rising segment, a peak, and a falling segment, representing the process of one contraction and relaxation of the heart. The peak part is usually sharp, corresponding to the rapid change of skin color caused by blood flow during heart contraction. The trough part is relatively gentle, representing the relative static state during heart relaxation. The period of the rPPG signal is directly related to the heart rate, that is, the faster the heart rate, the shorter the waveform period; the slower the heart rate, the longer the waveform period. By analyzing the waveform period, the number of heartbeats per minute (heart rate) can be inferred. The fluctuation amplitude of the rPPG signal is affected by many factors, including lighting conditions, skin reflectivity, and blood flow. A larger amplitude usually means a better signal with less noise interference. If the fluctuation amplitude is small or the signal contains more noise, it may be difficult to extract the heart rate information.

[0025] In the scenario of remote heart rate measurement, individuals can be in front of the camera in various postures. In this case, the method of using video for heart rate detection faces many challenges. For example, the rPPG signal may contain noise introduced by head movement, ambient light changes or other external factors. These noises usually appear as irregular waveform changes or additional fluctuations. In the prior art, the chrominance-based method (CHROM) mainly extracts heart rate-related signals directly from the video by analyzing the chrominance changes of the RGB channels. This method relies on the change of chromaticity signals to estimate the heart rate and is generally suitable for scenes with relatively stable lighting conditions. The principal component analysis method (PCA) extracts the main signal components from the video through statistical analysis for heart rate estimation. The PCA method relies on linear transformation to reduce the data dimension and extract the principal components related to the heart rate. It can be seen that there is no good solution in the prior art for the problem of unstable detection caused by changes in illumination, angle and background. As a basic guarantee for obtaining reliable measurement results, it is necessary to separate physiological signals (such as information related to pulse and heart rate) and non-physiological signals (such as hair, eyes, background and other information unrelated to pulse and heart rate) from the input.

[0026] Figure 2 It is a schematic diagram of a training method of a physiological feature extraction model according to an embodiment of the present disclosure.

[0027] Reference Figure 2 , the physiological feature extraction model includes a first encoder E1, a second encoder E2, a third encoder E3 and a generative adversarial network GAN.

[0028] The first encoder E1 is configured to separate a first feature f1 unrelated to the heart rate information from the image of the face region Ft input thereto. As an example, the first feature f1 may include ambient lighting, hair, eyes, background noise, and the like.

[0029] The second encoder E2 is configured to separate the second feature f2 including only the heart rate information from the image of the face region Ft input thereto. As an example, the second feature f2 may include a signal related to blood pulsation extracted from facial skin color changes, such as a slight change in skin color.

[0030] The third encoder E3 is configured to extract features from the rPPG signal input thereto. The third feature f3 is some high-dimensional features related to the heart rate waveform data extracted from the one-dimensional signal of the rPPG signal.

[0031] In an optional embodiment, the first feature f1, the second feature f2 and the third feature f3 may be in the form of a matrix or a vector. The second feature f2 and the third feature f3 may have the same or similar dimensions.

[0032] In the physiological feature extraction model, the generative adversarial network GAN consists of two parts: the generator (Generator) G and the discriminator (Discriminator) D.

[0033] In an embodiment, the task of the generator G is to generate a fake face region video based on the features f1, f2 and f3 extracted from the encoders E1, E2 and E3. The goal of the generator is to generate increasingly realistic fake face region videos so that the discriminator cannot distinguish them from real face regions. To this end, the generator maximizes the probability that the fake face region is judged as "real" by the discriminator by minimizing the loss function.

[0034] On the other hand, the task of the discriminator D is to distinguish whether the input video is a real face region Ft or a fake face region generated by the generator. The goal of the discriminator is to maximize the probability of outputting “true” for real face regions and to maximize the probability of outputting “false” for fake face regions.

[0035] Through the process of adversarial training of the generative adversarial network GAN, the pseudo-face area generated by the generator becomes more and more realistic, and the discrimination ability of the discriminator is continuously enhanced. At the same time, the first encoder E1, the second encoder E2 and the third encoder E3 are trained through back propagation, so that the feature extraction ability of the first encoder E1, the second encoder E2 and the third encoder E3 is optimized. Finally, the whole heart rate measurement model can maintain a high accuracy in complex environments.

[0036] In addition, although not shown, the physiological feature extraction model may also include a facial region extractor and a channel separator.

[0037] In an embodiment, a facial region extractor accurately locates and extracts a facial region Ft from an input video frame, thereby providing accurate facial region information for subsequent feature extraction and heart rate measurement. The facial region extractor separates the facial region from a complex background by detecting key feature points of the face or using a machine learning algorithm. As an example, computer vision and image processing algorithms can be used to perform facial region extraction. For example, Haar feature cascade classifier, HOG-based facial detection, YOLO algorithm, and facial key point detection based on Dlib, etc.

[0038] The channel separator separates the RGB channels of the image of the extracted facial area Ft. Different from the aforementioned embodiment in which the image of the facial area Ft is directly input into the first encoder E1 and the second encoder E2, in the embodiment including the channel separator, the separated red channel R, green channel G and blue channel B are input into different encoders for processing. Specifically, the red channel R and the blue channel B are input into the first encoder E1 to extract the first feature f1 that is not related to the heart rate. The green channel G is input into the second encoder E2 to extract the second feature f2 that represents the heart rate information. Studies have shown that the green channel G is most sensitive in capturing skin blood pulsation signals.

[0039] In an embodiment, the physiological feature extraction model may be trained by the following steps:

[0040] Step A1: Obtain a video containing a facial region Ft and a label corresponding to the video as heart rate waveform data.

[0041] In an embodiment, data collection can be achieved through the following steps: First, the data collection device generally includes a high-resolution camera and a heart rate measurement device connected thereto (such as a photoplethysmogram (PPG sensor, oximeter, ear-clip heart rate sensor), electrocardiograph, etc.). The camera is used to capture continuous video frames containing the facial area, while the heart rate measurement device synchronously records the heart rate waveform data corresponding to the video timestamp. During the video acquisition process, the heart rate measurement device is simultaneously used to directly measure the heart rate waveform data of the subject. The signal corresponds to the acquired video sequence on the time axis to form a synchronized heart rate waveform data sequence. These signals can be used as heart rate waveform data labels corresponding to the video. The acquired heart rate waveform data is matched with the corresponding video frame through timestamps or other synchronization techniques to generate a data set. In such a data set, each video has a corresponding heart rate waveform data label.

[0042] In an embodiment, at least a portion of the video is an unlabeled video without a corresponding heart rate waveform data label. In such an embodiment, step A1 may further include: using a chromaticity-based method or a principal component analysis method to perform heart rate estimation on the unlabeled video to obtain an estimated rPPG signal, and using the estimated rPPG signal as a label for the unlabeled video.

[0043] In the prior art, during the data collection process, problems such as the loss of the gold standard signal or misalignment with the facial video often occur. In addition, for existing unsupervised methods, it is difficult to obtain robust results in practical applications due to the lack of a learning model to address the impact of data changes in the real environment. In contrast, the physiological feature extraction model provided by the present disclosure uses a weakly supervised learning method to learn from a large amount of unlabeled data when labeled data is limited. That is, on the basis of making full use of existing high-quality labeled data, pseudo-label data is generated or directly combined with a large amount of unlabeled data that is easy to collect, which enriches the amount of data and expands the data scale on the one hand; on the other hand, it improves model performance. This can significantly reduce the need for labeled data, reduce the burden and cost of data labeling, and at the same time improve the generalization ability of the model.

[0044] Step A2: extracting the facial region Ft from the video. The facial region Ft can be extracted using the facial region extractor described above.

[0045] Step A2-1, perform channel separation on the face region Ft to obtain a red channel R, a blue channel B, and a green channel G. The channel separation step may use the channel separator described above. In addition, in an embodiment that does not include a channel separator, step A2-1 may be omitted.

[0046] Step A3: Separate the first feature f1 irrelevant to the heart rate information from the facial region Ft by the first encoder E1. In addition, in an embodiment including a channel separator, separate the first feature f1 irrelevant to the heart rate information from the red channel R and the blue channel B of the facial region Ft by the first encoder E1.

[0047] Step A4: Separate the second feature f2 including only the heart rate information from the facial region Ft by the second encoder E2. In addition, in an embodiment including a channel separator, separate the second feature f2 including only the heart rate information from the green channel G of the facial region by the second encoder E2.

[0048] Step A5: Encode the label (rPPG signal) through the third encoder E3 to obtain a third feature f3, where the third feature f3 is a feature used to characterize the heart rate waveform data.

[0049] Step A6: splice the first feature f1 and the second feature f2 to obtain a first spliced ​​feature c1, and splice the first feature f1 and the third feature f3 to obtain a second spliced ​​feature c2.

[0050] Step A7: Input the first stitching feature c1 and the second stitching feature c2 to the generator G of the generative adversarial network GAN to generate a pseudo face region, and use the face region Ft and the pseudo face region as inputs of the discriminator D of the generative adversarial network to discriminate the pseudo face region through the discriminator D. In the embodiment, the first stitching feature c1 is generated by the generator G to generate a first pseudo face region, and the second stitching feature c2 is generated by the generator G to generate a second pseudo face region.

[0051] Step A8, jointly train the first encoder E1, the second encoder E2, the third encoder E3, the generator G and the discriminator D until the discriminator D cannot distinguish between the facial region Ft and the pseudo-facial region.

[0052] In an embodiment, the step of joint training may be performed in the following manner:

[0053] Step B1, fix the first encoder E1, the second encoder E2, the third encoder E3 and the generator G, and iteratively execute steps A2 to A7 to generate the pseudo-face area required for training to train the discriminator D, so that the probability that the discriminator D judges the face area Ft as true and the probability that the pseudo-face area is judged as false are maximized.

[0054] Step B2, fix the discriminator D, generate the pseudo facial area required for training by iteratively executing steps A2 to A7, and train the first encoder E1, the second encoder E2, the third encoder E3 and the generator G by back propagation, so that the probability that the discriminator D judges the pseudo facial area as true is maximized.

[0055] Step B1 and step B2 are iteratively performed until the discriminator D cannot distinguish the face region Ft from the pseudo-face region.

[0056] In an embodiment, the first encoder E1, the second encoder E2 and the third encoder E3 may be convolutional neural networks. The loss function of the generator G is a negative log-likelihood loss function, and the loss function of the discriminator D is a binary cross entropy loss function.

[0057] During the training process, the generator generates more and more realistic face videos. At the same time, the loss function guides the third feature f3 generated by the third encoder E3 to extract physiological features. The generator G is used to generate face videos, which also constrains the first encoder E1 to only extract non-physiological features. Similarly, the second encoder E2 is also constrained to only extract physiological features. In other words, the trained second encoder E2 can extract the physiological features that best represent the heart rate information in the face video, thereby eliminating the interference of non-physiological signals (such as hair, eyes, background, and other information that is not related to pulse and heart rate).

[0058] Figure 3 is a flow chart of a remote heart rate measurement method according to an embodiment of the present disclosure.

[0059] Reference Figure 3 According to the embodiment, the remote heart rate measurement method performs the following steps:

[0060] In step S301, a video including a face region is collected. In an embodiment, the device for collecting the video including the face region generally includes a high-resolution camera to capture continuous video frames including the face region.

[0061] In step S303, a facial region is extracted from the video to be tested. In an embodiment, the step of extracting the facial region from the video to be tested may adopt the facial region extractor described above, and repeated description is omitted here.

[0062] In step S305, the second encoder E2 of the physiological feature extraction model is used to separate the physiological features including only the reactive heart rate information from the facial region. Figure 2 In the embodiment including the channel separator, the facial region is firstly channel-separated to obtain a green channel, and the green channel is input into the second encoder E2.

[0063] In step S307, based on the physiological characteristics, an estimator composed of a convolutional neural network (see Figure 4 rPPG estimator in ) to obtain estimated heart rate waveform data.

[0064] In the remote heart rate measurement method disclosed in the present invention, facial area detection and RGB channel separation are first performed on the collected video, and then the physiological features are extracted by the second encoder E2 of the trained physiological feature extraction model, so that the separated physiological features are more reliable, thereby eliminating the interference of non-physiological features. Finally, accurate estimated heart rate waveform data can be further obtained. This method solves the problem of detection instability caused by changes in illumination, angle and background, and improves the accuracy and generalization ability of remote heart rate measurement. Compared with the prior art, the remote heart rate measurement method disclosed in the present invention has better usability in actual application scenarios, and can achieve accurate heart rate measurement in a variety of complex environments.

[0065] Figure 4 is a schematic diagram of training an rPPG estimator according to an embodiment of the present disclosure.

[0066] Reference Figure 4 , we will describe how to use a small amount of labeled face videos to train the rPPG estimator to obtain a more accurate rPPG signal.

[0067] In an embodiment, the rPPG estimator includes an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and an output layer.

[0068] In an embodiment, the rPPG estimator is trained by the following steps:

[0069] Step C1, obtaining a second video including a facial region and a second label corresponding to the second video as heart rate waveform data. Here, the second video can be the same as or different from the video used in step A1 described above, as long as the training of the estimator can be achieved, and the scope of the present disclosure is not limited thereto.

[0070] In an embodiment, at least a portion of the second video is an unlabeled video without the corresponding second label, and wherein the remote heart rate measurement method further comprises: estimating the heart rate of the unlabeled video using a chromaticity-based method or a principal component analysis method to obtain an estimated remote photoplethysmography signal, and using the estimated rPPG signal as a label of the unlabeled video. This step is the same or similar to the step described above with reference to step A1, and repeated description is omitted here.

[0071] Through the above steps, the rPPG estimator can not only be effectively trained on labeled datasets, but also further improve its performance through weakly supervised learning on unlabeled datasets, thereby achieving accurate measurement and estimation of heart rate, which is suitable for remote heart rate detection in a variety of complex environments.

[0072] Step C2: extracting the facial region from the second video. The facial region can be extracted using the facial region extractor described above, and repeated descriptions are omitted here.

[0073] Step C3: Input the facial region to the second encoder E2 of the trained physiological feature extraction model to obtain physiological features. In addition, in an embodiment including a channel separator, the second encoder E2 separates the physiological features including only the heart rate information from the green channel of the facial region.

[0074] Step C4, providing the physiological features to the input layer, and minimizing the difference between the output of the output layer and the second label to train the rPPG estimator.

[0075] Figure 5 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0076] Figure 5 The electronic device 500 shown includes at least one memory 501 and at least one processor 502. The memory 501 stores a computer program. When the computer program is executed by the processor 502, it prompts the processor to implement the training method of the physiological feature extraction model and the remote heart rate measurement method as described above.

[0077] Figure 2 and Figure 4 Each unit shown in the figure may be configured as software, hardware, firmware, or any combination of the above items to perform a specific function. For example, each unit may correspond to a dedicated integrated circuit, or may correspond to a pure software code, or may correspond to a module that combines software and hardware. In addition, one or more functions implemented by each unit may also be uniformly executed by components in a physical entity device (e.g., a processor, a client, or a server, etc.).

[0078] In addition, refer to Figure 2 The training method of the described model and Figure 3 The remote heart rate measurement method described can be implemented by a program (or instruction) recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions can be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the training method of the physiological feature extraction model and the remote heart rate measurement method according to the present disclosure.

[0079] The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in computer devices such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above-mentioned steps or to perform more specific processing when executing the above-mentioned steps. The contents of these additional steps and further processing have been mentioned in the description of the relevant method with reference to the accompanying drawings, so they will not be repeated here to avoid repetition.

[0080] It should be noted that the various units of the training method of the physiological feature extraction model and the remote heart rate measurement method according to the exemplary embodiments of the present disclosure can completely rely on the operation of the computer program to realize the corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding function.

[0081] on the other hand, Figure 2 and Figure 4 The various units shown may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segment for performing the corresponding operation may be stored in a computer-readable medium such as a storage medium, so that the processor may perform the corresponding operation by reading and running the corresponding program code or code segment.

[0082] For example, an exemplary embodiment of the present disclosure may also be implemented as a computing device, which includes a storage component and a processor, wherein a set of computer executable instructions is stored in the storage component. When the set of computer executable instructions is executed by the processor, the training method of the physiological feature extraction model and the remote heart rate measurement method according to the exemplary embodiment of the present disclosure are executed.

[0083] Specifically, the computing device can be deployed in a server or client, or can be deployed on a node device in a distributed network environment. In addition, the computing device can be a PC, a tablet device, a personal digital assistant, a smart phone, a web application, or other device capable of executing the above instruction set.

[0084] Here, the computing device is not necessarily a single computing device, but may also be any device or circuit collection that can execute the above instructions (or instruction sets) individually or jointly. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0085] In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, a processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0086] According to the exemplary embodiments of the present disclosure, some operations described in the training method of the physiological feature extraction model and the remote heart rate measurement method can be implemented by software, some operations can be implemented by hardware, and in addition, these operations can also be implemented by a combination of software and hardware.

[0087] The processor may execute instructions or codes stored in one of the storage components, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0088] The storage component may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage component may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage component and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the storage component.

[0089] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the computing device may be connected to each other via a bus and / or a network.

[0090] The training method of the physiological feature extraction model and the remote heart rate measurement method according to the exemplary embodiment of the present disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logical device or operated according to non-exact boundaries.

[0091] Therefore, refer to Figure 2 The training method of the described model and Figure 3 The described remote heart rate measurement method may be implemented by a system including at least one computing device and at least one storage device storing instructions.

[0092] According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device for the training method of the physiological feature extraction model and the remote heart rate measurement method according to the exemplary embodiment of the present disclosure, and a computer executable instruction set is stored in the storage device. When the computer executable instruction set is executed by the at least one computing device, the training method of the physiological feature extraction model and the remote heart rate measurement method described above are executed.

[0093] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A training method for a physiological feature extraction model, characterized in that: The training method comprises: Step A1, obtaining a video containing a facial region and a label corresponding to the video as heart rate waveform data; Step A2, extracting a facial region from the video; Step A3, separating a first feature irrelevant to the heart rate information from the facial region by using a first encoder; Step A4, separating a second feature including only the heart rate information from the facial region by a second encoder; Step A5: encoding the tag by a third encoder to obtain a third feature, wherein the third feature is a feature used to characterize the heart rate waveform data; Step A6: splicing the first feature and the second feature to obtain a first splicing feature, and splicing the first feature and the third feature to obtain a second splicing feature; Step A7, inputting the first stitching feature and the second stitching feature into a generator of a generative adversarial network to generate a pseudo facial region, and using the facial region and the pseudo facial region as inputs of a discriminator of the generative adversarial network to discriminate the pseudo facial region through the discriminator; and Step A8: jointly training the first encoder, the second encoder, the third encoder, the generator, and the discriminator until the discriminator cannot distinguish between the facial region and the pseudo-facial region. The training method also includes: step A2-1, performing channel separation on the facial area to obtain a red channel, a blue channel and a green channel, wherein the first feature in step A3 is separated from the red channel and the blue channel by the first encoder, and the second feature in step A4 is separated from the green channel by the second encoder.

2. The training method according to claim 1, characterized in that: The step of jointly training the first encoder, the second encoder, the third encoder, the generator, and the discriminator comprises: Step B1, fixing the first encoder, the second encoder, the third encoder and the generator, and iteratively executing steps A2 to A7 to generate pseudo facial regions required for training to train the discriminator, so that the probability that the discriminator determines the facial region as true and the probability that the pseudo facial region is determined as false are maximized; Step B2, fixing the discriminator, iteratively executing steps A2 to A7 to generate a pseudo-face region required for training, and training the first encoder, the second encoder, the third encoder and the generator by back propagation, so that the probability that the discriminator determines the pseudo-face region as true is maximized; Step B1 and step B2 are iteratively performed until the discriminator cannot distinguish between the face region and the pseudo-face region.

3. The training method according to claim 1, characterized in that: At least a portion of the video is an unlabeled video that does not have the corresponding label, and Wherein, step A1 also includes: using a chromaticity-based method or a principal component analysis method to estimate the heart rate of the unlabeled video to obtain an estimated remote photoplethysmography signal, and using the estimated remote photoplethysmography signal as a label of the unlabeled video.

4. A remote heart rate measurement method, characterized in that: The remote heart rate measurement method comprises: Collect the video to be tested including the facial area; Extracting a facial region from the video to be tested; Separating physiological features including only reactive heart rate information from the facial region through a second encoder of the physiological feature extraction model; and Based on the physiological characteristics, an estimator composed of a convolutional neural network is used to obtain estimated heart rate waveform data. Wherein, the physiological feature extraction model is trained by the training method described in any one of claims 1 to 3.

5. The remote heart rate measurement method according to claim 4, characterized in that: The estimator includes an input layer, a convolution layer, an activation function, a pooling layer, a fully connected layer, and an output layer.

6. The remote heart rate measurement method according to claim 5, characterized in that: The remote heart rate measurement method further comprises training the estimator by the following steps: Acquire a second video including a facial area and a second label corresponding to the second video as heart rate waveform data; extracting a facial region from the second video; Inputting the facial region into the second encoder of the physiological feature extraction model to obtain the physiological feature; The physiological feature is provided to the input layer, and the estimator is trained by minimizing the difference between the output of the output layer and the second label.

7. The remote heart rate measurement method according to claim 6, characterized in that: At least a portion of the second video is an unlabeled video that does not have a corresponding second label, and The remote heart rate measurement method further includes: estimating the heart rate of the unlabeled video using a chromaticity-based method or a principal component analysis method to obtain an estimated remote photoplethysmography signal, and using the estimated remote photoplethysmography signal as a label of the unlabeled video.

8. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the training method of the physiological feature extraction model as described in any one of claims 1 to 3 or the remote heart rate measurement method as described in any one of claims 4 to 7.

9. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, they prompt the at least one computing device to execute the training method of the physiological feature extraction model as described in any one of claims 1 to 3 or the remote heart rate measurement method as described in any one of claims 4 to 7.

Images