A remote heart rate detection network training method and device based on passive domain adaptation

Abstract

The present invention discloses a remote heart rate detection network training method and device based on passive domain adaptation, wherein the method comprises: obtaining a target domain data set and preprocessing the target domain data set, the target domain data set comprising a plurality of target face image sequences without labels; obtaining weight parameters and a first remote heart rate detection network, the first remote heart rate detection network comprising a basic detection branch and a time detection branch and a space detection branch for supervised training, the basic detection branch comprising a first encoder and a first decoder, the time detection branch comprising a second encoder and a first decoder, and the space detection branch comprising a third encoder and a first decoder; initializing the first remote heart rate detection network using the weight parameters, and then training the initialized first remote heart rate detection network using the preprocessed target domain data set. The present invention can achieve excellent performance of migration from a source domain to a target domain, and ensure the accuracy and reliability of remote heart rate detection.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a remote heart rate detection network training method and device based on passive domain adaptation. Background Art

[0002] Remote Photoplethysmography (rPPG) based on vision is an effective, non-contact physiological index detection technology. The extraction of rPPG signals usually involves analyzing the slight changes in the skin color of the face in the video, and these changes are mainly reflected in the green channel in the RGB color space. However, this technology requires manual setting of the region of interest and performing some filtering operations when applied, and is easily disturbed by factors that affect the skin color, such as changes in illumination. To this end, some scholars have proposed the introduction of deep learning algorithms to overcome the influence of non-physiological factors in the video, and considering the problem of algorithm failure when dealing with unknown fields or encountering different data with field differences, it is further proposed to combine domain adaptation technology and domain generalization technology to reduce the field differences caused by non-physiological factors. However, both technologies require the use of source domain data and target domain data to train the same network model to perceive the gap between the source domain and the target domain in real time. This requirement cannot be met when the access to the source domain data is temporarily restricted, which easily leads to privacy and security issues of the source domain data. Summary of the invention

[0003] The present invention provides a remote heart rate detection network training method and device based on passive domain adaptation to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0004] In a first aspect, a remote heart rate detection network training method based on passive domain adaptation is provided, the method comprising:

[0005] Acquire a target domain dataset and perform preprocessing, wherein the target domain dataset includes a plurality of target face image sequences without labels;

[0006] Obtaining weight parameters and a first remote heart rate detection network, wherein the first remote heart rate detection network includes a basic detection branch and a time detection branch and a space detection branch for supervised training, wherein the basic detection branch includes a first encoder and a first decoder, the time detection branch includes a second encoder and the first decoder, and the space detection branch includes a third encoder and the first decoder;

[0007] The first remote heart rate detection network is initialized using the weight parameters, and the initialized first remote heart rate detection network is trained using the preprocessed target domain data set.

[0008] Furthermore, the weight parameter is obtained in the following manner:

[0009] Acquire a source domain data set and perform preprocessing, wherein the source domain data set includes a plurality of original face image sequences carrying labels, and the label carried by each original face image sequence is a real BVP signal;

[0010] A second remote heart rate detection network is obtained, which includes a fourth encoder and a second decoder, and the second remote heart rate detection network is pre-trained using the preprocessed source domain data set to obtain weight parameters.

[0011] Further, the weight parameters include the weight of the fourth encoder and the weight of the second decoder, and the initializing the first remote heart rate detection network using the weight parameters includes:

[0012] Using the weight of the fourth encoder as the initial weight of the first encoder, the second encoder and the third encoder, the network structures of these four encoders are the same;

[0013] The weight of the second decoder is used as the initial weight of the first decoder, and the network structures of the two decoders are the same.

[0014] Furthermore, when the preprocessed target domain data set is input into the basic detection branch, for each preprocessed target face image sequence, the first encoder is used to extract facial skin color continuous change features from the preprocessed target face image sequence to form first deep feature information, and the first decoder is used to reconstruct a first predicted rPPG signal based on the first deep feature information.

[0015] Furthermore, when the preprocessed target domain data set is input into the time detection branch, for each preprocessed target face image sequence, the preprocessed target face image sequence is frequency downsampled to obtain a first target face image sequence, the second encoder is used to extract the continuously changing features of facial skin color from the first target face image sequence to form second deep feature information, the first decoder is used to reconstruct the initial predicted rPPG signal according to the second deep feature information, and the initial predicted rPPG signal is frequency upsampled to obtain a second predicted rPPG signal.

[0016] Furthermore, when the preprocessed target domain data set is input into the spatial detection branch, for each preprocessed target face image sequence, the preprocessed target face image sequence is randomly spatially augmented to obtain a second target face image sequence, and the spatial augmentation operation includes at least one of image rotation, image horizontal flipping, and image vertical flipping. The third encoder is used to extract facial skin color continuous change features from the second target face image sequence to form third deep feature information, and the first decoder is used to reconstruct a third predicted rPPG signal based on the third deep feature information.

[0017] Furthermore, during the training process of the initialized first remote heart rate detection network, back propagation is performed using a spatiotemporal consistency loss function, and the spatiotemporal consistency loss function uses the Wasserstein distance to measure the distribution differences between all first predicted rPPG signals output by the basic detection branch and all second predicted rPPG signals output by the time detection branch and all third predicted rPPG signals output by the spatial detection branch.

[0018] Furthermore, the spatiotemporal consistency loss function is:

[0019]

[0020] in, is the spatiotemporal consistency loss function, is the temporal consistency loss function, is the spatial consistency loss function, F is the power spectrum density of all first predicted heart rates corresponding to all first predicted rPPG signals output by the basic detection branch, and F t are all second predicted heart rate power spectral densities corresponding to all second predicted rPPG signals output by the time detection branch, F s are all third predicted heart rate power spectral densities corresponding to all third predicted rPPG signals output by the spatial detection branch, softmax represents a normalization function, and WD represents the Wasserstein distance between two probability distributions.

[0021] Furthermore, the method also includes: obtaining a facial image sequence to be tested and inputting it into a trained basic detection branch for processing to obtain a final predicted rPPG signal.

[0022] In a second aspect, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the remote heart rate detection network training method based on passive domain adaptation as described in the first aspect.

[0023] The present invention has at least the following beneficial effects: by executing the network pre-training task based on the source domain data set and the network adaptation task based on the target domain data set in stages, the privacy security of the source domain data set can be protected; by directly applying the weight parameters of the second remote heart rate detection network obtained after pre-training to the first remote heart rate detection network for initialization, the knowledge learned in the source domain can be used to align the signal characteristics in the target domain in the network adaptation task based on the target domain data set, and the consistency of the data under time and space transformation can be fully utilized, so as to achieve excellent performance in migration from the source domain to the target domain, while ensuring the accuracy and reliability of remote heart rate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0025] Figure 1 It is a flowchart of a remote heart rate detection network training method based on passive domain adaptation in an embodiment of the present invention;

[0026] Figure 2 is a simplified schematic diagram of the training process of the first remote heart rate detection network in an embodiment of the present invention;

[0027] Figure 3 is a simplified schematic diagram of the training process of the second remote heart rate detection network in an embodiment of the present invention;

[0028] Figure 4 It is a schematic diagram of the hardware structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0032] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0033] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all content and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0034] Early vision-based remote photoplethysmography estimates the corresponding rPPG signal by analyzing the subtle color changes in the face area in the RGB video. The rPPG signal can reflect the patient's heart rate, respiratory rate, etc. It is usually assisted by color subspace transformation methods based on chromaticity and skin color orthogonal projection planes. However, these methods require manual setting of the region of interest and performance of some filtering operations, and are easily interfered by factors that affect skin color, such as changes in lighting.

[0035] Later, in the study of vision-based remote photoplethysmography, the application of deep learning technology, especially convolutional neural networks, became mainstream, which can overcome the influence of non-physiological factors in the video. Many researchers explored a variety of deep learning architectures including two-dimensional convolutional neural networks, three-dimensional convolutional neural networks, end-to-end spatiotemporal networks, and long short-term memory networks to encode time information in the video, which is particularly important for extracting continuous signals. Although these deep learning architectures can enhance the corresponding encoding representation by adaptively aggregating spatiotemporal features, they may fail when dealing with unknown fields or encountering different data with field differences. This situation often occurs in real application scenarios. For example, different ways of collecting data will also bring certain field differences.

[0036] For this reason, other researchers have proposed that the use of domain adaptation technology (such as domain replacement and domain expansion) and domain generalization technology (such as using a double bridge network to reduce target noise) can reduce domain differences caused by changes in non-physiological factors. However, both technologies require the use of source domain data and target domain data to train the same network model to perceive the gap between the source domain and the target domain in real time. This requirement cannot be met when access to source domain data is limited, and it is easy to cause privacy and security issues for source domain data. In existing domain adaptation technologies, domains are usually aligned based on the loss of the Kullback-Leibler divergence of single-point probabilities, which cannot fully take into account the global characteristics of rPPG signals similar to Gaussian distributions, nor can they fully utilize spatiotemporal consistency learning to align rPPG signals to reduce the differences between source and target domains. Based on this, an embodiment of the present invention provides a remote heart rate detection network training method and device based on passive domain adaptation, which introduces passive domain adaptation into the field of remote heart rate detection, and only uses an unlabeled target domain dataset and weight parameters pre-trained by a source domain dataset for network training, making full use of the consistency of data under time and space transformation, and can achieve excellent performance in migrating from the source domain to the target domain, reducing domain differences.

[0037] First, some of the terms involved in the present invention are explained as follows:

[0038] The MTCNN (Multi-task Cascaded Convolutional Neural Networks) algorithm is an efficient and accurate face detection and alignment method. It implements the functions of face detection and facial key point location by constructing three cascaded convolutional neural networks (i.e., P-Net, R-Net, and O-Net). It can detect all faces in the image and locate the key points of the face (such as eyes, nose, corners of mouth, etc.). The specific description is as follows:

[0039] P-Net (Proposal Network) is used to quickly generate candidate face boxes and boundary regression vectors (i.e., preliminarily screen out areas that may contain faces). Its input is an RGB image block, and its output includes face classification prediction, bounding box regression prediction, and key point position prediction;

[0040] R-Net (Refine Network) is used to screen and fine-tune the candidate face frames generated by P-Net (i.e., eliminate incorrect candidate face frames and accurately adjust the positions of face frames). Its input is the image blocks corresponding to the candidate face frames output by P-Net, and its output also includes face classification prediction, bounding box regression prediction, and key point position prediction;

[0041] O-Net (Output Network) is used to output the final face frame and key point positions. Its input is the image block corresponding to the candidate face frame output by R-Net. The output includes face classification prediction, bounding box regression prediction and position prediction of 5 facial key points.

[0042] The optimal transmission distance (WD, Wasserstein distance) can be understood as the minimum amount of work required to move one pile of stones to another pile of stones, and is further quantified as the sum of the products of the distance and number of stones that must be moved.

[0043] Please refer to Figure 1 , Figure 1 1 is a flow chart of a remote heart rate detection network training method based on passive domain adaptation provided by an embodiment of the present invention, the method comprising the following steps:

[0044] Step S110, obtaining a target domain data set and performing preprocessing;

[0045] Wherein, the target domain dataset includes a plurality of target face image sequences without labels;

[0046] Step S120, obtaining weight parameters and a first remote heart rate detection network;

[0047] The first remote heart rate detection network includes a basic detection branch and a time detection branch and a space detection branch for supervised training, the basic detection branch includes a first encoder and a first decoder, the time detection branch includes a second encoder and the first decoder, and the space detection branch includes a third encoder and the first decoder;

[0048] Step S130: Initialize the first remote heart rate detection network using the weight parameters, and then train the initialized first remote heart rate detection network using the preprocessed target domain data set.

[0049] It should be noted that each target facial image sequence mentioned in the above step S110 includes multiple frames of target facial images collected for the same person within a continuous period of time, and the frame width, frame height and number of channels of each frame of the target facial image are the same. The multiple frames of the target facial images are arranged in a certain time sequence, and can show the dynamic changes of the face of the person within the continuous period of time, such as skin color changes, etc., and different target facial image sequences are targeted at different people when they are collected; as a preferred implementation, the frame width of each frame of the target facial image is 128, the frame height is 128 and the number of channels is 3 (i.e., RGB channels), and the number of frames of all target facial images contained in each target facial image sequence is 220.

[0050] In some embodiments, the preprocessing method mentioned in the above step S110 can be to use the MTCNN algorithm to extract the region of interest for each target face image sequence, where the region of interest refers to a face rectangular frame in the target face image that can completely cover the entire face and part of the background area, so as to reduce the background noise in each target face image sequence.

[0051] More specifically, for each target face image sequence contained in the target domain dataset, a first frame of target face image with the earliest time is extracted from the target face image sequence, the first frame of target face image is detected using the MTCNN algorithm to extract a first face rectangular frame therefrom, the position of the first face rectangular frame in the first frame of target face image is obtained, and the first frame of target face image is cropped to retain only the image portion covered by the first face rectangular frame; considering that all target face images contained in the target face image sequence are allowed to use the same face rectangular frame at the same position, each other frame of target face image contained in the target face image sequence is cropped according to the position of the first face rectangular frame in the first frame of target face image to retain only the image portion covered by the first face rectangular frame.

[0052] In some embodiments, the method for obtaining the weight parameter mentioned in the above step S120 is described as follows:

[0053] (1) Obtain a source domain dataset, which includes multiple original face image sequences with labels;

[0054] Among them, the label carried by each original face image sequence is the real BVP (Blood Volume Pulse) signal corresponding to the original face image sequence, and each original face image sequence includes multiple frames of original face images collected for the same person within a continuous period of time. The frame width, frame height and number of channels of each frame of the original face image are the same. The multiple frames of the original face images are arranged in a certain time sequence, which can show the dynamic changes of the face of the person in the continuous period of time, such as skin color changes, etc., and different original face image sequences are collected for different people; as a preferred implementation, the frame width of each frame of the original face image is 128, the frame height is 128 and the number of channels is 3 (i.e., RGB channels), and the number of frames of all original face images contained in each original face image sequence is 300.

[0055] It should be noted that the real BVP signal corresponding to the original facial image sequence is obtained by directly collecting the fingertip area of ​​the person targeted by the original facial image sequence through a contact sensor (such as a photoplethysmogram sensor), which can also reflect the heart rate information of the person.

[0056] (2) Preprocessing the source domain dataset, preferably using the MTCNN algorithm to extract the region of interest from each original face image sequence, where the region of interest refers to a face rectangular frame in the original face image that can completely cover the entire face and part of the background area, so as to reduce the background noise in each original face image sequence;

[0057] More specifically, for each original face image sequence contained in the source domain dataset, the first frame of original face image with the earliest time is extracted from the original face image sequence, the first frame of original face image is detected using the MTCNN algorithm to extract a second face rectangular frame therefrom, the position of the second face rectangular frame in the first frame of original face image is obtained, and the first frame of original face image is cropped to retain only the image portion covered by the second face rectangular frame; considering that all original face images contained in the original face image sequence are allowed to use the same face rectangular frame at the same position, each other frame of original face image contained in the original face image sequence is cropped according to the position of the second face rectangular frame in the first frame of original face image to retain only the image portion covered by the second face rectangular frame.

[0058] (3) obtaining a second remote heart rate detection network, which mainly includes a second decoder and a fourth encoder;

[0059] In actual application, see Figure 3 As shown, when the preprocessed source domain data set is input into the second remote heart rate detection network, for each preprocessed original face image sequence therein, the fourth encoder extracts the continuously changing features of the facial skin color from the preprocessed original face image sequence and forms deep feature information, and then the second decoder analyzes the deep feature information to reconstruct the predicted rPPG signal; according to this embodiment, a plurality of predicted rPPG signals corresponding to the plurality of preprocessed original face image sequences can be obtained.

[0060] (4) inputting the preprocessed source domain data set into the second remote heart rate detection network for pre-training to obtain weight parameters, wherein the weight parameters mainly include the weight of the second decoder and the weight of the fourth encoder;

[0061] In this network pre-training process, it is preferred to use a label loss function for back propagation, in which the Wasserstein distance is used to measure the distribution differences between the multiple predicted rPPG signals output by the second remote heart rate detection network and the multiple real BVP signals corresponding to the preprocessed multiple original face image sequences. The label loss function can be expressed as follows:

[0062]

[0063] In the formula, is the label loss function, F source are multiple predicted heart rate power spectrum densities obtained after the multiple predicted rPPG signals are processed in the frequency domain, F lable are multiple real heart rate power spectrum densities obtained after the multiple real BVP signals are processed in the frequency domain. The frequency domain transformation process can be performed using the principle of fast Fourier transform. Softmax(·) refers to a normalization function, which is used to convert the relevant heart rate power spectrum density into a probability density function, so that the power of each frequency component is expressed as the relative probability of the frequency component in the entire power distribution. WD[softmax(F source ),softmax(F lable )] represents the probability distribution softmax(F source ) and probability distribution softmax(F lable ), CDF(·) refers to the cumulative distribution function, CDF i [softmax(F source )] represents the normalized cumulative distribution value of the ith predicted heart rate power spectrum density, CDF i [softmax(F lable )] represents the normalized cumulative distribution value of the ith true heart rate power spectrum density, and the ith predicted heart rate power spectrum density and the ith true heart rate power spectrum density are both directly associated with the ith original face image sequence after preprocessing.

[0064] It should be noted that in the prior art, the Kullback-Leibler divergence is usually chosen to study the differences between two distributions. However, the Kullback-Leibler divergence lacks global considerations and focuses more on the probability value of a single point, that is, it considers the probability situation between the two distributions vertically. The Wasserstein distance can not only solve the deviation between individual probabilities, but also capture the differences in the overall distribution, and is more suitable for rPPG signal alignment tasks that require global considerations.

[0065] In some embodiments, see Figure 2 As shown, the following description is made for the application process of the first remote heart rate detection network mentioned in the above step S120:

[0066] (1) When the preprocessed target domain data set is input into the basic detection branch, for each preprocessed target face image sequence therein, the first encoder extracts the continuously changing features of the facial skin color from the preprocessed target face image sequence and forms the first deep feature information, and then the first decoder analyzes the first deep feature information to reconstruct the first predicted rPPG signal; according to this embodiment, a plurality of first predicted rPPG signals corresponding to the plurality of preprocessed target face image sequences can be obtained.

[0067] (2) When the preprocessed target domain data set is input into the time detection branch, for each preprocessed target face image sequence therein, the preprocessed target face image sequence is processed by the first frequency resampler to obtain a first target face image sequence, and then the second encoder is used to extract the continuously changing features of the facial skin color from the first target face image sequence and form second deep feature information, and then the first decoder is used to analyze the second deep feature information to reconstruct an initial predicted rPPG signal, and finally the initial predicted rPPG signal is processed by the second frequency resampler to obtain a second predicted rPPG signal; according to this embodiment, a plurality of second predicted rPPG signals corresponding to the plurality of preprocessed target face image sequences can be obtained.

[0068] Among them, the first frequency resampler is used to perform a frequency downsampling operation and the first resampling factor used internally is r∈[0.66,0.80], and the second frequency resampler is used to perform a frequency upsampling operation and the second resampling factor used internally is 1 / r; by setting the first resampling factor and the second resampling factor to be inversely related to each other, using the first frequency resampler to change the data scale, and then using the second frequency resampler to restore the data scale, introducing two frequency resamplers in the time detection branch can enhance the feature representation of the second predicted rPPG signal in the time dimension.

[0069] (3) When the preprocessed target domain data set is input into the spatial detection branch, for each preprocessed target face image sequence therein, spatial augmentation is randomly performed on the preprocessed target face image sequence to obtain a second target face image sequence, and then the third encoder is used to extract the continuously changing features of the facial skin color from the second target face image sequence and form third deep feature information, and finally the first decoder is used to analyze the third deep feature information to reconstruct a third predicted rPPG signal; according to this embodiment, multiple third predicted rPPG signals corresponding to the multiple preprocessed target face image sequences can be obtained.

[0070] Among them, the spatial augmentation operation mainly includes at least one of image vertical flipping, image horizontal flipping and image rotation, and image rotation includes image 0° rotation, image 90° rotation, image 180° rotation and image 270° rotation; by randomly performing different spatial augmentation operations on different frame target face images contained in the preprocessed target face image sequence, the feature representation of the third predicted rPPG signal in the spatial dimension can be enhanced.

[0071] In some embodiments, the network initialization method mentioned in the above step S130 includes the following: defining the weight of the fourth encoder as the initial weight of the first encoder, the initial weight of the second encoder and the initial weight of the third encoder, and defining the weight of the second decoder as the initial weight of the first decoder;

[0072] It should be noted that the first encoder, the second encoder, the third encoder and the fourth encoder are all built with the same network structure, and the first decoder and the second decoder are all built with the same network structure, wherein any encoder and any decoder are preferably decomposed from the existing PhysNet-3DCNN model (i.e., an end-to-end deep spatiotemporal network), which is recorded in an English document entitled “Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks”.

[0073] In some embodiments, in the network training process mentioned in the above step S130, it is preferred to use a spatiotemporal consistency loss function for back propagation, in which the distribution differences between the multiple first predicted rPPG signals output by the basic detection branch and the multiple second predicted rPPG signals output by the time detection branch, and the distribution differences between the multiple first predicted rPPG signals output by the basic detection branch and the multiple third predicted rPPG signals output by the space detection branch are measured by the Wasserstein distance. The spatiotemporal consistency loss function can be expressed as follows:

[0074]

[0075] In the formula, is the spatiotemporal consistency loss function, is the temporal consistency loss function, is a spatial consistency loss function, F is a plurality of first predicted heart rate power spectrum densities obtained after the plurality of first predicted rPPG signals are processed in the frequency domain, and F t are multiple second predicted heart rate power spectrum densities obtained after the multiple second predicted rPPG signals are processed in the frequency domain, s are multiple third predicted heart rate power spectral densities obtained after the multiple third predicted rPPG signals are processed in the frequency domain. The frequency domain transformation process can be performed using the fast Fourier transform principle. Softmax(·) refers to a normalization function, which is used to convert the relevant predicted heart rate power spectral density into a probability density function, so that the power of each frequency component represents the relative probability of the frequency component in the entire power distribution. WD[softmax(F),softmax(F t )] represents the probability distribution softmax(F) and the probability distribution softmax(F t ), WD[softmax(F),softmax(F s )] represents the probability distribution softmax(F) and the probability distribution softmax(F s ), CDF(·) refers to the cumulative distribution function, CDF k [softmax(F)] represents the normalized cumulative distribution value of the kth first predicted heart rate power spectrum density, CDF k [softmax(F t )] represents the normalized cumulative distribution value of the kth second predicted heart rate power spectrum density, CDF k [softmax(F s )] represents the normalized cumulative distribution value of the kth third predicted heart rate power spectrum density, and the kth first predicted heart rate power spectrum density, the kth second predicted heart rate power spectrum density and the kth third predicted heart rate power spectrum density are all directly associated with the preprocessed kth target face image sequence.

[0076] In some embodiments, after completing the training task of the first remote heart rate detection network through the above steps S110 to S130, the trained basic detection branch can be put into formal application, which specifically includes the following steps:

[0077] (1) obtaining a sequence of face images to be tested, which includes multiple frames of face images to be tested collected from the same target person within a continuous period of time;

[0078] As a preferred implementation, each frame of the face image to be tested has a frame width of 128, a frame height of 128, and a channel number of 3 (i.e., RGB channels), and the number of the multiple frames of face images to be tested is 220;

[0079] (2) Inputting the face image sequence to be tested into the trained basic detection branch for analysis to obtain the final predicted rPPG signal.

[0080] Of course, the final predicted rPPG signal may be further transformed in the frequency domain to obtain the final predicted heart rate power spectrum density, so that the user can analyze the specific heart rate information of the target person according to the final predicted rPPG signal and the final predicted heart rate power spectrum density.

[0081] In an embodiment of the present invention, by performing a network pre-training task based on a source domain data set and a network adaptation task based on a target domain data set in stages, the privacy security of the source domain data set can be protected; by directly applying the weight parameters of the second remote heart rate detection network obtained after pre-training to the first remote heart rate detection network for initialization, the knowledge learned in the source domain can be used to align the signal characteristics in the target domain in the network adaptation task based on the target domain data set, and the consistency of the data under time and space transformation can be fully utilized, so as to achieve excellent performance in migration from the source domain to the target domain and ensure the accuracy and reliability of remote heart rate detection.

[0082] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, a remote heart rate detection network training method based on passive domain adaptation in the above embodiment is implemented. Wherein, the computer-readable storage medium includes but is not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. In other words, the storage device includes any medium that stores or transmits information in a readable form by a device (such as a computer, a mobile phone, etc.), which can be a read-only memory, a disk or an optical disk, etc.

[0083] also, Figure 42 is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention, wherein the computer device includes a processor 220, a memory 230, an input unit 240, a display unit 250 and other devices. Those skilled in the art can understand that Figure 4 The device structure device shown does not constitute a limitation on all devices, and may include more or fewer components than shown, or combine certain components. The memory 230 can be used to store the computer program 210 and various functional modules, and the processor 220 runs the computer program 210 stored in the memory 230, thereby executing various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include an internal memory and an external memory. The internal memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory. The external memory may include a hard disk, a floppy disk, a U disk, a magnetic tape, etc. The memory 230 disclosed in the embodiment of the present invention includes but is not limited to the above-mentioned types of memory. The memory 230 disclosed in the embodiment of the present invention is only an example and not a limitation.

[0084] The input unit 240 is used to receive the input of the signal and the keyword input by the user. The input unit 240 may include a touch panel and other input devices. The touch panel can collect the user's touch operation on or near it (such as the user's operation on or near the touch panel using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program; other input devices may include but are not limited to one or more of a physical keyboard, a function key (such as a playback control key, a switch key, etc.), a trackball, a mouse, a joystick, etc. The display unit 250 can be used to display the information input by the user or the information provided to the user and various menus of the terminal device. The display unit 250 can be in the form of a liquid crystal display, an organic light emitting diode, etc. The processor 220 is the control center of the terminal device, which uses various interfaces and lines to connect various parts of the entire device, and executes various functions and processes data by running or executing software programs and / or modules stored in the memory 230, and calling data stored in the memory 230.

[0085] As an embodiment, the computer device includes a processor 220, a memory 230 and a computer program 210, wherein the computer program 210 is stored in the memory 230 and is configured to be executed by the processor 220, and the computer program 210 is configured to execute a remote heart rate detection network training method based on passive domain adaptation in the above-mentioned embodiment.

[0086] Although the description of the present application has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims by reference to the attached claims, taking into account the prior art, so as to effectively cover the intended scope of the present application. In addition, the above description of the present application is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present application that have not yet been foreseen may still represent equivalent changes to the present application.

Claims

1. A remote heart rate detection network training method based on passive domain adaptation, characterized in that: The method comprises: Acquire a target domain dataset and perform preprocessing, wherein the target domain dataset includes a plurality of target face image sequences without labels; Obtaining weight parameters and a first remote heart rate detection network, wherein the first remote heart rate detection network includes a basic detection branch and a time detection branch and a space detection branch for supervised training, wherein the basic detection branch includes a first encoder and a first decoder, the time detection branch includes a second encoder and the first decoder, and the space detection branch includes a third encoder and the first decoder; Initializing the first remote heart rate detection network using the weight parameters, and then training the initialized first remote heart rate detection network using the preprocessed target domain data set; When the preprocessed target domain data set is input into the basic detection branch, for each preprocessed target face image sequence, the first encoder is used to extract facial skin color continuous change features from the preprocessed target face image sequence to form first deep feature information, and the first decoder is used to reconstruct a first predicted rPPG signal according to the first deep feature information; When the preprocessed target domain data set is input into the time detection branch, for each preprocessed target face image sequence, the preprocessed target face image sequence is frequency downsampled to obtain a first target face image sequence, the second encoder is used to extract facial skin color continuous change features from the first target face image sequence to form second deep feature information, the first decoder is used to reconstruct an initial predicted rPPG signal according to the second deep feature information, and the initial predicted rPPG signal is frequency upsampled to obtain a second predicted rPPG signal; When the preprocessed target domain data set is input into the spatial detection branch, for each preprocessed target face image sequence, the preprocessed target face image sequence is randomly spatially augmented to obtain a second target face image sequence, the spatial augmentation operation includes at least one of image rotation, image horizontal flipping and image vertical flipping, the third encoder is used to extract facial skin color continuous change features from the second target face image sequence to form third deep feature information, and the first decoder is used to reconstruct a third predicted rPPG signal according to the third deep feature information; During the training process of the initialized first remote heart rate detection network, back propagation is performed using a spatiotemporal consistency loss function, and the spatiotemporal consistency loss function uses the Wasserstein distance to measure the distribution differences between all first predicted rPPG signals output by the basic detection branch and all second predicted rPPG signals output by the time detection branch and all third predicted rPPG signals output by the space detection branch.

2. The remote heart rate detection network training method based on passive domain adaptation according to claim 1 is characterized in that: The weight parameters are obtained in the following way: Acquire a source domain data set and perform preprocessing, wherein the source domain data set includes a plurality of original face image sequences carrying labels, and the label carried by each original face image sequence is a real BVP signal; A second remote heart rate detection network is obtained, which includes a fourth encoder and a second decoder, and the second remote heart rate detection network is pre-trained using the preprocessed source domain data set to obtain weight parameters.

3. The remote heart rate detection network training method based on passive domain adaptation according to claim 2 is characterized in that: The weight parameters include the weight of the fourth encoder and the weight of the second decoder, and the initializing the first remote heart rate detection network using the weight parameters includes: Using the weight of the fourth encoder as the initial weight of the first encoder, the second encoder and the third encoder, the network structures of these four encoders are the same; The weight of the second decoder is used as the initial weight of the first decoder, and the network structures of the two decoders are the same.

4. The remote heart rate detection network training method based on passive domain adaptation according to claim 1 is characterized in that: The spatiotemporal consistency loss function is: in, is the spatiotemporal consistency loss function, is the temporal consistency loss function, is the spatial consistency loss function, are all first predicted heart rate power spectrum densities corresponding to all first predicted rPPG signals output by the basic detection branch, are all second predicted heart rate power spectrum densities corresponding to all second predicted rPPG signals output by the time detection branch, are all third predicted heart rate power spectrum densities corresponding to all third predicted rPPG signals output by the spatial detection branch, represents the normalization function, Represents the Wasserstein distance between two probability distributions.

5. The remote heart rate detection network training method based on passive domain adaptation according to claim 1 is characterized in that: The method further comprises: acquiring a sequence of face images to be tested and inputting the sequence into a trained basic detection branch for processing to obtain a final predicted rPPG signal.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: The processor executes the computer program to implement the remote heart rate detection network training method based on passive domain adaptation as described in any one of claims 1 to 5.

Images