Method and device for extracting rPPG signal, medium, equipment and model training method
By acquiring the target face video and determining the difference characteristics and space-time fusion characteristics, the encoder decoder structure and jump connection are used to solve the problem of weak noise interference and color changes in the video, and high-precision rPPG signal extraction is achieved.
Patent Information
- Application Number
- CN202411100283.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-29
AI Technical Summary
In complex environments, the current deep learning framework is difficult to effectively eliminate noise that is irrelevant to rPPG signals in videos, and the face color changes in the video sequence are weak, making it difficult to extract rPPG signals.
By obtaining the target face video, determining the difference characteristics and space-time fusion characteristics, the rPPG signal is extracted using the encoder decoder structure and jump connection, reducing the noise influence and enhancing the perception of rPPG signal fluctuations.
Effectively extract the rPPG signal in the video, reduce noise interference, enhance the perception of rPPG signal, and improve the extraction accuracy.
Smart Images

Figure CN119049106A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of remote photoplethysmography, and specifically relates to an rPPG signal extraction method, device, medium, equipment and model training method. Background Art
[0002] Heart rate is an important physiological indicator, which can be used to analyze human health status, emotional changes, and stress values. However, traditional physiological signal measurement technologies such as Photoplethysmograph (PPG) usually require contact with human fingers and other parts, which has certain limitations in some scenarios where it is inconvenient to contact the skin (such as neonatal monitoring, burn patients, and sleep detection). To solve this problem, remote photoplethysmography (rPPG) has attracted much attention in recent years. It provides a new detection method. It only needs to shoot a face video with a camera, and the rPPG signal can be extracted by analyzing the tiny color changes in the face video to extract physiological indicators such as heart rate. This color change originates from the blood circulation caused by the beating of the heart, which causes the blood volume of the microvascular tissue bed under the skin to change, and then causes periodic color fluctuations in the skin area such as the face, which cannot be detected by the human eye. Compared with traditional methods, remote photoplethysmography (rPPG) is more economical and faster, so it has attracted more and more attention.
[0003] It is challenging to extract rPPG signals with high accuracy in complex environments. Current deep learning frameworks cannot effectively remove noise from videos that are irrelevant to rPPG signals. In addition, the color changes of faces in video sequences are subtle, making it very difficult to extract rPPG signals from them. Summary of the invention
[0004] In view of the above technical problems, the present invention proposes a method, device, medium, equipment and model training method for extracting rPPG signals. The present application obtains a target face video, and determines the difference features of the target face video according to the target face video; determines the spatiotemporal fusion features of the target face video according to the difference features and the target face video; determines the decoder features according to the spatiotemporal fusion features; and extracts the rPPG signal from the decoder features. The noise effect is reduced, and the network's perception of rPPG signal fluctuations is enhanced, so that the rPPG signal in the video can be effectively extracted.
[0005] To solve the above technical problems, the technical solution provided by the present invention includes five aspects.
[0006] In a first aspect, the present application provides a method for extracting an rPPG signal, comprising: obtaining a target face video, and determining a difference feature of the target face video based on the target face video; determining a spatiotemporal fusion feature of the target face video based on the difference feature and the target face video; determining a decoder feature based on the spatiotemporal fusion feature; and extracting an rPPG signal from the decoder feature.
[0007] In some embodiments, determining the difference features of the target face video based on the target face video includes: generating multiple time-shifted videos based on the target face video; and determining the difference features of the target face video based on the multiple time-shifted videos.
[0008] In some embodiments, determining the spatiotemporal fusion features of the target face video based on the difference features and the target face video includes: determining a first waveform feature based on the difference features; determining a second waveform feature based on the target face video; determining a merged waveform feature based on the first waveform feature and the second waveform feature; determining a first enhancement feature based on the merged waveform feature; determining a second enhancement feature based on the second waveform feature; and determining the spatiotemporal fusion features based on the first enhancement feature and the second enhancement feature.
[0009] In some embodiments, determining the decoder features based on the spatiotemporal fusion features includes: obtaining encoder features through multiple encoder modules of the spatiotemporal fusion features; extracting time dimension features from the encoder features; fusing features obtained during the encoding process with the time dimension features and having the same duration as the time dimension features to obtain time-based enhanced features; and obtaining the decoder features through a decoder.
[0010] In a second aspect, the present application proposes a training method for a neural network model, wherein the neural network model is used to execute any of the steps described in the first aspect, and the method includes: obtaining a training video to be processed; cropping the training video to obtain a training set; inputting the training set into the neural network model to obtain an rPPG signal output by the neural network model; determining the self-similarity loss of the neural network model based on the rPPG signal and the label of the training set; determining the Pearson loss of the neural network model based on the rPPG signal and the true value of the training set; and adjusting the neural network model based on the Pearson loss and the self-similarity loss.
[0011] In some embodiments, the cropping of the training video to obtain a training set includes: determining facial landmarks based on the training video; determining a facial center point of each frame based on the facial landmarks; determining an initial bounding box for each frame based on the facial center point; and cropping the facial area of the training video based on the facial center point and the initial bounding box to form a training set.
[0012] In some embodiments, determining the self-similarity loss of the neural network model based on the rPPG signal and the label of the training set includes: randomly sampling the rPPG signal and the label through a time sampler to obtain a plurality of signal samples and label samples with the same duration; determining a signal power spectral density based on the signal samples; determining a label power spectral density based on the label samples; and determining the self-similarity loss based on the signal power spectral density and the label power spectral density.
[0013] In a third aspect, the present application proposes a device for extracting rPPG signals, comprising: a first execution module, used to obtain a target face video, and determine the difference features of the target face video based on the target face video; a first determination module, used to determine the spatiotemporal fusion features of the target face video based on the difference features and the target face video; a second determination module, used to determine the decoder features based on the spatiotemporal fusion features; and a second execution module, used to extract rPPG signals from the decoder features.
[0014] In a fourth aspect, the present application proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.
[0015] In a fifth aspect, the present application proposes a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.
[0016] Beneficial effects of the invention: The invention obtains a target face video, determines the difference features of the target face video according to the target face video, determines the spatiotemporal fusion features of the target face video according to the difference features and the target face video, determines the decoder features according to the spatiotemporal fusion features, and extracts the rPPG signal from the decoder features. The noise effect is reduced, and the network's perception of the fluctuation of the rPPG signal is enhanced, so that the rPPG signal in the video can be effectively extracted. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The scope of the present disclosure may be better understood by reading the following detailed description of exemplary embodiments in conjunction with the accompanying drawings. The drawings included are:
[0018] Figure 1 An overall flow chart of a method for extracting rPPG signals provided in an embodiment of the present application;
[0019] Figure 2 An overall flow chart of a neural network training method provided in an embodiment of the present application;
[0020] Figure 3 A structural block diagram of a spatiotemporal feature fusion module provided in an embodiment of the present application;
[0021] Figure 4 A flow chart of a spatiotemporal feature fusion module provided in an embodiment of the present application;
[0022] Figure 5 A flow chart of a convolution module provided in an embodiment of the present application;
[0023] Figure 6 This is a structural block diagram of a device for extracting rPPG signals provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0025] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0026] If similar descriptions of "first\second\third" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0028] Embodiment 1:
[0029] It is challenging to extract rPPG signals with high accuracy in complex environments. Current deep learning frameworks cannot effectively remove noise from videos that are irrelevant to rPPG signals. In addition, the color changes of faces in video sequences are subtle, making it very difficult to extract rPPG signals from them.
[0030] In view of the problems existing in the prior art, such as Figure 1 As shown, the present application provides a method for extracting rPPG signals, and the method is applied to electronic devices, and the electronic devices can be servers, mobile terminals, computers, cloud platforms, neural network models, etc. The functions implemented by the device data processing provided in the embodiments of the present application can be implemented by calling program codes by the processor of the electronic device, wherein the program codes can be stored in a computer storage medium, and the method for extracting rPPG signals includes:
[0031] Step S1: Acquire a target face video, and determine a difference feature of the target face video according to the target face video.
[0032] In video sequences, various noises are relatively strong, while facial color changes are very subtle. How to effectively deal with this challenge is a key issue in the rPPG task. We observed that difference frames can effectively remove the influence of inherent noise (such as skin color, video artifacts). This is because the color changes in face videos are mainly reflected in the slight differences between adjacent frames, and difference frames can help neural networks better perceive changes in rPPG signals. However, using only difference frames as input can easily make the model focus on noise such as motion artifacts that are irrelevant to the signal. Therefore, a difference feature is proposed in this application, and it is fused with the original video based on the difference feature to improve the perception of rPPG at the frame level.
[0033] In some embodiments, step S1 “determining the difference features of the target face video according to the target face video” includes:
[0034] Step S11: generating a plurality of time-series displacement videos according to the target face video.
[0035] Step S12: determining the difference frames of the target face video according to the multiple time-series shifted videos.
[0036] Step S13: Connect the plurality of difference frames in the channel dimension to form the difference feature.
[0037] Suppose the dimension of the input RGB video sequence x is N×3×T×128×128, where N is the batch size, 3 is the number of channels, T is the time, and 128 is the spatial size. The input video sequence is first obtained by temporal displacement (t-2) 、x (t-1) 、x (t) 、x(t+1) 、x (t+2) Next, calculate the difference frames in time order, that is, calculate x (t+2) -x (t+1) 、x (t+1) -x (t) 、x (t) -x (t-1) 、x (t-1) -x (t-2) , D1, D2, D3 and D4 are obtained respectively, and the obtained difference frames are connected in the channel dimension to form the difference features.
[0038] Among them, x t-2 Indicates moving the video sequence backward by two frames in the time dimension, and filling the two empty frames with the first frame. t-1 Similar. t is the original input video sequence, x t+1 Indicates moving the video sequence forward one frame in the time dimension, and filling the empty frame with the last frame. t+2 similar.
[0039] Step S2: determining the spatiotemporal fusion features of the target face video according to the difference frame and the target face video.
[0040] In some embodiments, step S2 "determining the spatiotemporal fusion features of the target face video according to the difference features and the target face video" includes:
[0041] Step S21: determining a first waveform feature according to the difference feature.
[0042] Step S22: determining a second waveform feature according to the target face video.
[0043] Step S23: determining a combined waveform feature according to the first waveform feature and the second waveform feature.
[0044] Step S24: determining a first enhanced feature according to the combined waveform feature.
[0045] Step S25: determining a second enhanced feature according to the second waveform feature.
[0046] Step S26: Determine the spatiotemporal fusion feature according to the first enhancement feature and the second enhancement feature.
[0047] In the present application, the difference features and the target face video can be simultaneously subjected to a series of operations such as 2D convolution with a convolution kernel size of (5×5), batch normalization (BN), Tanh activation function and MaxPool2D to obtain various first waveform features and second waveform features.
[0048] Then, the merged waveform feature formed by the first waveform feature and the second waveform feature and the second waveform feature are subjected to 2D convolution with a convolution kernel size of (3×3), batch normalization (BN), Tanh activation function and MaxPool2D to obtain the corresponding first enhanced feature and second enhanced feature, and then the spatiotemporal fusion feature is determined by combining the first enhanced feature and the second enhanced feature.
[0049] The calculation process that will obtain the first main waveform characteristic and the second waveform characteristic is called Stem1, and the calculation process that will obtain the first enhanced characteristic and the second enhanced characteristic is called Stem2.
[0050] Therefore, the calculation formula of step S21 to step S26 is as follows:
[0051] X dif =Stem1(Concat(D1,D2,D3,D4));
[0052] X origin =Stem1(X t );
[0053] X fusion =(1-λ)Stem2(λ×X origin +(1-λ)×X dif )+λ×Stem2(X origin )
[0054] X dif Represents the first waveform feature, X origin represents the second waveform feature, λ represents the weight coefficient, X fusion Represents the spatiotemporal fusion characteristics.
[0055] Steps S1 to S26 solve the problem that using only difference frames as input easily causes the model to focus on noise such as motion artifacts that are irrelevant to the signal, making it possible to perceive changes in rPPG signals at the frame level, effectively enhancing the features of rPPG while only increasing a small amount of computational cost.
[0056] Step S3: Determine decoder features according to the spatiotemporal fusion features.
[0057] In some embodiments, step S3 “determining decoder features according to the spatiotemporal fusion features” includes:
[0058] Step S31: Passing the spatiotemporal fusion features through multiple encoder modules to obtain encoder features.
[0059] Step S32: extracting time dimension features from the encoder features.
[0060] Step S33: The feature obtained in the encoding process and having the same duration as the time dimension feature is merged with the time dimension feature to obtain a time self-enhanced feature.
[0061] Step S34: Pass the enhanced features through a decoder to obtain the decoder features.
[0062] In the prior art, 2D convolution and time shift module TSM are used to jointly model spatiotemporal features to make up for the lack of 2D convolution's focus on features in the time dimension. However, this method has a flaw in that it can only focus on local period features, but cannot take into account long-term periodic features. 3D convolution can take into account features in both time and space dimensions at the same time. Previous work has shown that the encoder-decoder structure has a good effect on long-term periodic feature extraction, but because the structure compresses features in the time dimension, it leads to the loss of short-term time information. To solve this problem, we designed a jump-connected 3D convolutional network with an encoder-decoder structure. This convolutional network can focus on both short-term features of rPPG signals and long-term periodic features.
[0063] First, the spatiotemporal fusion features are compressed through multiple encoder modules to obtain the features X encoder , which is the encoder feature. The encoder module includes encoder1 and encoder2, which are composed of a pooling layer and two convolutional layers. Unlike encoder1, the convolution kernel size of the pooling layer of encoder2 is (2×2×2), and the convolution kernel size in the convolution layer is (3×3×3). Since compression in the time dimension may cause loss of short-term information, we use skip connections to solve this problem. Before the feature passes through the decoder, we add the encoder part feature with the same time length to the current feature and pass the average pooling layer to the encoder feature X encoder Perform spatial scale reduction to match the decoder feature X decoder Alignment. The decoder features obtained can express the short-term characteristics of the rPPG signal and take into account the long-term periodic characteristics.
[0064] Step S4: extracting rPPG signal from the decoder features.
[0065] There are many prior art methods for extracting rPPG signals from decoder features, which will not be repeated here. The present application uses a Predictor module to extract rPPG signals from the decoder features.
[0066] This application enhances the perception of rPPG signals by fusing the original frames and difference frames of facial videos, reducing the impact of noise on signal extraction, thereby solving the problem of weak signals and severe noise interference in videos. It also uses jump connections to repair the feature loss caused by the encoder during the compression process of the time dimension.
[0067] Embodiment 2:
[0068] like Figure 2 As shown, the present application also discloses a neural network training method, which can be used to execute the steps described in Example 1 to achieve the technical effect of the method described in Example 1, and the neural network model is only a comprehensive expression and application of the method described in Example 1, and Example 1 can also have other expressions and implementation methods. The training method includes:
[0069] Step S5: Obtain the training video to be processed.
[0070] Step S6: cropping the training video to obtain a training set.
[0071] In some embodiments, step S6 of “cropping the training video to obtain a training set” includes:
[0072] Step S61: determining facial landmarks according to the training video.
[0073] Step S62: Determine the facial center point of each frame according to the facial landmarks.
[0074] Step S63: determining an initial border of each frame according to the facial center point.
[0075] Step S64: cropping the facial region of the training video according to the facial center point and the initial frame to form a training set.
[0076] Since the videos in the dataset contain a lot of background noise that is unrelated to the rPPG signal, the dataset is preprocessed. The datasets used in the present invention are PURE, UBFC-rPPG and COHFACE. In order to reduce the computational cost and improve the efficiency of model training, we cropped the video sequence. First, we used OpenFace to generate 68 facial landmarks and determined the center point of each frame of the face by calculating the minimum bounding rectangle of these landmarks. Then, we established the initial bounding box based on these center points and a fixed bounding box size. In order to ensure that the bounding box can contain key parts such as the chin, we set the size of the bounding box to 1.2 times the vertical coordinate range and keep it unchanged throughout the processing. Next, in each subsequent frame, we cropped the facial area according to the determined facial center point and the fixed bounding box size. Finally, we adjusted the cropped facial area to a size of 128×128 and input it into our model for processing. Through such preprocessing steps, we can ensure the consistency and standardization of the input data and provide better conditions for model training.
[0077] Step S7: input the training set into the neural network model to obtain the output rPPG signal of the neural network model.
[0078] The neural network model mainly includes: spatiotemporal feature fusion module and convolution module.
[0079] like Figure 3 As shown, the moment feature fusion module includes: a difference feature module, a first signal capture module, a merging module, a second signal capture module and a fusion module.
[0080] The difference feature module is used to generate multiple time-series displacement videos based on the face video, determine the difference frame of two time-series displacement videos that are temporally adjacent, and connect the multiple difference frames in the channel dimension to form a difference feature.
[0081] Suppose the dimension of the input RGB video sequence x is N×3×T×128×128, where N is the batch size, 3 is the number of channels, T is the time, and 128 is the spatial size. The difference feature module first obtains x by temporal displacement of the input video sequence. (t-2) 、x (t-1) 、x (t) 、x (t+1) 、x (t+2) Next, calculate the difference frames in time order, that is, calculate x (t+2) -x (t+1) 、x (t+1) -x (t) 、x (t) -x (t-1) 、x(t-1) -x (t-2) , D1, D2, D3 and D4 are obtained respectively, and the obtained difference frames are connected in the channel dimension to form the difference features.
[0082] Among them, x t-2 Indicates moving the video sequence backward by two frames in the time dimension, and filling the two empty frames with the first frame. t-1 Similar. t is the original input video sequence, x t+1 Indicates moving the video sequence forward one frame in the time dimension, and filling the empty frame with the last frame. t+2 similar.
[0083] The first signal capture module is connected to the difference feature module, and is used to obtain a first waveform feature according to the difference feature, and to form a second waveform feature according to the face video. The merging module is connected to the first signal capture module, and is used to merge the first waveform feature and the second waveform feature to form a merged waveform feature. The second signal capture module is connected to the merging module and the first signal capture module, and is used to obtain a first enhanced feature according to the merged waveform feature, and to obtain a second enhanced feature according to the second waveform feature. The fusion module is connected to the second signal capture module, and is used to fuse the first enhanced feature and the second enhanced feature to obtain a spatiotemporal fusion feature.
[0084] The first signal capture module is a collection of operations including 2D convolution with a convolution kernel size of (5×5), batch normalization (BN), Tanh activation function, and MaxPool2D. The second signal capture module is a collection of operations including 2D convolution with a convolution kernel size of (3×3), batch normalization (BN), Tanh activation function, and MaxPool2D.
[0085] The calculation of the fusion module and the first difference module is as follows:
[0086] X dif =Stem1(Concat(D1,D2,D3,D4));
[0087] X origin =Stem1(X t );
[0088] X fusion =(1-λ)Stem2(λ×X origin +(1-λ)×X dif )+λ×Stem2(X origin)
[0089] X difRepresents the first waveform feature, X origin represents the second waveform feature, λ represents the weight coefficient, X fusion Represents the spatiotemporal fusion feature. Stem1 represents the first signal capture module, and Stem2 represents the second signal capture module.
[0090] like Figure 4 As shown, the workflow of the spatiotemporal feature fusion module is as follows:
[0091] The difference feature module forms multiple time-series videos based on the face video, forms multiple difference frames based on the multiple time-series videos, and then connects the multiple difference frames in the channel dimension to form a difference feature. Then, the first waveform feature and the second waveform feature are calculated based on the difference feature and the face video respectively by the first signal capture module. Then, the first waveform feature and the second waveform feature are merged and input into the second signal capture module to obtain the first enhanced feature, and the second waveform feature is input into the second signal capture module to obtain the second enhanced feature, and then the first enhanced feature and the second enhanced feature are fused to obtain the spatiotemporal fusion feature.
[0092] like Figure 5 As shown, the convolution module includes: a first encoding module, a second encoding module, an alignment encoding module, a first upsampling module, a first encoding module, a second upsampling module and a second encoding module.
[0093] The first encoding module is connected to the fusion module, and is used to obtain the first encoding feature according to the fusion module. The second encoding module is connected to the first encoding module, and is used to obtain the second encoding feature according to the first encoding feature. The alignment encoding module is connected to the second encoding module, and is used to spatially compress the second encoding feature to align it with the decoding feature to generate a third encoding feature. The first upsampling module is connected to the comparison encoding module, and extracts the first time dimension feature from the third encoding feature. The first decoding module is connected to the first upsampling module and the second encoding module, and is used to generate the first decoding feature according to the first time dimension feature and the second encoding feature. The second upsampling module is connected to the first decoding module, and is used to extract the second time dimension feature from the first decoding feature. The second decoding module is connected to the second upsampling module and the first encoding module, and is used to determine the second decoding feature according to the first encoding feature and the second time dimension feature.
[0094] Take the spatiotemporal fusion feature with a dimension of N×32×T×64×64 as an example, where N represents the batch size, 32 represents the number of channels, T represents time, and 64 represents the spatial size. The dimension of the first encoded feature obtained by the first encoding module is N×64×T×32×32, the dimension of the second encoded feature obtained by the second encoding module is N×64×T / 2×16×16, and the dimension of the third encoded feature after the alignment encoding module is N×64×T / 4×4×4.
[0095] After sampling by the first upsampling module, the dimension of the first time dimension feature is N×64×T / 2×4×4. At this time, the duration of the first time dimension feature is the same as the duration of the second coding feature, so the two can be merged and then input into the first decoding module, and then the first decoding feature with a dimension of N×64×T / 2×4×4 is obtained. Then the second upsampling module upsamples the first decoding feature to obtain a second time dimension feature with a dimension of N×64×T×4×4. At this time, the duration of the second time dimension feature is the same as the duration of the first coding feature, so the two can be merged and then input into the second coding module, and then the second decoding feature is obtained.
[0096] Therefore, in the convolution module, the spatiotemporal fusion features are compressed through multiple encoder modules to obtain the features X encoder , which is the encoder feature. The encoder module includes encoder1 and encoder2, which are composed of a pooling layer and two convolutional layers. Unlike encoder1, the convolution kernel size of the pooling layer of encoder2 is (2×2×2), and the convolution kernel size in the convolution layer is (3×3×3). Since compression in the time dimension may cause loss of short-term information, we use skip connections to solve this problem. Before the feature passes through the decoder, we add the encoder part feature with the same time length to the current feature and pass the average pooling layer to the encoder feature X encoder Perform spatial scale reduction to match the decoder feature X decoder Alignment. The decoder features obtained can express the short-term characteristics of the rPPG signal and take into account the long-term periodic characteristics.
[0097] Step S8: Determine the self-similarity loss of the neural network model based on the rPPG signal and the label of the training set.
[0098] In some embodiments, step S8 “determining the self-similarity loss of the neural network model according to the rPPG signal and the label of the training set” includes:
[0099] Step S81: randomly sampling the rPPG signal and the label through a time sampler to obtain a plurality of signal samples and label samples with the same duration.
[0100] Step S82: Determine the signal power spectrum density according to the signal samples.
[0101] Step S83: determining the tag power spectrum density according to the tag samples.
[0102] Step S84: Determine the self-similarity loss according to the signal power spectrum density and the tag power spectrum density.
[0103] There is a phase shift between the measured rPPG signal and the label. This phase shift will reduce the accuracy of the model and the speed of model convergence. By observing the power spectral density (PSD) of the extracted rPPG signal and the label on random segments, we found that there is a strong similarity between them.
[0104] Therefore, in order to reduce the impact of phase offset on model training, we proposed a self-similar loss to assist model training. First, we perform spatiotemporal sampling on the measured rPPG signals and labels. Specifically, for rPPG signals and labels of length T, we randomly sample through a time sampler to obtain K rPPG signal samples of length Δt and K label samples of length Δt. The start time of each sample is random, and Δt represents the duration of the sampling, which is T / 2. After K samplings, we obtain two groups of samples, namely [S1,...,Sk] and [S'1,...,S'K]. Subsequently, these two groups of samples are converted into power spectral density (PSD), namely [p1,...,pk] and [p'1,...,p'k], respectively.
[0105] In order to reduce the impact of the phase offset model training in the presence of labels, we designed a self-similarity loss, which uses the mean square error to measure similarity. For rPPG signals and labels, we calculated their power spectral density (PSD) in different time periods respectively. Under the similarity assumption, the PSD should be close in different time periods. Therefore, we measure the similarity of PSD using a loss function. Specifically, for rPPG signals, we calculated the PSD in different time periods and compared the similarities between them. The same operation is performed for labels. Finally, we averaged the self-similarity losses of rPPG signals and labels as the total similarity loss.
[0106] The specific self-similar loss is calculated as follows:
[0107] For rPPG signals:
[0108] PSD{S(t1→t1+Δt)}≈PSD{S(t2→t2+Δt)}→p i ≈p j , i≠j
[0109]
[0110] For the tags:
[0111] PSD{S′(t1→t1+Δt)}≈PSD{S′(t2→t2+Δt)}→p i ′≈p j ′, i≠j
[0112]
[0113] Therefore, the total self-similarity loss can be expressed as:
[0114]
[0115] A self-similar mechanism is introduced, which includes rPPG self-similar loss and label self-similar loss. Through the self-similar mechanism, the impact of phase shift on network training can be effectively reduced.
[0116] Step S9: Determine the Pearson loss of the neural network model according to the rPPG signal and the true value of the training set.
[0117] Step S10: adjusting the neural network model according to the Pearson loss and the self-similarity loss.
[0118] The design of the overall loss function (Overal|LossFunction) is as follows:
[0119] Loss=Loss SS +Loss NP
[0120]
[0121] Where T is the signal length, S is the extracted rPPG signal, and S′ is the true value. SS Represents self-similarity loss, Loss NP stands for negative Pearson loss.
[0122] Therefore, the training method of the present application and the neural network model obtained by the training method can utilize the spatiotemporal feature fusion module to greatly reduce the impact of noise on signal extraction and enhance the network's perception of rPPG fluctuations. This module significantly improves the performance of extracting rPPG signals. At the same time, a self-similar mechanism is introduced, including rPPG self-similar loss and label self-similar loss. Through the self-similar mechanism, the impact of phase shift on network training can be effectively reduced.
[0123] Embodiment 3:
[0124] Based on the foregoing embodiments, the embodiments of the present application provide an apparatus for extracting rPPG signals. The modules included in the apparatus and the units included in the modules can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.
[0125] like Figure 6 As shown, a device for extracting rPPG signals includes: a first execution module 1, a first determination module 2, a second determination module 3 and a second execution module 4.
[0126] The first execution module 1 is used to obtain a target face video and determine the difference features of the target face video according to the target face video. The first determination module 2 is used to determine the spatiotemporal fusion features of the target face video according to the difference features and the target face video. The second determination module 3 is used to determine the decoder features according to the spatiotemporal fusion features. The second execution module 4 is used to extract the rPPG signal from the decoder features.
[0127] Each module in the above-mentioned rPPG signal extraction device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the device in the form of hardware, or can be stored in the memory in the processing device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation.
[0128] Embodiment 4:
[0129] In the first aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.
[0130] Embodiment 5:
[0131] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.
[0132] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0133] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.
[0134] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0135] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0136] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. They may be located in one place or distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0137] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit. The above integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0138] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the above method embodiments are executed. The above storage medium includes: a mobile storage device, a read-only memory (ROM), a magnetic disk or an optical disk, and other media that can store program codes.
[0139] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a controller to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0140] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for extracting rPPG signals, characterized in that: include: Acquire a target face video, and determine a difference feature of the target face video based on the target face video; Determining the spatiotemporal fusion features of the target face video according to the difference features and the target face video; Determining decoder features according to the spatiotemporal fusion features; An rPPG signal is extracted from the decoder features.
2. The method according to claim 1, characterized in that The determining the difference feature of the target face video according to the target face video includes: Generate multiple time-series displacement videos according to the target face video; Determine the difference frame of the target face video according to the multiple time-series shifted videos; A plurality of the difference frames are connected in the channel dimension to form the difference feature.
3. The method according to claim 1, characterized in that The step of determining the spatiotemporal fusion features of the target face video according to the difference features and the target face video includes: Determine a first waveform feature according to the difference feature; Determining a second waveform feature according to the target face video; Determine a combined waveform feature according to the first waveform feature and the second waveform feature; Determining a first enhanced feature according to the combined waveform feature; determining a second enhanced feature according to the second waveform feature; The spatiotemporal fusion feature is determined according to the first enhanced feature and the second enhanced feature.
4. The method according to claim 1, characterized in that: The determining of decoder features according to the spatiotemporal fusion features comprises: Passing the spatiotemporal fusion features through a plurality of encoder modules to obtain encoder features; Extracting time dimension features from the encoder features; The feature with the same duration as the time dimension feature obtained in the encoding process is fused with the time dimension feature to obtain a time self-reading enhanced feature; The enhanced feature is passed through a decoder to obtain the decoder feature.
5. A training method for a neural network model, characterized in that: The neural network model is used to perform the steps described in any one of claims 1 to 4, and the method includes: Get the training video to be processed; Crop the training video to obtain a training set; Input the training set into the neural network model to obtain an output rPPG signal of the neural network model; Determining a self-similarity loss of the neural network model based on the rPPG signal and the label of the training set; Determining a Pearson loss of the neural network model according to the rPPG signal and the true value of the training set; The neural network model is adjusted according to the Pearson loss and the self-similarity loss.
6. The method according to claim 5, characterized in that The step of cropping the training video to obtain a training set includes: determining facial landmarks based on the training video; Determine a facial center point of each frame according to the facial landmarks; Determine an initial border of each frame according to the facial center point; The facial region of the training video is cropped according to the facial center and the initial frame to form a training set.
7. The method according to claim 5, characterized in that Determining the self-similarity loss of the neural network model according to the rPPG signal and the label of the training set includes: Randomly sampling the rPPG signal and the label through a time sampler to obtain a plurality of signal samples and label samples with the same duration; determining a signal power spectral density based on the signal samples; Determining a tag power spectral density according to the tag sample; The self-similarity loss is determined according to the signal power spectral density and the tag power spectral density.
8. A device for extracting rPPG signals, characterized in that: include: A first execution module is used to obtain a target face video and determine a difference feature of the target face video according to the target face video; A first determination module, configured to determine the spatiotemporal fusion features of the target face video according to the difference features and the target face video; A second determination module, configured to determine decoder features according to the spatiotemporal fusion features; The second execution module is used to extract the rPPG signal from the decoder feature.
9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 4.
10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.
Citation Information
Patent Citations
End-to-end remote heart rate detection method based on channel enhanced space-time attention network
CN114912487A
Self-supervision pre-training method for remote physiological measurement based on mask auto-encoder
CN116385837A
Non-contact physiological signal detection method and device based on fusion feature enhancement
CN117694845A
Cited By
Method, device and equipment for extracting rPPG signal, medium and program product
CN119672612A
Remote photoplethysmography method and system based on long and short term space-time convolution network
CN120299077A
Remote photoplethysmography method and system based on long short-term spatio-temporal convolutional network
CN120299077B
