Face video-based pulse waveform measurement method and device, and electronic equipment

Abstract

The application discloses a pulse waveform measurement method and device based on a face video and electronic equipment, and comprises the following steps: extracting an original rPPG signal from a face video; constructing and training a neural network based on a Transformer, wherein the neural network is mainly stacked vertically by a plurality of Transformer modules, and the Transformer module comprises: one-dimensional Patch Embedding, which is used for converting the dimension of the original rPPG signal to generate tokens that can be accepted by a subsequent Transformer encoder; a Transformer encoder containing a multi-head self-attention module, which is used for establishing a global connection of a signal time domain; and a signal restoration transformation, which is used for converting the output of the Transformer encoder and restoring it to a one-dimensional time signal; and the original rPPG signal is input into the trained neural network based on the Transformer for enhancement and noise reduction processing, so as to obtain a one-dimensional time signal, namely a pulse waveform, after processing. The application improves the measurement accuracy.

Description

Technical Field

[0001] This application relates to the field of rPPG signal processing, and in particular to a method, apparatus, and electronic device for measuring pulse waveforms based on face video. Background Technology

[0002] The periodic beating of the heart causes periodic changes in blood volume throughout the body's blood vessels. Remote photoplethysmography (rPPG) is a method for extracting physiological signals from facial videos. This method estimates changes in blood volume in facial capillaries by analyzing subtle color changes in the facial skin region of the video, thereby extracting pulse signals. rPPG signals can be further used to predict physiological signals such as heart rate, heart rate variability, respiratory rate, blood oxygen saturation, and blood pressure.

[0003] Because the skin color changes caused by heartbeats are very subtle, the rPPG signal extracted from facial videos usually contains a lot of noise, typically from lighting variations and motion artifacts. The accuracy of rPPG signal extraction plays a crucial role in predicting heart rate, heart rate variability, respiratory rate, blood oxygen saturation, and blood pressure. However, most existing methods are directly applied to tasks such as heart rate measurement, respiratory rate measurement, and liveness detection, with relatively little work done on enhancing or denoising the rPPG signal itself. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, and electronic device for measuring pulse waveforms based on face video, which solves the problem of excessive noise in rPPG signals, improves the correlation between rPPG signals and real signals, and can be combined with traditional methods to further improve the prediction accuracy of rPPG signals.

[0005] According to a first aspect of the embodiments of this application, a method for measuring pulse waveform based on face video is provided, comprising:

[0006] Extracting raw rPPG signals from face videos;

[0007] A Transformer-based neural network is constructed and trained. The neural network is mainly composed of multiple Transformer modules stacked vertically. The Transformer modules include: a one-dimensional patch embedding, used to transform the dimension of the original rPPG signal to generate tokens that can be accepted by the subsequent Transformer encoder; a Transformer encoder containing a multi-head self-attention module, used to enable the network to establish global connections in the time domain of the signal; and a signal restoration transformation, used to transform the dimension of the output of the Transformer encoder to restore it to a one-dimensional time signal, i.e., a pulse waveform.

[0008] The original rPPG signal is input into a trained Transformer-based neural network for enhancement and noise reduction to obtain a processed one-dimensional time signal, i.e., a pulse waveform.

[0009] Furthermore, the raw rPPG signal is extracted from the face video, including:

[0010] Use OpenFace to extract facial keypoint coordinates from video;

[0011] Based on the coordinates of the key points, a region of interest (ROI) is constructed.

[0012] Calculate the average pixel color in each ROI of each frame in the video to generate a temporal signal;

[0013] The time-domain signal is processed using a filter to obtain the original rPPG signal.

[0014] Furthermore, when training the Transformer-based neural network, the shift consistency correlation coefficient (shift CCC) is used as a metric and the loss function is constructed.

[0015] Furthermore, the shift consistency correlation coefficient (shift CCC) is used as a metric and a loss function is constructed, including:

[0016]

[0017] In the formula, Indicates will The vector is cyclically shifted to the right t times, that is...

[0018] ρ y Let μ be the standard deviation of vector y. y Let ρ be the mean of vector y, and let ρ(·,·) be the correlation coefficient between the two variables.

[0019] According to a second aspect of the embodiments of this application, a pulse waveform measurement device based on face video is provided, comprising:

[0020] The extraction module is used to extract the raw rPPG signal from face videos;

[0021] A construction and training module is used to construct and train a Transformer-based neural network. The neural network is mainly composed of multiple Transformer modules stacked vertically. The Transformer modules include: a one-dimensional PatchEmbedding, used to transform the dimension of the original rPPG signal to generate tokens that can be accepted by the subsequent Transformer encoder; a Transformer encoder containing a multi-head self-attention module, used to enable the network to establish global connections in the time domain of the signal; and a signal restoration transformation, used to transform the dimension of the output of the Transformer encoder to restore it to a one-dimensional time signal, i.e., a pulse waveform.

[0022] The processing module is used to input the original rPPG signal into a trained Transformer-based neural network for enhancement and noise reduction processing to obtain a processed one-dimensional time signal, i.e., a pulse waveform.

[0023] According to a second aspect of the embodiments of this application, an electronic device is provided, comprising:

[0024] One or more processors;

[0025] Memory, used to store one or more programs;

[0026] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0027] According to a second aspect of the embodiments of this application, a computer-readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0028] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0029] As can be seen from the above embodiments, this application is the first to apply the Transformer network to the direct processing of rPPG signals. By establishing a global connection between the rPPG signals in the time dimension through the Transformer, the prediction accuracy of pulse waveform signals is improved. The Pulse Transformer network proposed in this method can be combined with traditional rPPG methods such as CHROM and 2SR to further reduce noise in the rPPG signals extracted by traditional methods, thereby improving accuracy. The Pulse Transformer network proposed in this method can also be added as a module to tasks such as heart rate prediction based on rPPG, providing clean rPPG signals for downstream rPPG tasks and improving the accuracy of downstream rPPG tasks such as heart rate prediction. The motion consistency correlation coefficient proposed in this method takes into account the time difference between the heartbeat and arterial pulsations at different parts of the body, and is more suitable for measuring the accuracy of rPPG signals than the ordinary consistency correlation coefficient.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0032] Figure 1 This is a flowchart illustrating a pulse waveform measurement method based on a face video according to an exemplary embodiment.

[0033] Figure 2 This is the architecture diagram of Vision Transformer, a popular application in the field of image processing.

[0034] Figure 3 This is a diagram of the Pulse Transformer architecture shown according to an exemplary embodiment.

[0035] Figure 4 This is an encoder structure of the Pulse Transformer illustrated according to an exemplary embodiment.

[0036] Figure 5 This is a general rPPG task flowchart, demonstrating the universality and scalability of the invention. It can be combined with traditional rPPG methods such as CHROM and 2SR, or used in downstream tasks such as heart rate prediction, respiratory rate prediction, and blood oxygen saturation prediction.

[0037] Figure 6 This is a block diagram illustrating a pulse waveform measurement device based on a face video according to an exemplary embodiment. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0039] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0040] Figure 1 This is a flowchart illustrating a pulse waveform measurement method based on a face video according to an exemplary embodiment, such as... Figure 1 As shown, the method may include the following steps:

[0041] S1: Extract the raw rPPG signal from the face video;

[0042] S2: Construct and train a Transformer-based neural network (Pulse Transformer network). The neural network is mainly composed of multiple Transformer modules stacked vertically. The Transformer modules include: a one-dimensional PatchEmbedding, used to transform the dimension of the original rPPG signal to generate tokens that can be accepted by the subsequent Transformer encoder; a Transformer encoder containing a multi-head self-attention module, used to enable the network to establish global connections in the time domain of the signal; and a signal restoration transformation, used to transform the dimension of the output of the Transformer encoder and restore it to a one-dimensional time signal, i.e., a pulse waveform.

[0043] S3: Input the original rPPG signal into the trained Transformer-based neural network for enhancement and noise reduction to obtain the processed one-dimensional time signal, i.e., the pulse waveform.

[0044] As can be seen from the above technical solutions, this application is the first to apply the Transformer network to the direct processing of rPPG signals. By establishing a global connection between the rPPG signals in the time dimension through the Transformer, the prediction accuracy of the pulse waveform signal is improved. The Pulse Transformer network proposed in this method can be combined with traditional rPPG methods such as CHROM and 2SR to further reduce noise in the rPPG signals extracted by traditional methods and improve accuracy. The Pulse Transformer network proposed in this method can also be added as a module to tasks such as heart rate prediction based on rPPG, providing clean rPPG signals for downstream rPPG tasks and improving the accuracy of downstream rPPG tasks such as heart rate prediction.

[0045] In the specific implementation of S1: the raw rPPG signal is extracted from the face video;

[0046] Specifically, firstly, OpenFace is used to extract the key point coordinates of the face from the video. In this step, the video containing the face is input frame by frame into the OpenFace software package, and the key point coordinates of the face in each frame of the video image are extracted, including the positions of various organ outlines such as eyes, nose, and lips, totaling 68 key points.

[0047] Then, based on the keypoint coordinates, a Region of Interest (ROI) is constructed. Since the rPPG signal needs to be extracted from the skin, a region containing skin needs to be designated as the ROI. The principle for selecting the ROI is to include as flat as possible skin areas, as such areas are less affected by lighting and movement in terms of color change.

[0048] Next, the average pixel color of each ROI in each frame of the video is calculated to generate a temporal signal. Averaging is a common way to reduce random errors because each ROI contains many pixels, and calculating the average pixel color can reduce color variations caused by uneven facial skin.

[0049] Finally, the time-domain signal is processed using a filter to obtain the original rPPG signal. Bandpass filtering allows a portion of the signal's frequency components to pass through. Bandpass filtering preserves the 0.67–4 Hz portion of the signal, as this frequency range contains the most information in the rPPG signal and in the heart rate prediction task.

[0050] In the specific implementation of S2: a Transformer-based neural network (PulseTransformer network) is constructed and trained. The neural network is mainly composed of multiple Transformer modules stacked vertically. The Transformer modules include: a one-dimensional patch embedding, used to transform the dimension of the original rPPG signal to generate tokens that can be accepted by the subsequent Transformer encoder; a Transformer encoder containing a multi-head self-attention module, used to enable the network to establish a global connection of the signal in the time domain; and a signal restoration transformation, used to transform the dimension of the output of the Transformer encoder and restore it to a one-dimensional time signal, i.e., a pulse waveform.

[0051] Specifically, such as Figure 2 As shown, this invention provides a Pulse Transformer network. This network takes the original rPPG signal as input, segments the signal in the time dimension, performs linear mapping on the segmented time-series signal, adds position embedding to the obtained tokens, and obtains a feature sequence, which is then input into the subsequent Transformer encoder. Specifically, for the original rPPG signal X, its dimension is B×C×T×ROI, where B is the batch size, C is the number of channels, T is the time series length, and ROI is the number of regions of interest. We also consider the number of regions of interest as a type of channel, merging these two dimensions, i.e., B×C′×T. After this one-dimensional signal is segmented, the size of each segment is P, i.e. Become The segmented signal is then input into the linear mapping layer, transforming the dimension into... Where N is the token size, obtained from the linear mapping layer. Finally, positional embedding is introduced to obtain the feature sequence that can be input to the Transformer encoder.

[0052] like Figure 3As shown, the Transformer encoder used in Pulse Transformer is a pyramidal hierarchical network structured into four stages. Each stage contains a Stage Embedding layer and several recurrent Transformer Block layers. The Stage Embedding processes the feature sequence input to this stage, containing two fully connected layers to handle the token size of the feature sequence. The recurrent Transformer Blocks within each stage are identical and used to establish global attention. As the network deepens, the computation becomes increasingly complex in later stages, reflected in the number of multi-head attention heads in the Transformer encoder, the depth of the recurrent Transformer Block network layers, the token size, and the hidden layer dimensions. Specific network parameters are as follows: Figure 3 As shown. Similar to the design of most convolutional neural networks, the connections between network relationships are mainly in the earlier stages, while the computational load of the network is mainly in the later stages.

[0053] When training a Transformer-based neural network, the shift consistency correlation coefficient (shift CCC) is used as a metric and to construct the loss function.

[0054] This method proposes a metric for measuring the accuracy of pulse waveform measurement methods, called the Shift Consistency Correlation Coefficient (Shift CCC). Its key feature is that it considers the time difference between the heartbeat and arterial pulsations at different locations in the body. Therefore, it calculates the correlation between the measured signal and the actual signal after cyclically shifting the measured signal along the time dimension, thus avoiding the influence of pulse time difference on the results. Specifically, the calculation method is as follows:

[0055]

[0056] in, Indicates will The vector is cyclically shifted to the right t times, that is:

[0057]

[0058] CCC(·,·) is the consistency correlation coefficient between two variables, and its calculation method is as follows:

[0059]

[0060] Where, σ y Let μ be the standard deviation of vector y. y Let ρ be the mean of vector y, and ρ(·,·) be the correlation coefficient between the two variables. Substituting the above formula, we get:

[0061]

[0062] Since a cyclic right shift in the time dimension does not change the mean and standard deviation of the vector, the coefficients in the formula... It is unrelated to t. And... It can be directly calculated by constructing a correlation matrix:

[0063]

[0064] In the specific implementation of S3: the original rPPG signal is input into a trained Transformer-based neural network for enhancement and noise reduction processing to obtain a processed one-dimensional time signal, i.e., a pulse waveform.

[0065] Specifically, based on experimental results, the Transformer-based neural network proposed in this invention can enhance and reduce noise in the original rPPG signal.

[0066] Compared with traditional methods and deep learning methods, the method of this invention achieves excellent results in pulse waveform measurement tasks based on face videos. Experiments were conducted on two datasets, MAHNOB-HCI and VIPL-HR, using the motion consistency correlation coefficient (sCCC) as a metric. The experimental results are shown in the following tables. Table 1 shows the experimental results of pulse waveform measurement on the MAHNOB-HCI dataset, where sCCC is the motion consistency correlation coefficient, with higher values ​​indicating better results. The experiments demonstrate that the present invention achieves excellent results, and its performance is further enhanced when combined with traditional methods. Table 2 shows the experimental results of pulse waveform measurement and heart rate measurement on the VIPL-HR dataset. The experiments show that the present invention can be applied to various downstream rPPG tasks, such as heart rate prediction, and its performance is better than traditional methods such as CHROM and 2SR.

[0067] Table 1 shows the experimental results of pulse waveform measurement on the MAHNOB-HCI dataset.

[0068]

[0069]

[0070] Table 2 shows the experimental results of pulse waveform measurement and heart rate measurement on the VIPL-HR dataset.

[0071]

[0072] Corresponding to the aforementioned embodiments of the pulse waveform measurement method based on face video, this application also provides embodiments of a pulse waveform measurement device based on face video.

[0073] Figure 6 This is a block diagram illustrating a pulse waveform measurement device based on a face video, according to an exemplary embodiment. (Refer to...) Figure 6 The device includes:

[0074] Extraction module 1 is used to extract the raw rPPG signal from the face video;

[0075] Module 2 is used to build and train a Transformer-based neural network. The neural network is mainly composed of multiple Transformer modules stacked vertically. The Transformer modules include: a one-dimensional PatchEmbedding, used to transform the dimension of the original rPPG signal to generate tokens that can be accepted by the subsequent Transformer encoder; a Transformer encoder containing a multi-head self-attention module, used to enable the network to establish global connections in the time domain of the signal; and a signal restoration transformation, used to transform the dimension of the output of the Transformer encoder to restore it to a one-dimensional time signal, i.e., a pulse waveform.

[0076] Processing module 3 is used to input the original rPPG signal into a trained Transformer-based neural network for enhancement and noise reduction processing to obtain a processed one-dimensional time signal, i.e., a pulse waveform.

[0077] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0078] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0079] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the pulse waveform measurement method based on face video as described above.

[0080] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the pulse waveform measurement method based on face video as described above.

[0081] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0082] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for measuring pulse waveform based on facial video, characterized in that, include: Extracting raw rPPG signals from face videos; A Transformer-based neural network is constructed and trained. The neural network is mainly composed of multiple Transformer modules stacked vertically. The Transformer modules include: a one-dimensional patch embedding, used to transform the dimension of the original rPPG signal to generate tokens that can be accepted by the subsequent Transformer encoder; a Transformer encoder containing a multi-head self-attention module, used to enable the network to establish global connections in the time domain of the signal; and a signal restoration transformation, used to transform the dimension of the output of the Transformer encoder to restore it to a one-dimensional time signal, i.e., a pulse waveform. The original rPPG signal is input into a trained Transformer-based neural network for enhancement and noise reduction to obtain a processed one-dimensional time signal, i.e., a pulse waveform. When training a Transformer-based neural network, the shift consistency correlation coefficient (shift CCC) is used as a metric and to construct the loss function, specifically including: ； In the formula, Indicates will Vector Circular Right Shift Next, that is ; For vectors standard deviation For vectors The mean, The correlation coefficient between the two variables is denoted as .

2. The method according to claim 1, characterized in that, Extracting raw rPPG signals from face videos includes: Use OpenFace to extract facial keypoint coordinates from a video; Based on the coordinates of the key points, a region of interest (ROI) is constructed. Calculate the average pixel color in each ROI of each frame in the video to generate a temporal signal; The time-domain signal is processed using a filter to obtain the original rPPG signal.

3. A pulse waveform measurement device based on facial video, characterized in that, include: An extraction module is used to extract the raw rPPG signal from face videos; A construction and training module is used to construct and train a Transformer-based neural network. The neural network is mainly composed of multiple Transformer modules stacked vertically. The Transformer modules include: a one-dimensional PatchEmbedding, used to transform the dimension of the original rPPG signal to generate tokens that can be accepted by the subsequent Transformer encoder; a Transformer encoder containing a multi-head self-attention module, used to enable the network to establish global connections in the time domain of the signal; and a signal restoration transformation, used to transform the dimension of the output of the Transformer encoder to restore it to a one-dimensional time signal, i.e., a pulse waveform. The processing module is used to input the original rPPG signal into a trained Transformer-based neural network for enhancement and noise reduction processing to obtain a processed one-dimensional time signal, i.e., a pulse waveform. When training a Transformer-based neural network, the shift consistency correlation coefficient (shift CCC) is used as a metric and to construct the loss function, specifically including: ； In the formula, Indicates will Vector Circular Right Shift Next, that is ; For vectors standard deviation For vectors The mean, The correlation coefficient between the two variables is denoted as .

4. The apparatus according to claim 3, characterized in that, Extracting raw rPPG signals from face videos includes: Use OpenFace to extract facial keypoint coordinates from a video; Based on the coordinates of the key points, a region of interest (ROI) is constructed. Calculate the average pixel color in each ROI of each frame in the video to generate a temporal signal; The time-domain signal is processed using a filter to obtain the original rPPG signal.

5. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-2.

6. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-2.

Images