Non-contact rPPG signal extraction method and system based on space-time attention

Abstract

The invention discloses a non-contact rPPG signal extraction method and system based on space-time attention. The method comprises the following steps: 1, collecting video data; 2, constructing a signal extraction model; the signal extraction model comprises a time-space attention module, a visual encoder, a self-attention feature extraction module, a time-space feedforward module and a full connection layer which are connected in sequence. The space-time attention module comprises a first space feature extraction branch and a first time sequence feature extraction branch which are parallel. And 3, outputting an rPPG signal. According to the method, the face video data is processed by using a double-path parallel space-time attention mechanism, the time sequence branch performs convolution on the adjacent frame difference features to obtain the time sequence data, and the space branch extracts the space features through deep convolution, point-by-point convolution and 2D convolution, so that the signal-to-noise ratio of the face features is improved, and the accuracy of the face feature extraction is improved. Therefore, the model focuses on the tiny change of facial physiological information, and a more accurate rPPG signal is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of electrocardiogram data collection, and in particular relates to a non-contact rPPG signal extraction method and system based on spatiotemporal attention. Background Art

[0002] With the advancement of deep learning technology, there are more and more studies combining non-contact heart rate detection with deep learning. The non-end-to-end model Synrhythm proposed by Niuxue Song et al. uses a camera to capture face videos, segments the facial information of the test subject through a face recognition model, selects a region of interest (ROI), and converts the RGB signal of the region into a space-time graph representation as the input of the deep network. However, non-end-to-end models often require some manual feature extraction and prior knowledge, and have poor generalization. The HR-CNN proposed by Špetlík et al. first used convolutional neural networks to develop two key components: a signal extractor and a heart rate meter. The method first processes the facial video sequence through a 2DCNN signal extractor to extract rPPG signals, and then the heart rate meter calculates the heart rate based on these continuous rPPG signals. Considering that 2DCNN has good results in spatial feature extraction, but ignores the time domain information, many researchers have begun to propose 3DCNN networks, such as the physformer proposed by Zitong Yu et al. In order to provide transformer with sufficient spatiotemporal information, 3D CNN is used to extract facial features through three stages, and the extracted facial features are handed over to the transformer for a longer-distance self-attention mechanism. However, at the same time, 3D CNN also brings high performance overhead, which is not conducive to mobile deployment. At the same time, the input low signal-to-noise ratio face video increases the difficulty of the model to extract signals, resulting in the transformer being unable to better focus on the weak facial rPPG signal.

[0003] The original face video contains not only weak facial physiological information, but also other information such as ambient light, motion artifacts, camera noise, etc. How to extract weak rPPG signals from face videos has become a very valuable topic. The ROI feature extraction method is often based on prior knowledge and cannot be generalized in multiple scenarios. In addition, an additional key point detection model needs to be deployed, which increases the task overhead. If the input is the original video, the low signal-to-noise ratio video makes it very difficult to extract model features. Neither 2D CNN nor 3D CNN has demonstrated strong feature extraction capabilities. Summary of the invention

[0004] The object of the present invention is to provide a non-contact rPPG signal extraction method and system based on spatiotemporal attention.

[0005] In a first aspect, the present invention provides a non-contact rPPG signal extraction method based on spatiotemporal attention, which comprises the following steps: Step 1: Collect video data of the subject's face; the video data is used to extract the subject's blood volume pulse signal.

[0006] Step 2, construct a signal extraction model. The signal extraction model includes a spatiotemporal attention module, a visual encoder, a self-attention feature extraction module, a spatiotemporal feedforward module and a fully connected layer connected in sequence. The spatiotemporal attention module includes a parallel first spatial feature extraction branch and a first temporal feature extraction branch. The first spatial feature extraction branch extracts the spatial features of the video data; the first temporal feature extraction branch takes the difference between adjacent frames of the video data to obtain the differential features of adjacent frames, and extracts temporal features based on the differential features of adjacent frames. The visual encoder is used to encode the output features of the spatiotemporal attention module; the spatiotemporal feedforward network includes a parallel second spatial feature extraction branch and a second temporal feature extraction branch. The fully connected layer is used to output the rPPG signal.

[0007] Step 3: Input the video data collected in step 1 into the signal extraction model to obtain the rPPG signal.

[0008] Preferably, the spatiotemporal attention module further comprises a first feature fusion module. The first feature fusion module combines the features extracted by the first spatial feature extraction branch and the first temporal feature extraction branch to obtain a preliminary fusion feature F concat Initial fusion feature F concat Perform 2D convolution to obtain the high-level features x of the input visual encoder STBlock .

[0009] Preferably, the first spatial feature extraction branch sequentially performs depth convolution processing, point-by-point convolution processing and 2D convolution processing on the input video sequence, and each convolution is followed by normalization and / or activation function.

[0010] Preferably, the first temporal feature extraction branch calculates the difference between adjacent frames of the input video sequence to obtain adjacent frame differential features; the adjacent frame differential features are sequentially output through two rounds of 2D convolution.

[0011] Preferably, the visual encoder adopts a 3D convolutional neural network.

[0012] Preferably, the self-attention feature extraction module maps the output features of the visual encoder through 3×3×3 convolution to obtain the query vector Q and the key vector K; maps the numerical vector V through 1×1×1 convolution; performs attention calculation on the query vector Q and the key vector K, and obtains the attention weight after normalization; and multiplies the attention weight with the numerical vector V to obtain the multi-head attention output feature.

[0013] Preferably, the second spatial feature extraction branch uses 1×3×3 deep convolution with batch normalization and GELU activation function to extract feature information of the spatial dimension. The second temporal feature extraction branch uses 3×1×1 deep convolution to process temporal information to extract feature information of the temporal dimension. The spatiotemporal feedforward network also includes a second feature fusion module. The second feature fusion module fuses the output features of the second spatial feature extraction branch and the second temporal feature extraction branch through addition, and then undergoes 1×1×1 convolution and batch normalization.

[0014] In a second aspect, the present invention provides a non-contact rPPG signal extraction system, which is used to perform the aforementioned non-contact rPPG signal extraction method based on spatiotemporal attention; the non-contact rPPG signal extraction system includes a video acquisition module, a feature extraction module and a display feedback module. The video acquisition module is used to acquire face video data; the feature extraction module is used to process the face video data through a signal extraction model and output an rPPG signal. The display feedback module is used to display the rPPG signal.

[0015] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the aforementioned non-contact rPPG signal extraction method based on spatiotemporal attention.

[0016] In a fourth aspect, the present invention provides a readable storage medium storing a computer program; when the computer program is executed by a processor, the aforementioned non-contact rPPG signal extraction method based on spatiotemporal attention is implemented.

[0017] The present invention has the following beneficial effects: 1. The present invention uses a dual-path parallel spatiotemporal attention mechanism to process face video data. Its temporal branch convolves the differential features of adjacent frames to obtain temporal data, and the spatial branch extracts spatial features through deep convolution, point-by-point convolution and 2D convolution to improve the signal-to-noise ratio of facial features, so that the model focuses on subtle changes in facial physiological information and obtains more accurate rPPG signals.

[0018] 2. The deep separable convolution in the spatial branch of the spatiotemporal attention mechanism of the present invention can better retain the subtle blood flow characteristics in each color channel by first performing channel-independent deep convolution and then integrating channel information with point-by-point convolution, while filtering out irrelevant environmental interference, thereby effectively extracting weak rPPG signals in the original face video and reducing the influence of interference factors such as ambient light, motion artifacts and camera noise.

[0019] 3. The present invention can significantly reduce the number of parameters of the signal extraction model, making the model of the present invention lightweight and suitable for real-time rPPG signal extraction on mobile devices. The dual-path parallel spatiotemporal attention mechanism design avoids the direct use of 3D convolution, thereby reducing the dimension of the convolution kernel. At the same time, the depthwise separable convolution decomposes the standard 2D convolution into depthwise convolution and pointwise convolution, while maintaining good feature extraction capabilities, it also significantly reduces the number of parameters and calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a system block diagram of the signal extraction model used in the present invention.

[0021] Figure 2 Schematic diagram of the network framework of the spatiotemporal attention module in the present invention.

[0022] Figure 3 Schematic diagram of the network framework of the self-attention feature extraction module in the present invention.

[0023] Figure 4 Schematic diagram of the network framework of the spatiotemporal feedforward module in the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings.

[0025] like Figure 1 As shown, a non-contact rPPG signal extraction method based on spatiotemporal attention includes the following steps: Step 1: Build a dataset containing face videos and rPPG signals.

[0026] Step 2: Build a signal extraction model.

[0027] The signal extraction model includes a spatiotemporal attention module, a visual encoder, a self-attention feature extraction module, a spatiotemporal feedforward module and a fully connected layer connected in sequence. The self-attention feature extraction module adopts a transformer model. During the signal extraction process, the spatiotemporal attention module first focuses on the changes in facial physiological signals over time, and then the output of the spatiotemporal attention module is handed over to the visual encoder for encoding to facilitate the subsequent self-attention feature extraction module to better perform the self-attention mechanism. Since the spatiotemporal attention module pays more attention to physiological signals, the self-attention feature extraction module's attention to other information is greatly reduced, and the extraction efficiency of physiological signals is improved; the spatiotemporal feedforward module will further process the high-level features output by the self-attention feature extraction module, perform the last step of feature analysis on the physiological signals, and finally output the rPPG signal through a fully connected layer.

[0028] like Figure 2 As shown, the spatiotemporal attention module adopts a dual-path parallel architecture design, including a first spatial feature extraction branch, a first temporal feature extraction branch and a first feature fusion module. The first spatial feature extraction branch and the first temporal feature extraction branch are parallel structures, and the output features are fused through the first feature fusion module.

[0029] The first spatial feature extraction branch and the first temporal feature extraction branch both receive an input video sequence ; Where N is the batch size; D is the video sequence length; C is the number of input channels; H and W are the height and width of the feature map, respectively.

[0030] In the first spatial feature extraction branch, the input video sequence is first subjected to 3×3 deep convolution and normalization and activation function, and then 1×1 point-by-point convolution is used to adjust the number of channels and perform cross-channel information fusion; followed by normalization and 3×3 2D convolution, and finally output after normalization and activation function.

[0031] In the first temporal feature extraction branch, the differential feature extraction mechanism is used to differenciate adjacent frames of the input video sequence to obtain adjacent frame differential features; the adjacent frame differential features are sequentially output after two rounds of 3×3 2D convolution, normalization and activation function to achieve further extraction of temporal features.

[0032] The differential feature extraction mechanism is specifically for each frame image x in the input video sequence t , both of which are calculated with the previous frame image x t-1 The differential characteristic D t as follows:

[0033] The differential feature extraction mechanism captures the information that changes over time in the video by taking the difference between two adjacent frames; the information that changes over time contains the rPPG signal of the face.

[0034] The first feature fusion module concatenates and fuses the features extracted by the first spatial feature extraction branch and the first temporal feature extraction branch to obtain a preliminary fusion feature F concat as follows:

[0035] F spatial is the output feature of the first spatial feature extraction branch; F temp is the output feature of the first temporal feature extraction branch; concat(·,·) is the concatenation operation.

[0036] The feature fusion module sequentially fused the initial features F concat Perform a round of 1×1 2D convolution, normalization and activation function, and a round of 3×3 2D convolution, normalization and activation function to achieve the initial fusion feature F concat Further strengthening of STBlock as follows:

[0037] Advanced Featuresx STBlock The dimension is .

[0038] The high-level features x output by the feature fusion module STBlock Enter the visual encoder. The visual encoder includes a 1×4×4 3D convolutional neural network, whose main function is to encode the features, which is more conducive to the calculation of the subsequent self-attention feature extraction module.

[0039] The visual encoder outputs features x encoder The dimension is .

[0040] like Figure 3 As shown, the visual encoder outputs the feature x encoder Input the self-attention feature extraction module, and map the query vector Q, key vector K, and value vector V as follows:

[0041]

[0042] Among them, proj q (·) and proj k (·) are linear transformations for obtaining query vector Q and key vector K respectively; proj q (·) and projk The structure of (·) is the same, specifically, the mapping is established through a 3×3×3 convolution.

[0043] The numerical vector V is obtained by encoder A 1×1×1 convolution is performed to establish a mapping. An attention calculation is performed on the query vector Q and the key vector K. The attention weight is calculated through a Softmax function and then multiplied with the numerical vector V to obtain the multi-head attention output feature of the self-attention feature extraction module.

[0044] like Figure 4 As shown, the spatiotemporal feedforward network adopts a dual-path parallel architecture design of spatiotemporal attention, including a second spatial feature extraction branch, a second temporal feature extraction branch and a second feature fusion module.

[0045] In the second spatial feature extraction branch, 1×3×3 deep convolution is used for processing, combined with batch normalization (BatchNorm) and GELU activation function, mainly to capture feature information in the spatial dimension.

[0046] In the second temporal feature extraction branch, a 3×1×1 deep convolution is used to process the temporal information and effectively capture the time-varying features in the video sequence.

[0047] The second feature fusion module performs feature fusion on the output features of the second spatial feature extraction branch and the second temporal feature extraction branch through addition operation, and then performs 1×1×1 convolution and batch normalization to obtain the output of the spatiotemporal feedforward network.

[0048] Finally, the high-level features output by the spatiotemporal feedforward network are input into the fully connected layer to obtain the rPPG signal.

[0049] Step 3: Use the data set in step 1 to train the signal extraction model.

[0050] Step 4: Collect the subject's facial video and input the obtained facial video into the signal extraction model. The signal extraction model outputs the subject's rPPG signal.

[0051] The reason why the dual-path feature fusion structure used by the spatiotemporal attention module and the spatiotemporal feedforward module in this embodiment is particularly effective in face video heart rate detection is mainly because the uniqueness of this task perfectly matches the advantages of the dual-path architecture. First, in view of the extremely weak characteristics of the rPPG signal, which only accounts for about 0.1% of the original pixel value, this embodiment simultaneously enhances the signal from both spatial and temporal dimensions. The spatial path can accurately locate the optimal signal acquisition locations in the areas with rich facial blood vessels, while the temporal path focuses on amplifying the subtle changes in these areas over time. At the same time, there are inevitably interference sources such as ambient light changes and human body movements in the process of face video acquisition, and these interferences have different manifestations in spatial and temporal dimensions. The dual-path architecture can accurately extract weak heart rate signals in a strong noise environment by processing these interferences of different natures separately, and then combining the inherent periodic characteristics of the heart rate signal (0.7Hz-4Hz) and the anatomical characteristics of the distribution of facial blood vessels. This feature makes the dual-path feature fusion show unique advantages in rPPG tasks, far exceeding its general application in other fields.

[0052] In the existing extraction methods, some people have tried to use a fixed window difference calculation method to extract time series features, which may lose important time series information. The rPPG signal is essentially a continuous physiological signal, and the change in blood volume is gradual frame by frame; therefore, this embodiment calculates the difference between adjacent frames, which is more streamlined and can better capture this gradual feature. It can also process video sequences of any length without being limited by the number of input video frames.

[0053] In addition, the deep separable convolution in the spatial channel has unique advantages in rPPG signal extraction, which stems from the special nature of the rPPG signal: it reflects the small periodic changes in blood volume in the skin, which present different but related patterns in the three RGB channels. Traditional convolution processes the characteristics of spatial and channel dimensions at the same time, which easily blurs the subtle differences between channels; while the deep separable convolution can better preserve the subtle blood flow characteristics in each color channel while filtering out irrelevant environmental interference by first performing channel-independent deep convolution and then integrating channel information with point-by-point convolution. This ability to accurately model signal features, coupled with the advantage of lightweight models brought about by reducing the number of parameters to 1 / N of traditional convolution, makes it particularly suitable for real-time rPPG signal extraction on mobile devices.

[0054] In order to test the effectiveness of the rPPG signal extraction method provided by the present invention, four comparative experiments were carried out in this embodiment: Experiment 1: training on PURE data and testing on UBFC-rPPG data set; Experiment 2: training on PURE data and testing on MMPD data set; Experiment 3: training on UBFC-rPPG data and testing on UPURE data set; Experiment 4: training on UBFC-rPPG data and testing on MMPD data set.

[0055] Four existing extraction methods (Deepphys, Physnet, TS-CAN, and Physformer) were set as a control group; the experimental results are shown in Table 1 below; it can be seen that the effect of this embodiment in extracting human rPPG signals using face videos is better than that of the existing extraction methods.

[0056] Table 1 Cross-dataset test comparison table

[0057] In addition, the comparison of the computational overhead of this embodiment and the existing extraction method is shown in Table 2 below; it can be seen from Table 2 that this embodiment significantly reduces the computational overhead while accurately extracting the rPPG signal.

[0058] Much smaller than past inventions.

[0059] Table 2 Computational cost comparison table

Claims

1. A non-contact rPPG signal extraction method based on spatiotemporal attention, characterized in that: The following steps are involved: Step 1: Collect video data; Step 2, construct a signal extraction model; the signal extraction model includes a spatiotemporal attention module, a visual encoder, a self-attention feature extraction module, a spatiotemporal feedforward module and a fully connected layer connected in sequence; the spatiotemporal attention module includes a parallel first spatial feature extraction branch and a first temporal feature extraction branch; the first spatial feature extraction branch extracts the spatial features of the video data; the first temporal feature extraction branch obtains the difference features of adjacent frames of the video data by taking the difference, and extracts the temporal features for the adjacent frame differential features; the spatiotemporal feedforward network includes a parallel second spatial feature extraction branch and a second temporal feature extraction branch; The fully connected layer is used to output the rPPG signal; Step 3: Input the video data collected in step 1 into the signal extraction model to obtain the rPPG signal.

2. The non-contact rPPG signal extraction method based on spatiotemporal attention according to claim 1, characterized in that: The spatiotemporal attention module also includes a first feature fusion module; the first feature fusion module splices the features extracted by the first spatial feature extraction branch and the first temporal feature extraction branch to obtain a preliminary fusion feature F concat ; Initial fusion feature F concat Perform 2D convolution to obtain high-level features for input into the visual encoder.

3. The non-contact rPPG signal extraction method based on spatiotemporal attention according to claim 1, characterized in that: The first spatial feature extraction branch sequentially performs depth convolution processing, point-by-point convolution processing and 2D convolution processing on the input video sequence; each convolution is followed by normalization and / or activation function.

4. The non-contact rPPG signal extraction method based on spatiotemporal attention according to claim 1, characterized in that: The first temporal feature extraction branch calculates the difference between adjacent frames of the input video sequence to obtain adjacent frame differential features; The differential features of adjacent frames are output through two rounds of 2D convolution in sequence.

5. The non-contact rPPG signal extraction method based on spatiotemporal attention according to claim 1, characterized in that: The visual encoder adopts a 3D convolutional neural network.

6. The non-contact rPPG signal extraction method based on spatiotemporal attention according to claim 1, characterized in that: The self-attention feature extraction module maps the output features of the visual encoder through 3×3×3 convolution to obtain the query vector Q and the key vector K; and maps the output features of the visual encoder through 1×1×1 convolution to obtain the numerical vector V; Attention is calculated for the query vector Q and the key vector K, and the attention weight is obtained after normalization; the multi-head attention output feature is obtained by multiplying the attention weight with the numerical vector V.

7. The non-contact rPPG signal extraction method based on spatiotemporal attention according to claim 1, characterized in that: The second spatial feature extraction branch uses 1×3×3 deep convolution with batch normalization and GELU activation function to extract feature information of the spatial dimension; the second temporal feature extraction branch uses 3×1×1 deep convolution to process temporal information to extract feature information of the temporal dimension; the spatiotemporal feedforward network also includes a second feature fusion module; the second feature fusion module fuses the output features of the second spatial feature extraction branch and the second temporal feature extraction branch through addition, and then undergoes 1×1×1 convolution and batch normalization.

8. A non-contact rPPG signal extraction system, characterized in that: Used to execute a non-contact rPPG signal extraction method based on spatiotemporal attention as described in claim 1; the non-contact rPPG signal extraction system includes a video acquisition module, a feature extraction module and a display feedback module; the video acquisition module is used to acquire face video data; the feature extraction module is used to process the face video data through a signal extraction model and output an rPPG signal; the display feedback module is used to display the rPPG signal.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The memory stores a computer program; the processor executes a non-contact rPPG signal extraction method based on spatiotemporal attention as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program; characterized in that: When the computer program is executed by a processor, it is used to implement a non-contact rPPG signal extraction method based on spatiotemporal attention as described in any one of claims 1 to 7.

Images