Time-space domain weighted photoelectric volume pulse wave recovery method based on face video

Abstract

The invention discloses a time-space domain weighted photoelectric volume pulse wave recovery method based on a face video. A model involved in the method mainly comprises a signal extraction module, a signal quality scoring module and a signal enhancement module. The method comprises the following steps: firstly, performing face detection and cutting on a face video to obtain a video only containing a face as input; the signal extraction module calculates an input frame difference map, so that a spatial domain weight is extracted to weight an input feature, a face non-motion region is concerned from a spatial domain, and a pulse wave signal is further preliminarily extracted from a weighted signal; the signal quality scoring module scores the pulse wave signals in a segmented manner; and the signal enhancement module distributes a higher attention weight value to the signal segment with the low score, pays attention to the part with poor signal quality from a time domain, and outputs a recovered high-quality pulse wave signal. According to the method, the motion interference in the imaging type photoelectric volume pulse wave tracing method can be relieved, effective signals are extracted from the time domain and the space domain, and high-quality pulse wave signals are provided for follow-up physiological parameter estimation.

Description

Technical Field

[0001] The invention relates to the field of digital image processing technology and digital signal processing technology, and in particular to a spatiotemporal domain weighted photoplethysmogram restoration method based on facial video. Background Art

[0002] Physiological parameters such as heart rate, blood oxygen saturation, respiratory rate, and blood pressure reflect the physiological state of the human body and provide important information about health status and potential risks. Real-time and accurate monitoring of these physiological parameters can help detect diseases early, and can also help individuals conduct personalized health management and prevent the occurrence of diseases. Traditional physiological signal measurement is usually contact measurement, which relies on direct contact between the sensor and the skin. However, it is inconvenient to use for long-term monitoring and can easily cause discomfort to patients with sensitive skin or trauma. Image Photoplethysmography (IPPG) technology, as a non-contact method, has become a popular development direction for future physiological parameter monitoring equipment. Its principle is similar to that of photoplethysmography. Due to the periodic contraction and relaxation of the heart, the blood filling degree in the blood vessels changes accordingly. When ambient light is irradiated on the skin surface, the skin's absorption of light will also produce corresponding pulsations due to changes in the blood filling degree, resulting in corresponding changes in the intensity of the reflected light. By analyzing the changes in the light intensity of each frame captured by the camera, the physiological characteristics of the human body can be extracted. Compared with traditional contact methods, this technology has the advantages of easy operation, low cost, and no need for direct contact with the skin, thereby improving the real-time and convenience of monitoring. It has practical value and important prospects for telemedicine and daily monitoring.

[0003] However, the IPPG technology is easily affected by various interference factors such as movement during measurement, and performs poorly in the reconstruction of photoelectric volume pulse waves. It can basically only be used to estimate heart rate, and the detection effect of other physiological parameters is poor. Currently, most related applications focus on how to extract original signals from a limited number of facial regions of interest. However, due to the irregular movement of the head, the signal quality varies at different times and in different facial regions. Selecting a fixed number of regions of interest cannot fully utilize all high-quality signal areas on the face.

[0004] In order to solve this problem, a spatiotemporal weighted photoplethysmography recovery method based on facial video was invented. Starting from the spatial domain and temporal domain, the input video is weighted in the spatial domain using the frame difference map, and the facial parts with obvious movement are given low weight values, so that the network pays more attention to the stable facial area when restoring the signal; the signal quality score is used for weighting in the temporal domain, and the parts with low signal quality are given high weights, so that the network focuses on repairing the fragments with low signal quality during the signal enhancement process. The present invention can dynamically restore the photoplethysmography signal, which helps to improve the accuracy of estimation of various physiological indicators. Summary of the invention

[0005] In order to solve the problems existing in the background technology, the purpose of the present invention is to provide a spatiotemporal domain weighted photoplethysmography restoration method based on facial video, which can effectively reduce the influence of motion interference and improve the reliability of the restored photoplethysmography signal.

[0006] The specific technical solution for achieving the purpose of the present invention is:

[0007] A spatiotemporal weighted photoplethysmogram recovery method based on facial video, the method comprising the following steps:

[0008] Step 1: Build a pulse wave recovery model

[0009] The constructed pulse wave recovery model consists of a signal extraction module, a signal quality scoring module and a signal enhancement module; wherein the signal extraction module consists of a convolution block and an average pooling, calculates the frame difference map of the input video, extracts features of the input video and the calculated frame difference map respectively, uses the frame difference map features as spatial domain weights to weight the features of the input video, and preliminarily extracts a one-dimensional pulse wave signal; the signal quality scoring module consists of a linear layer and an LSTM network, is responsible for giving the quality score of the one-dimensional pulse wave signal in different time periods, and uses this as the time domain attention weight; the signal enhancement module is an encoder block and a decoder block structure, is responsible for multiplying the attention weight with the preliminarily extracted one-dimensional pulse wave signal, that is, performing time domain weighting on the preliminarily extracted one-dimensional pulse wave signal, and assigning a high attention weight value to the signal segment with a low quality score, so as to improve the quality of the pulse wave signal after the time domain weighting;

[0010] Step 2: Train the pulse wave recovery model

[0011] The training of the pulse wave recovery model is divided into two stages. First, the signal extraction module, the signal quality scoring module, and the signal enhancement module are pre-trained separately; the video input signal extraction module containing only the face is trained to obtain a motion weighted signal extraction model, and a preliminary extracted one-dimensional pulse wave signal is output; the preliminary extracted one-dimensional pulse wave signal is input into the signal quality scoring module for training to obtain a signal quality scoring model, and a signal quality score is output; the signal quality score and the preliminary extracted one-dimensional pulse wave signal are input into the signal enhancement module for training to obtain a signal enhancement model, and a restored pulse wave signal with higher quality is output;

[0012] Secondly, the pre-trained motion weighted signal extraction model, signal quality scoring model, and signal enhancement model are trained and fine-tuned together to obtain a usable pulse wave recovery model;

[0013] Step 3: For the collected facial video, use the Mediapipe Face Detection method to detect the face position in the facial video, and crop the facial video to obtain a video containing only the face, and input the available pulse wave recovery model to obtain the restored final pulse wave signal.

[0014] Further, the signal extraction module is trained, and the specific process is as follows:

[0015] After the video containing only human faces is input into the signal extraction module, the frame difference map DIF is first calculated. The process is as follows: for the input video containing only human faces, the green channel image corresponding to each frame is extracted. The i-th frame green channel image frame i , calculate the green channel image frame of the adjacent frame i+1 The pixel-by-pixel normalized difference value is denoted as dif i , and its calculation formula is as follows:

[0016]

[0017] Also calculate the motion suppression normalized difference value Because the more intense the movement, the dif i The larger the value, the more motion suppression normalized difference value is calculated. Make the weight of the parts corresponding to the intense movement closer to 0;

[0018] The motion suppression normalized difference value The image is scaled to the range of [0, 255], and then binarized, where the pixel values ​​exceeding 220 are set to 255, indicating an area with a high spatial attention weight, and the pixel values ​​less than or equal to 220 are set to 0, indicating an area with a low spatial attention weight, thereby obtaining a binary frame difference map DIF between the i-th frame image and the adjacent image in the input video i ;

[0019] The binary frame difference map corresponding to each frame of the input video is calculated in sequence. If the i-th frame image is the last frame image of the input video, its binary frame difference map is the same as the i-1-th frame binary frame difference map, thereby forming a frame difference map DIF. If the input video contains m frames, then DIF = {DIF 1 ,DIF 2 ,……,DIF m-1 ,DIF m-1};

[0020] The signal extraction module extracts the input video features that only contain human faces;

[0021] The signal extraction module extracts the frame difference image DIF feature as a spatial domain weight, weights the video feature containing only the face, focuses on the non-moving area of ​​the face from the spatial domain, obtains a spatially weighted signal, and preliminarily extracts a one-dimensional pulse wave signal;

[0022] The signal extraction module is trained to obtain a motion weighted signal extraction model. The training process is specifically as follows:

[0023] The signal extraction module uses the photoplethysmogram (PPG) collected synchronously with the input video as a label for training, and the loss function is Pearson Loss, and the formula is as follows:

[0024]

[0025] Where N is the number of samples, y is the i-th signal value of a certain PPG signal, is the mean value of a certain segment of PPG signal, is the i-th signal value of a certain pulse wave signal, is the mean value of a certain pulse wave signal, and n is the number of signal values ​​of a signal; when training the signal extraction module, a video containing only a face is used as input, and Pearson Loss is used as a loss function to control the training, thereby obtaining the motion weighted signal extraction model.

[0026] Furthermore, the signal quality scoring module is trained, and the specific process is as follows:

[0027] For each of the one-dimensional pulse wave signals, the standard deviation of the difference between the frame numbers corresponding to each adjacent maximum peak point is calculated. width , the standard deviation of the difference between the signal value of each minimum point and the adjacent maximum point height , normalize both to between 0 and 1, then the label point label The calculation formula is as follows:

[0028] point label =0.3*point height +0.7*point width ;

[0029] The training loss function is L1_Loss, and the formula is as follows:

[0030]

[0031] Where N is the number of samples, y is the actual label of the i-th sample, is the predicted value of the i-th sample;

[0032] The signal quality scoring module uses a one-dimensional pulse wave signal as input during training, and L1_Loss is used as a loss function to control the training, thereby obtaining the signal quality scoring model.

[0033] Furthermore, the signal enhancement model training process is specifically as follows:

[0034] The signal enhancement module uses the signal quality score and the initially extracted one-dimensional pulse wave signal as input, the PPG waveform collected synchronously with the input video as a label, and Pearson Loss as a loss function to control training to obtain the signal enhancement model.

[0035] Furthermore, the training process of the available pulse wave recovery model is specifically as follows:

[0036] Importing the motion weighted signal extraction model, signal quality scoring model, and signal enhancement model weights into their corresponding signal extraction module, signal quality scoring module, and signal enhancement module respectively;

[0037] Except for the decoder block part of the signal enhancement module, the weights of the rest are frozen, all modules are trained jointly, and the decoder block part is fine-tuned;

[0038] The joint training loss function includes two parts: signal quality loss and heart rate loss; the signal quality loss uses the PPG waveform collected synchronously with the input video as a label, compares the difference between the final pulse wave signal and the PPG waveform collected synchronously with the input video, and the loss function is Pearson Loss; the heart rate loss uses the heart rate data collected synchronously with the input video as a label, compares the difference between the heart rate calculated by the final pulse wave signal and the synchronously collected heart rate data, and the loss function is L1_Loss;

[0039] When training the pulse wave restoration model, a video containing only a human face is used as input, and the sum of Pearson Loss and L1_Loss is used as a loss function to control the training, thereby obtaining the usable pulse wave restoration model and outputting the final restored pulse wave signal.

[0040] The beneficial effects of the technical solution provided by the present invention are as follows: the proposed method of spatiotemporal weighted photoplethysmography restoration based on facial video calculates the frame difference map corresponding to the input facial video frame sequence, extracts its features as the spatial domain weight, and weights the input facial video, assigning a lower attention weight value to the region with large inter-frame motion, but not limiting the size and number of the region of interest, effectively improving the problem of limited quality of source signal extraction caused by the selection of fixed regions of interest under motion; at the same time, from the perspective of improving the quality of signal recovery, by giving a one-dimensional pulse wave signal quality score by time period, assigning a higher attention weight value to the signal segment with a low quality score, effectively improving the quality and reliability of pulse wave signal recovery. The present invention can dynamically restore the photoplethysmography signal, which helps to improve the accuracy of estimation of various physiological indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the present invention;

[0042] Figure 2 is a schematic diagram of a facial frame and a frame difference map in the present invention;

[0043] Figure 3 A neural network structure diagram of the present invention;

[0044] Figure 4 This is a diagram of the signal quality scoring network structure of the present invention. DETAILED DESCRIPTION

[0045] In order to more clearly illustrate the technical means, technical improvements and beneficial effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings.

[0046] Example

[0047] This embodiment takes a facial video of an adult male with a duration of 4 seconds and a frame rate of 30 frames per second as an example.

[0048] See also Figure 1-4 The present invention provides a method for restoring a weighted photoplethysmogram in spatiotemporal domain based on facial video, comprising the following specific steps:

[0049] Use the Mediapipe Face Detection method to detect the face position for each frame in the video. If a face is detected in the current frame, extract the face bounding box information and calculate the face position according to the bounding box coordinates x. min ,y min and width, height to crop the face image from the current frame, see Figure 2 (a)(b) Traverse all frames in the video to perform cropping, and scale all cropped face images to 198*198 to obtain a video frame sequence containing only faces as the input video F = {frame 1 ,frame 2 ,……,frame 120};

[0050] Among them, the Mediapipe Face Detection method is a well-known algorithm [Lugaresi C, Tang J, Nash H, et al. Mediapipe: A framework for building perception pipelines[J]. arXiv preprint arXiv: 1906.08172, 2019.] and will not be introduced in detail here.

[0051] Calculate the frame difference map of the input video. The specific process is as follows:

[0052] After the video containing only human faces is input into the signal extraction module, the frame difference map DIF is first calculated. The process is as follows: for the input video containing only human faces, the green channel image corresponding to each frame is extracted. For the i-th frame green channel image frame i , calculate the green channel image frame of the adjacent frame i+1 The pixel-by-pixel normalized difference value is denoted as dif i , and its calculation formula is as follows:

[0053]

[0054] For dif i The more intense the movement, the closer the corresponding normalized difference value is to 1. In order to suppress movement, the movement suppression normalized difference value is calculated. Make the weight of the parts corresponding to the intense movement closer to 0;

[0055] The motion suppression normalized difference value The image is scaled to the range of [0, 255], and then binarized. The pixel values ​​exceeding 220 are set to 255, indicating the area with high spatial attention weight, and the pixel values ​​less than or equal to 220 are set to 0, indicating the area with low spatial attention weight. Thus, the binary frame difference map DIF between the i-th frame image and the adjacent image in the input video is obtained. i ,See Figure 2 (c);

[0056] The binary frame difference map corresponding to each frame of the input video is calculated in sequence. If the i-th frame image is the last frame image of the input video, its binary frame difference map is the same as the i-1-th frame binary frame difference map, thereby forming a frame difference map DIF. If the input video contains m frames, then DIF = {DIF 1 ,DIF 2 ,……,DIF m-1 ,DIF m-1};

[0057] The signal extraction module includes two groups of convolution blocks, average pooling (AvgPool3D) and a layer of three-dimensional convolutional neural network (Conv3D). The convolution block mainly includes a layer of maximum pooling (MaxPool3D) and two layers of three-dimensional convolutional neural network (Conv3D). Figure 3 (a);

[0058] The signal extraction module extracts features of the input video;

[0059] The signal extraction module extracts the frame difference image DIF feature as the spatial domain weight weight spatial , and weight the features of the input video, focus on the non-moving area of ​​the face from the spatial domain, obtain the spatially weighted signal, and preliminarily extract the one-dimensional pulse wave signal IPPG pre ;

[0060] The signal extraction module is trained to obtain a motion weighted signal extraction model. The training process is specifically as follows:

[0061] The signal extraction module uses the PPG waveform collected synchronously with the input video as a label for training, and the loss function is Pearson Loss, and the formula is as follows:

[0062]

[0063] Where N is the number of samples, y i is the i-th signal value of a certain PPG signal, is the mean value of a certain segment of PPG signal, is the i-th signal value of a certain pulse wave signal, is the mean value of a certain pulse wave signal, and n is the number of signal values ​​in a signal;

[0064] The signal extraction module uses a video containing only a human face as input during training, uses Pearson Loss as a loss function to control training, and outputs a preliminary extracted one-dimensional pulse wave signal IPPG. Ore , and obtain the motion weighted signal extraction model.

[0065] Furthermore, the signal quality scoring module is trained, and the specific process is as follows:

[0066] For each of the one-dimensional pulse wave signals, the standard deviation of the difference between the frame numbers corresponding to each adjacent maximum peak point is calculated. width , the standard deviation of the difference between the signal value of each minimum point and the adjacent maximum point heighi , normalize both to between 0 and 1, then the label point label The calculation formula is as follows:

[0067] point label =0.3*point height +0.7*point width ;

[0068] The signal quality scoring module extracts the one-dimensional pulse wave signal features;

[0069] The signal quality scoring network is a Bi-LSTM structure, which mainly includes three linear layer blocks and one LSTM block. Except for the last linear layer block, the remaining linear layer blocks include a linear layer (Linear) and a SELU activation function. The last linear layer block includes a linear layer (Linear) and a Sigmoid activation function. Figure 4 , the training loss function is L1_Loss, the formula is as follows:

[0070]

[0071] Where N is the number of samples, y i is the actual label of the i-th sample, is the predicted value of the i-th sample;

[0072] Cut the 120-data-point signal into three segments of 60 data points each with a sliding window of 30 data points. 1 、signal 2 、signal 3, input these three overlapping signal segments into the quality scoring module to obtain three normalized scoring points 1 、point 2 、point 3 ;

[0073] Divide the original signal into three non-overlapping signal segments of 40 data points in length. 1* 、signal 2* 、signal 3* , the score of each signal segment is the mean score of the coverage, denoted as point 1* 、point 2* 、point 3* ,Right now

[0074] Use 1-the average score to get each signal segment signal 1* 、signal 2* 、signal 3* The signal quality score, weight 1 、weight 2 、weight 3 .

[0075] The signal quality scoring module uses a one-dimensional pulse wave signal as input during training, uses L1_Loss as a loss function to control training, outputs a signal quality score for each signal segment, and obtains the signal quality scoring model.

[0076] Furthermore, the signal enhancement model training process is specifically as follows:

[0077] The signal quality score weight time The IPPG pulse wave signal extracted from the pre Input signal enhancement module, which is specifically:

[0078] The signal enhancement network structure mainly includes six encoder blocks and six decoder blocks. Each encoder block includes a one-dimensional convolutional neural network (Conv1D) and a PReLU activation function. Each decoder block includes a one-dimensional deconvolutional neural network (ConvTranspose1D), a PReLU activation function and a jump connection layer. The last decoder includes a one-dimensional deconvolutional neural network (ConvTranspose1D) and a Tanh activation function. Figure 3 (b);

[0079] The signal enhancement module uses the signal quality score and the initially extracted one-dimensional pulse wave signal as input, extracts the time-domain weighted one-dimensional pulse wave signal features, uses the PPG waveform collected synchronously with the input video as a label, uses Pearson Loss as a loss function to control training, and outputs a restored pulse wave signal with higher quality to obtain the signal enhancement model.

[0080] Furthermore, the training process of the available pulse wave recovery model is specifically as follows:

[0081] Importing the motion weighted signal extraction model, signal quality scoring model, and signal enhancement model weights into their corresponding signal extraction module, signal quality scoring module, and signal enhancement module respectively;

[0082] Except for the decoder block part of the signal enhancement module, the weights of the rest are frozen, all modules are trained jointly, and the decoder block part is fine-tuned;

[0083] The joint training loss function includes two parts: signal quality loss and heart rate loss. The signal quality loss uses the PPG waveform collected synchronously with the input video as a label, compares the difference between the final pulse wave signal and the synchronously collected PPG waveform, and the loss function is Pearson Loss; the heart rate loss uses the heart rate data collected synchronously with the input video as a label, compares the difference between the heart rate calculated by the final pulse wave signal and the heart rate data collected simultaneously, and the loss function is L1_Loss. The sum of Pearson Loss and L1_Loss is used to jointly control the training to obtain the usable pulse wave recovery model.

[0084] Input the video containing only the face into the available pulse wave recovery model to obtain the final recovered pulse wave signal IPPG pd .

[0085] In summary, the present invention proposes a spatiotemporal domain weighted photoplethysmography recovery method based on facial video, extracts the frame difference map corresponding to the input facial video frame sequence as the spatial domain weight, and weights the input facial video, assigning a lower attention weight value to the region with large inter-frame motion, but does not limit the size and number of the region of interest, effectively improving the problem of limited quality of source signal extraction caused by the selection of fixed regions of interest under motion conditions; at the same time, from the perspective of improving the quality of signal recovery, the one-dimensional pulse wave signal quality score is given by time period, and a higher attention weight value is assigned to the signal segment with a low quality score, effectively improving the quality and reliability of pulse wave signal recovery. The method of the present invention provides effective data for non-contact physiological signal measurement.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A spatiotemporal weighted photoplethysmography recovery method based on facial video, characterized in that: The method comprises the following steps: Step 1: Build a pulse wave recovery model The constructed pulse wave recovery model consists of a signal extraction module, a signal quality scoring module and a signal enhancement module; wherein the signal extraction module consists of a convolution block and an average pooling, calculates the frame difference map of the input video, extracts features of the input video and the calculated frame difference map respectively, uses the frame difference map features as spatial domain weights to weight the features of the input video, and preliminarily extracts a one-dimensional pulse wave signal; the signal quality scoring module consists of a linear layer and an LSTM network, is responsible for giving the quality score of the one-dimensional pulse wave signal in different time periods, and uses this as the time domain attention weight; the signal enhancement module is an encoder block and a decoder block structure, is responsible for multiplying the attention weight with the preliminarily extracted one-dimensional pulse wave signal, that is, performing time domain weighting on the preliminarily extracted one-dimensional pulse wave signal, and assigning a high attention weight value to the signal segment with a low quality score, so as to improve the quality of the pulse wave signal after the time domain weighting; Step 2: Train the pulse wave recovery model The training of the pulse wave recovery model is divided into two stages. First, the signal extraction module, the signal quality scoring module, and the signal enhancement module are pre-trained separately; the video input signal extraction module containing only the face is trained to obtain a motion weighted signal extraction model, and a preliminarily recovered one-dimensional pulse wave signal is output; the preliminarily recovered one-dimensional pulse wave signal is input into the signal quality scoring module for training to obtain a signal quality scoring model, and a signal quality score is output; the signal quality score and the preliminarily recovered one-dimensional pulse wave signal are input into the signal enhancement module for training to obtain a signal enhancement model, and a restored pulse wave signal with higher quality is output; Secondly, the pre-trained motion weighted signal extraction model, signal quality scoring model, and signal enhancement model are trained and fine-tuned together to obtain a usable pulse wave recovery model; Step 3: For the collected facial video, use the Mediapipe Face Detection method to detect the face position in the facial video, and crop the facial video to obtain a video containing only the face, and input the available pulse wave recovery model to obtain the restored final pulse wave signal.

2. The spatiotemporal weighted photoplethysmography recovery method according to claim 1, characterized in that: The signal extraction module is trained, and the specific process is as follows: After the video input signal containing only the face is extracted into the module, the frame difference map DIF is first calculated, and the process is as follows: For the input video containing only human faces, extract the green channel image corresponding to each frame, the i-th frame green channel image frame i , calculate the green channel image frame of the adjacent frame i+1 The pixel-by-pixel normalized difference value is denoted as dif i , and its calculation formula is as follows: Also calculate the motion suppression normalized difference value dif i * , because the more intense the movement, the greater the dif i The larger the value, the more motion suppression normalized difference value dif is calculated. i * =1-dif i Make the weight of the parts corresponding to the intense movement closer to 0; The motion suppression normalized difference value dif i * The image is scaled to the range of [0, 255], and then binarized, where the pixel values ​​exceeding 220 are set to 255, indicating an area with a high spatial attention weight, and the pixel values ​​less than or equal to 220 are set to 0, indicating an area with a low spatial attention weight, thereby obtaining a binary frame difference map DIF between the i-th frame image and the adjacent image in the input video i ; The binary frame difference map corresponding to each frame of the input video is calculated in sequence. If the i-th frame image is the last frame image of the input video, its binary frame difference map is the same as the i-1-th frame binary frame difference map, thereby forming a frame difference map DIF. If the input video contains m frames, DIF = {DIF1, DIF2, ..., DIF m-1 ,DIF m-1 }; The signal extraction module extracts the input video features that only contain human faces; The signal extraction module extracts the frame difference image DIF feature as a spatial domain weight, weights the video feature containing only the face, focuses on the non-moving area of ​​the face from the spatial domain, obtains a spatially weighted signal, and preliminarily extracts a one-dimensional pulse wave signal; The signal extraction module is trained to obtain a motion weighted signal extraction model. The training process is specifically as follows: The signal extraction module uses the photoplethysmogram (PPG) collected synchronously with the input video as a label for training, and the loss function is Pearson Loss, and the formula is as follows: Where N is the number of samples, y is the i-th signal value of a certain PPG signal, is the mean value of a certain segment of PPG signal, is the i-th signal value of a certain pulse wave signal, is the mean value of a certain pulse wave signal, and n is the number of signal values ​​in a signal; The signal extraction module uses a video containing only a human face as input during training, and uses Pearson Loss as a loss function to control the training, thereby obtaining the motion weighted signal extraction model.

3. The spatiotemporal weighted photoplethysmography recovery method according to claim 1, characterized in that: The signal quality scoring module is trained, and the specific process is as follows: For each of the one-dimensional pulse wave signals extracted initially, the standard deviation of the difference between the frame numbers corresponding to each adjacent maximum peak point is calculated. width , the standard deviation of the difference between the signal value of each minimum point and the adjacent maximum point height , normalize both to between 0 and 1, then the label point label The calculation formula is as follows: point label =0.3*point heihht +0.7*point width ; The training loss function is L1_Loss, and the formula is as follows: Where N is the number of samples, y is the actual label of the i-th sample, is the predicted value of the i-th sample; The signal quality scoring module uses a one-dimensional pulse wave signal as input during training, and L1_Loss is used as a loss function to control the training, thereby obtaining the signal quality scoring model.

4. The spatiotemporal weighted photoplethysmography recovery method according to claim 1, characterized in that: The signal enhancement model training process is specifically as follows: The signal enhancement module uses the signal quality score and the initially extracted one-dimensional pulse wave signal as input, the PPG waveform collected synchronously with the input video as a label, and Pearson Loss as a loss function to control training to obtain the signal enhancement model.

5. The spatiotemporal weighted photoplethysmography recovery method according to claim 1, characterized in that: The training process of the available pulse wave recovery model is as follows: Importing the motion weighted signal extraction model, signal quality scoring model, and signal enhancement model weights into their corresponding signal extraction module, signal quality scoring module, and signal enhancement module respectively; Except for the decoder block part of the signal enhancement module, the weights of the rest are frozen, all modules are trained jointly, and the decoder block part is fine-tuned; The joint training loss function includes two parts: signal quality loss and heart rate loss; the signal quality loss uses the PPG waveform collected synchronously with the input video as a label, compares the difference between the final pulse wave signal and the PPG waveform collected synchronously with the input video, and the loss function is Pearson Loss; the heart rate loss uses the heart rate data collected synchronously with the input video as a label, compares the difference between the heart rate calculated by the final pulse wave signal and the synchronously collected heart rate data, and the loss function is L1_Loss; When training the pulse wave restoration model, a video containing only a human face is used as input, and the sum of Pearson Loss and L1_Loss is used as a loss function to control the training, thereby obtaining the usable pulse wave restoration model and outputting the final restored pulse wave signal.

Images