Method, device and equipment for extracting rPPG signal, medium and program product

Abstract

The embodiment of the invention relates to the technical field of rPPG signal extraction, in particular to an rPPG signal extraction method, device and equipment, a medium and a program product. The rPPG signal extraction method comprises the following steps: acquiring an original video of a to-be-extracted rPPG signal; performing channel enhancement on the original video to obtain a channel enhanced video; extracting a space-time fusion feature of the rPPG signal from the channel enhanced video; segmenting the space-time fusion features into feature sequences with different frame rates; extracting rPPG sub-signals of the original video at different frame rates from each feature sequence; and determining the rPPG signal of the original video according to the rPPG sub-signal under each frame rate. The precision of the extracted rPPG signal is improved under the condition that the calculation cost is reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of rPPG signal extraction, and in particular to a method, device, equipment, medium and program product for extracting rPPG signals. Background Art

[0002] Heart rate is an important indicator of human health. At present, the commonly used traditional heart rate measurement methods include electrocardiogram (ECG) and photoplethysmography (PPG). These two methods calculate heart rate by collecting ECG or PPG signals. However, since both technologies require direct contact with the skin for measurement, they have limitations in some scenarios where contact is not suitable (such as skin burns, neonatal monitoring, sleep monitoring, etc.). Remote photoplethysmography (rPPG) provides a non-contact alternative. The beating of the heart circulates blood, causing changes in the blood volume of the microvascular bed under the skin, which triggers periodic color fluctuations in skin areas such as the face. rPPG remotely analyzes these color fluctuations and extracts the periodic physiological signals, namely rPPG signals. This means that even with an ordinary webcam, rPPG signals can be easily obtained to calculate heart rate information. Compared with traditional contact methods, rPPG technology is more economical and convenient and suitable for a variety of scenarios. However, rPPG signals are easily interfered by a variety of strong noises in the video, and the signal itself is weak, which poses a severe challenge to the accuracy and stability of heart rate measurement. Summary of the invention

[0003] The purpose of the present invention is to at least provide a method, device, equipment, medium and program product for extracting rPPG signals. The present application provides a method for extracting rPPG signals, including: obtaining an original video of the rPPG signal to be extracted; performing channel enhancement on the original video to obtain a channel enhanced video; extracting the spatiotemporal fusion features of the rPPG signal from the channel enhanced video; dividing the spatiotemporal fusion features into feature sequences with different frame rates; extracting rPPG sub-signals of the original video at different frame rates from each of the feature sequences; and determining the rPPG signal of the original video according to the rPPG sub-signals at each frame rate. The method improves the accuracy of the extracted rPPG signal while reducing the computational cost.

[0004] To solve the above technical problems, this application proposes five aspects.

[0005] In a first aspect, the present application provides a method for extracting an rPPG signal, comprising: obtaining an original video of an rPPG signal to be extracted; performing channel enhancement on the original video to obtain a channel enhanced video; extracting spatiotemporal fusion features of the rPPG signal from the channel enhanced video; dividing the spatiotemporal fusion features into feature sequences with different frame rates; extracting rPPG sub-signals of the original video at different frame rates from each of the feature sequences; and determining the rPPG signal of the original video according to the rPPG sub-signals at each frame rate.

[0006] In some embodiments, performing channel enhancement on the original video to obtain the channel enhanced video includes: generating a Boolean image and a chromatogram based on the original video; and generating the channel enhanced video based on the Boolean image, the chromatogram and the original video.

[0007] In some embodiments, the extracting the spatiotemporal fusion features of the rPPG signal from the channel-enhanced video includes: obtaining a plurality of time-shifted videos by performing a time-shift operation on the channel-enhanced video; determining a difference frame between two adjacent time-shifted videos based on the plurality of time-shifted videos; performing difference frame splicing on the plurality of difference frames, and extracting the spatiotemporal fusion features of the rPPG signal from the splicing result.

[0008] In some embodiments, extracting the rPPG sub-signal of the original video at different frame rates from each of the feature sequences includes: performing the following operations for multiple feature sequences at any frame rate: determining the spatiotemporal attention features of each of the feature sequences according to each of the feature sequences; determining the rPPG sub-signal of the original video at the current frame rate according to each of the spatiotemporal attention features.

[0009] In some embodiments, determining the spatiotemporal attention features of each feature sequence according to each feature sequence includes: performing the following operations for any feature sequence: determining the attention weight of the feature sequence according to the feature sequence; determining the time channel attention features of the feature sequence according to the attention weight and the feature sequence; determining the spatiotemporal attention weight of the feature sequence according to the time channel attention features; determining the spatiotemporal attention features of the feature sequence according to the spatiotemporal attention weight and the time channel attention features.

[0010] In some embodiments, determining the rPPG signal of the original video according to the rPPG sub-signals at each frame rate includes: determining the self-attention weight of the original video according to the spatiotemporal fusion feature; and fusing the rPPG sub-signals of different frame rates according to the self-attention weight to determine the rPPG signal of the original video.

[0011] In the second aspect, the present application provides an rPPG signal extraction device, comprising: a first acquisition module, used to acquire an original video of the rPPG signal to be extracted; a first enhancement module, used to perform channel enhancement on the original video to obtain a channel enhanced video; a first fusion module, used to extract the spatiotemporal fusion features of the rPPG signal from the channel enhanced video; a first segmentation module, used to segment the spatiotemporal fusion features into feature sequences with different frame rates; a first extraction module, used to extract rPPG sub-signals of the original video at different frame rates from each of the feature sequences; a first determination module, used to determine the rPPG signal of the original video according to the rPPG sub-signals at each frame rate.

[0012] In a third aspect, the present application proposes a computer electronic production device, characterized in that it includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0013] In a fourth aspect, the present application proposes a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0014] In a fifth aspect, the present application proposes a computer program product, comprising a computer program, which implements the steps of any one of the methods described in the first aspect when executed by a processor.

[0015] The present application provides a method for extracting rPPG signals, including: obtaining an original video of the rPPG signal to be extracted; performing channel enhancement on the original video to obtain a channel enhanced video; extracting the spatiotemporal fusion features of the rPPG signal from the channel enhanced video; dividing the spatiotemporal fusion features into feature sequences with different frame rates; extracting rPPG sub-signals of the original video at different frame rates from each of the feature sequences; and determining the rPPG signal of the original video according to the rPPG sub-signals at each frame rate. The method improves the accuracy of the extracted rPPG signal while reducing the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.

[0017] Figure 1 A main flow chart of a method for extracting rPPG signals provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a channel strengthening process provided in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of a process for extracting spatiotemporal fusion features provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a process for extracting channel attention features provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of a process for extracting spatiotemporal attention features provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0023] Figure 7 This is a main structural block diagram of an rPPG signal extraction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0025] Heart rate is an important indicator of human health. At present, the commonly used traditional heart rate measurement methods include electrocardiogram (ECG) and photoplethysmography (PPG). These two methods calculate heart rate by collecting ECG or PPG signals. However, since both technologies require direct contact with the skin for measurement, they have limitations in some scenarios where contact is not suitable (such as skin burns, neonatal monitoring, sleep monitoring, etc.). Remote photoplethysmography (rPPG) provides a non-contact alternative. The beating of the heart circulates blood, causing changes in the blood volume of the microvascular bed under the skin, which triggers periodic color fluctuations in skin areas such as the face. rPPG remotely analyzes these color fluctuations and extracts the periodic physiological signals, namely rPPG signals. This means that even with an ordinary webcam, rPPG signals can be easily obtained to calculate heart rate information. Compared with traditional contact methods, rPPG technology is more economical and convenient and suitable for a variety of scenarios. However, rPPG signals are easily interfered by a variety of strong noises in the video, and the signal itself is weak, which poses a severe challenge to the accuracy and stability of heart rate measurement.

[0026] In recent years, rPPG research has made significant progress. Especially driven by deep learning, many data-driven rPPG signal extraction methods have emerged. However, how to suppress noise such as motion artifacts, video compression artifacts and illumination changes has always been a core challenge facing rPPG methods. In order to improve the robustness of rPPG signal extraction, early researchers used manual modeling methods to separate noise from useful rPPG signals to extract purer rPPG signals. Although this method works well in dealing with noise, its limitations are also very obvious. In data-driven methods, some studies have tried to combine the prior knowledge of manual modeling to perform spatial transformation on RGB videos to improve signal extraction performance. However, since this direct conversion method may lead to the loss of original video features, its effect is still restricted by the limitations of traditional methods. At the same time, this type of method fails to give full play to the advantages of deep neural networks in automatic feature extraction.

[0027] Although deep learning-based methods have achieved significant performance improvements over manual modeling, their high computational cost limits the feasibility of practical applications. To improve the operating efficiency of the model, a two-stream network structure is proposed to accelerate model reasoning through downsampled differential frames. However, since this method uses a 2D convolutional network, it cannot simultaneously process the temporal and spatial features in face videos, resulting in less than ideal performance. In addition, attempts were made to simplify the high-computational-cost PhysNet network and downsample the input face video in the spatial dimension to reduce the network depth and input data volume, thereby increasing the signal extraction speed. However, this operation inevitably affects the stability and final performance of the model.

[0028] In order to solve the above technical problems, the present invention proposes a method for extracting rPPG signals. The implementation details of the method for determining the bandwidth of the present embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of the present solution.

[0029] Embodiment 1:

[0030] like Figure 1 As shown, the present application provides a method for extracting rPPG signals, and the method system is applied to electronic production equipment, and the electronic production equipment can be a server, a mobile terminal, a computer, a cloud platform, etc. The function implemented by the production equipment data processing provided in the embodiment of the present application can be implemented by calling the program code by the processor of the electronic production equipment, wherein the program code can be stored in a computer storage medium. The method for extracting rPPG signals includes:

[0031] Step S1: Obtain the original video of the rPPG signal to be extracted.

[0032] Step S2: performing channel enhancement on the original video to obtain a channel enhanced video.

[0033] In some embodiments, step S2 of “performing channel enhancement on the original video to obtain a channel enhanced video” includes:

[0034] Step S21: Generate a Boolean image and a chromatogram according to the original video.

[0035] Step S22: Generate the channel enhanced video according to the Boolean image, the chromatogram and the original video.

[0036] The POS method based on prior knowledge can effectively reduce the noise caused by motion. This method generates a two-channel image for extracting rPPG signals by projecting the RGB image (original video image) onto a plane orthogonal to the vector [1,1,1]. However, since this method directly performs spatial transformation on the original RGB image, some information in the original video frame is lost. Figure 2 As shown, in this application, the original RGB video is converted into a two-channel POS image (Boolean image) and a CHROM image (chromatogram). The conversion method is shown in the following formula.

[0037]

[0038] Where X is the input of channel expansion. Using the prior knowledge of manual modeling, the input X is multiplied by the matrix to obtain the channel enhancement X', that is, the channel enhanced video. Through steps S21-S22, the enhanced video features are added to the channel dimension, so that the useful features are effectively separated from the noise, thereby significantly improving the characterization ability of the rPPG signal.

[0039] Step S3: extracting spatiotemporal fusion features of the rPPG signal from the channel enhanced video.

[0040] Since facial color changes are very subtle, how to effectively express rPPG signals in enhanced video sequences becomes a key issue. In previous methods, intrinsic noise such as background noise and skin color changes were effectively removed by calculating difference frames in the time dimension. This is because rPPG signals mainly come from slight changes in the color of adjacent frames in face videos, and difference frames can help neural networks capture these changes more keenly by subtracting the pixel values ​​of adjacent frames, thereby better extracting rPPG signals.

[0041] In some embodiments, step S3 “extracting spatiotemporal fusion features of the rPPG signal from the channel enhanced video” includes:

[0042] Step S31: obtaining a plurality of time-shifted videos by performing a time-shift operation on the channel-enhanced video.

[0043] Step S32: determining a difference frame between two adjacent time-shifted videos according to the plurality of time-shifted videos.

[0044] Step S33: performing difference frame splicing on the plurality of difference frames, and extracting the spatiotemporal fusion features of the rPPG signal from the splicing result.

[0045] like Figure 3 As shown in the figure, it is assumed that the dimension of the channel-enhanced video is N×7×T×128×128, where N is the batch size, 7 is the number of channels, T is the number of video frames, and 128 is the spatial resolution of the video frame. First, the channel-enhanced video is subjected to a time displacement operation to generate x(t-1), x(t), x(t+1), and the corresponding difference frames D1 and D2 are calculated in chronological order. Next, these difference frames are concatenated in the channel dimension to form a feature representation containing multi-scale information. These features are then sent to two sub-networks, Stem1 and Stem2, respectively, to extract the main rPPG signal waveform features. Finally, through the feature fusion step, the features of different channels are integrated to further enhance the representation capability of the rPPG signal. The process can be expressed as follows:

[0046] x fusion =Stem2(Stem1(Concat(D1,D2)))

[0047] Among them, x fusion Step S3-Step S3 enhances the model's perception of rPPG signals and can effectively reduce noise interference, ensuring that the extracted signal is more stable and accurate.

[0048] Step S4: Segment the spatiotemporal fusion features into feature sequences with different frame rates.

[0049] Because using low frame rate videos to extract rPPG signals can sometimes achieve better results. This application constructs a time feature pyramid in the time dimension to capture multi-scale rPPG signal features through videos with different frame rates. Specifically, this application processes the spatiotemporal fusion features according to multiple frame rate slices, such as 30FPS, 15FPS and lower frame rates. Due to the use of different frame rate slices, the spatiotemporal fusion features can form different numbers of feature sequences at different frame rate levels. Then the feature sequence of each frame rate level is passed through a special rPPG signal extraction module to capture the multi-scale features in the time series and ensure that information at different time scales is fully utilized.

[0050] Step S5: extracting rPPG sub-signals of the original video at different frame rates from each of the feature sequences.

[0051] In some embodiments, step S5 of “extracting rPPG sub-signals of the original video at different frame rates from each of the feature sequences” includes:

[0052] Step S51: Determine the spatiotemporal attention features of each feature sequence according to each feature sequence.

[0053] In some embodiments, step S51 "determining the spatiotemporal attention features of each of the feature sequences according to each of the feature sequences" includes:

[0054] Step S511: Determine the attention weight of the feature sequence according to the feature sequence.

[0055] Step S512: Determine the time channel attention feature of the feature sequence according to the attention weight and the feature sequence.

[0056] Step S513: Determine the spatiotemporal attention weight of the feature sequence according to the temporal channel attention feature.

[0057] Step S514: Determine the spatiotemporal attention features of the feature sequence based on the spatiotemporal attention weights and the temporal channel attention features.

[0058] Step S52: Determine the rPPG sub-signal of the original video at the current frame rate according to each of the spatiotemporal attention features.

[0059] In order to fully extract the rPPG signal in the video, this application designs a sophisticated rPPG signal extraction module. The module mainly consists of two parts, each of which introduces different attention mechanisms in the structure to capture the signal features more comprehensively. The first part consists of a 3D convolution layer, a temporal-channel attention (CTA), and a normalization layer. CTA focuses on the feature information of the channel dimension and the time dimension, and can effectively capture the subtle signals of different color channels that change over time. This helps the model accurately separate the features that reflect physiological information while suppressing errors introduced by noise or illumination changes. The second part consists of a 3D convolution layer, a temporal-spatial attention (STA), and a normalization layer. STA focuses on the spatial and temporal dimensions, and can capture the weak signal fluctuations of the face area over time, and effectively perceive the periodic changes of facial pixels, which is crucial to enhancing the stability of the rPPG signal. The difference between the two parts lies mainly in the different attention mechanisms introduced. By combining CTA and STA in the extraction module, we can fully utilize the features of the channel, spatial, and temporal dimensions of the video. This design improves the model's perception of rPPG signals and achieves more stable and efficient signal extraction in complex scenarios.

[0060] Temporal channel attention aims to optimize the weight differences of rPPG signals in different channels in face videos. This module is designed to extract the attention weights of both time and channel dimensions simultaneously to achieve feature association between the two dimensions. Figure 4 As shown. Since its main goal is to capture information in the time and channel dimensions, the input features are globally averaged in the spatial dimension. Then, the processed features are convolved using convolutions with three different dilation rates. For an input of shape C×T×H×W, an output result Di of C×T×1×1 will be obtained. The calculation formula is as follows:

[0061] D i =DilConv i (Ave 3,4 (CT in ))

[0062] Dilated convolution helps capture a wider range of contextual information by expanding the receptive field. This application sets the dilation rate of the dilated convolution to 1, 2, and 4. The obtained features D1, D2, and D4 are concatenated in the first dimension, and then the attention weights are calculated through a group convolution with a group set to 3, a 1×1×1 convolution kernel, and a sigmoid function. Finally, the attention weights are multiplied with the original input to obtain the temporal channel attention feature CTout, where CT in is the input feature sequence, and its calculation formula is as follows:

[0063] W ct=σ(Conv(concat(D1,D2,D4)))

[0064]

[0065] The facial features in the video will change over time, so the spatiotemporal attention module aims to jointly extract the attention weights of the spatial dimension and the temporal dimension to more accurately capture these dynamic changes. Figure 5 As shown in the figure, the module consists of two branches: the first branch performs average pooling in the channel dimension to extract spatiotemporal features; the second branch focuses on learning the characteristics of the time dimension by performing average pooling in the channel and spatial dimensions. The outputs of the two branches are processed by convolution and Sigmoid activation functions respectively, and then multiplied element by element to generate the final temporal and spatial joint attention weight Wst. The specific calculation formula is as follows:

[0066]

[0067] Among them, the convolution kernel size of the first branch is 3×3×3, which is used to capture spatiotemporal features; the convolution kernel size of the second branch is 1×1×1, which is specifically used for temporal feature extraction. Finally, the generated temporal and spatial attention weights Wst are multiplied element by element with the input feature STin (channel attention feature) to obtain the final spatiotemporal attention feature STout. The specific calculation formula is as follows:

[0068]

[0069] After each feature sequence passes through the time channel attention module and the time-space attention module, the spatiotemporal attention features of the feature sequence can be obtained. By fusing the spatiotemporal attention features of each feature sequence at each frame rate, the rPPG signal at that frame rate can be obtained, that is, the rPPG sub-signal of the original video at that frame rate.

[0070] Step S6: determining the rPPG signal of the original video according to the rPPG sub-signals at each frame rate.

[0071] In some embodiments, step S6 “determining the rPPG signal of the original video according to the rPPG sub-signals at each frame rate” includes:

[0072] Step S61: Determine the self-attention weight of the original video according to the spatiotemporal fusion features.

[0073] Step S62: According to the self-attention weight, the rPPG sub-signals of different frame rates are fused to determine the rPPG signal of the original video.

[0074] Since the rPPG sub-signals at different frame rates have different characteristics, in order to obtain a higher-precision rPPG signal, it is necessary to fuse the rPPG sub-signals of different frame rates. In the fusion process, it is necessary to consider the proportion of rPPG sub-signals of different frame rates, so it is necessary to first determine the self-attention weight of the original video. The present application determines the self-attention weight of the original video by globally pooling the spatiotemporal fusion features, and then fuses the rPPG sub-signals corresponding to each frame rate according to the self-attention weight to obtain a high-precision rPPG signal.

[0075] Since the rPPG signal itself is weak and easily interfered by various strong noises in the video, it is difficult to obtain a high-precision rPPG signal in the prior art, and to obtain a high-precision rPPG signal, complex calculations must be performed, which requires a large amount of calculations. However, the present application achieves a high-precision rPPG signal with a low amount of calculations through the above method.

[0076] In addition, the method of the present application can also be implemented by a neural network model. In some embodiments, such as Figure 6 As shown in the figure, the neural network model includes: a channel enhancement module, a feature fusion module, and a temporal feature pyramid rPPG signal extraction module. In view of the characteristics of strong noise and weak signals in the video, the channel enhancement module uses POS and CHROM to enhance the video features. The enhanced video features are added to the channel dimension, so that useful features are effectively separated from noise, thereby significantly improving the representation ability of rPPG signals. In order to further improve the effect of signal extraction, we designed a feature fusion module, which enhances the network's perception of rPPG signals by analyzing differential frames. Finally, we downsample the features in the time dimension to obtain multi-scale feature representations to construct a temporal feature pyramid, extract rPPG signals of multiple scale features respectively, and obtain the final rPPG signal after fusion. In the rPPG signal extraction module, the channel-time attention (CTA) and space-time attention (STA) mechanisms we proposed can focus on the characteristics of rPPG signals in multiple dimensions.

[0077] Embodiment 2:

[0078] Based on the foregoing embodiments, the embodiments of the present application provide an apparatus for extracting rPPG signals. The modules included in the apparatus and the units included in the modules can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0079] like Figure 7 As shown, a device for extracting rPPG signals includes: a first acquisition module 1, a first enhancement module 2, a first fusion module 3, a first segmentation module 4, a first extraction module 5 and a first determination module 6.

[0080] The first acquisition module 1 is used to obtain the original video of the rPPG signal to be extracted. The first enhancement module 2 is used to perform channel enhancement on the original video to obtain a channel enhanced video. The first fusion module 3 is used to extract the spatiotemporal fusion features of the rPPG signal from the channel enhanced video. The first segmentation module 4 is used to segment the spatiotemporal fusion features into feature sequences with different frame rates. The first extraction module 5 is used to extract the rPPG sub-signals of the original video at different frame rates from each of the feature sequences. The first determination module 6 is used to determine the rPPG signal of the original video according to the rPPG sub-signals at each frame rate.

[0081] Each module in the above-mentioned rPPG signal extraction device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the device in the form of hardware, or can be stored in the memory in the processing device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation.

[0082] Embodiment 3:

[0083] In a third aspect, the present application provides a computer electronic production device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0084] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0085] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0086] Embodiment 4:

[0087] In a fourth aspect, the present application proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0088] Embodiment 5:

[0089] In a fifth aspect, the present application proposes a computer program product, comprising a computer program / instructions, characterized in that when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0090] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as: ROM), random access memory (Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

[0091] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for extracting rPPG signals, characterized in that: include: Obtain the original video of the rPPG signal to be extracted; Performing channel enhancement on the original video to obtain a channel enhanced video; Extracting spatiotemporal fusion features of rPPG signals from the channel enhanced video; Segmenting the spatiotemporal fusion features into feature sequences with different frame rates; Extracting rPPG sub-signals of the original video at different frame rates from each of the feature sequences; The rPPG signal of the original video is determined according to the rPPG sub-signals at each frame rate.

2. The method according to claim 1, characterized in that The step of performing channel enhancement on the original video to obtain the channel enhanced video includes: generating a Boolean image and a color spectrum according to the raw video; The channel-enhanced video is generated based on the Boolean image, the chromatogram, and the original video.

3. The method according to claim 1, characterized in that The extracting the spatiotemporal fusion features of the rPPG signal from the channel enhanced video comprises: Obtaining a plurality of time-shifted videos by performing a time-shift operation on the channel-enhanced video; Determine a difference frame between two adjacent time-shifted videos according to the plurality of time-shifted videos; The plurality of difference frames are spliced, and the spatiotemporal fusion features of the rPPG signal are extracted from the splicing results.

4. The method according to claim 1, characterized in that The extracting the rPPG sub-signals of the original video at different frame rates from each of the feature sequences includes: The following operations are performed for multiple feature sequences at any frame rate: Determine the spatiotemporal attention features of each of the feature sequences according to each of the feature sequences; Determine the rPPG sub-signal of the original video at the current frame rate according to each of the spatiotemporal attention features.

5. The method according to claim 4, characterized in that Determining the spatiotemporal attention features of each of the feature sequences according to each of the feature sequences includes: For any feature sequence, perform the following operations: Determining an attention weight of the feature sequence according to the feature sequence; Determine a time channel attention feature of the feature sequence according to the attention weight and the feature sequence; Determining the spatiotemporal attention weight of the feature sequence according to the temporal channel attention feature; The spatiotemporal attention features of the feature sequence are determined according to the spatiotemporal attention weights and the temporal channel attention features.

6. The method according to claim 1, characterized in that The determining the rPPG signal of the original video according to the rPPG sub-signals at various frame rates includes: Determining a self-attention weight of the original video according to the spatiotemporal fusion feature; The rPPG signal of the original video is determined by fusing the rPPG sub-signals of different frame rates according to the self-attention weight.

7. A device for extracting rPPG signals, characterized in that: include: A first acquisition module is used to acquire the original video of the rPPG signal to be extracted; A first enhancement module, used for performing channel enhancement on the original video to obtain a channel enhanced video; A first fusion module, used for extracting spatiotemporal fusion features of rPPG signals from the channel enhanced video; A first segmentation module, used to segment the spatiotemporal fusion features into feature sequences with different frame rates; A first extraction module, used to extract rPPG sub-signals of the original video at different frame rates from each of the feature sequences; The first determination module is used to determine the rPPG signal of the original video according to the rPPG sub-signals at various frame rates.

8. A computer electronic production equipment, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images