Remote heart rate measurement method based on color amplification and space-time self-attention

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a remote heart rate measurement method based on color amplification and space-time self-attention. The method comprises the following steps: step 1, processing each frame of image of a video through a face key point detection technology to realize cutting and face alignment operations, secondly, an end-to-end trainable heart rate detection neural network model combining video color amplification, space-time self-attention feature extraction and comparative learning is constructed; 3, inputting the training set into the center rate detection neural network model in the step 2 for training, and verifying by using the test set; step 4, performing remote heart rate measurement on the new video by using the trained heart rate detection neural network model; according to the model provided by the invention, the challenge of how to accurately model the fine change of the face color in remote heart rate measurement is solved, and the accuracy and effectiveness of measurement are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a remote heart rate measurement method based on color amplification and spatiotemporal self-attention. Background Art

[0002] As a contactless physiological monitoring technology, remote heart rate measurement has great potential in various fields such as healthcare, health tracking and affective computing. By analyzing the facial video feed of the monitored individual, the technology can extract the heart rate signal without the need for direct skin contact, providing a convenient and comfortable means for individuals to monitor their heart activity.

[0003] In recent years, models based on self-attention mechanisms have become the dominant deep neural network architecture in computer vision, outperforming convolution operations in modeling long-term dependencies. However, deep neural networks based on pure self-attention mechanisms have deficiencies in temporal and spatial modeling and cannot effectively extract temporal and spatial features of video sequence signals, which is still a certain drawback for video sequences. The facial color changes caused by blood flow are usually subtle and imperceptible to the human eye. Video magnification technology can be used to highlight the periodic color changes of the face, thereby reducing the complexity of feature extraction. Although traditional Euler video magnification can amplify weak color signals, it relies on manually designed spatial decomposition filters and uses Gaussian pyramid decomposition methods to separate color signals and motion signals, resulting in simultaneous amplification of color and motion, which may affect the final prediction results.

[0004] In view of this, a remote heart rate measurement method based on deep learning video color amplification and spatiotemporal self-attention is proposed to solve the above problems. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a remote heart rate measurement method based on color amplification and spatiotemporal self-attention in view of the deficiencies in the prior art, so as to improve the accuracy and effectiveness of remote heart rate measurement.

[0006] In order to solve the above technical problems, the present invention discloses a remote heart rate measurement method based on color amplification and spatiotemporal self-attention, comprising the following steps:

[0007] Step 1: Process each frame of the video using facial key point detection technology to achieve cropping and face alignment operations, and then divide the cropped and aligned video sequence into a training set and a test set;

[0008] Step 2: Build an end-to-end trainable heart rate detection neural network model that combines video color amplification, spatiotemporal self-attention feature extraction, and contrastive learning;

[0009] Step 3, input the training set into the heart rate detection neural network model in step 2 for training, and use the test set for verification to obtain the trained heart rate detection neural network model;

[0010] Step 4: Use the trained heart rate detection neural network model to remotely measure the heart rate of the new video.

[0011] Wherein, the step 2 specifically includes the following steps:

[0012] Step 2-1, construct a dataset that simulates slight color changes, and use this dataset to pre-train the video color amplification module;

[0013] Step 2-2, input the processed data set in step 1 into the color amplification module based on deep learning, amplify the subtle color changes of the face to obtain the color amplification features;

[0014] Step 2-3, input the color amplification features obtained in step 2-2 into the spatiotemporal self-attention feature extraction module, which models the local and global spatiotemporal relationship between the color amplification features and learns the long-range dependency between the amplified features;

[0015] Step 2-4: The color amplification features obtained in step 2-2 need to be input into the contrast learning module to further enhance the network's ability to extract features;

[0016] Step 2-5: After the operation of step 2-4, the heart rate signal output by the spatiotemporal self-attention feature extraction module in step 2-3 is calculated to obtain the heart rate signal value of the video sequence.

[0017] Wherein, in step 2, the color amplification module includes a representation separation unit, a bandpass filter and an amplification processing unit; wherein the representation separation unit includes three layers of convolutional layers and five layers of residual networks, which are used to separate the color representation and motion representation of each frame of the image; the bandpass filter filters the color signal synthesized by the color representation of these video frames to obtain a color signal within the frequency range of interest; the amplification processing unit uses a combination of convolutional layers, residual blocks and linear amplification to amplify the color signal to obtain the final color amplification feature.

[0018] Wherein, in step 2, the spatiotemporal self-attention feature extraction module includes spatiotemporal feature pipeline, spatiotemporal feature extraction and post-processing unit; wherein, the spatiotemporal feature pipeline includes three layers of neural network layers, each layer consists of three-dimensional convolution, batch normalization, activation layer and three-dimensional maximum pooling, which is used to capture information of different scales for the obtained color amplification features and enhance feature representation; the spatiotemporal feature extraction unit includes a block separation layer and two layers of spatiotemporal feature extraction blocks composed of a multi-head self-attention mechanism, the first layer of spatiotemporal feature extraction block includes a linear mapping layer and a multi-head self-attention block, which are used to capture the local and global spatiotemporal relationship between the amplified features, and the second layer of spatiotemporal feature extraction block includes a three-dimensional block merging layer for integrating local block information and a multi-head self-attention block, which is used to further refine and extract the spatiotemporal features; the post-processing unit includes a deconvolution, a batch normalization, an activation layer, an adaptive average pooling and a three-dimensional convolution, which are used to generate a single-channel feature map of the final output.

[0019] Wherein, in step 2, the contrast learning module includes a frequency up-sampling unit, a frequency down-sampling unit and a reverse sampling unit; wherein the frequency up-sampling unit and the frequency down-sampling unit are used to perform sampling processing on the color amplification feature, and the sampled signal is input into the spatiotemporal self-attention feature extraction module to obtain the extracted up-sampling feature and down-sampling feature, and the up-sampling feature and the down-sampling feature are added to obtain a positive sample; the reverse sampling unit is used to perform reverse sampling processing on the down-sampling feature to obtain a negative sample;

[0020] Preferably, each sampling unit includes a sampling layer, and the target size is adjusted by different frequency factors. The frequency factors in the upsampling and downsampling units are random values ​​between 1.25 and 1.5 and 0.5 and 0.75, respectively, and are directly adjusted to the original values ​​in the desampling unit.

[0021] Preferably, in step 2, the output obtained by inputting the color amplification feature in step 2-3 into the spatiotemporal self-attention feature extraction module is used as an anchor sample, and a contrast loss function is applied between the positive sample, the negative sample and the anchor point sample; wherein the contrast loss function uses a power spectral density mean square error function;

[0022] More preferably, the contrast loss function uses a power spectral density mean square error function, first calculating the power spectral density of each positive sample, negative sample, and anchor point sample, then calculating the mean square error value between the positive sample and the anchor point sample, and the mean square error value between the negative sample and the anchor point sample, and finally adding the two values. By utilizing the similarity between positive samples and the difference between negative samples, the model can learn more semantic feature representations and improve the accuracy of detection.

[0023] Preferably, the specific steps of step 4 are: processing the new video according to step 1 to obtain the corresponding video sequence, inputting it into the trained and verified heart rate detection neural network obtained in step 3, and finally obtaining the heart rate signal measurement value.

[0024] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of the present invention.

[0025] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of the present invention are implemented.

[0026] Beneficial effects: The present invention proposes an end-to-end learnable neural network model for heart rate detection, which solves the challenge of how to accurately model subtle changes in facial color in remote heart rate measurement. The color amplification module amplifies subtle color changes in facial videos, and then inputs the amplified color features into the spatiotemporal self-attention feature extraction module for spatiotemporal feature extraction to model global and local spatiotemporal relationships. In addition, we construct positive and negative sample pairs based on the principle of video frequency resampling for contrastive learning, and introduce contrastive learning to enhance the effectiveness of learning heart rate information from weak data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0028] Figure 1 Schematic diagram of the flow of a remote heart rate measurement method based on color amplification and spatiotemporal self-attention in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail in combination with the following specific embodiments and with reference to the accompanying drawings, but it is not intended to limit the scope of the present invention.

[0030] Embodiment 1:

[0031] The present invention is a remote heart rate measurement method based on color amplification and spatiotemporal self-attention, such as Figure 1 As shown, it mainly includes the following steps:

[0032] Step 1: Select multiple videos, use facial key point detection technology to process each frame of the image, implement cropping and face alignment operations, and finally divide the cropped videos into training sets and test sets;

[0033] Step 2: Build an end-to-end trainable heart rate detection neural network model that combines video color amplification, spatiotemporal self-attention feature extraction, and contrastive learning;

[0034] Step 3: Input the training set processed by step 1 into the network described in step 2 for training to obtain a trained heart rate detection neural network. Input the test set processed by step 1 into the trained heart rate detection neural network for testing;

[0035] Step 4: remotely measure the heart rate of the new video, including: processing the new video according to step 1 to obtain a corresponding video sequence, inputting the trained and verified heart rate detection neural network obtained in step 3, and finally obtaining a heart rate signal measurement value.

[0036] In this embodiment, the process of constructing an end-to-end trainable heart rate detection neural network model combining video color amplification, spatiotemporal self-attention feature extraction, and contrastive learning described in step 2 specifically includes:

[0037] Step 2-1, pre-training the video color amplification module: constructing a dataset simulating small color changes based on the existing simulated small motion dataset and its design concept, and using this dataset to pre-train the video color amplification module in step 2-2;

[0038] Step 2-2, input the processed data set in step 1 into the color amplification module based on deep learning, amplify the subtle color changes of the face to obtain the color amplification features;

[0039] Step 2-3, input the color amplification features obtained in step 2-2 into the spatiotemporal self-attention feature extraction module, which models the local and global spatiotemporal relationship between the color amplification features and learns the long-range dependency between the amplified features;

[0040] Step 2-4: The color amplification features obtained in step 2-2 need to be input into the contrast learning module to further enhance the network's ability to extract features;

[0041] Step 2-5: After the operation of step 2-4, the heart rate signal output by the spatiotemporal self-attention feature extraction module in step 2-3 is calculated to obtain the heart rate signal value of the video sequence.

[0042] Specifically, the video color amplification module includes three units: representation separation, bandpass filtering, and amplification processing. The representation separation unit separates the color representation and motion representation of each frame of the image. The module is fully convolutional and includes three layers of convolutional layers and five layers of residual networks. Compared with the motion representation, the color representation undergoes two downsampling operations implemented by strided convolution to reduce the spatial frequency and reduce noise. The bandpass filter filters the color signal synthesized by the color representation of these video frames to obtain the color signal within the frequency range of interest. The amplification module uses a combination of convolutional layers, residual blocks, and linear amplification to amplify the color signal to obtain the final color amplification feature.

[0043] Next, the amplified color features are sent to the spatiotemporal self-attention feature extraction module, which includes spatiotemporal feature pipeline, spatiotemporal feature extraction and post-processing units. The spatiotemporal feature pipeline unit includes three layers of neural network layers, each of which consists of three-dimensional convolution, batch normalization, activation layer and three-dimensional maximum pooling, which will capture information of different scales for the color amplification features and enhance feature representation. The spatiotemporal feature extraction unit includes a block separation layer for converting the pipelined features into a series of three-dimensional blocks of the same size and two layers of spatiotemporal feature extraction blocks composed of multi-head self-attention mechanisms. The first layer of spatiotemporal feature extraction blocks includes a linear mapping layer and a multi-head self-attention block to capture the local and global spatiotemporal relationships between the amplified features. The second layer of spatiotemporal feature extraction blocks includes a three-dimensional block merging layer for integrating local block information and a multi-head self-attention block to further refine and extract the spatiotemporal features. Each multi-head self-attention block includes a window multi-head self-attention layer, a shifted window multi-head self-attention layer, and two multi-layer perceptron layers. Each self-attention layer and multi-layer perceptron layer contains layer normalization and residual connections. The post-processing module includes a deconvolution, a batch normalization, an activation layer, an adaptive average pooling, and a 3D convolution to generate the final output single-channel feature map.

[0044] The amplified color features are also fed into the contrastive learning module. The contrastive learning module generates the positive and negative sample pairs required for contrastive learning by resampling the video sequence with different frequency factors based on the principle of video frequency resampling. The module includes a frequency upsampling unit, a frequency downsampling unit and a desampling unit. Each sampling unit includes a sampling layer. The target size is adjusted by different frequency factors. The frequency factors in the upsampling and downsampling units are random values ​​between 1.25 and 1.5 and 0.5 and 0.75, respectively. The desampling unit directly adjusts them to the original values. The output of the obtained color amplification features after upsampling and downsampling is input into the spatiotemporal self-attention feature extraction module in step 2 to obtain the extracted upsampled features and downsampled features. The downsampled features are desampled to obtain negative samples, and the upsampled features and downsampled features are added to obtain positive samples. The obtained color amplification features are output from the spatiotemporal self-attention feature extraction module as anchor samples. The contrast loss function is applied between positive samples, negative samples and anchor point samples. The contrast loss function uses the power spectral density mean square error function. First, the power spectral density of each positive sample, negative sample, and anchor point sample is calculated, and then the mean square error value between the positive sample and the anchor point sample, and the mean square error value between the negative sample and the anchor point sample are calculated respectively, and finally the two values ​​are added together. By utilizing the similarity between positive samples and the difference between negative samples, the model can learn more semantic feature representations and improve the accuracy of detection.

[0045] In summary, a remote heart rate measurement method based on color amplification and spatiotemporal self-attention of the present invention mainly includes three modules: video color amplification, spatiotemporal self-attention feature extraction and contrastive learning. The video color amplification module is implemented by a deep neural network, which initially amplifies subtle facial color changes in the input video. The amplified color features are input into the spatiotemporal self-attention feature extraction module. This module uses a multi-head self-attention mechanism and a convolutional neural network to model the information exchange between video amplification frames locally and globally, capture long-range dependencies and extract spatiotemporal features. In addition, the model also introduces a contrastive learning module to improve weak signal detection in facial videos. By generating positive and negative samples based on video frequency resampling, the model captures the similarities and differences between input samples, thereby learning more robust semantic feature representations.

[0046] Embodiment 2:

[0047] The computer-readable storage medium of this embodiment stores a computer program thereon, which, when executed by a processor, implements the steps in the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of Embodiment 1.

[0048] The computer-readable storage medium of this embodiment may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal; the computer-readable storage medium of this embodiment may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device.

[0049] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0050] Embodiment 3:

[0051] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of Embodiment 1 are implemented.

[0052] In this embodiment, the processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, readily available programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0053] Those skilled in the art will appreciate that the disclosed content of the embodiments may be provided as methods, systems, or computer program products. Therefore, the present solution may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Moreover, the present solution may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0054] The present solution is described with reference to the flowchart and / or block diagram of the method and computer program product according to the embodiment of the present solution. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions; these computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0057] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0058] The present invention provides a concept and method for remote heart rate measurement based on color amplification and spatiotemporal self-attention. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A remote heart rate measurement method based on color amplification and spatiotemporal self-attention, characterized in that: The following steps are involved: Step 1: Process each frame of the video using facial key point detection technology to achieve cropping and face alignment operations, and then divide the cropped and aligned video sequence into a training set and a test set; Step 2: Build an end-to-end trainable heart rate detection neural network model that combines video color amplification, spatiotemporal self-attention feature extraction, and contrastive learning; Step 3, input the training set into the heart rate detection neural network model in step 2 for training, and use the test set for verification to obtain the trained heart rate detection neural network model; Step 4: Use the trained heart rate detection neural network model to remotely measure the heart rate of the new video.

2. The measuring method according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2-1, pre-training video color amplification module: construct a data set that simulates slight color changes, and use this data set to pre-train the video color amplification module; Step 2-2, input the processed data set in step 1 into the color amplification module based on deep learning, amplify the subtle color changes of the face to obtain color amplification features; Step 2-3, input the color amplification features obtained in step 2-2 into the spatiotemporal self-attention feature extraction module, which models the local and global spatiotemporal relationship between the color amplification features and learns the long-range dependency between the amplified features; Step 2-4: The color amplification features obtained in step 2-2 need to be input into the contrast learning module to further enhance the network's ability to extract features; Step 2-5: After the operation of step 2-4, the heart rate signal output by the spatiotemporal self-attention feature extraction module in step 2-3 is calculated to obtain the heart rate signal value of the video sequence.

3. The measuring method according to claim 2, characterized in that: In step 2, the color amplification module includes a representation separation unit, a bandpass filter and an amplification processing unit; wherein the representation separation unit includes three layers of convolutional layers and five layers of residual networks to separate the color representation and motion representation of each frame of the image; the bandpass filter filters the color signal synthesized by the color representation of these video frames to obtain a color signal within the frequency range of interest; the amplification processing unit uses a combination of convolutional layers, residual blocks and linear amplification to amplify the color signal to obtain the final color amplification feature.

4. The measuring method according to claim 2, characterized in that: In step 2, the spatiotemporal self-attention feature extraction module includes spatiotemporal feature pipeline, spatiotemporal feature extraction and post-processing unit; wherein the spatiotemporal feature pipeline includes three layers of neural network layers, each layer consists of three-dimensional convolution, batch normalization, activation layer and three-dimensional maximum pooling, which is used to capture information of different scales for the obtained color amplification features and enhance feature representation; the spatiotemporal feature extraction unit includes a block separation layer and two layers of spatiotemporal feature extraction blocks composed of multi-head self-attention mechanisms, the first layer of spatiotemporal feature extraction blocks includes a linear mapping layer and a multi-head self-attention block, which are used to capture the local and global spatiotemporal relationship between the amplified features, and the second layer of spatiotemporal feature extraction blocks includes a three-dimensional block merging layer for integrating local block information and a multi-head self-attention block, which is used to further refine and extract the spatiotemporal features; the post-processing unit includes a deconvolution, a batch normalization, an activation layer, an adaptive average pooling and a three-dimensional convolution, which are used to generate a single-channel feature map of the final output.

5. The measuring method according to claim 2, characterized in that: In step 2, the contrastive learning module includes a frequency upsampling unit, a frequency downsampling unit and a reverse sampling unit; wherein the frequency upsampling unit and the frequency downsampling unit are used to perform sampling processing on the color amplification feature, and the sampled signal is input into the spatiotemporal self-attention feature extraction module to obtain the extracted upsampling feature and downsampling feature, and the upsampling feature and the downsampling feature are added to obtain a positive sample; the reverse sampling unit is used to perform reverse sampling processing on the downsampling feature to obtain a negative sample.

6. The measuring method according to claim 5, characterized in that: In the step 2, the output obtained by inputting the color amplification feature in step 2-3 into the spatiotemporal self-attention feature extraction module is used as an anchor sample, and the contrast loss function is applied between the positive sample, the negative sample and the anchor point sample; wherein the contrast loss function uses a power spectral density mean square error function.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a remote heart rate measurement method based on color amplification and spatiotemporal self-attention is implemented as described in any one of claims 1 to 6.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the remote heart rate measurement method based on color amplification and spatiotemporal self-attention is implemented as described in any one of claims 1 to 6.

Images