Remote heart rate measurement method based on color magnification and spatio-temporal self-attention

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a remote heart rate measurement method based on color amplification and space-time self-attention; the method comprises the following steps: step 1, processing each frame of image of a video through a face key point detection technology, realizing a cutting and face alignment operation, and then dividing the video sequence after cutting and alignment into a training set and a test set; step 2, constructing an end-to-end trainable heart rate detection neural network model combined with video color amplification, space-time self-attention feature extraction and contrast learning; step 3, inputting the training set into the heart rate detection neural network model of step 2 for training, and verifying by using the test set; step 4, using the trained heart rate detection neural network model to measure the remote heart rate of a new video; the model proposed in the present application solves the challenge of how to accurately model the subtle changes of facial color in remote heart rate measurement, and improves the accuracy and effectiveness of measurement.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote heart rate measurement method based on color amplification and spatiotemporal self-attention. Background Art

[0002] As a contactless physiological monitoring technology, remote heart rate measurement has enormous potential in various fields, including healthcare, health tracking, and affective computing. By analyzing a facial video feed of the monitored individual, the technology can extract heart rate signals without the need for direct skin contact, providing a convenient and comfortable means for individuals to monitor their heart activity.

[0003] In recent years, models based on self-attention mechanisms have outperformed convolutional operations in modeling long-term dependencies and have become the dominant deep neural network architecture in computer vision. However, deep neural networks based solely on self-attention mechanisms have shortcomings in temporal and spatial modeling, making them ineffective in extracting temporal and spatial features of video sequence signals. This still presents a drawback for video sequences. Facial color changes caused by blood flow are often subtle and imperceptible to the human eye. Video upscaling techniques can be used to highlight the periodic color changes of the face, thereby reducing the complexity of feature extraction. While traditional Eulerian video upscaling can amplify weak color signals, it relies on manually designed spatial decomposition filters and uses Gaussian pyramid decomposition methods to separate color and motion signals, resulting in simultaneous amplification of both color and motion, which can affect the final prediction results.

[0004] In view of this, a remote heart rate measurement method based on deep learning video color amplification and spatiotemporal self-attention is proposed to solve the above problems. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a remote heart rate measurement method based on color amplification and spatiotemporal self-attention in response to the shortcomings of the existing technology, so as to improve the accuracy and effectiveness of remote heart rate measurement.

[0006] In order to solve the above technical problems, the present invention discloses a remote heart rate measurement method based on color amplification and spatiotemporal self-attention, comprising the following steps:

[0007] Step 1: Use facial key point detection technology to process each frame of the video to achieve cropping and face alignment operations, and then divide the cropped and aligned video sequence into a training set and a test set;

[0008] Step 2: Build an end-to-end trainable heart rate detection neural network model that combines video color amplification, spatiotemporal self-attention feature extraction, and contrastive learning;

[0009] Step 3, input the training set into the step 2 heart rate detection neural network model for training, and verify by using the test set, to obtain the trained heart rate detection neural network model;

[0010] Step 4, using the trained heart rate detection neural network model to measure the remote heart rate of the new video.

[0011] The step 2 specifically comprises the following steps:

[0012] Step 2-1, a data set simulating micro color change is constructed, and the data set is used for pre-training video color amplification module;

[0013] Step 2-2, the data set processed in step 1 is input into the color amplification module based on deep learning, and the subtle color change of the face is amplified to obtain color amplification features;

[0014] Step 2-3, the color amplification features obtained in step 2-2 are input into the spatio-temporal self-attention feature extraction module, which models the local and global spatio-temporal relationship between the color amplification features and learns the long-range dependency relationship between the amplification features;

[0015] Step 2-4, the color amplification features obtained in step 2-2 also need to be input into the contrast learning module to further enhance the ability of the network to extract features;

[0016] Step 2-5, after the operation in step 2-4, the heart rate signal output by the spatio-temporal self-attention feature extraction module in step 2-3 is calculated, and the heart rate signal value of the video sequence can be obtained.

[0017] In step 2, the color amplification module comprises a representation separation unit, a band-pass filter and an amplification processing unit; wherein the representation separation unit comprises three convolutional layers and five residual networks, which are used to separate the color representation and motion representation of each frame of image; the band-pass filter filters the color signal synthesized by the color representation of the video frames, to obtain the color signal in the frequency range of interest; the amplification processing unit uses convolutional layers, residual blocks and linear amplification to amplify the color signal, to obtain the final color amplification features.

[0018] Wherein, in step 2, the spatiotemporal self-attention feature extraction module includes spatiotemporal feature pipeline, spatiotemporal feature extraction and post-processing unit; wherein, the spatiotemporal feature pipeline includes three layers of neural network layers, each layer consists of three-dimensional convolution, batch normalization, activation layer and three-dimensional maximum pooling, which is used to capture information of different scales of the obtained color amplification features and enhance feature representation; the spatiotemporal feature extraction unit includes a block separation layer and two layers of spatiotemporal feature extraction blocks composed of a multi-head self-attention mechanism, the first layer of spatiotemporal feature extraction block includes a linear mapping layer and a multi-head self-attention block to capture the local and global spatiotemporal relationship between the amplified features, the second layer of spatiotemporal feature extraction block includes a three-dimensional block merging layer for integrating local block information and a multi-head self-attention block to further refine and extract the spatiotemporal features; the post-processing unit includes a deconvolution, a batch normalization, an activation layer, an adaptive average pooling and a three-dimensional convolution to generate a single-channel feature map for the final output.

[0019] Wherein, in step 2, the contrastive learning module includes a frequency upsampling unit, a frequency downsampling unit and a desampling unit; wherein the frequency upsampling unit and the frequency downsampling unit are used to sample the color amplification feature, input the sampled signal into the spatiotemporal self-attention feature extraction module to obtain the extracted upsampling feature and downsampling feature, and add the upsampling feature and the downsampling feature to obtain a positive sample; the desampling unit is used to desampling the downsampling feature to obtain a negative sample;

[0020] Preferably, each sampling unit includes a sampling layer, and the target size is adjusted by different frequency factors. The frequency factors in the upsampling and downsampling units are random values ​​between 1.25 and 1.5 and 0.5 and 0.75, respectively, and are directly adjusted to the original values ​​in the desampling unit.

[0021] Preferably, in step 2, the output obtained by inputting the color amplification feature in step 2-3 into the spatiotemporal self-attention feature extraction module is used as an anchor sample, and a contrast loss function is applied between the positive sample, the negative sample and the anchor point sample; wherein the contrast loss function uses a power spectral density mean square error function;

[0022] More preferably, the contrastive loss function uses a power spectral density mean squared error function. This function first calculates the power spectral density of each positive, negative, and anchor point sample, then calculates the mean squared error between the positive sample and the anchor point sample, and the mean squared error between the negative sample and the anchor point sample, and finally adds the two values ​​together. By leveraging the similarities between positive samples and the differences between negative samples, the model can learn more semantic feature representations, improving detection accuracy.

[0023] Preferably, the specific steps of step 4 are: processing the new video according to step 1 to obtain a corresponding video sequence, inputting it into the trained and verified heart rate detection neural network obtained in step 3, and finally obtaining a heart rate signal measurement value.

[0024] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of the present invention.

[0025] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of the present invention are implemented.

[0026] Beneficial Effects: This paper proposes an end-to-end learnable neural network model for heart rate detection, addressing the challenge of accurately modeling subtle facial color changes during remote heart rate measurement. The color amplification module amplifies subtle color changes in facial videos, then feeds the amplified color features into the spatiotemporal self-attention feature extraction module for spatiotemporal feature extraction, modeling global and local spatiotemporal relationships. Furthermore, we construct a contrastive learning algorithm for positive and negative sample pairs based on the principle of video frequency resampling, introducing contrastive learning to enhance the effectiveness of learning heart rate information from weak data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0028] Figure 1 Schematic diagram of a flow chart of a remote heart rate measurement method based on color amplification and spatiotemporal self-attention in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail with reference to the following specific embodiments and accompanying drawings, but this does not limit the scope of the present invention.

[0030] Example 1:

[0031] The present invention is a remote heart rate measurement method based on color amplification and spatiotemporal self-attention, such as Figure 1 As shown, it mainly includes the following steps:

[0032] Step 1: Select multiple videos, use facial key point detection technology to process each frame of the image, implement cropping and face alignment operations, and finally divide the cropped videos into training and test sets;

[0033] Step 2: Build an end-to-end trainable heart rate detection neural network model that combines video color amplification, spatiotemporal self-attention feature extraction, and contrastive learning;

[0034] Step 3: Input the training set processed in step 1 into the network described in step 2 for training to obtain a trained heart rate detection neural network. Input the test set processed in step 1 into the trained heart rate detection neural network for testing;

[0035] Step 4: remotely measure the heart rate of the new video, including: processing the new video according to step 1 to obtain a corresponding video sequence, inputting the trained and verified heart rate detection neural network obtained in step 3, and finally obtaining a heart rate signal measurement value.

[0036] In this embodiment, the process of constructing an end-to-end trainable heart rate detection neural network model combining video color amplification, spatiotemporal self-attention feature extraction, and contrastive learning described in step 2 specifically includes:

[0037] Step 2-1, pre-training the video color amplification module: Based on the existing simulated small motion dataset and its design ideas, a dataset simulating small color changes is constructed, and this dataset is used to pre-train the video color amplification module in step 2-2;

[0038] Step 2-2: Input the processed data set in step 1 into the deep learning-based color amplification module to amplify the subtle color changes on the face to obtain color amplification features;

[0039] Step 2-3: Input the color amplification features obtained in step 2-2 into the spatiotemporal self-attention feature extraction module, which models the local and global spatiotemporal relationships between the color amplification features and learns the long-range dependencies between the amplified features;

[0040] Step 2-4: The color amplification features obtained in step 2-2 need to be input into the contrast learning module to further enhance the network's ability to extract features;

[0041] Step 2-5: After the operation of step 2-4, the heart rate signal output by the spatiotemporal self-attention feature extraction module in step 2-3 is calculated to obtain the heart rate signal value of the video sequence.

[0042] Specifically, the video color amplification module includes three units: representation separation, bandpass filtering, and amplification processing. The representation separation unit separates the color representation and motion representation of each frame of the image. The module is fully convolutional and includes three layers of convolutional layers and a five-layer residual network. Compared with the motion representation, the color representation undergoes a downsampling operation achieved by two strided convolutions to reduce the spatial frequency and reduce noise. The bandpass filter filters the color signal synthesized by the color representation of these video frames to obtain the color signal within the frequency range of interest. The amplification module uses a combination of convolutional layers, residual blocks, and linear amplification to amplify the color signal to obtain the final color amplification feature.

[0043] Next, the amplified color features are fed into a spatiotemporal self-attention feature extraction module, which includes a spatiotemporal feature pipeline, spatiotemporal feature extraction, and post-processing unit. The spatiotemporal feature pipeline unit comprises three neural network layers, each consisting of a 3D convolution, batch normalization, activation layer, and 3D max pooling. This layer captures information at different scales for the amplified color features and enhances feature representation. The spatiotemporal feature extraction unit includes a block separation layer for converting the pipelined features into a series of 3D blocks of equal size, and two spatiotemporal feature extraction blocks composed of a multi-head self-attention mechanism. The first spatiotemporal feature extraction block includes a linear mapping layer and a multi-head self-attention block to capture local and global spatiotemporal relationships between the amplified features. The second spatiotemporal feature extraction block includes a 3D block merging layer for integrating local block information and a multi-head self-attention block to further refine and extract the spatiotemporal features. Each multi-head self-attention block consists of a windowed multi-head self-attention layer, a shifted windowed multi-head self-attention layer, and two multi-layer perceptron layers. Each self-attention layer and multi-layer perceptron layer is followed by layer normalization and residual connections. The post-processing module includes a deconvolution, a batch normalization, an activation layer, an adaptive average pooling, and a 3D convolution to generate the final single-channel feature map.

[0044] The amplified color features are also fed into the contrastive learning module. Based on the principle of video frequency resampling, this module resamples the video sequence at different frequency factors to generate the positive and negative sample pairs required for contrastive learning. This module comprises an upsampling unit, a downsampling unit, and a desampling unit. Each sampling unit includes a sampling layer that adjusts the target size using different frequency factors. The frequency factors in the upsampling and downsampling units are random values ​​ranging from 1.25 to 1.5 and 0.5 to 0.75, respectively. The desampling unit directly adjusts these factors back to their original values. The outputs of the up- and down-sampling color amplification features are fed into the spatiotemporal self-attention feature extraction module in step 2, yielding the extracted upsampled and downsampled features, respectively. The downsampled features are desampled to generate negative samples. The upsampled and downsampled features are summed to generate positive samples. The resulting color amplification features are then passed through the spatiotemporal self-attention feature extraction module as anchor samples. A contrastive loss function is applied between the positive, negative, and anchor samples. The contrastive loss function uses the power spectral density mean squared error function. It first calculates the power spectral density of each positive sample, negative sample, and anchor point sample, then calculates the mean squared error between the positive sample and the anchor point sample, and the mean squared error between the negative sample and the anchor point sample, and finally adds the two values. By leveraging the similarities between positive samples and the differences between negative samples, the model can learn more semantic feature representations, improving detection accuracy.

[0045] In summary, the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of the present invention mainly includes three modules: video color amplification, spatiotemporal self-attention feature extraction and contrastive learning. The video color amplification module is implemented by a deep neural network, which initially amplifies subtle facial color changes in the input video. The amplified color features are input into the spatiotemporal self-attention feature extraction module. This module uses a multi-head self-attention mechanism and a convolutional neural network to model the information exchange between the magnified video frames locally and globally, capture long-range dependencies and extract spatiotemporal features. In addition, the model also introduces a contrastive learning module to improve weak signal detection in facial videos. By generating positive and negative samples based on video frequency resampling, the model captures the similarities and differences between input samples, thereby learning more robust semantic feature representations.

[0046] Example 2:

[0047] The computer-readable storage medium of this embodiment stores a computer program thereon, which, when executed by a processor, implements the steps of the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of embodiment 1.

[0048] The computer-readable storage medium of this embodiment may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal; the computer-readable storage medium of this embodiment may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device.

[0049] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0050] Example 3:

[0051] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the remote heart rate measurement method based on color amplification and spatiotemporal self-attention of embodiment 1 are implemented.

[0052] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The memory can include read-only memory and random access memory, and provide instructions and data to the processor. A part of the memory can also include non-volatile random access memory. For example, the memory can also store information about the device type.

[0053] Those skilled in the art will appreciate that the disclosed contents of the embodiments may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0054] The present invention is described with reference to the flowcharts and / or block diagrams of the methods and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of the processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions; these computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0057] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0058] The present invention provides a concept and method for remote heart rate measurement based on color amplification and spatiotemporal self-attention. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also considered to be within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A remote heart rate measurement method based on color amplification and spatiotemporal self-attention, characterized in that: The following steps are involved: Step 1: Use facial key point detection technology to process each frame of the video to achieve cropping and face alignment operations, and then divide the cropped and aligned video sequence into a training set and a test set; Step 2: Build an end-to-end trainable heart rate detection neural network model that combines video color amplification, spatiotemporal self-attention feature extraction, and contrastive learning; Step 3: Input the training set into the heart rate detection neural network model in step 2 for training, and use the test set for verification to obtain the trained heart rate detection neural network model; Step 4: Use the trained heart rate detection neural network model to remotely measure the heart rate of the new video; The step 2 specifically includes the following steps: Step 2-1, pre-training the video color amplification module: construct a dataset that simulates slight color changes and use this dataset to pre-train the video color amplification module; Step 2-2: Input the processed data set in step 1 into the deep learning-based color amplification module to amplify the subtle color changes on the face to obtain color amplification features; Step 2-3: Input the color amplification features obtained in step 2-2 into the spatiotemporal self-attention feature extraction module, which models the local and global spatiotemporal relationships between the color amplification features and learns the long-range dependencies between the amplified features; Step 2-4: The color amplification features obtained in step 2-2 need to be input into the contrast learning module to further enhance the network's ability to extract features; Step 2-5: After the operation after step 2-4, the heart rate signal output by the spatiotemporal self-attention feature extraction module in step 2-3 is calculated to obtain the heart rate signal value of the video sequence; In step 2, the color amplification module includes a representation separation unit, a bandpass filter, and an amplification processing unit; wherein the representation separation unit includes three convolutional layers and a five-layer residual network to separate the color representation and motion representation of each frame of image; the bandpass filter filters the color signal synthesized by the color representations of these video frames to obtain a color signal within the frequency range of interest; the amplification processing unit uses a combination of convolutional layers, residual blocks, and linear amplification to amplify the color signal to obtain the final color amplification feature; In step 2, the spatiotemporal self-attention feature extraction module includes spatiotemporal feature pipeline, spatiotemporal feature extraction and post-processing unit; wherein, the spatiotemporal feature pipeline includes three layers of neural network layers, each layer consists of three-dimensional convolution, batch normalization, activation layer and three-dimensional maximum pooling, which is used to capture information of different scales of the obtained color amplification features and enhance feature representation; the spatiotemporal feature extraction unit includes a block separation layer and two layers of spatiotemporal feature extraction blocks composed of a multi-head self-attention mechanism, the first layer of spatiotemporal feature extraction block includes a linear mapping layer and a multi-head self-attention block to capture the local and global spatiotemporal relationship between the amplified features, the second layer of spatiotemporal feature extraction block includes a three-dimensional block merging layer for integrating local block information and a multi-head self-attention block to further refine and extract the spatiotemporal features; the post-processing unit includes a deconvolution, a batch normalization, an activation layer, an adaptive average pooling and a three-dimensional convolution to generate a single-channel feature map for the final output; In step 2, the contrastive learning module includes a frequency upsampling unit, a frequency downsampling unit and a desampling unit; wherein the frequency upsampling unit and the frequency downsampling unit are used to perform sampling processing on the color amplification feature, and the sampled signal is input into the spatiotemporal self-attention feature extraction module to obtain the extracted upsampling feature and downsampling feature, and the upsampling feature and the downsampling feature are added to obtain a positive sample; the desampling unit is used to perform desampling processing on the downsampling feature to obtain a negative sample.

2. The measuring method according to claim 1, wherein In step 2, the output obtained by inputting the color amplification feature in step 2-3 into the spatiotemporal self-attention feature extraction module is used as the anchor sample, and the contrast loss function is applied between the positive sample, the negative sample and the anchor point sample; wherein the contrast loss function uses the power spectral density mean square error function.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the remote heart rate measurement method based on color amplification and spatiotemporal self-attention as described in any one of claims 1 or 2 is implemented.

4. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the remote heart rate measurement method based on color amplification and spatiotemporal self-attention according to any one of claims 1 or 2 is implemented.

Images