Non-contact heart rate calculation method and system based on dual-channel input

Abstract

The present invention provides a non-contact heart rate calculation method, system, storage medium and electronic device based on dual-channel input, and relates to the field of non-contact heart rate calculation. The present invention uses facial visible light video and facial differential image sequence as inputs of a non-contact heart rate extraction model based on dual-stream input, respectively, and uses the powerful self-attention mechanism of the Transformer model to capture the correlation between each position between the video and the video differential, obtain the rPPG signal, and obtain the heart rate value of the test subject through fast Fourier transform. By designing a dual-stream input to better aggregate fine-grained information related to motion, and using the self-attention mechanism to fuse global spatiotemporal information, the robustness of the algorithm to motion is improved. In addition, a gradient dynamic adjustment strategy is designed during the model training process. The gradient direction will be dynamically adjusted according to the data in each mini batch, which effectively solves the problem of unstable optimization direction during training and effectively enhances the generalization performance of the model.

Description

Technical Field

[0001] The present invention relates to the field of non-contact heart rate calculation, and in particular to a non-contact heart rate calculation method, system, storage medium and electronic equipment based on dual-channel input. Background Art

[0002] As one of the important physiological parameters of the human body, heart rate can directly and effectively reflect the health status of the human body. Therefore, heart rate monitoring plays an irreplaceable role in monitoring human health. Traditional heart rate monitoring methods are mainly contact-based. With the development of technology, non-contact heart rate detection tasks have become a new hotspot in this field.

[0003] In the related technology, contactless heart rate extraction based on the Transformer model is a feasible contactless heart rate detection solution. This solution directly divides the video into basic units of fixed size (token, W×H×t) in the spatial and temporal dimensions, and then uses the powerful self-attention mechanism of the Transformer model to calculate feature vectors with global representation capabilities, thereby realizing effective aggregation of spatial and temporal information in the video.

[0004] However, the above scheme directly divides tokens for self-attention calculation, which is difficult to have excellent performance when the subject is moving, and thus cannot guarantee the accuracy of the non-contact heart rate detection results. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a non-contact heart rate calculation method, system, storage medium and electronic device based on dual-channel input, which solves the technical problem that it is difficult to have excellent performance when the subject is exercising.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A non-contact heart rate calculation method based on dual-channel input, a non-contact heart rate extraction model based on dual-stream input, the non-contact heart rate extraction model includes a dual-stream feature extractor, a channel attention module and an rPPG predictor based on a Transformer module; the non-contact heart rate calculation method includes:

[0010] Obtaining the original visible light video of the subject;

[0011] Detecting the human face region in the original visible light video and locating the facial landmark points on the human face region to obtain a facial visible light video; calculating the pixel difference between each frame and the previous frame based on the facial visible light video to obtain a facial differential image sequence having the same size and length as the facial visible light video;

[0012] Taking the facial visible light video and facial difference image sequence as inputs of corresponding feature extractors, respectively obtaining visible light spatiotemporal blocks and video difference spatiotemporal blocks, and splicing them to obtain original spatiotemporal blocks;

[0013] Taking the original spatiotemporal block as the input of the channel attention module, obtaining a channel weight matrix, and multiplying the channel attention matrix with the original spatiotemporal block to obtain a channel attention spatiotemporal block;

[0014] After segmenting the channel attention spatiotemporal block and fusing the linear position encoding, the feature vector containing global and local fine-grained information is obtained as the input of the Transformer module; the feature vector is used as the input of the upsampling block of the rPPG predictor, and the final rPPG signal is obtained after linear transformation;

[0015] A fast Fourier transform is performed on the rPPG signal to calculate and obtain the heart rate value of the test subject.

[0016] Preferably, the dual-stream feature extractor comprises two feature extractors of the same structure;

[0017] The feature extractor is composed of three stacked convolution modules with the same structure, and the convolution module is composed of a 3D convolution layer, a 3D batch normalization layer, a ReLU layer and a maximum pooling layer.

[0018] Preferably, the channel attention module has two channel branch networks and a Sigmoid layer built in;

[0019] Among them, the first channel branch network is composed of an average pooling layer, a 3D convolution layer, a Relu layer, and a 3D convolution layer; the second channel branch network is composed of a maximum pooling layer, a 3D convolution layer, a Relu layer, and a 3D convolution layer.

[0020] Preferably, the rPPG predictor based on the Transformer module includes a Transformer module, two upsampling blocks with the same structure and a linear layer;

[0021] The Transformer module is composed of three standard Vision Transformer modules cascaded together; the upsampling block is composed of an upsampling layer, a 3D convolution layer, a 3D batch normalization layer and an ELU layer.

[0022] Preferably, the setting process of the loss function of the contactless heart rate extraction model includes:

[0023] Based on the rPPG signal obtained by the model and the pre-acquired gold standard PPG signal, the time domain loss L is set t :

[0024] L t =1-PGor(rPPG,PPG)

[0025] Where PCor(·) represents Pearson correlation;

[0026] And based on the power spectral density of the rPPG signal obtained by the model and the pre-acquired gold standard heart rate, the frequency domain loss L is set f :

[0027] L f =CE(PSD rPPG ,hr gt )

[0028] Among them, CE(·) represents the cross entropy loss, PSD rPPG Represents the power spectral density of the rPPG signal, hr gt represents the gold standard heart rate;

[0029] The overall loss function L total Set to:

[0030] L total =L t +aL f

[0031] Among them, a is a configurable parameter.

[0032] Preferably, a gradient dynamic adjustment strategy is introduced to update parameters of the non-contact heart rate extraction model.

[0033] Preferably, the non-contact heart rate calculation method further includes:

[0034] At least the heart rate value of the test subject is visualized.

[0035] A non-contact heart rate calculation system based on dual-channel input, a non-contact heart rate extraction model based on dual-stream input, the non-contact heart rate extraction model includes a dual-stream feature extractor, a channel attention module and an rPPG predictor based on a Transformer module; the non-contact heart rate calculation system includes:

[0036] A video acquisition module is used to acquire the original visible light video of the subject;

[0037] A video preprocessing module is used to detect the face area in the original visible light video and locate the facial landmark points on the face area to obtain a facial visible light video; based on the facial visible light video, calculate the pixel difference between each frame and the previous frame to obtain a facial differential image sequence with the same size and length as the facial visible light video;

[0038] A spatiotemporal information extraction module, used to use the facial visible light video and facial differential image sequence as inputs of corresponding feature extractors, respectively obtain visible light spatiotemporal blocks and video differential spatiotemporal blocks, and obtain original spatiotemporal blocks after splicing;

[0039] A channel attention fusion module is used to take the original spatiotemporal block as the input of the channel attention module, obtain a channel weight matrix, and multiply the channel attention matrix with the original spatiotemporal block to obtain a channel attention spatiotemporal block;

[0040] An rPPG signal acquisition module is used to segment the channel attention spatiotemporal block and fuse the linear position encoding as the input of the Transformer module to obtain a feature vector containing global and local fine-grained information; the feature vector is used as the input of the upsampling block of the rPPG predictor, and the final rPPG signal is obtained after linear transformation;

[0041] The heart rate value acquisition module is used to perform fast Fourier transform on the rPPG signal to calculate and obtain the heart rate value of the test subject.

[0042] A storage medium, characterized in that it stores a computer program for non-contact heart rate calculation based on dual-channel input, wherein the computer program enables a computer to execute the non-contact heart rate calculation method as described above.

[0043] An electronic device, comprising:

[0044] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the non-contact heart rate calculation method as described above.

[0045] (III) Beneficial effects

[0046] The present invention provides a non-contact heart rate calculation method, system, storage medium and electronic device based on dual-channel input. Compared with the prior art, it has the following beneficial effects:

[0047] The present invention is based on a non-contact heart rate extraction model with dual-stream input. It performs face segmentation on the original visible light video to obtain the visible light video of the face. It also calculates the image difference between adjacent frames of the facial video, aggregates the fine-grained spatiotemporal information between the two frames, and obtains the facial differential image sequence. Then, the facial visible light video and the facial differential image sequence are used as the input of the model respectively, and the channel attention is fused. The powerful self-attention mechanism of the Transformer model is used to capture the correlation between the positions of the video and the video differential to obtain the final rPPG signal. Finally, the heart rate value of the subject is obtained through fast Fourier transform. By designing a dual-stream input to better aggregate fine-grained information related to motion, and using the self-attention mechanism to fuse global spatiotemporal information, the robustness of the algorithm to motion is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A block diagram of a non-contact heart rate calculation method based on dual-channel input provided by an embodiment of the present invention;

[0050] Figure 2 A block diagram of a non-contact heart rate calculation method based on dual-channel input provided by an embodiment of the present invention;

[0051] Figure 3 A network architecture diagram of a contactless heart rate extraction model based on dual-stream input provided by an embodiment of the present invention;

[0052] Figure 4 A network architecture diagram of a dual-stream feature extractor provided in an embodiment of the present invention;

[0053] Figure 5 A network architecture diagram of a channel attention module provided by an embodiment of the present invention;

[0054] Figure 6 A network architecture diagram of an rPPG predictor based on a Transformer module provided in an embodiment of the present invention;

[0055] Figure 7 An example diagram of a heart rate result display interface provided by an embodiment of the present invention;

[0056] Figure 8 A schematic diagram of an adaptive gradient adjustment strategy provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The embodiment of the present application solves the technical problem that it is difficult to have excellent performance when the subject is exercising by providing a non-contact heart rate calculation method, system, storage medium and electronic device based on dual-channel input.

[0059] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0060] The applicant recognizes that the existing technology directly divides tokens for self-attention calculation, which makes it difficult to extract the fine-grained information contained in the two adjacent frames within each token. Since the two frames within the video contain important information such as rPPG and movement, in order to extract high-quality physiological signals and effectively aggregate the fine-grained information between the two frames with the global information within the video, a dual-channel input Transformer model is designed to aggregate the fine-grained information between the two frames through video difference, and then fuse the fine-grained information with the global information through feature fusion and the Vision Transformer module.

[0061] In addition, most of the existing non-contact heart rate extraction methods based on deep learning can only be trained on the same dataset. However, simultaneous training on multiple datasets can enhance the generalization of the model in different scenarios. Due to the differences in data distribution between different datasets, the existing training mode may lead to unstable optimization direction during model training.

[0062] Therefore, the embodiment of the present invention also introduces an adaptive gradient adjustment strategy to project the example gradients in each mini batch into non-conflicting directions, so that the model can be trained on data sets with different data distributions, and the optimization direction still has good stability.

[0063] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0064] Embodiment 1:

[0065] like Figure 1 As shown, an embodiment of the present invention provides a non-contact heart rate calculation method based on dual-channel input, a non-contact heart rate extraction model based on dual-stream input, the non-contact heart rate extraction model includes a dual-stream feature extractor, a channel attention module and an rPPG predictor based on a Transformer module; the non-contact heart rate calculation method includes:

[0066] S1. Obtain the original visible light video of the subject;

[0067] S2, detecting the face area in the original visible light video, and locating the facial landmark points on the face area to obtain a facial visible light video; based on the facial visible light video, calculating the pixel difference between each frame and the previous frame to obtain a facial differential image sequence with the same size and length as the facial visible light video;

[0068] S3, using the facial visible light video and facial difference image sequence as inputs of corresponding feature extractors, respectively obtaining visible light spatiotemporal blocks and video difference spatiotemporal blocks, and splicing them to obtain original spatiotemporal blocks;

[0069] S4, taking the original spatiotemporal block as the input of the channel attention module, obtaining a channel weight matrix, and multiplying the channel attention matrix with the original spatiotemporal block to obtain a channel attention spatiotemporal block;

[0070] S5, after segmenting the channel attention spatiotemporal block and fusing the linear position encoding, the feature vector containing global and local fine-grained information is obtained as the input of the Transformer module; the feature vector is used as the input of the upsampling block of the rPPG predictor, and the final rPPG signal is obtained after linear transformation;

[0071] S6. Perform fast Fourier transform on the rPPG signal to calculate and obtain the heart rate value of the test subject.

[0072] The embodiment of the present invention designs a dual-stream input to better aggregate fine-grained information related to motion and utilizes a self-attention mechanism to fuse global spatiotemporal information, thereby improving the robustness of the algorithm to motion.

[0073] In an optional embodiment, if Figure 2 As shown, the non-contact heart rate calculation method also includes:

[0074] S7. At least visualize the heart rate value of the test subject.

[0075] The embodiment of the present invention converts the data into a visual form so as to intuitively and clearly display the data analysis results such as the heart rate value.

[0076] like Figure 3 As shown, Figure 3 The network architecture diagram of the contactless heart rate extraction model based on dual-stream input is given. Figure 3 The following are the steps of the above solution:

[0077] In step S1, an original visible light video of a subject is obtained;

[0078] To ensure that the face video of the subject can be captured, the visible light camera aimed at the face area is kept 0.5m to 0.8m away from the subject in this step. After clicking the start acquisition button, the acquisition device records the original visible light video of the subject.

[0079] In step S2, the face area in the original visible light video is detected, and the facial landmark points on the face area are located to obtain a facial visible light video; based on the facial visible light video, the pixel difference between each frame and the previous frame is calculated to obtain a facial differential image sequence that is consistent with the size and length of the facial visible light video.

[0080] Considering that the original facial video contains a lot of environmental noise, in order to improve the accuracy of heart rate monitoring, the embodiment of the present invention first locates the facial area of ​​the subject in the video and removes the remaining background area to reduce the impact of environmental noise on the accuracy of the algorithm. At the same time, in order to better integrate the spatiotemporal information between two frames of the video, the embodiment of the present invention also calculates the difference between the two frames of the facial video and constructs a facial difference image sequence as one of the model inputs.

[0081] (1) Face region segmentation

[0082] This step uses the open source 3DDFA_V2 algorithm to implement face detection in the video and locate the 68 facial landmark points of the face. The 68 facial landmark points are output as a matrix of size 68×3 (number of landmark points × (x coordinate, y coordinate, z depth coordinate)), and the marked parts include the facial contour, eyes, nose and mouth. The facial area can be obtained by comparing the coordinates of the 68 landmark points. The specific coordinate position is output as a 5×1 matrix ([xmin, xmax, ymax, ymin, score], where xmin represents the left boundary of the face, xmax represents the right boundary of the face, ymax represents the lower boundary of the face, ymin represents the upper boundary of the face, and score represents the possibility of successful prediction). Through the facial coordinate position, the video background area is cropped, the face area in the video is segmented, the interference of the background area in the video is reduced, and the facial visible light video is obtained.

[0083] (2) Calculate facial difference image sequence

[0084] Based on the facial visible light video obtained in (1), the pixel difference between each frame and the previous frame is calculated. To ensure that the image pixel value is greater than 0, the absolute value of the difference is taken. The difference of the first frame is the frame itself. Therefore, after this step of calculation, a facial difference image sequence with the same size and length as the facial video is obtained. The video difference contains the difference information between the two frames of the video. While containing the physiological information of the human body, it also contains information such as the movement of the human body and the change of lighting between the two frames. Introducing video difference as input can effectively aggregate the fine-grained spatiotemporal information between the two frames and enhance the performance of the model.

[0085] In step S3, the facial visible light video and facial difference image sequence are used as inputs of corresponding feature extractors to obtain visible light spatiotemporal blocks and video difference spatiotemporal blocks respectively, and the original spatiotemporal blocks are obtained after splicing.

[0086] like Figure 4 As shown, the dual-stream feature extractor includes two feature extractors with the same structure. The feature extractor is formed by stacking three convolution modules with the same structure, and the convolution module is formed by stacking a 3D convolution layer, a 3D batch normalization layer, a ReLU layer, and a maximum pooling layer.

[0087] In this step, the facial visible light video and facial differential image sequence are respectively input into two feature extractors with the same structure, aiming to extract the coarse-grained spatiotemporal features of the two inputs and realize the effect of feature dimensionality reduction, thereby reducing the subsequent calculation amount and saving computing resources to a certain extent.

[0088] After the video and video difference pass through the dual-stream feature extractor, the visible light spatiotemporal block and the video difference spatiotemporal block are obtained respectively, which contain coarse-grained spatiotemporal information. Then the two spatiotemporal blocks are spliced ​​together in the channel dimension to obtain the original spatiotemporal block, realizing the fusion of two-channel data.

[0089] In step S4, the original spatiotemporal block is used as the input of the channel attention module to obtain the channel weight matrix, and the channel attention matrix is ​​multiplied by the original spatiotemporal block to obtain the channel attention spatiotemporal block.

[0090] Since there may be a lot of redundant information in the channel dimension, which slows down the model training efficiency, this step inputs the original spatiotemporal block into the channel attention module.

[0091] like Figure 5 As shown in the figure, the channel attention module has two built-in channel branch networks and a Sigmoid layer. The first channel branch network is composed of an average pooling layer, a 3D convolution layer, a Relu layer, and a 3D convolution layer; the second channel branch network is composed of a maximum pooling layer, a 3D convolution layer, a Relu layer, and a 3D convolution layer.

[0092] See also Figure 3 , the original spatiotemporal block is calculated through two channel networks to obtain two features, and then the two output features are added to obtain the fused features. Finally, the fused features are input into a Sigmoid layer to obtain the final channel weight matrix. The channel attention matrix is ​​multiplied by the original spatiotemporal block to obtain the final channel attention spatiotemporal block that integrates the channel attention.

[0093] After this operation, the redundant information of the feature in the channel dimension is effectively reduced, while the robustness of the model is improved to a certain extent, and the convergence speed of the model is accelerated to reduce the computational cost.

[0094] In step S5, the channel attention spatiotemporal block is segmented and fused with linear position coding, and then used as the input of the Transformer module to obtain a feature vector containing global and local fine-grained information; the feature vector is used as the input of the upsampling block of the rPPG predictor, and the final rPPG signal is obtained after linear transformation.

[0095] After fusing channel attention, this step divides the spatiotemporal blocks into tokens of size 4×4×16. After fusing linear position encoding and retaining position information, it is input into the rPPG predictor based on the Transformer module.

[0096] like Figure 6As shown, the rPPG predictor based on the Transformer module includes a Transformer module, two upsampling blocks with the same structure, and a linear layer. The Transformer module is composed of three standard Vision Transformer modules in cascade; the upsampling block is composed of an upsampling layer, a 3D convolution layer, a 3D batch normalization, and an ELU layer.

[0097] Through the powerful self-attention mechanism of Transformer, a powerful feature representation that integrates global spatiotemporal information is obtained. This feature effectively combines the global spatiotemporal features in the video, and also contains the fine-grained spatiotemporal difference information between two frames extracted from the video differential. After the above operations, a feature vector containing global and local fine-grained information is obtained. Then, the feature vector is passed through the upsampling block, and the feature is upsampled to the length of the video in the time dimension. Finally, through the linear layer, the feature is reduced to one in the channel and other dimensions, thereby obtaining the final image photoplethysmography (iPPG) signal.

[0098] In step S6, a fast Fourier transform is performed on the rPPG signal to calculate and obtain the heart rate value of the test subject.

[0099] In this step, the rPPG signal extracted by the model is subjected to a fast Fourier transform, and the time domain signal is transformed into the frequency domain to obtain the power spectral density PSD corresponding to the rPPG signal. rPPG In the power spectrum density function, the frequency value at the maximum power (amplitude) corresponds to the heart rate. Max ×60, you can get the heart rate value hr corresponding to the rPPG signal rPPG .

[0100] In step S7, at least the heart rate value of the test subject is visualized.

[0101] For example, the heart rate result display interface of the embodiment of the present invention can refer to Figure 7 .

[0102] So far, the embodiment of the present invention has completed the entire process of the non-contact heart rate calculation method based on dual-channel input.

[0103] In particular, when training the contactless heart rate extraction model, in terms of the loss function:

[0104] In order to ensure that the extracted rPPG signal is as close as possible to the gold standard in the time domain, the time domain loss L is set based on the rPPG signal obtained by the model and the pre-acquired gold standard PPG signal. t :

[0105] L t =1-PGor(rPPG,PPG)

[0106] Where PCor(·) represents Pearson correlation;

[0107] In order to ensure that the extracted rPPG signal can extract accurate heart rate values, the frequency domain loss L is set based on the power spectral density of the rPPG signal obtained by the model and the pre-acquired gold standard heart rate. f :

[0108] L f =CE(PSD rPPG ,hr gt )

[0109] Among them, CE(·) represents the cross entropy loss, PSD rPPG Represents the power spectral density of the rPPG signal, hr gt represents the gold standard heart rate;

[0110] The overall loss function L total Set to:

[0111] L total =L t +aL f

[0112] Among them, a is a configurable parameter and can be set according to the situation.

[0113] In addition, due to the large amount of data in the dataset, in order to speed up the convergence of the model, the dataset is divided into mini batches for training. During each training process, the gradient of each instance in the mini batch needs to be calculated according to the overall loss function. After calculating the gradient, the conventional practice is to calculate the average gradient of all instances in each mini batch for parameter update. However, since the gradient directions of different instances may conflict, the efficiency of model parameter update is slowed down, and even the parameter optimization direction is unstable.

[0114] Therefore, during the training process, the embodiment of the present invention also introduces a gradient dynamic adjustment strategy. Figure 8 As shown in the figure, in each mini batch, the gradient G1 of an instance is randomly selected, and the gradients Gn of other instances are traversed. If the gradient directions of two instances conflict, Gn is projected onto the orthogonal plane corresponding to G1, thereby offsetting the conflicting parts between the two gradients, making the training parameter optimization direction process more stable and accelerating the model convergence speed.

[0115] Embodiment 2:

[0116] The embodiment of the present invention provides a non-contact heart rate calculation system based on dual-channel input, a non-contact heart rate extraction model based on dual-stream input, the non-contact heart rate extraction model includes a dual-stream feature extractor, a channel attention module and an rPPG predictor based on a Transformer module; the non-contact heart rate calculation system includes:

[0117] A video acquisition module is used to acquire the original visible light video of the subject;

[0118] A video preprocessing module is used to detect the face area in the original visible light video and locate the facial landmark points on the face area to obtain a facial visible light video; based on the facial visible light video, calculate the pixel difference between each frame and the previous frame to obtain a facial differential image sequence with the same size and length as the facial visible light video;

[0119] A spatiotemporal information extraction module, used to use the facial visible light video and facial differential image sequence as inputs of corresponding feature extractors, respectively obtain visible light spatiotemporal blocks and video differential spatiotemporal blocks, and obtain original spatiotemporal blocks after splicing;

[0120] A channel attention fusion module is used to take the original spatiotemporal block as the input of the channel attention module, obtain a channel weight matrix, and multiply the channel attention matrix with the original spatiotemporal block to obtain a channel attention spatiotemporal block;

[0121] An rPPG signal acquisition module is used to segment the channel attention spatiotemporal block and fuse the linear position encoding as the input of the Transformer module to obtain a feature vector containing global and local fine-grained information; the feature vector is used as the input of the upsampling block of the rPPG predictor, and the final rPPG signal is obtained after linear transformation;

[0122] The heart rate value acquisition module is used to perform fast Fourier transform on the rPPG signal to calculate and obtain the heart rate value of the test subject.

[0123] In an optional implementation, the contactless heart rate calculation system further includes:

[0124] The result visualization module is used to at least visualize the heart rate value of the test subject.

[0125] Embodiment 3:

[0126] An embodiment of the present invention provides a storage medium, characterized in that it stores a computer program for non-contact heart rate calculation based on dual-channel input, wherein the computer program enables a computer to execute the non-contact heart rate calculation method as described in Example 1.

[0127] Embodiment 4:

[0128] An embodiment of the present invention provides an electronic device, including:

[0129] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the non-contact heart rate calculation method as described in Example 1.

[0130] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0131] 1. The embodiment of the present invention is based on a non-contact heart rate extraction model with dual-stream input, which performs face segmentation on the original visible light video to obtain the visible light video of the face, and simultaneously calculates the image difference between adjacent frames of the facial video, aggregates the fine-grained spatiotemporal information between the two frames, and obtains the facial differential image sequence; then the facial visible light video and the facial differential image sequence are used as the input of the model respectively, the channel attention is fused, and the powerful self-attention mechanism of the Transformer model is used to capture the correlation between the positions of the video and the video differential, and the final rPPG signal is obtained; finally, the heart rate value of the subject is obtained through fast Fourier transform. By designing a dual-stream input to better aggregate fine-grained information related to motion, and using the self-attention mechanism to fuse global spatiotemporal information, the robustness of the algorithm to motion is improved.

[0132] 2. The embodiment of the present invention designs a gradient dynamic adjustment strategy during the model training process. The gradient direction is dynamically adjusted according to the data in each mini batch, which effectively solves the problem of unstable optimization direction during the training process and effectively enhances the generalization performance of the model.

[0133] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0134] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact heart rate calculation method based on dual-channel input, characterized in that: A contactless heart rate extraction model based on dual-stream input, the contactless heart rate extraction model comprising a dual-stream feature extractor, a channel attention module, and an rPPG predictor based on a Transformer module; The non-contact heart rate calculation method comprises: Obtaining the original visible light video of the subject; Detecting the human face region in the original visible light video and locating the facial landmark points on the human face region to obtain a facial visible light video; calculating the pixel difference between each frame and the previous frame based on the facial visible light video to obtain a facial differential image sequence having the same size and length as the facial visible light video; Taking the facial visible light video and facial difference image sequence as inputs of corresponding feature extractors, respectively obtaining visible light spatiotemporal blocks and video difference spatiotemporal blocks, and splicing them to obtain original spatiotemporal blocks; Taking the original spatiotemporal block as the input of the channel attention module, obtaining a channel weight matrix, and multiplying the channel attention matrix with the original spatiotemporal block to obtain a channel attention spatiotemporal block; After segmenting the channel attention spatiotemporal block and fusing the linear position encoding, the feature vector containing global and local fine-grained information is obtained as the input of the Transformer module; the feature vector is used as the input of the upsampling block of the rPPG predictor, and the final rPPG signal is obtained after linear transformation; A fast Fourier transform is performed on the rPPG signal to calculate and obtain the heart rate value of the test subject.

2. The non-contact heart rate calculation method according to claim 1, characterized in that: The dual-stream feature extractor includes two feature extractors with the same structure; The feature extractor is composed of three stacked convolution modules with the same structure, and the convolution module is composed of a 3D convolution layer, a 3D batch normalization layer, a ReLU layer and a maximum pooling layer.

3. The non-contact heart rate calculation method according to claim 1, characterized in that: The channel attention module has two channel branch networks and a Sigmoid layer built in; Among them, the first channel branch network is composed of an average pooling layer, a 3D convolution layer, a Relu layer, and a 3D convolution layer; the second channel branch network is composed of a maximum pooling layer, a 3D convolution layer, a Relu layer, and a 3D convolution layer.

4. The non-contact heart rate calculation method according to claim 1, characterized in that: The rPPG predictor based on the Transformer module includes a Transformer module, two upsampling blocks with the same structure, and a linear layer; The Transformer module is composed of three standard Vision Transformer modules cascaded together; the upsampling block is composed of an upsampling layer, a 3D convolution layer, a 3D batch normalization layer and an ELU layer.

5. The non-contact heart rate calculation method according to claim 1, characterized in that: The setting process of the loss function of the contactless heart rate extraction model includes: Based on the rPPG signal obtained by the model and the pre-acquired gold standard PPG signal, the time domain loss L is set t : L t =1-PGor(rPPG,PPG) Where PCor(·) represents Pearson correlation; And based on the power spectral density of the rPPG signal obtained by the model and the pre-acquired gold standard heart rate, the frequency domain loss L is set f : L f =CE(PSD rPPG ,hr gt ) Among them, CE(·) represents the cross entropy loss, PSD rPPG Represents the power spectral density of the rPPG signal, hr gt represents the gold standard heart rate; The overall loss function L total Set to: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> total <h2 style=";text-align:left;direction:ltr"> =L<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> +aL<h2 style=";text-align:left;direction:ltr"> f Among them, a is a configurable parameter.

6. The non-contact heart rate calculation method according to claim 1, characterized in that: A gradient dynamic adjustment strategy is introduced to update the parameters of the non-contact heart rate extraction model.

7. The non-contact heart rate calculation method according to any one of claims 1 to 6, characterized in that: Also includes: At least the heart rate value of the test subject is visualized.

8. A non-contact heart rate calculation system based on dual-channel input, characterized in that: A contactless heart rate extraction model based on dual-stream input, the contactless heart rate extraction model comprising a dual-stream feature extractor, a channel attention module, and an rPPG predictor based on a Transformer module; The non-contact heart rate calculation system comprises: A video acquisition module is used to acquire the original visible light video of the subject; A video preprocessing module is used to detect the face area in the original visible light video and locate the facial landmark points on the face area to obtain a facial visible light video; based on the facial visible light video, calculate the pixel difference between each frame and the previous frame to obtain a facial differential image sequence with the same size and length as the facial visible light video; A spatiotemporal information extraction module, used to use the facial visible light video and facial differential image sequence as inputs of corresponding feature extractors, respectively obtain visible light spatiotemporal blocks and video differential spatiotemporal blocks, and obtain original spatiotemporal blocks after splicing; A channel attention fusion module is used to take the original spatiotemporal block as the input of the channel attention module, obtain a channel weight matrix, and multiply the channel attention matrix with the original spatiotemporal block to obtain a channel attention spatiotemporal block; An rPPG signal acquisition module is used to segment the channel attention spatiotemporal block and fuse the linear position encoding as the input of the Transformer module to obtain a feature vector containing global and local fine-grained information; the feature vector is used as the input of the upsampling block of the rPPG predictor, and the final rPPG signal is obtained after linear transformation; The heart rate value acquisition module is used to perform fast Fourier transform on the rPPG signal to calculate and obtain the heart rate value of the test subject.

9. A storage medium, characterized in that: The computer stores a computer program for non-contact heart rate calculation based on dual-channel input, wherein the computer program enables the computer to execute the non-contact heart rate calculation method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the non-contact heart rate calculation method according to any one of claims 1 to 7.

Images