Contactless heart rate variability detection method and system based on joint attention

Abstract

The present invention provides a contactless heart rate variability detection method and system based on joint attention, and relates to the field of contactless heart rate variability detection. The present invention is based on a monitoring network of a time-channel-space joint attention mechanism, and the monitoring network includes a time-channel joint attention layer, a time-space joint attention layer, a channel shuffle layer, and a U-Net network. Based on the aforementioned monitoring network, high-quality, high-signal-to-noise ratio rPPG signals are extracted to reduce the non-contact heart rate variability extraction error, thereby reducing the measurement error of heart rate variability. In addition, the present invention also considers long-term rPPG signals to extract non-contact heart rate variability features, thereby enhancing the accuracy and stability of monitoring. The present invention constructs a loss function that simultaneously considers the overall trend of the signal and the global and local features, attaches importance to the peak recovery position and peak interval in the rPPG signal, effectively improves the signal-to-noise ratio and reduces the error of heart rate variability.

Description

Technical Field

[0001] The present invention relates to the field of non-contact heart rate variability detection, and in particular to a non-contact heart rate variability detection method, system, storage medium and electronic device based on joint attention. Background Art

[0002] Heart Rate Variability (HRV) is a measure of the autonomic nervous system and is widely considered one of the best objective indicators of physical health and readiness.

[0003] The non-contact heart rate variability detection technology is mainly based on remote photoplethysmography signals. By analyzing the pre-acquired rPPG signal related indicators, the non-contact heart rate variability detection results are obtained. The existing heart rate variability detection methods are based on signal processing methods (such as ICA, CHROM, etc.). Such methods are based on certain assumptions or models, focusing on specific scenarios, and denoising the rPPG signal according to the noise characteristics, thereby improving the accuracy of heart rate variability extraction. At the same time, the existing non-contact heart rate variability detection methods mainly consider the fitting of the overall trend of the rPPG signal, and mainly extract time domain indicators such as AVNN and SDNN.

[0004] However, in real scenarios, the complexity of the noise involved in the rPPG signal far exceeds the preset situation. The detection model of the existing non-contact heart rate variability detection method is simple and poorly generalized. It is easy to lack attention to important information, resulting in poor accuracy in heart rate variability extraction. In addition, the peak recovery position and peak interval in the rPPG signal have a key impact on the calculation of heart rate variability indicators. The existing rPPG signal extraction technology only considers the overall trend fitting of the rPPG signal, which will cause the heart rate variability detection results to deviate from the true value. In addition, the existing deep learning method is limited by the attention module. The attention module focuses on spatial feature extraction, and has less focus on channels and time, lacking the integrity between time information and channel information. In addition, compared with time domain indicators such as AVNN and SDNN, frequency domain indicators such as LF, HF, LF / HF and nonlinear indicators such as SD1 and SD2 are more important in specific scenarios. The existing rPPG signal extraction technology only considers time domain indicators when extracting heart rate variability, and the non-contact heart rate variability indicator detection is not comprehensive. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a non-contact heart rate variability detection method, system, storage medium and electronic device based on joint attention, which solves the technical problem of large errors in the existing non-contact heart rate variability extraction.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A non-contact heart rate variability detection method based on joint attention, a monitoring network based on a time-channel-space joint attention mechanism, the monitoring network comprising a time-channel joint attention layer, a time-space joint attention layer, a channel shuffle layer and a U-Net network; the non-contact heart rate variability detection method comprises:

[0010] Obtain a facial video of the subject;

[0011] Detecting a face and locating key regions of interest in the facial video, segmenting a skin region from each of the key regions of interest, and converting the RGB color value of the skin region into a YCbCr color space;

[0012] Each converted skin segmentation image is used as the input of the monitoring network, and is sequentially passed through the time channel joint attention layer and the time space joint attention layer to obtain the corresponding attention weight matrix and perform residual connection; the feature map output by the time space joint attention layer is passed through the channel shuffle layer to perform channel information fusion, so as to finally obtain a one-dimensional short-time rPPG signal;

[0013] Each of the short-term rPPG signals is spliced ​​in time sequence to obtain a long-term rPPG signal, and the long-term rPPG signal is used as the input of the U-Net network to obtain the final rPPG signal;

[0014] Based on the rPPG signal, a non-contact heart rate variability feature of the test subject is extracted.

[0015] Preferably, the monitoring network consists of a first convolutional layer, several groups of joint attention layers and channel shuffling layers, a global average pooling layer, a second convolutional layer and a U-Net network; wherein the joint attention layer sequentially includes a time-channel joint attention layer, a third convolutional layer and a time-space joint attention layer.

[0016] Preferably, the time channel joint attention layer includes a spatial average layer, a convolution layer, a batch normalization layer, and a rectified linear unit activation layer; through the time channel joint attention layer, the corresponding attention weights are obtained and residual connections are performed, including:

[0017] The first feature map of the input is sequentially passed through the spatial averaging layer, the convolution layer, the batch normalization layer, and the rectified linear unit activation layer to obtain a first attention weight matrix;

[0018] The first attention weight matrix is ​​residually connected and multiplied with the first feature map to obtain a second feature map.

[0019] Preferably, the temporal and spatial joint attention layer includes two parallel branches and a channel weight expansion layer, wherein the two parallel branches include a first branch and a second branch, the first branch includes a global average layer, a convolution layer, a batch normalization layer, and a rectified linear unit activation layer, and the second branch includes a channel average layer, a convolution layer, a batch normalization layer, and a rectified linear unit activation layer; through the temporal and spatial joint attention layer, the corresponding attention weights are obtained and residual connections are performed, including:

[0020] The input third feature map is sequentially passed through the global average layer, the convolution layer, the batch normalization layer, and the rectified linear unit activation layer to obtain a third attention weight matrix, and the input third feature map is sequentially passed through the channel average layer, the convolution layer, the batch normalization layer, and the rectified linear unit activation layer to obtain a fourth attention weight matrix;

[0021] After multiplying the third attention weight matrix and the fourth attention weight matrix, a second attention weight matrix is ​​obtained through the channel weight expansion layer;

[0022] The second attention weight matrix is ​​residually connected and multiplied with the third feature map to obtain a fourth feature map.

[0023] Preferably, any of the channel shuffling layers includes a channel expansion layer, a channel transposition layer, a channel tiling layer, a channel recovery layer, a convolution layer and a rectified linear unit in sequence; the feature map output by the time-space joint attention layer is fused through the channel shuffling layer to finally obtain a one-dimensional short-time rPPG signal, including:

[0024] Pass the fourth feature map through the channel expansion layer, the channel transposition layer, the channel tiling layer, the channel restoration layer, the convolution layer and the rectified linear unit in sequence to obtain a fifth feature map;

[0025] The fifth feature map output by the last channel shuffling layer is sequentially passed through the global average pooling layer and the second convolutional layer to obtain a one-dimensional short-time rPPG signal.

[0026] Preferably, the loss function of the monitoring network in the training phase is constructed as follows:

[0027] (1) Constructing negative Pearson correlation loss of time domain signal :

[0028]

[0029] in, N Indicates the length of the rPPG signal output by the monitoring network, represents the first in the time domain signal of the pre-acquired gold standard PPG signal i values, The first time domain signal of the rPPG signal i Values;

[0030] (2) Constructing the negative Pearson correlation coefficient loss of frequency domain signal :

[0031]

[0032] in, The frequency domain signal representing the gold standard PPG signal i values, The frequency domain signal representing the rPPG signal i Values;

[0033] (3) Constructing peak loss :

[0034]

[0035] in, Indicates the first i The location of the peak, express forward n A peak, express back n A wave peak; Indicates the first i The location of the peak, express forward n A peak, express back n A wave peak; L Represents the time interval between different wave peak positions; α、β is the corresponding weight value;

[0036] (4) Constructing the overall loss function L :

[0037]

[0038] in, ω 1 ,ω 2 ,ω3 are the weight values ​​of the corresponding losses.

[0039] Preferably, the non-contact heart rate variability characteristics include time domain characteristics, frequency domain characteristics and nonlinear characteristics; wherein:

[0040] The time domain feature is one or a combination of any of AVNN, SDNN, and RMSSD;

[0041] The frequency domain feature is one or a combination of any of HF, LF, and LF / HF;

[0042] The nonlinear feature is one or a combination of any two of SD1 and SD2.

[0043] A non-contact heart rate variability detection system based on joint attention, a monitoring network based on a time-channel-space joint attention mechanism, the monitoring network comprising a time-channel joint attention layer, a time-space joint attention layer, a channel shuffling layer and a U-Net network; the heart rate variability detection system comprises:

[0044] A facial video acquisition module is used to acquire the facial video of the subject;

[0045] A facial video preprocessing module, for detecting a face and locating a key region of interest in the facial video, segmenting a skin region from each key region of interest, and converting the RGB color value of the skin region into a YCbCr color space;

[0046] The rPPG signal extraction module is used to use each converted skin segmentation image as the input of the monitoring network, and sequentially pass through the time channel joint attention layer and the time space joint attention layer to obtain the corresponding attention weight matrix and perform residual connection; the feature map output by the time space joint attention layer is fused through the channel shuffling layer to finally obtain a one-dimensional short-time rPPG signal;

[0047] And performing time-series splicing on each of the short-time rPPG signals to obtain a long-time rPPG signal, and using the long-time rPPG signal as an input of a U-Net network to obtain a final rPPG signal;

[0048] The non-contact heart rate variability feature extraction module is used to extract the non-contact heart rate variability feature of the test subject based on the rPPG signal.

[0049] A storage medium stores a computer program for non-contact heart rate variability detection based on joint attention, wherein the computer program enables a computer to execute the non-contact heart rate variability detection method as described above.

[0050] An electronic device, comprising:

[0051] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the non-contact heart rate variability detection method as described above.

[0052] (III) Beneficial effects

[0053] The present invention provides a non-contact heart rate variability detection method, system, storage medium and electronic device based on joint attention. Compared with the prior art, it has the following beneficial effects:

[0054] The present invention is based on a monitoring network of a time-channel-space joint attention mechanism, and the monitoring network includes a time-channel joint attention layer, a time-space joint attention layer, a channel shuffle layer, and a U-Net network. Based on the aforementioned monitoring network, high-quality, high-signal-to-noise ratio rPPG signals are extracted to reduce the non-contact heart rate variability extraction error, thereby reducing the measurement error of heart rate variability. In addition, the present invention also considers long-term rPPG signals to extract non-contact heart rate variability features, thereby enhancing the accuracy and stability of monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A block diagram of a non-contact heart rate variability detection method based on joint attention provided by an embodiment of the present invention;

[0057] Figure 2 A complete flow chart of a non-contact heart rate variability detection method based on joint attention provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the structure of a temporal channel joint attention layer (TCA) provided in an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of the structure of a temporal-spatial joint attention layer (TSA) provided in an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of the structure of a channel shuffling layer provided by an embodiment of the present invention;

[0061] Figure 6 A schematic diagram of the structure of a U-Net network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The embodiments of the present application solve the technical problem of large errors in the existing non-contact heart rate variability extraction by providing a non-contact heart rate variability detection method, system, storage medium and electronic device based on joint attention.

[0064] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0065] In addition to solving the noise problem mentioned in the background art, the non-contact heart rate variability detection method based on joint attention provided by the embodiment of the present invention has at least the following technical improvements:

[0066] (1) Compared with the existing non-contact method which mainly focuses on fitting the overall rPPG signal trend to obtain the heart rate, the embodiments of the present invention focus on obtaining the heart rate variability characteristics through the rPPG signal.

[0067] (2) Compared with the existing non-contact heart rate variability methods, the detection model is simple, the generalization is poor, and the attention to the quality of rPPG signal recovery is insufficient. The embodiment of the present invention constructs a loss function that considers the overall trend of the signal and the global and local characteristics at the same time, pays attention to the peak recovery position and peak interval in the rPPG signal, effectively improves the signal-to-noise ratio and reduces the error of heart rate variability.

[0068] (3) Compared with the existing non-contact heart rate variability monitoring methods, which mainly focus on extracting time domain indicators such as RR interval mean, SDNN, and pNN50, there are few studies on the extraction of frequency domain indicators such as LF, HF, and LF / HF. Based on the recognition that frequency domain indicators are as important as time domain indicators, the embodiments of the present invention perform heart rate variability analysis based on the acquired high-precision rPPG signal from different angles, including time domain features (such as AVNN, SDNN, RMSSD, etc.), frequency domain features (such as HF, LF, LF / HF, etc.), and nonlinear features (such as SD1, SD2, etc.).

[0069] (4) Compared with the existing non-contact heart rate variability detection, most of them are based on the study of short-term rPPG signals (e.g., 128 frames). Considering that the study of heart rate variability calculated from ECG signals and PPG signals usually requires at least 5 minutes of data to obtain accurate results, the shortest time is not less than 1 minute. Therefore, in the non-contact heart rate variability detection, the embodiment of the present invention selects long-term (e.g., 1 minute) rPPG signals to extract heart rate variability features to ensure the reliability of the results.

[0070] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0071] Embodiment 1:

[0072] An embodiment of the present invention provides a contactless heart rate variability detection method based on joint attention, and a monitoring network based on a time-channel-space joint attention mechanism, wherein the monitoring network includes a time-channel joint attention layer, a time-space joint attention layer, a channel shuffling layer, and a U-Net network.

[0073] like Figure 1 As shown, the non-contact heart rate variability detection method includes:

[0074] S1, obtain the facial video of the subject;

[0075] S2, detecting the face and locating the key regions of interest in the facial video, segmenting the skin region from each of the key regions of interest, and converting the RGB color value of the skin region into a YCbCr color space;

[0076] S3, taking each converted skin segmentation image as the input of the monitoring network, sequentially passing through the time channel joint attention layer and the time space joint attention layer, obtaining the corresponding attention weight matrix and performing residual connection; the feature map output by the time space joint attention layer is subjected to channel information fusion through the channel shuffling layer to finally obtain a one-dimensional short-time rPPG signal;

[0077] S4, performing time-series splicing on each of the short-term rPPG signals to obtain a long-term rPPG signal, and using the long-term rPPG signal as the input of the U-Net network to obtain a final rPPG signal;

[0078] S5. Extracting non-contact heart rate variability characteristics of the test subject based on the rPPG signal.

[0079] The embodiment of the present invention is based on the above monitoring network to extract high-quality, high signal-to-noise ratio rPPG signals, reduce the non-contact heart rate variability extraction error, and thus reduce the measurement error of heart rate variability. In addition, the embodiment of the present invention also considers long-term rPPG signals to extract non-contact heart rate variability features, thereby enhancing the accuracy and stability of monitoring.

[0080] like Figure 2 As shown, Figure 2 A complete flowchart of the joint attention-based contactless heart rate variability detection method is given. Figure 2 It can be seen that the monitoring network provided by the embodiment of the present invention consists of a first convolutional layer, several groups of joint attention layers and a channel shuffling layer ( Figure 2 3 groups are exemplarily set up in the example, a global average pooling layer, a second convolutional layer and a U-Net network; wherein the joint attention layer includes a time channel joint attention layer (TCA), a third convolutional layer and a time-space joint attention layer (TSA) in sequence.

[0081] Next, we will combine Figure 2 The following are the steps of the above solution:

[0082] In step S1, a facial video of a subject is obtained.

[0083] To ensure that the subject's facial information can be captured, the visible light camera aimed at the facial area is kept 0.5m~0.8m away from the subject in this step. After clicking the start acquisition button, the acquisition device records the subject's facial video.

[0084] In step S2, the face is detected and key regions of interest are located for the facial video, a skin region is segmented from each of the key regions of interest, and the RGB color value of the skin region is converted into a YCbCr color space.

[0085] This step performs preprocessing operations on facial videos, such as Figure 2 As shown, specifically including:

[0086] (1) Select the area of ​​interest

[0087] Facial landmark localization and face detection algorithms are used to detect faces and locate key regions of interest from video sequences.

[0088] Exemplarily, the facial detection model integrated in the mediapipe library is used to detect facial feature points to extract and save the entire facial image.

[0089] (2) Segmentation of skin areas

[0090] Skin segmentation aims to separate the skin area from the facial image for more accurate analysis. Since the non-rigid motion of the eyes, mouth and other parts (blinking and facial expressions, etc.) has a great impact on the extraction of rPPG signals, these areas need to be covered or segmented out. The captured facial image is subjected to skin segmentation, and the eyes, mouth and other parts are covered to obtain the facial image after skin segmentation. At this time, the facial image only retains the skin area, and the obtained skin segmentation images are arranged in time series.

[0091] (3) Convert color space

[0092] The RGB color value of the skin area is converted to the YCbCr color space, that is, the color space of the obtained skin segmentation image is converted from the BGR three-channel to the Y, Cb, Cr three-channel, and then through the appropriate Y, Cb and Cr channel thresholds, the threshold is set to determine whether each pixel in the selected face image belongs to the skin. If the pixel belongs to the skin, it is retained, and if the pixel does not belong to the skin, it is covered with black.

[0093] Among them, Y, Cb, and Cr are brightness component, blue chroma component, and red chroma component respectively; the channel thresholds of Y, Cb, and Cr are set as follows: 137 <Cr<177,77<Cb<127,190<Cb+0.6Cr<215。

[0094] In order to focus on the time information and frequency domain information of the rPPG signal and reduce the amount of network calculation to achieve the purpose of lightweight, the embodiment of the present invention proposes a monitoring network based on the time-channel-space joint attention mechanism, and considers the problem of computational cost, integrates the channel shuffling module, greatly reduces the computational cost, and effectively promotes the information flow and fusion between different channels, thereby improving the feature extraction ability of the network. For details, please refer to the contents of steps S3~S4 below.

[0095] In step S3, each converted skin segmentation image is used as the input of the monitoring network, and passes through the time-channel joint attention layer and the time-space joint attention layer in sequence to obtain the corresponding attention weight matrix and perform residual connection; the feature map output by the time-space joint attention layer is fused with channel information through the channel shuffling layer to finally obtain a one-dimensional short-time rPPG signal.

[0096] See also Figure 2 , each converted skin segmentation image is used as the input of the monitoring network:

[0097] First, the first convolutional layer is used to extract preliminary features and obtain the first feature map M1 (B*C*T*H*W).

[0098] Secondly, the first feature map M1 (B*C*T*H*W) is used as the input of the temporal channel joint attention layer (TCA) to jointly learn the attention weights in the time and channel dimensions to connect the information between different channels in the time dimension.

[0099] Specifically, if Figure 3 As shown, any of the temporal channel joint attention layers (TCA) includes a spatial averaging layer, a convolutional layer, a batch normalization layer, and a rectified linear unit activation layer (ReLU).

[0100] Accordingly, the time channel is combined with the attention layer to obtain the corresponding attention weights and perform residual connection, including:

[0101] The first feature map M1 (B*C*T*H*W), the spatial dimension of the feature map is averaged through the spatial averaging layer, and the size of the feature map is reduced to obtain the feature vector M CT (B*C*T), after M CT Enter the convolution layer to implement the convolution operation, then enter the batch normalization layer to implement the normalization operation, and finally enter the rectified linear unit activation layer (ReLU) to obtain the first attention weight matrix M Weight .

[0102] The first attention weight matrix M Weight With the first feature map M1 Perform residual connection multiplication to obtain the second feature map M2 (B*C*T*H*W).

[0103] Again, the second feature map M2 (B*C*T*H*W) is used as the input of the third convolutional layer to further extract feature information in TCA and TSA to obtain the third feature map M3 (B*C*T*H*W).

[0104] Next, the third feature map M3 (B*C*T*H*W) is used as the input of the temporal-spatial joint attention layer (TSA) to jointly learn the attention weights in the spatial and temporal dimensions to better capture the features related to the pulse signal.

[0105] Specifically, if Figure 4As shown, any of the temporal-spatial joint attention layers (TSA) includes two parallel branches and a channel weight expansion layer, wherein the two parallel branches include a first branch and a second branch, the first branch includes a global average layer, a convolution layer, a batch normalization layer, and a rectified linear unit (ReLU), and the second branch includes a channel average layer, a convolution layer, a batch normalization layer, and a rectified linear unit (ReLU).

[0106] It can be understood that the above two branches are used to learn weights of different dimensions, so that complementary information can be learned.

[0107] Accordingly, the corresponding attention weights are obtained through the temporal-spatial joint attention layer and residual connections are performed, including:

[0108] The third feature map M3 (B*C*T*H*W), and input the global average layer and the channel average layer to obtain the corresponding feature vectors respectively. Then the two enter the corresponding convolution layer to implement the convolution operation, enter the corresponding batch normalization layer to implement the normalization operation, and enter the corresponding rectified linear unit activation layer (ReLU), and finally get the third attention weight matrix G Weight and the fourth attention weight matrix C Weight .

[0109] The third attention weight matrix G Weight and the fourth attention weight matrix C Weight After multiplication, the second attention weight matrix is ​​obtained through the channel weight expansion layer M Weight ;

[0110] The second attention weight matrix M Weight With the third characteristic diagram M3 (B*C*T*H*W) performs residual connection multiplication to obtain the fourth feature M4 (B*C*T*H*W).

[0111] It should be noted that the design of TCA and TSA allows the network to capture information related to the pulse signal in different feature dimensions on the facial frame sequence, rather than just focusing on a single dimension. In this way, the network can more effectively extract the pulse signal from the facial video, even in the presence of motion artifacts and other interference factors can also ensure high accuracy.

[0112] Then, the fourth feature map M4(B*C*T*H*W), as the input of the channel shuffle layer of the same group, effectively promotes the information flow and fusion between different channels without increasing too much computation, thereby improving the feature extraction capability of the network. That is, the embodiment of the present invention proposes a three-dimensional channel shuffle suitable for 3DCNN, which can reduce the computation in 3DCNN.

[0113] The lack of information interaction and integration between channels will inevitably have a negative impact on the feature extraction ability of the network. A more economical channel shuffle is used, which allows information fusion between different channels without convolution. Specifically, Figure 5 As shown, any of the channel shuffling layers includes a channel expansion layer, a channel transposition layer, a channel tiling layer, a channel restoration layer, a convolutional layer and a rectified linear unit in sequence.

[0114] Accordingly, the feature map output by the temporal-spatial joint attention layer is subjected to channel information fusion through the channel shuffling layer, including:

[0115] The fourth feature map M4 (B*C*T*H*W) is input to the channel expansion layer, that is M The second dimension C is expanded into two dimensions g×n M expand (B*g*n*T*H*W), then M expand The input channel transpose layer is to transpose along the g-axis and n-axis of the matrix to obtain M transpose (B*n*g*T*H*W), M transpose Input channel tiling layer, that is, tiling the second dimension n and the third dimension g, can be restored to C dimension, and get M rocver (B*C*T*H*W), then the input convolution layer implements the convolution operation, and finally enters the rectified linear unit activation layer (ReLU) to obtain the fifth feature map M5 (B*C*T*H*W).

[0116] Finally, after being processed by the above three sets of joint attention layers and channel shuffle layers, the fifth feature map output by the last channel shuffle layer is M5 (B*C*T*H*W), the spatial dimensions are averaged through the global average pooling layer, and the convolution channels are convolved through the second convolution layer, so as to finally obtain a one-dimensional short-time rPPG signal.

[0117] In step S4, each of the short-time rPPG signals is spliced ​​in time sequence to obtain a long-time rPPG signal, and the long-time rPPG signal is used as the input of the U-Net network to obtain the final rPPG signal.

[0118] In order to denoise and restore long-term rPPG signals, the embodiment of the present invention uses a U-Net network to encode and decode long-term rPPG signals, which can effectively remove noise, fill missing data, and restore the original characteristics of the signal, thereby improving signal quality and reliability.

[0119] Specifically, due to limited computing resources, the time dimension T obtained above is relatively small (only 160 frames), and since the heart rate variability detection requires a long time (it takes at least 1 minute to calculate the result to be reliable), it is chosen to splice the aforementioned multiple short-term rPPG signals with a time length of T in time series to form a long-term rPPG signal, and input the long-term rPPG signal into the U-Net network for denoising and recovery of the long-term rPPG signal, so as to reduce the noise after the long-term rPPG signal splicing and effectively calculate the heart rate variability index.

[0120] For example, Figure 6 As shown, in the embodiment of the present invention, it is limited that the U-Net network adopts two pooling layers and two upsampling layers.

[0121] In particular, the loss function of the monitoring network in the training phase in the embodiment of the present invention is constructed as follows:

[0122] (1) To restore the rPPG signal, we need to pay attention to the periodicity of the signal and the quality of the signal restoration. Therefore, we need to calculate the negative Pearson correlation loss of the time domain signal between the rPPG signal and the gold standard PPG signal. :

[0123]

[0124] in, N Indicates the length of the rPPG signal output by the monitoring network, represents the first in the time domain signal of the pre-acquired gold standard PPG signal i values, The first time domain signal of the rPPG signal i value.

[0125] (2) The loss in the frequency domain is also extremely important. The rPPG signal and the gold standard PPG signal are converted to the frequency domain, and the negative Pearson correlation coefficient loss of the frequency domain signal between the two is constructed. :

[0126]

[0127] in, The frequency domain signal representing the gold standard PPG signal ivalues, The frequency domain signal representing the rPPG signal i value.

[0128] (3) Considering that the peak position and spacing play a very important role in the detection of heart rate variability, it is necessary to pay attention to the local and global characteristics of the peak of the rPPG signal. The local feature refers to the , which is close to the peak n The peak is a local feature, while the m A peak is the peak of the whole global The influence is called global feature.

[0129] Specifically, the defined peak consists of three attributes: i Location ,crest i With the previous n The interval between the peaks , and after n The interval between the peaks . These are n Different weight values ​​are assigned to the time intervals between the peaks. Since the peak at the far end has less influence than the peak at the near end, the normal distribution probability model is selected for n The weight values ​​between the peaks are assigned, where the weight assignment conforms to the normal distribution.

[0130] Thus, the peak loss is constructed :

[0131]

[0132] in, Indicates the first i The location of the peak, express forward n A peak, express back n A wave peak; Indicates the first i The location of the peak, express forward n A peak, express back n A wave peak; L Represents the time interval between different wave peak positions; α、β is the corresponding weight value.

[0133] (4) Constructing the overall loss functionL :

[0134]

[0135] in, ω 1 ,ω 2 ,ω 3 are the weight values ​​of the corresponding losses.

[0136] In step S5, the contactless heart rate variability characteristics of the subject are extracted based on the rPPG signal.

[0137] The non-contact HRV features in the embodiment of the present invention mainly include three aspects, namely time domain features (such as AVNN, SDNN, RMSSD, etc.), frequency domain features (such as HF, LF, LF / HF, etc.) and nonlinear features (such as SD1, SD2, etc.). The details are as follows:

[0138] (1) Time domain feature analysis

[0139] After using the deep learning network to obtain high-precision rPPG time domain signals, the mean (AVNN), overall standard deviation (SDNN), and mean square standard deviation (RMSSD) of the differences are obtained using the following calculation formula:

[0140]

[0141] in, N Indicates the location of the peak in the time domain signal. NN i Indicates the length of time between two adjacent peaks.

[0142] (2) Frequency domain feature analysis

[0143] The obtained time domain signal is converted into a frequency domain signal using Fourier transform, and the low frequency index LF in the range of 0.04-0.15Hz and the high frequency index HF in the range of 0.15-0.4Hz are selected from the frequency domain signal, and finally the ratio LF / HF between the low frequency index and the high frequency index is obtained. The calculation formula is as follows:

[0144]

[0145] (3) Nonlinear characteristic analysis

[0146] For a continuous heartbeat interval, i The heartbeat interval is taken as the horizontal axis, i+1With the heartbeat interval as the vertical coordinate, an ellipse-like figure can be drawn in a two-dimensional plane. The center of the ellipse is located at the coordinate point determined by (the average heartbeat interval on the x-axis and the average heartbeat interval on the y-axis). The semi-major axis SD1 and the semi-minor axis SD2 of the ellipse are the nonlinear indicators of heart rate variability.

[0147] So far, the embodiment of the present invention has completed the entire process of the non-contact heart rate variability detection method based on joint attention.

[0148] Embodiment 2:

[0149] The embodiment of the present invention provides a non-contact heart rate variability detection system based on joint attention, a monitoring network based on a time-channel-space joint attention mechanism, the monitoring network includes a time-channel joint attention layer, a time-space joint attention layer, a channel shuffling layer and a U-Net network; the heart rate variability detection system includes:

[0150] A facial video acquisition module is used to acquire the facial video of the subject;

[0151] A facial video preprocessing module, for detecting a face and locating a key region of interest in the facial video, segmenting a skin region from each key region of interest, and converting the RGB color value of the skin region into a YCbCr color space;

[0152] The rPPG signal extraction module is used to use each converted skin segmentation image as the input of the monitoring network, and sequentially pass through the time channel joint attention layer and the time space joint attention layer to obtain the corresponding attention weight matrix and perform residual connection; the feature map output by the time space joint attention layer is fused through the channel shuffling layer to finally obtain a one-dimensional short-time rPPG signal;

[0153] And performing time-series splicing on each of the short-time rPPG signals to obtain a long-time rPPG signal, and using the long-time rPPG signal as an input of a U-Net network to obtain a final rPPG signal;

[0154] The non-contact heart rate variability feature extraction module is used to extract the non-contact heart rate variability feature of the test subject based on the rPPG signal.

[0155] Embodiment 3:

[0156] An embodiment of the present invention provides a storage medium storing a computer program for non-contact heart rate variability detection based on joint attention, wherein the computer program enables a computer to execute the non-contact heart rate variability detection method as described in Example 1.

[0157] Embodiment 4:

[0158] An embodiment of the present invention provides an electronic device, including:

[0159] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the non-contact heart rate variability detection method as described above.

[0160] It can be understood that the non-contact heart rate variability detection system based on joint attention, storage medium and electronic device provided in the embodiments of the present invention correspond to the non-contact heart rate variability detection method based on facial video provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the non-contact heart rate variability detection method, which will not be repeated here.

[0161] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0162] 1. The present invention is based on a monitoring network of a time-channel-space joint attention mechanism, and the monitoring network includes a time-channel joint attention layer, a time-space joint attention layer, a channel shuffle layer, and a U-Net network. Based on the aforementioned monitoring network, high-quality, high-signal-to-noise ratio rPPG signals are extracted to reduce the non-contact heart rate variability extraction error, thereby reducing the measurement error of heart rate variability. In addition, the present invention also considers long-term rPPG signals to extract non-contact heart rate variability features, thereby enhancing the accuracy and stability of monitoring.

[0163] 2. The embodiment of the present invention constructs a loss function that considers the overall trend of the signal and the global and local features at the same time, pays attention to the peak recovery position and peak interval in the rPPG signal, effectively improves the signal-to-noise ratio and reduces the error of heart rate variability. Compared with the traditional method, it overcomes the defect of only focusing on the overall trend of the signal and the quality fitting effect, while ignoring local features such as peak position and interval.

[0164] 3. The embodiments of the present invention realize lightweight model, so that it can be deployed and used in special scenarios with limited computing resources, requiring only less computing resources.

[0165] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0166] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact heart rate variability detection method based on joint attention, characterized in that: A monitoring network based on a time-channel-space joint attention mechanism, the monitoring network comprising a time-channel joint attention layer, a time-space joint attention layer, a channel shuffle layer, and a U-Net network; The non-contact heart rate variability detection method comprises: Obtain the facial video of the subject; Detecting a face and locating key regions of interest in the facial video, segmenting a skin region from each of the key regions of interest, and converting the RGB color value of the skin region into a YCbCr color space; Each converted skin segmentation image is used as the input of the monitoring network, and is sequentially passed through the time channel joint attention layer and the time space joint attention layer to obtain the corresponding attention weight matrix and perform residual connection; the feature map output by the time space joint attention layer is passed through the channel shuffle layer to perform channel information fusion, so as to finally obtain a one-dimensional short-time rPPG signal; Each of the short-term rPPG signals is spliced ​​in time sequence to obtain a long-term rPPG signal, and the long-term rPPG signal is used as the input of the U-Net network to obtain the final rPPG signal; Based on the rPPG signal, a non-contact heart rate variability feature of the test subject is extracted.

2. The non-contact heart rate variability detection method according to claim 1, characterized in that: The monitoring network consists of a first convolutional layer, several groups of joint attention layers and channel shuffling layers, a global average pooling layer, a second convolutional layer and a U-Net network; wherein the joint attention layer sequentially includes a time-channel joint attention layer, a third convolutional layer and a time-space joint attention layer.

3. The non-contact heart rate variability detection method according to claim 2, characterized in that: The temporal channel joint attention layer includes a spatial averaging layer, a convolutional layer, a batch normalization layer, and a rectified linear unit activation layer; By combining the time channel with the attention layer, the corresponding attention weights are obtained and residual connections are performed, including: The first feature map of the input is sequentially passed through the spatial averaging layer, the convolution layer, the batch normalization layer, and the rectified linear unit activation layer to obtain a first attention weight matrix; The first attention weight matrix is ​​residually connected and multiplied with the first feature map to obtain a second feature map.

4. The non-contact heart rate variability detection method according to claim 2, characterized in that: The temporal-spatial joint attention layer includes two parallel branches and a channel weight expansion layer, wherein the two parallel branches include a first branch and a second branch, the first branch includes a global average layer, a convolution layer, a batch normalization layer, and a rectified linear unit activation layer, and the second branch includes a channel average layer, a convolution layer, a batch normalization layer, and a rectified linear unit activation layer; Through the temporal and spatial joint attention layer, the corresponding attention weights are obtained and residual connections are performed, including: The input third feature map is sequentially passed through the global average layer, the convolution layer, the batch normalization layer, and the rectified linear unit activation layer to obtain a third attention weight matrix, and the input third feature map is sequentially passed through the channel average layer, the convolution layer, the batch normalization layer, and the rectified linear unit activation layer to obtain a fourth attention weight matrix; After multiplying the third attention weight matrix and the fourth attention weight matrix, a second attention weight matrix is ​​obtained through the channel weight expansion layer; The second attention weight matrix is ​​residually connected and multiplied with the third feature map to obtain a fourth feature map.

5. The non-contact heart rate variability detection method according to claim 4, characterized in that: Any of the channel shuffling layers includes a channel expansion layer, a channel transposition layer, a channel tiling layer, a channel recovery layer, a convolution layer and a rectified linear unit in sequence; the feature map output by the time-space joint attention layer is fused through the channel shuffling layer to finally obtain a one-dimensional short-time rPPG signal, including: Pass the fourth feature map through the channel expansion layer, the channel transposition layer, the channel tiling layer, the channel restoration layer, the convolution layer and the rectified linear unit in sequence to obtain a fifth feature map; The fifth feature map output by the last channel shuffling layer is sequentially passed through the global average pooling layer and the second convolutional layer to obtain a one-dimensional short-time rPPG signal.

6. The non-contact heart rate variability detection method according to claim 1, characterized in that: The loss function of the monitoring network in the training phase is constructed as follows: (1) Constructing negative Pearson correlation loss of time domain signal : in, N Indicates the length of the rPPG signal output by the monitoring network, represents the first in the time domain signal of the pre-acquired gold standard PPG signal i values, The first time domain signal of the rPPG signal i values; (2) Constructing the negative Pearson correlation coefficient loss of frequency domain signal : in, The frequency domain signal representing the gold standard PPG signal i values, The frequency domain signal representing the rPPG signal i values; (3) Constructing peak loss : in, Indicates the first i The location of the peak, express forward n A peak, express back n A wave peak; Indicates the first i The location of the peak, express forward n A peak, express back n A wave peak; L Represents the time interval between different wave peak positions; α、β is the corresponding weight value; (4) Constructing the overall loss function L : in, ω 1 ,ω 2 ,ω 3 are the weight values ​​of the corresponding losses.

7. The non-contact heart rate variability detection method according to claim 1, characterized in that: The non-contact heart rate variability characteristics include time domain characteristics, frequency domain characteristics and nonlinear characteristics; wherein: The time domain feature is one or a combination of any of AVNN, SDNN, and RMSSD; The frequency domain feature is one or a combination of any of HF, LF, and LF / HF; The nonlinear feature is one or a combination of any two of SD1 and SD2.

8. A non-contact heart rate variability detection system based on joint attention, characterized in that: A monitoring network based on a time-channel-space joint attention mechanism, the monitoring network comprising a time-channel joint attention layer, a time-space joint attention layer, a channel shuffle layer, and a U-Net network; The heart rate variability detection system comprises: A facial video acquisition module is used to acquire the facial video of the subject; A facial video preprocessing module, for detecting a face and locating a key region of interest in the facial video, segmenting a skin region from each key region of interest, and converting the RGB color value of the skin region into a YCbCr color space; The rPPG signal extraction module is used to use each converted skin segmentation image as the input of the monitoring network, and sequentially pass through the time channel joint attention layer and the time space joint attention layer to obtain the corresponding attention weight matrix and perform residual connection; the feature map output by the time space joint attention layer is fused through the channel shuffling layer to finally obtain a one-dimensional short-time rPPG signal; And performing time-series splicing on each of the short-time rPPG signals to obtain an rPPG signal, and using the rPPG signal as an input of a U-Net network to obtain a final rPPG signal; The non-contact heart rate variability feature extraction module is used to extract the non-contact heart rate variability feature of the test subject based on the rPPG signal.

9. A storage medium, characterized in that: It stores a computer program for non-contact heart rate variability detection based on joint attention, wherein the computer program enables a computer to execute the non-contact heart rate variability detection method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a program for executing the non-contact heart rate variability detection method as described in any one of claims 1 to 7.

Images