A non-contact vital signs detection method based on deep learning

Abstract

The present invention relates to the field of medical care information technology of artificial intelligence, and specifically discloses a non-contact vital sign detection method based on deep learning, including frame processing of visible light video of human face; obtaining color face image and frame difference image; extracting color face image and frame difference image by multi-channel input, and forming stacked feature image; extracting spatial features; weighting spatial feature image; performing channel transformation fusion processing on weighted spatial feature image; transforming the fusion feature image dimension into a feature image vector; storing the feature image vector in sequence; and capturing the heart rate timing characteristics, respiratory rate timing characteristics, and blood oxygen saturation timing characteristics in rPPG signals respectively. The present invention greatly reduces parameters, saves the amount of calculation parameters and memory consumption, and has the technical characteristics of relatively simple analysis process, easy implementation, and fast calculation response speed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence medical care information technology, and in particular to a non-contact vital sign detection method based on deep learning. Background Art

[0002] Human vital characteristics such as heart rate, respiratory rate, and blood oxygen saturation are important medical and health care information that reflects the physiological health status of the human body, and have vital technical significance for health monitoring of human vital characteristics.

[0003] For a long time, the acquisition of human vital signs such as heart rate, respiratory rate, and blood oxygen saturation has been achieved by direct contact with the human body or intrusion into the body by relevant monitoring instruments. This has technical problems such as limited operating scenarios (including but not limited to inability to be carried out at a long distance or remotely), cumbersome operation process (including but not limited to contact connection of monitoring instruments, adjustment of monitoring instruments, manual reading and recording), lack of timeliness of operation (including but not limited to long detection cycle and inability to respond to emergency detection), and low objective accuracy (including but not limited to errors in contact connection of monitoring instruments, manual reading and recording errors).

[0004] In recent years, with the rapid development of artificial intelligence technology, various non-contact vital signs detection technologies that are independent of traditional detection modes have emerged in the prior art. Non-contact vital signs detection technology is based on the acquired human body video. It obtains the corresponding heart rate value, respiratory rate value and other human life characteristics by analyzing and capturing the skin color changes in the human body video to achieve the technical purpose of detection (measurement), namely rPPG (remote photoplethysmography) technology. For specific implementation methods, please refer to the detailed description of the technology disclosed in the Chinese patent document entitled "Non-contact Physiological and Psychological Multi-parameter Detection Method and System", with the publication number CN116959062A and the publication date of October 27, 2023.

[0005] However, this type of artificial intelligence analysis technology collects the original human body video by collecting visible light facial video and thermal infrared video in real time respectively. As a result, the data that needs to be analyzed in the process is heterogeneous, the number of calculation parameters is large, and the memory consumption is large, resulting in high complexity of the analysis process, high technical difficulty in implementation, and slow calculation response speed. Summary of the invention

[0006] The technical purpose of the present invention is to provide a non-contact vital signs detection method based on visible light video of the human face, based on deep learning, with a relatively simple analysis process, easy to implement, and fast calculation response speed, in view of the particularity of the above-mentioned non-contact vital signs detection technology and the shortcomings of the existing technology.

[0007] The technical purpose of the present invention is achieved by the following technical solution: a non-contact vital sign detection method based on deep learning, the detection method comprising:

[0008] Step 1. Obtain a visible light video of a human face, and process the video into frames to obtain an image frame sequence;

[0009] Step 2. Obtain a number of three-dimensional matrix color face images frame by frame and store them in a face image queue;

[0010] Each frame of the color face image is grayed out in turn, and two adjacent frames of grayed face images are differentially processed in turn to obtain a corresponding frame difference image, and the image is stored in a frame difference queue;

[0011] Step 3. The S_Net architecture extracts the corresponding color face images and frame difference images from the face image queue and the frame difference queue in batches in a multi-channel input manner in a first-in-first-out order, and forms a stacked feature map;

[0012] Step 4. The S_Conv neural network in the S_Net architecture extracts spatial features from the input stacked feature map to obtain a spatial feature map;

[0013] Step 5. The spatial self-attention mechanism in the S_Net architecture performs weighted processing on the spatial feature map extracted by the S_Conv neural network to focus on important spatial features;

[0014] Step 6. The 1x1 convolution mechanism in the S_Net architecture performs channel transformation and fusion processing on the weighted spatial feature map;

[0015] Step 7. The shape adjustment mechanism in the S_Net architecture transforms the fused feature map dimension into a vector of feature maps;

[0016] Step 8. The feature sequence collection mechanism in the S_Net architecture stores the feature map vector according to the time position information;

[0017] Step 9. The predicted rPPG mechanism in the S_Net architecture captures the heart rate timing features in the rPPG signal through the long short-term memory network LSTM1 to obtain the predicted heart rate value P_HR, captures the respiratory rate timing features in the rPPG signal through the long short-term memory network LSTM2 to obtain the predicted respiratory rate value P_RR, and captures the blood oxygen saturation timing features in the rPPG signal through the long short-term memory network LSTM3 to obtain the predicted blood oxygen saturation value P_SPO2.

[0018] As a preferred technical solution, in step 2, the color face image is obtained by performing face positioning on each video frame to set pixels to obtain a three-dimensional matrix color face image Face;

[0019] Each color face image Face is cached in the face image queue Q1 in a first-in-first-out order. The cache in the face image queue Q1 satisfies the following relationship:

[0020] Q1=[Face 0 ,Face 1 ,...,Face i ,...,Face N-1 ];

[0021] Where Q1 represents the face image queue;

[0022] Face 0 、Face 1 、Face i 、Face N-1 Indicates the position of the color face image Face in the video frame in the corresponding index order;

[0023] Wherein, i is the index number of any set sequential position in the video frame, 0≤i<N;

[0024] 0, 1, N-1 are the quantitative assignments for the i index number;

[0025] N represents the length of the total video frame;

[0026] The grayscale processing of the color face image of each video frame satisfies the following relationship:

[0027] ;

[0028] In the formula, represents the grayscale face image of the i-th video frame;

[0029] X represents the matrix of grayscale face image;

[0030] i is the index number of any set sequential position in the video frame, 0≤i<N;

[0031] Represents the pixel value of the grayscale face image in row 0 and column 0 of the i-th video frame;

[0032] represents the pixel value of the grayscale face image in the ith video frame at row 0 and column M-1, where M is the size of the matrix;

[0033] The difference processing of two adjacent grayscale face images satisfies the following relationship:

[0034] ;

[0035] in,

[0036] ;

[0037] In the formula, A frame difference image representing the grayscale face image of the i-th video frame and the grayscale face image of the adjacent previous video frame;

[0038] represents the grayscale face image of the i-th video frame;

[0039] Indicates that it is adjacent to the current The grayscale face image of the previous video frame of the video frame grayscale face image;

[0040] Each frame difference image △FaceGr is cached in the frame difference queue Q2 in a first-in-first-out order. The cache in the frame difference queue Q2 satisfies the following relationship:

[0041] Q2=[△FaceGr 0 ,△FaceGr 1 ,...,△FaceGr i ,...,△FaceGr N-1 ];

[0042] Where, Q2 represents the frame difference queue;

[0043] △FaceGr 0 , △FaceGr 1 , △FaceGr i , △FaceGr N-1 Indicates the position of the frame difference map △FaceGr corresponding to the index order in the video frame;

[0044] Wherein, i is the index number of any set sequential position in the video frame, 0≤i<N;

[0045] 0, 1, N-1 are the quantitative assignments for the i index number;

[0046] N represents the total video frame length.

[0047] As a preferred technical solution, in step 3, the S_Net architecture separates each extracted color face image Face into a red channel FaceR, a green channel FaceG, and a blue channel FaceB according to the RGB three channels;

[0048] The S_Net architecture extracts the atlas information in each batch in a multi-channel input manner to satisfy the following relationship:

[0049] ;

[0050] in, ;

[0051] In the formula, Represents all inputs of the b-th batch, 0≤b<N1;

[0052] Wherein, N1=[N / B], N1 represents the total batch number of video frames; N represents the length of the total video frames; B represents the set number of video frames extracted in each batch;

[0053] I b0 ,I b1 ,I bj ,I b，B-1 Represents the 4-channel stacked feature map of the corresponding sequential input in the b-th batch; the 4-channel stacked feature map is a three-channel face image matrix of the color face image Face The corresponding sequence frame difference map The stacking structure of

[0054] Represents the red channel matrix of the j-th input face image in the b-th batch;

[0055] Represents the green channel matrix of the j-th input face image in the b-th batch;

[0056] Represents the blue channel matrix of the j-th input face image in the b-th batch;

[0057] represents the frame difference map of the jth input in the bth batch;

[0058] Wherein, j is the index number of any set sequential position in the bth batch, 0≤j<B;

[0059] 0, 1, and B-1 are quantitative assignments for the j index number.

[0060] As a preferred technical solution, in step 4, the S_Conv neural network includes a convolution calculation layer, an activation calculation layer and a pooling calculation layer that are iterated three times in sequence for each batch of input data, and uses the calculation output of the current layer as the calculation input of the next layer to obtain a spatial feature map;

[0061] The size of the feature map changes before and after the convolution calculation. The convolution calculation layer calculates the width and height of the feature map size respectively, and the calculation process satisfies the following relationship:

[0062] ;

[0063] In the formula, Indicates the width / height of the feature map size output by the convolution calculation;

[0064] Indicates the width / height of the feature map size of the convolution calculation input;

[0065] w represents the width / height of the convolution kernel;

[0066] p represents the number of circles filled with 0;

[0067] Indicates the step size of the convolution kernel sliding;

[0068] The size of the feature map before and after the activation calculation remains unchanged, and the calculation process of the feature map by the activation calculation layer satisfies the following relationship:

[0069] f(x)=max(0,x);

[0070] Where x represents each pixel value in the feature map of the activation calculation input; if x≤0, the value is 0, and if x>0, the value remains unchanged;

[0071] The size of the feature map changes before and after the pooling calculation. The pooling calculation layer calculates the width and height of the feature map size respectively, and the calculation process satisfies the following relationship:

[0072] ;

[0073] In the formula, Indicates the width / height of the feature map size output after pooling calculation;

[0074] Indicates the width / height of the feature map size of the pooling calculation input;

[0075] k represents the width / height of the pooling kernel;

[0076] Indicates the step size of the pooling kernel sliding.

[0077] As a preferred technical solution, in step 5, the spatial self-attention mechanism focuses on the weighted processing of the important spatial features of the spatial feature map X8 extracted by the S_Conv neural network, and the attention calculation satisfies the following relationship:

[0078] ;

[0079] When Key_X8=Value_X8=Query_X8, the calculation of self-attention distribution and the weighting of self-attention satisfy the following relationship:

[0080] Self_A X =Attention(Query_X8,X8);

[0081] The self-attention weights of the spatial feature map satisfy the following relationship:

[0082] X9=X8⊙Self_A X ;

[0083] Where X9 represents the weighted spatial feature map calculated by self-attention;

[0084] X8 represents the spatial feature map extracted by the S_Conv neural network;

[0085] Self_A x Represents the self-attention distribution of the spatial feature map X8;

[0086] Attention represents the self-attention distribution function of spatial features;

[0087] Query_X8 represents the query matrix corresponding to the spatial feature map X8, and its value is any element in the spatial feature map X8;

[0088] Key_X8 represents the key matrix in the key-value pair corresponding to the spatial feature map X8;

[0089] The association between Query_X8 and Key_X8 is Sim(Query_X8,key_X8)=Query_X8.Key_X8;

[0090] Value_X8 represents the value matrix of the key-value pairs corresponding to the spatial feature graph X8, which is the weight coefficient;

[0091] Lx represents the length of the spatial feature map X8.

[0092] As a preferred technical solution, in step 6, the 1x1 convolution mechanism performs channel transformation fusion on the weighted spatial feature map without changing the width and height of the spatial feature map size;

[0093] The specific channel transformation fusion process satisfies the following relationship:

[0094] ;

[0095] Where u and v represent the horizontal and vertical coordinates of the pixels in the output fusion feature map X10 respectively;

[0096] C9 represents the number of channels of the input weighted spatial feature map X9;

[0097] c represents the channel number of the pixel currently being convolved;

[0098] K represents the matrix expression of the 1x1 convolution kernel.

[0099] As a preferred technical solution, in step 7, the shape adjustment mechanism transforms the fusion feature map dimension into a feature map vector, satisfying the following relationship:

[0100] X11=flat(X10);

[0101] Where X11 represents the vector of the output feature map;

[0102] flat() is a dimension transformation function;

[0103] X10 represents the input fused feature map; wherein the channel of the fused feature map X10 is 1, the width is W10, and the height is H10.

[0104] As a preferred technical solution, in step 8, the feature sequence collection mechanism stores the order of each batch of feature graph vectors to satisfy the following relationship:

[0105] seqB = flat(seq);

[0106] Where, seq=[seq 0, seq 1,.. seq n,..， seq B-1 ];

[0107] seq n =M n ;

[0108] In the formula, seqB means that the feature sequence seq is transformed into a vector with a length of B*W10*H10;

[0109] B represents the set number of video frames extracted in each batch;

[0110] flat() is a dimension transformation function;

[0111] seq represents the feature sequence of the input video frame. The shape of the feature sequence seq is N rows and W10*H10 columns;

[0112] seq 0 、seq 1 、seq n 、seq B-1 Represents the feature vector obtained by inputting the corresponding order in the feature sequence seq;

[0113] Where n is the index number of any set sequential position in the feature sequence seq, 0≤n<B;

[0114] 0, 1, B-1 are quantitative assignments for the n index number;

[0115] M n Represents the final feature matrix of the face image of the nth video frame in this batch.

[0116] As a preferred technical solution, in step 9, the rPPG prediction mechanism obtains the rPPG signal for batch prediction from the feature sequence set mechanism through the fully connected FC layer, and the acquisition process satisfies the following relationship:

[0117] P_rPPG T =seq F;

[0118] Where, P_rPPG T A vector representing the rPPG signal used for prediction;

[0119] seq represents the feature sequence of the input video frame;

[0120] F represents the learnable fully connected matrix;

[0121] Represents matrix multiplication calculation.

[0122] As a preferred technical solution, in step 9, the rPPG prediction mechanism iteratively processes the model for obtaining the rPPG signal task for prediction according to the input batches of video frames;

[0123] The iterative processing of each batch of video frames includes a prediction calculation process and a training process with the goal of reducing the loss;

[0124] The loss of the input video is the average loss of all its batches, which is obtained by averaging the losses of each batch. The loss of the input video satisfies the following relationship:

[0125]

[0126] Where, LOSS represents the loss of the input video;

[0127] T represents the video frame sequence of the Tth batch, 0≤T<N2;

[0128] N2= [N / B]; N2 represents the total number of batches of the input video after batching, N represents the length of the total video frames, and B represents the set number of video frames extracted in each batch;

[0129] Represents the total model fusion loss function of the T-th batch frame sequence, which satisfies the following relationship:

[0130] ;

[0131] In the formula, α, β, γ, and θ are the corresponding weight coefficients set, and the sum is 1;

[0132] Indicates that the Tth batch frame sequence obtains rPPG t The S_Net architecture loss value of the signal;

[0133] Indicates that the heart rate value P_HR is obtained for the Tth batch of frame sequencest The LSTM1 model loss value;

[0134] Indicates that the Tth batch frame sequence obtains the breathing rate value P_RR t The LSTM2 model loss value;

[0135] Indicates that the Tth batch of frame sequences obtains the blood oxygen saturation value P_SPO2 t LSTM3 model loss value;

[0136] The predicted rPPG mechanism The training of loss value satisfies the following relationship:

[0137]

[0138] Where B represents the set number of video frames extracted in each batch;

[0139] t represents the tth frame of the Tth batch;

[0140] Represents the t-th frame prediction rPPG of the T-th batch frame sequence t Signal;

[0141] Represents the true rPPG of the tth frame of the Tth batch frame sequence t Signal;

[0142] The predicted rPPG mechanism The training of loss value satisfies the following relationship:

[0143] ;

[0144] Where B represents the set number of video frames extracted in each batch;

[0145] t represents the tth frame of the Tth batch;

[0146] Represents the predicted heart rate value of the tth frame of the Tth batch frame sequence;

[0147] Indicates the annotated heart rate value of the tth frame of the Tth batch frame sequence;

[0148] The predicted rPPG mechanism The training of loss value satisfies the following relationship:

[0149] ;

[0150] Where B represents the set number of video frames extracted in each batch;

[0151] t represents the tth frame of the Tth batch;

[0152] represents the predicted breathing rate value of the tth frame of the Tth batch frame sequence;

[0153] Represents the breathing rate value of the tth frame of the Tth batch frame sequence;

[0154] The predicted rPPG mechanism The training of loss value satisfies the following relationship:

[0155] ;

[0156] Where B represents the set number of video frames extracted in each batch;

[0157] t represents the tth frame of the Tth batch;

[0158] represents the predicted blood oxygen saturation value of the tth frame of the Tth batch frame sequence;

[0159] It represents the annotated blood oxygen saturation value of the tth frame of the Tth batch frame sequence;

[0160] Furthermore, the rPPG prediction mechanism is based on the predicted sequences Calculate the average value to obtain the final predicted heart rate value P_HR; the predicted heart rate value P_HR is obtained to satisfy the following relationship:

[0161]

[0162] Where, P_HR represents the predicted heart rate value;

[0163] N2 represents the total number of batches of input videos after batching;

[0164] B represents the set number of video frames extracted in each batch;

[0165] T represents the video frame sequence of the Tth batch, 0≤T<N2;

[0166] t represents the tth frame of the Tth batch;

[0167] represents the predicted heart rate value of the tth frame of the Tth batch frame sequence;

[0168] The rPPG prediction mechanism is to predict each sequence Calculate the average value to obtain the final predicted respiratory rate value P_RR; the predicted respiratory rate value P_RR is obtained to satisfy the following relationship:

[0169]

[0170] Where, P_RR predicts the respiratory rate value;

[0171] N2 represents the total number of batches of input videos after batching;

[0172] B represents the set number of video frames extracted in each batch;

[0173] T represents the video frame sequence of the Tth batch, 0≤T<N2;

[0174] t represents the tth frame of the Tth batch;

[0175] represents the predicted breathing rate value of the tth frame of the Tth batch frame sequence;

[0176] The rPPG prediction mechanism is to predict each sequence The average is calculated to obtain the final predicted blood oxygen saturation value P_SPO2; the predicted blood oxygen saturation value P_SPO2 is obtained to satisfy the following relationship:

[0177]

[0178] In the formula, P_SPO2 predicts the blood oxygen saturation value;

[0179] N2 represents the total number of batches of input videos after batching;

[0180] B represents the set number of video frames extracted in each batch;

[0181] T represents the video frame sequence of the Tth batch, 0≤T<N2;

[0182] t represents the tth frame of the Tth batch;

[0183] represents the predicted blood oxygen saturation value of the t-th frame of the T-th batch frame sequence.

[0184] The beneficial technical effect of the present invention is that the above-mentioned technical measures are aimed at rPPG technology for non-contact vital signs detection, which effectively solves the technical problems existing in traditional contact detection of human life characteristics such as heart rate, respiratory rate, blood oxygen saturation, etc., such as operating scene limitations, cumbersome operation process, lack of operation timeliness, and low objective accuracy.

[0185] The above technical measures are based on visible light video of human face, and support real-time acquisition and timely processing of video data through color face image queue and frame difference image queue; through the deep neural network combined with the spatiotemporal network, a powerful spatiotemporal feature extraction capability is formed, and the self-attention mechanism is used in the spatial feature extraction network to focus on the important features in the space, and the 1x1 convolution kernel is used to replace the full connection, which greatly reduces the parameters, saves the amount of calculation parameters and memory consumption, and has a fast calculation speed, which is effectively adapted to the non-invasive detection and real-time detection of human vital signs; the time series features are analyzed in a multi-task manner. The rPPG signal, each branch has the same structure and universal method, but the specific parameters of the network are different, and a universal method is used to achieve reliable prediction of heart rate, respiratory rate and blood oxygen saturation values, which reduces the complexity of the processing process and is easy to implement. Therefore, the above technical measures have the technical characteristics of a relatively simple analysis process, easy implementation, and fast calculation response speed.

[0186] Compared with the technology of publication number CN116959062A, the above technical measures have the following technical advantages:

[0187] First, high real-time processing capability;

[0188] The present invention eliminates the disadvantage of heterogeneous data that requires analysis of both visible light video and infrared video collected in real time, and only uses visible light video of human face as the basis, supports real-time acquisition and timely processing of video data through color face image queue and frame difference image queue, has high real-time processing capability, and is more convenient to use.

[0189] Second, the ability to extract important features is high;

[0190] The present invention abandons the time-consuming manual sequence method of obtaining facial interest area images. The present invention adopts a self-attention mechanism in the spatial feature extraction network to focus on important features in the space, and replaces the full connection with a 1x1 convolution kernel, which greatly reduces the parameters, saves the calculation parameters and memory consumption, and has a fast calculation speed.

[0191] Third, there is a lot of feature information;

[0192] The present invention adopts a deep neural network combined with a spatiotemporal network, which has a powerful spatiotemporal feature extraction capability. The acquired rPPG signal contains more feature information and can reliably predict human vital characteristics such as blood oxygen saturation values ​​in addition to heart rate values ​​and respiratory rate values, thereby enhancing the coverage of detecting human vital characteristics.

[0193] Fourth, it reduces the complexity of prediction;

[0194] The timing characteristics of the present invention adopt a multi-task method to analyze rPPG signals. Each branch has the same structure and a universal method, except that the specific parameters of the network are different. The prediction of heart rate, respiratory rate and blood oxygen saturation values ​​is achieved in a universal method, which reduces the complexity of the processing process and is easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0195] Figure 1 The figure is a principle block diagram of the present invention.

[0196] Figure 2 It is a calculation flow chart of the S_Conv neural network of the present invention. DETAILED DESCRIPTION

[0197] The present invention relates to the field of artificial intelligence medical care information technology, and specifically to a non-contact vital sign detection method based on deep learning. Figure 1 and Figure 2 The technical solution of the present invention is explained clearly and in detail.

[0198] It should be noted that the drawings of the present invention are schematic, and unnecessary details have been simplified to clarify the technical purpose of the present invention, so as to avoid blurring the technical solution of the present invention that contributes to the prior art. In addition, the expressions "approximately", "substantially" and the like regarding quantity or matching relationship in the following text mean that reasonable assembly errors and processing errors in the industry are allowed to exist, and do not literally express absolute quantities or matching relationships.

[0199] See also Figure 1 As shown, the non-contact vital sign detection method of the present invention includes the following nine execution steps and corresponding specific steps.

[0200] Step 1. With the human head remaining still for a set period of time, a visible light video of the human face is obtained by recording under visible light conditions using a recording device such as a mobile phone.

[0201] When recording a visible light video of a human face, the human head is required to remain still for a set time of m seconds (generally set to 10 seconds). Assuming that the frame rate of the video recording device is f (generally 30 frames / second), the length of the recorded visible light video of the human face satisfies the following relationship:

[0202] N = fm;

[0203] Where N represents the total frame length of the current human face video;

[0204] f represents the frame rate of the video recording device;

[0205] m represents the recording time of the human face.

[0206] The acquired human face video is processed by a frame segmentation program to obtain an image frame sequence.

[0207] Step 2. According to the set width M pixels and height H pixels (usually the width M and the height H are equal), the face is located for each picture in the image frame sequence frame by frame, and a number of three-dimensional matrix color face images Face with set width M pixels (for example, set to 127) and height H pixels (for example, set to 127) are obtained respectively, and are stored in the face image queue in sequence.

[0208] Specifically, each color face image Face is cached in the face image queue Q1 in a first-in-first-out order, and the cache in the face image queue Q1 satisfies the following relationship:

[0209] Q1=[Face 0 ,Face 1 ,...,Face i ,...,Face N-1 ];

[0210] Where Q1 represents the face image queue;

[0211] Face 0 、Face 1 、Face i 、Face N-1 Indicates the position of the color face image Face in the video frame in the corresponding index order;

[0212] Wherein, i is the index number of any set sequential position in the video frame, 0≤i<N;

[0213] 0, 1, N-1 are the quantitative assignments for the i index number;

[0214] N represents the total video frame length.

[0215] In order to obtain the change characteristics of adjacent frame images, each color face image Face is first grayed out in turn, and then each adjacent two grayed face images are differentially processed in turn to obtain the differential sequence between adjacent frames, that is, the corresponding frame difference image information, and stored in the frame difference queue.

[0216] Specifically, the color face image Face of each video frame i The grayscale processing satisfies the following relationship:

[0217] ;

[0218] In the formula, represents the grayscale face image of the i-th video frame;

[0219] X represents the matrix of grayscale face image;

[0220] i is the index number of any set sequential position in the video frame, 0≤i<N;

[0221] Represents the pixel value of the grayscale face image in row 0 and column 0 of the i-th video frame;

[0222] represents the pixel value of the grayscale face image in the ith video frame at row 0 and column M-1, where M is the size of the matrix;

[0223] Represents the pixel value of the grayscale face image in the ith video frame at row 1 and column 0;

[0224] represents the pixel value of the grayscale face image in the i-th video frame in row 1 and column M-1, where M is the size of the matrix;

[0225] represents the pixel value of the grayscale face image in the i-th video frame in the M-1 row and 1 column, where M is the size of the matrix;

[0226] Represents the pixel value of the grayscale face image in the i-th video frame in the M-1 row and M-1 column, where M is the size of the matrix.

[0227] The difference processing of two adjacent grayscale face images satisfies the following relationship:

[0228] ;

[0229] in,

[0230] ;

[0231] In the formula, A frame difference image representing the grayscale face image of the i-th video frame and the grayscale face image of the adjacent previous video frame;

[0232] represents the grayscale face image of the i-th video frame;

[0233] Indicates that it is adjacent to the current The grayscale face image of the previous video frame of the video frame grayscale face image;

[0234] Each frame difference image △FaceGr is cached in the frame difference queue Q2 in a first-in-first-out order. The cache in the frame difference queue Q2 satisfies the following relationship:

[0235] Q2=[△FaceGr 0 ,△FaceGr 1 ,...,△FaceGr i,...,△FaceGr N-1 ];

[0236] Where, Q2 represents the frame difference queue;

[0237] △FaceGr 0 , △FaceGr 1 , △FaceGr i , △FaceGr N-1 Indicates the position of the frame difference map △FaceGr corresponding to the index order in the video frame;

[0238] Wherein, i is the index number of any set sequential position in the video frame, 0≤i<N;

[0239] 0, 1, N-1 are the quantitative assignments for the i index number;

[0240] N represents the total video frame length.

[0241] Step 3. The input data of the S_Net architecture is extracted from the face image queue Q1 and the frame difference queue Q2 in a first-in-first-out order.

[0242] Each face image stored in the face image queue Q1 is a color image. The color face image Face is an RGB three-channel image, namely, a red channel FaceR, a green channel FaceG, and a blue channel FaceB.

[0243] The frame difference queue Q2 stores the frame difference image ∆FaceGr corresponding to the color face image Face.

[0244] The S_Net architecture extracts the corresponding color face images and frame difference images from the face image queue Q1 and the frame difference queue Q2 in batches in a multi-channel input manner in a first-in-first-out order, and forms a stacked feature map. The S_Net architecture separates the extracted color face image Face into the red channel FaceR, the green channel FaceG, and the blue channel FaceB according to the RGB three channels.

[0245] Specifically, if the batch size of the input data of the S_Net architecture is represented by B (generally B=30), it means that B face images are taken out from the face image queue Q1 in each batch, and B frame difference images are taken out from the frame difference queue Q2 synchronously to be sent to the S_Net architecture for calculation.

[0246] The S_Net architecture extracts the graphs in each batch in a multi-channel input manner to satisfy the following relationship:

[0247] ;

[0248] in, ;

[0249] In the formula, Represents all inputs of the b-th batch, 0≤b<N1;

[0250] Wherein, N1=[N / B], N1 represents the total batch number of video frames; N represents the length of the total video frames; B represents the set number of video frames extracted in each batch;

[0251] I b0 ,I b1 ,I bj ,I b，B-1 Represents the 4-channel stacked feature map of the corresponding sequential input in the b-th batch; the 4-channel stacked feature map is a three-channel face image matrix of the color face image Face The corresponding sequence frame difference map The stacking structure of

[0252] Represents the red channel matrix of the j-th input face image in the b-th batch;

[0253] Represents the green channel matrix of the j-th input face image in the b-th batch;

[0254] Represents the blue channel matrix of the j-th input face image in the b-th batch;

[0255] represents the frame difference map of the jth input in the bth batch;

[0256] Wherein, j is the index number of any set sequential position in the bth batch, 0≤j<B;

[0257] 0, 1, and B-1 are quantitative assignments for the j index number (i.e., B-1 is the last input in the bth batch).

[0258] The total number of batches for each video clip satisfies the following relationship: ;

[0259] In the formula, Indicates the batch number of the current video clip; where b represents the bth batch;

[0260] N represents the total frame length of the current human face video;

[0261] B represents the size of each batch, that is, the set number of video frames extracted in each batch.

[0262] The S_Net architecture, which serves as a spatial feature extraction network, is divided into six parts: the S_Conv neural network for feature extraction; the spatial self-attention mechanism for weighted calculation; the 1x1 convolution mechanism for transform fusion processing; the shape adjustment mechanism for dimensional transformation; the feature sequence collection mechanism for time order storage; and the predictive rPPG mechanism for rPPG signal prediction.

[0263] The following is a detailed description of the various parts of the S_Net architecture.

[0264] Step 4. The S_Conv neural network in the S_Net architecture is responsible for extracting spatial features from the input graph.

[0265] For details, see Figure 2 As shown in the figure, the S_Conv neural network in the S_Net architecture extracts spatial features from the stacked feature maps of each batch input. It includes three sequential iterations of convolution calculation layers, activation calculation layers, and pooling calculation layers. The calculation output of the current layer is used as the calculation input of the next layer. After three iterations, the activation calculation layer of the third iteration outputs the spatial feature map.

[0266] Among them, the size of the feature map input after the convolution calculation layer will change, that is, the size of the feature map before and after the convolution calculation changes. The convolution calculation layer calculates the width and height of the feature map size separately, but the calculation method of the width and height of the feature map size is the same. The calculation process of the width / height of the feature map size by the convolution calculation layer satisfies the following relationship (for the convenience of subsequent reference, this relationship is listed as relationship 1):

[0267] ;

[0268] In the formula, Indicates the width / height of the feature map size output by the convolution calculation;

[0269] Indicates the width / height of the feature map size of the convolution calculation input;

[0270] w represents the width / height of the convolution kernel;

[0271] p represents the number of circles filled with 0, that is, the number of circles filled with 0 to avoid losing the edge features of the feature map during convolution;

[0272] Indicates the step size of the convolution kernel sliding.

[0273] Since the convolutional calculation layer is a linear calculation with insufficient expressive power, and the activation function adds nonlinear transformation to enhance the expressive power of the model, the output of the convolutional calculation layer is processed by the activation calculation layer.

[0274] The size of the feature map before and after the activation calculation does not change. The calculation process of the feature map by the activation calculation layer satisfies the following relationship (for the convenience of subsequent reference, this relationship is listed as relationship 2):

[0275] f(x)=max(0,x);

[0276] Where x represents each pixel value in the feature map of the activation calculation input; if x≤0, the value is 0, and if x>0, the value remains unchanged.

[0277] The pooling calculation layer can reduce the data scale while maintaining the translation invariance of the features. The size of the feature map after the pooling calculation layer will change, but the number of channels of the feature map will not change. Since the size of the feature map before and after the pooling calculation will change, the pooling calculation layer calculates the width and height of the feature map size separately, but the calculation method for the width and height of the feature map size is the same. The calculation process of the width / height of the feature map size by the pooling calculation layer satisfies the following relationship (for the convenience of subsequent reference, this relationship is listed as relationship 3):

[0278] ;

[0279] In the formula, Indicates the width / height of the feature map size output after pooling calculation;

[0280] Indicates the width / height of the feature map size of the pooling calculation input;

[0281] k represents the width / height of the pooling kernel;

[0282] Indicates the step size of the pooling kernel sliding.

[0283] In the sequential iterative calculation process of the convolution calculation layer, activation calculation layer and pooling calculation layer of the S_Conv neural network, the size of each calculated input feature map and output feature map is of great significance to the calculation and analysis results.

[0284] The following describes in detail the sequential iterative calculation process of the convolution calculation layer, activation calculation layer, and pooling calculation layer of the S_Conv neural network to illustrate the specific changes in the input and output of each layer.

[0285] See also Figure 2 As shown, the input of the S_Conv neural network is 4 channels, which are matrices with a width of 128 pixels and a height of 128 pixels, expressed as 4@128x128, that is, =127.

[0286] The first convolution calculation layer 1 performs convolution calculation. 64 convolution kernels with a width of 5 pixels and a height of 5 pixels are used for convolution calculation, which is represented by w=5. The sliding step size of this convolution calculation is is 2. If the input is not padded, p=0. Substituting it into the above equation 1, we get =62. Since the number of convolution kernels is 64, and since one convolution kernel performs one convolution on the input feature map of one channel to obtain one output feature map, the output of the first convolution calculation layer is 64 feature maps, which are represented as matrix X1. That is, the number of channels of the output matrix X1 is 64, and the size of the matrix X1 is 64@62x62.

[0287] The first activation calculation layer 1 performs activation function calculation. The input feature map is matrix X1, then substitute it into the above relationship 2, and the input element x=X1 cij ; Where c represents the channel of matrix X1, c=[0,1,....,63]; i represents the row of matrix X1, i=[0,1,...,61]; j represents the column of matrix X1, j=[0,1,...,61]; Substitute into the above equation 2 to calculate the matrix X2 of the output feature map. Since the activation calculation layer does not change the size of the feature map, the size of matrix X2 is the same as that of matrix X1, still expressed as 64@62x62.

[0288] The first pooling calculation layer (i.e., maximum pooling calculation) 1 performs pooling calculation. The matrix X2 is used as the input feature map. The pooling kernel of this pooling calculation layer is 2 pixels wide and 2 pixels high, expressed as k=2, and the sliding step length of the pooling kernel is is 2 pixels, expressed as = 2. The size of each feature map in the matrix X2 is 62 pixels in width and height. =62, and these parameters are put into the above equation 3 to calculate =32, indicating that the width and height of the feature map matrix X3 output after pooling calculation are 32 pixels. Since the pooling calculation layer only changes the width and height of the feature map but does not change the number of channels, the size of the matrix X3 is expressed as 64@32x32.

[0289] The second convolution calculation layer 2 performs convolution calculation. 128 convolution kernels with a width of 3 pixels and a height of 3 pixels are used for convolution calculation, which is represented by w=3. The sliding step size of this convolution is The input is 2, and a circle of 0 is filled in the outermost layer of the image to avoid edge feature loss during convolution, that is, p=1. Substituting it into the above relationship 1, we get =32. Since the number of convolution kernels is 128, the output of the second convolution calculation layer is 128 feature maps. The output feature map is represented by matrix X4, and the size is 128@32x32.

[0290] The second activation calculation layer 2 performs activation function calculation. The input feature map is matrix X4, then substitute it into the above relationship 2, and the input element x=X4 cij ; Where c represents the channel of matrix X4, c=[0,1,....127]; i represents the row of matrix X4, i=[0,1,...,31]; j represents the column of matrix X4, j=[0,1,...,31]; Substitute into the above equation 2 to calculate the matrix X5. Since the activation calculation layer does not change the input matrix size, the size of matrix X5 is the same as that of matrix X4, still expressed as 128@32x32.

[0291] The second pooling calculation layer 2 performs pooling calculations. The matrix X5 is used as input. The pooling kernel of this pooling calculation layer is 2 pixels wide and 2 pixels high, expressed as k=2, and the sliding step length of the pooling kernel is is 2 pixels, expressed as = 2, the size of each feature map in the input matrix X5 is 32 pixels in width and height. =32, and these parameters are put into the above equation 3 to calculate =16, indicating that the width and height of the feature map matrix X6 output after pooling calculation are both 16 pixels. Since the pooling calculation layer only changes the width and height of the feature map but does not change the number of channels, the size of the matrix X6 is expressed as 128@16x16.

[0292] The third convolution calculation layer 3 performs convolution calculation. 32 convolution kernels with a width of 3 pixels and a height of 3 pixels are used for convolution calculation, which is expressed as w=3. The sliding step size of this convolution is is 2, and the input fills the outermost layer of the image with a circle of 0s to avoid edge feature loss during convolution, then p=1. Substituting into the above equation 1, we get = 16. Since the number of convolution kernels is 32, the output after calculation of the third convolution calculation layer is 32 feature maps, represented as matrix X7, and the size is 32@16x16.

[0293] The third activation calculation layer 3 performs activation function calculation. The input feature map is matrix X7, then substitute it into the above relationship 2, and the input element x=X7 cij ; Where c represents the channel of matrix X7, c=[0,1,....,31]; i represents the row of matrix X7, i=[0,1,...,15]; j represents the column of matrix X7, j=[0,1,...,15]; Substituting into the above equation 2, the matrix X8 is calculated. Since the activation calculation layer does not change the input matrix size, the size of matrix X8 is the same as that of matrix X7, expressed as 32@16x16.

[0294] Step 5. The spatial self-attention mechanism in the S_Net architecture performs weighted processing on the spatial feature map matrix X8 extracted by the S_Conv neural network to focus on important spatial features.

[0295] By using spatial self-attention, we can focus on the correlation between different positions within the feature map matrix X8 extracted by the S_Conv neural network, enhance the model's ability to capture important features in the feature map, and thus extract richer and more meaningful features.

[0296] Specifically, the spatial self-attention mechanism focuses on the weighted processing of the important spatial features of the spatial feature map extracted by the S_Conv neural network, and the attention calculation satisfies the following relationship:

[0297] ;

[0298] When Key_X8=Value_X8=Query_X8, the calculation of self-attention distribution and the weighting of self-attention satisfy the following relationship:

[0299] Self_A X =Attention(Query_X8,X8);

[0300] The self-attention weights of the spatial feature map satisfy the following relationship:

[0301] X9=X8⊙Self_A X ;

[0302] Where X9 represents the weighted spatial feature map calculated by self-attention;

[0303] X8 represents the spatial feature map extracted by the S_Conv neural network;

[0304] Self_A x Represents the self-attention distribution of the spatial feature map X8;

[0305] Attention represents the self-attention distribution function of spatial features;

[0306] Query_X8 represents the query matrix corresponding to the spatial feature map X8, and its value is any element in the spatial feature map X8;

[0307] Key_X8 represents the key matrix in the key-value pair corresponding to the spatial feature map X8;

[0308] The association between Query_X8 and Key_X8 is Sim(Query_X8,key_X8)=Query_X8.Key_X8;

[0309] Value_X8 represents the value matrix of the key-value pairs corresponding to the spatial feature graph X8, which is the weight coefficient;

[0310] Lx represents the length of the spatial feature map X8;

[0311] The calculation symbol “⊙” represents the dot product calculation of the matrix.

[0312] Step 6. After obtaining important features through the spatial self-attention mechanism, the 1x1 convolution mechanism is used for calculation. The 1x1 convolution calculation is a special form of convolution, which can play a channel transformation role for the input feature map, that is, the 1x1 convolution mechanism in the S_Net architecture, which performs channel transformation fusion processing on the weighted spatial feature map.

[0313] Specifically, the matrix X9 obtained after the spatial self-attention calculation is expressed as (C9, W9, H9). Here, C9 1x1 convolution matrices X9 are used for convolution calculation, and the matrix X10 is obtained after calculation. Here, C9 1x1 matrices X9 play a dimensionality reduction role in reducing the number of channels. Here, a 1x1 convolution kernel is used to replace the fully connected calculation in the traditional convolutional neural network, which greatly reduces the number of parameters and memory consumption and improves the efficiency of calculation.

[0314] The 1x1 convolution mechanism transforms and fuses the channels of the weighted spatial feature map without changing the width and height of the spatial feature map. The 1x1 convolution mechanism transforms and fuses the channels of the weighted spatial feature map to satisfy the following relationship:

[0315] ;

[0316] Where u and v represent the horizontal and vertical coordinates of the pixels in the output fusion feature map X10 respectively;

[0317] C9 represents the number of channels of the input weighted spatial feature map X9;

[0318] c represents the channel number of the pixel currently being convolved;

[0319] K represents the matrix expression of 1x1 convolution kernel;

[0320] The calculation symbol “⊙” represents the dot product calculation of the matrix.

[0321] After calculation, the number of feature map channels is reduced from C9 of matrix X9 to C10=1 of matrix X10. This calculation does not change the width and height of the feature map, that is, the width of the feature map W10=W9, and the height of the feature map H10=H9.

[0322] Step 7. The shape adjustment mechanism in the S_Net architecture is to construct the feature map matrix into a feature vector. In other words, the shape adjustment mechanism in the S_Net architecture transforms the fused feature map dimension into a vector of feature maps.

[0323] The specific implementation method is to flatten the feature map matrix into a vector through dimensional transformation. The shape adjustment mechanism transforms the fusion feature map dimension into a feature map vector, satisfying the following relationship:

[0324] X11=flat(X10);

[0325] Where X11 represents the vector of the output feature map, and its length is 1*W10*H10;

[0326] flat() is a dimension transformation function;

[0327] X10 represents the input fused feature map; wherein the channel of the fused feature map X10 is 1, the width is W10, and the height is H10.

[0328] Step 8. Considering that the data is input in batches, if multiple threads are concurrent, the execution order sent to the S_Net architecture will be disrupted, and the feature maps output by the S_Net architecture may also be out of order. In order to record their correct time order, a feature sequence collection mechanism is formed in the S_Net architecture, which stores each batch of feature map vectors according to the time position information, that is, the feature map matrix after shape adjustment is placed in the collection according to the time position information, and the time position information corresponds to the index number of the frame.

[0329] The feature sequence collection mechanism stores the order of each batch of feature graph vectors, satisfying the following relationship:

[0330] seqB = flat(seq);

[0331] Where, seq=[seq 0, seq 1,.. seq n,..， seq B-1 ];

[0332] seq n =M n ;

[0333] In the formula, seqB indicates that the feature sequence seq is transformed into a vector of length B*W10*H10; in order to facilitate the subsequent full connection calculation, it is expressed as a matrix of 1 row and N*W10*H10 columns;

[0334] B represents the set number of video frames extracted in each batch;

[0335] flat() is a dimension transformation function;

[0336] seq represents the feature sequence of the input video frame. The shape of the feature sequence seq is N rows and W10*H10 columns;

[0337] seq 0 、seq 1 、seq n 、seq B-1 Represents the feature vector obtained by inputting the corresponding order in the feature sequence seq; that is, seq 0 It represents the feature vector obtained after the previous series of calculations of the 0th input in the feature sequence seq. 1 It represents the feature vector obtained after a series of calculations of the first input in the feature sequence seq. B-1 Indicates that it is the first B-1 The feature vector obtained after the previous series of calculations for the input (the last input of the batch);

[0338] Where n is the index number of any set sequential position in the feature sequence seq, 0≤n<B;

[0339] 0, 1, B-1 are quantitative assignments for the n index number;

[0340] M n Represents the final feature matrix of the face image of the nth video frame in this batch.

[0341] Step 9. In the collected data set, the rPPG signal annotated for the video clip is a one-dimensional vector T_rPPG, whose shape is represented by D_rPPG. The rPPG prediction mechanism obtains the rPPG signal for prediction from the feature sequence set mechanism through the fully connected FC layer.

[0342] Specifically, the prediction rPPG mechanism obtains the prediction rPPG signal from the feature sequence set mechanism through the fully connected FC layer, satisfying the following relationship:

[0343] P_rPPG T =seq F;

[0344] Where, P_rPPG T The vector representing the rPPG signal used for prediction; wherein the shape of P_rPPG is consistent with that of T_rPPG;

[0345] seq represents the feature sequence of the input video frame;

[0346] F represents a learnable fully connected matrix; the shape of the fully connected matrix F is (N*W10*H10,D_rPPG), that is, N*W10*H10 pixels in rows and D_rPPG pixels in columns;

[0347] Represents matrix multiplication calculation.

[0348] Since the S_Net architecture uses multi-task learning to obtain the rPPG signal for prediction, three parallel long short-term memory networks (LSTM) are used to extract the temporal dependency features in the rPPG signal, and finally the heart rate, respiratory rate and blood oxygen saturation values ​​are predicted. Therefore, the rPPG prediction mechanism iteratively processes the model of the task of obtaining the rPPG signal for prediction according to the input batches of video frames; the iterative processing of each batch of video frames includes a prediction calculation process and a training process with the goal of reducing losses.

[0349] The loss of the input video is the average loss of all its batches, which is obtained by averaging the losses of each batch. The loss of the input video satisfies the following relationship:

[0350]

[0351] Where, LOSS represents the loss of the input video;

[0352] T represents the video frame sequence of the Tth batch, 0≤T<N2;

[0353] N2=[N / B]; N2 represents the total number of batches of the input video after batching, N represents the length of the total video frames, and B represents the set number of video frames extracted in each batch;

[0354] represents the total model fusion loss function of the T-th batch frame sequence; it satisfies the following relationship:

[0355] ;

[0356] In the formula, , β, , They are the corresponding weight coefficients set, the sum of which is 1, and usually each takes a value of 0.25;

[0357] Indicates that the Tth batch frame sequence obtains rPPG t The S_Net architecture loss value of the signal;

[0358] Indicates that the heart rate value P_HR is obtained for the Tth batch of frame sequences t The LSTM1 model loss value;

[0359] Indicates that the Tth batch frame sequence obtains the breathing rate value P_RR t The LSTM2 model loss value;

[0360] Indicates that the Tth batch of frame sequences obtains the blood oxygen saturation value P_SPO2 t The LSTM3 model loss value.

[0361] Predicting rPPG mechanisms, The training of the loss value (i.e. the training of the S_Net architecture) uses the Pearson correlation coefficient as the loss function. The Pearson correlation coefficient is used to reflect the linear correlation between two random variables. The closer it is to 0, the weaker the correlation, and the closer it is to 1, the stronger the correlation. Specifically, it satisfies the following relationship:

[0362]

[0363]

[0364] Where B represents the set number of video frames extracted in each batch;

[0365] t represents the tth frame of the Tth batch;

[0366] Represents the t-th frame prediction rPPG of the T-th batch frame sequence t Signal;

[0367] Represents the true rPPG of the tth frame of the Tth batch frame sequence t Signal.

[0368] The rPPG prediction mechanism captures the heart rate timing characteristics in the rPPG signal through the long short-term memory network LSTM1 to obtain the predicted heart rate value P_HR.

[0369] The predicted heart rate value P_HR obtained by the rPPG mechanism needs to be trained through the loss function of the sub-model LSTM1 for obtaining the heart rate value P_HR. The prediction rPPG mechanism uses the mean square error loss function to train the sub-model LSTM1 for obtaining the heart rate value P_HR. That is, the prediction rPPG mechanism The training of loss value should satisfy the following relationship:

[0370] ;

[0371] Where B represents the set number of video frames extracted in each batch;

[0372] t represents the tth frame of the Tth batch;

[0373] Represents the predicted heart rate value of the tth frame of the Tth batch frame sequence;

[0374] Indicates the annotated heart rate value of the tth frame of the Tth batch frame sequence.

[0375] The rPPG prediction mechanism is to predict each sequence Calculate the average value to obtain the final predicted heart rate value P_HR. The predicted heart rate value P_HR satisfies the following relationship:

[0376]

[0377] Where, P_HR represents the predicted heart rate value;

[0378] N2 represents the total number of batches of input videos after batching;

[0379] B represents the set number of video frames extracted in each batch;

[0380] T represents the video frame sequence of the Tth batch, 0≤T<N2;

[0381] t represents the tth frame of the Tth batch;

[0382] Represents the predicted heart rate value of the tth frame of the Tth batch frame sequence.

[0383] The rPPG prediction mechanism captures the respiratory rate timing characteristics in the rPPG signal through the long short-term memory network LSTM2 to obtain the predicted respiratory rate value P_RR.

[0384] The predicted respiratory rate value P_RR obtained by the rPPG mechanism needs to be trained through the loss function of the sub-model LSTM2 that obtains the respiratory rate value P_RR. The prediction rPPG mechanism uses the mean square error loss function to train the sub-model LSTM2 that obtains the respiratory rate value P_RR. That is, the prediction rPPG mechanism The training of loss value should satisfy the following relationship:

[0385] ;

[0386] Where B represents the set number of video frames extracted in each batch;

[0387] t represents the tth frame of the Tth batch;

[0388] represents the predicted breathing rate value of the tth frame of the Tth batch frame sequence;

[0389] Represents the annotated breathing rate value of the tth frame of the Tth batch frame sequence.

[0390] The rPPG prediction mechanism is to predict each sequence Calculate the average value to obtain the final predicted respiratory rate value P_RR. The predicted respiratory rate value P_RR satisfies the following relationship:

[0391]

[0392] Where, P_RR predicts the respiratory rate value;

[0393] N2 represents the total number of batches of input videos after batching;

[0394] B represents the set number of video frames extracted in each batch;

[0395] T represents the video frame sequence of the Tth batch, 0≤T<N2;

[0396] t represents the tth frame of the Tth batch;

[0397] Represents the predicted breathing rate value of the tth frame of the Tth batch frame sequence.

[0398] The rPPG prediction mechanism captures the blood oxygen saturation time series characteristics in the rPPG signal through the long short-term memory network LSTM3 to obtain the predicted blood oxygen saturation value P_SPO2.

[0399] The predicted blood oxygen saturation value P_SPO2 obtained by the rPPG mechanism needs to be trained through the loss function of the sub-model LSTM3 for obtaining the blood oxygen saturation value P_SPO2. The prediction rPPG mechanism uses the mean square error loss function to train the sub-model LSTM3 for obtaining the blood oxygen saturation value P_SPO2. That is, the prediction rPPG mechanism The training of loss value should satisfy the following relationship:

[0400] ;

[0401] Where B represents the set number of video frames extracted in each batch;

[0402] t represents the tth frame of the Tth batch;

[0403] represents the predicted blood oxygen saturation value of the tth frame of the Tth batch frame sequence;

[0404] Indicates the annotated blood oxygen saturation value of the tth frame of the Tth batch frame sequence.

[0405] The rPPG prediction mechanism is to predict each sequence Calculate the average value to obtain the final predicted blood oxygen saturation value . Predicted blood oxygen saturation value The acquisition satisfies the following relationship:

[0406]

[0407] In the formula, Predict blood oxygen saturation values;

[0408] N2 represents the total number of batches of input videos after batching;

[0409] B represents the set number of video frames extracted in each batch;

[0410] T represents the video frame sequence of the Tth batch, 0≤T<N2;

[0411] t represents the tth frame of the Tth batch;

[0412] represents the predicted blood oxygen saturation value of the t-th frame of the T-th batch frame sequence.

[0413] As for the above-mentioned long short-term memory network LSTM, it is a publicly available relational expression that can be directly called.

[0414] The above specific technical solutions are only used to illustrate the present invention rather than to limit it.

[0415] Although the present invention has been described in detail with reference to the above-mentioned specific technical solutions, those skilled in the art should understand that they can still modify the above-mentioned specific technical solutions, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the present invention.

Claims

1. A non-contact vital sign detection method based on deep learning, characterized in that: The detection method comprises: Step 1. Obtain a visible light video of a human face, and process the video into frames to obtain an image frame sequence; Step 2. Obtain a number of three-dimensional matrix color face images frame by frame and store them in a face image queue; Each frame of the color face image is grayed out in turn, and two adjacent frames of grayed face images are differentially processed in turn to obtain a corresponding frame difference image, and the image is stored in a frame difference queue; Step 3. The S_Net architecture extracts the corresponding color face images and frame difference images from the face image queue and the frame difference queue in batches in a multi-channel input manner in a first-in-first-out order, and forms a stacked feature map; Step 4. The S_Conv neural network in the S_Net architecture extracts spatial features from the input stacked feature map to obtain a spatial feature map; Step 5. The spatial self-attention mechanism in the S_Net architecture performs weighted processing on the spatial feature map extracted by the S_Conv neural network to focus on important spatial features; Step 6. The 1x1 convolution mechanism in the S_Net architecture performs channel transformation and fusion processing on the weighted spatial feature map; Step 7. The shape adjustment mechanism in the S_Net architecture transforms the fused feature map dimension into a vector of feature maps; Step 8. The feature sequence collection mechanism in the S_Net architecture stores the feature map vector according to the time position information; Step 9. The prediction rPPG mechanism in the S_Net architecture captures the heart rate timing features in the rPPG signal through the long short-term memory network LSTM1 to obtain the predicted heart rate value P_HR, captures the respiratory rate timing features in the rPPG signal through the long short-term memory network LSTM2 to obtain the predicted respiratory rate value P_RR, and captures the blood oxygen saturation timing features in the rPPG signal through the long short-term memory network LSTM3 to obtain the predicted blood oxygen saturation value P_SPO2; The rPPG prediction mechanism iteratively processes the model for the task of obtaining the rPPG signal for prediction according to the input batches of video frames; The iterative processing of each batch of video frames includes a prediction calculation process and a training process with the goal of reducing the loss; The loss of the input video is the average loss of all its batches, which is obtained by averaging the losses of each batch. The loss of the input video satisfies the following relationship: Where LOSS represents the loss of the input video; T represents the video frame sequence of the Tth batch, 0≤T<N2; Wherein, N2 represents the total number of batches of the input video after batching, N represents the length of the total video frames, and B represents the set number of video frames extracted in each batch; Loss T Represents the total model fusion loss function of the T-th batch frame sequence, which satisfies the following relationship: Loss T =∝.L_rPPG T +b.L_HR T +c.L_RR T +θ.L_SPO2 T ; In the formula, ∝, β, γ, and θ are the corresponding weight coefficients set, and the sum is 1; L_rPPG T Indicates that the Tth batch frame sequence obtains rPPG t The S_Net architecture loss value of the signal; L_HR T Indicates that the heart rate value P_HR is obtained for the Tth batch of frame sequences t The LSTM1 model loss value; L_RR T Indicates that the Tth batch frame sequence obtains the breathing rate value P_RR t The LSTM2 model loss value; L_SPO2 T Indicates that the Tth batch of frame sequences obtains the blood oxygen saturation value P_SPO2 t LSTM3 model loss value; The predicted rPPG mechanism for L_rPPG T The training of loss value satisfies the following relationship: Where B represents the set number of video frames extracted in each batch; t represents the tth frame of the Tth batch; P_rPPG Tt Represents the t-th frame prediction rPPG of the T-th batch frame sequence t Signal; T_rPPG Tt Represents the true rPPG of the tth frame of the Tth batch frame sequence t Signal; The predicted rPPG mechanism for L_HR T The training of loss value satisfies the following relationship: Where B represents the set number of video frames extracted in each batch; t represents the tth frame of the Tth batch; P_HR Tt Represents the predicted heart rate value of the tth frame of the Tth batch frame sequence; T_HR Tt Indicates the annotated heart rate value of the tth frame of the Tth batch frame sequence; The predicted rPPG mechanism for L_RR T The training of loss value satisfies the following relationship: Where B represents the set number of video frames extracted in each batch; t represents the tth frame of the Tth batch; P_RR Tt represents the predicted breathing rate value of the tth frame of the Tth batch frame sequence; T_RR Tt Represents the breathing rate value of the tth frame of the Tth batch frame sequence; The predicted rPPG mechanism for L_SPO2 T The training of loss value satisfies the following relationship: Where B represents the set number of video frames extracted in each batch; t represents the tth frame of the Tth batch; P_SPO2 Tt represents the predicted blood oxygen saturation value of the tth frame of the Tth batch frame sequence; T_SPO2 Tt It represents the annotated blood oxygen saturation value of the tth frame of the Tth batch frame sequence; Furthermore, the rPPG prediction mechanism predicts each sequence P_HR by Tt Calculate the average value to obtain the final predicted heart rate value P_HR; the predicted heart rate value P_HR is obtained to satisfy the following relationship: Where P_HR represents the predicted heart rate value; N2 represents the total number of batches of input videos after batching; B represents the set number of video frames extracted in each batch; T represents the video frame sequence of the Tth batch, 0≤T<N2; t represents the tth frame of the Tth batch; P_HR Tt represents the predicted heart rate value of the tth frame of the Tth batch frame sequence; The rPPG prediction mechanism is to predict each sequence P_RR Tt Calculate the average value to obtain the final predicted respiratory rate value P_RR; the predicted respiratory rate value P_RR satisfies the following relationship: Where, P_RR predicts the respiratory rate value; N2 represents the total number of batches of input videos after batching; B represents the set number of video frames extracted in each batch; T represents the video frame sequence of the Tth batch, 0≤T<N2; t represents the tth frame of the Tth batch; P_RR Tt represents the predicted breathing rate value of the tth frame of the Tth batch frame sequence; The rPPG prediction mechanism is based on the predicted sequence P_SPO2 Tt The average is calculated to obtain the final predicted blood oxygen saturation value P_SPO2; the predicted blood oxygen saturation value P_SPO2 is obtained to satisfy the following relationship: In the formula, P_SPO2 predicts the blood oxygen saturation value; N2 represents the total number of batches of input videos after batching; B represents the set number of video frames extracted in each batch; T represents the video frame sequence of the Tth batch, 0≤T<N2; t represents the tth frame of the Tth batch; P_SPO2 Tt represents the predicted blood oxygen saturation value of the t-th frame of the T-th batch frame sequence.

2. The non-contact vital sign detection method based on deep learning according to claim 1, characterized in that: In step 2, the color face image is obtained by performing face positioning on each video frame to set pixels to obtain a three-dimensional matrix color face image Face; Each color face image Face is cached in the face image queue Q1 in a first-in-first-out order. The cache in the face image queue Q1 satisfies the following relationship: Q1=[Face0,Face1,...,Face i ,...,Face N-1 ]; Where, Q1 represents the face image queue; Face0, Face1, Face i 、Face N-1 Indicates the position of the color face image Face in the video frame in the corresponding index order; Wherein, i is the index number of any set sequential position in the video frame, 0≤i<N; 0, 1, N-1 are the quantitative assignments for the i index number; N represents the length of the total video frame; The grayscale processing of the color face image of each video frame satisfies the following relationship: In the formula, FaceGr i represents the grayscale face image of the i-th video frame; X represents the matrix of grayscale face image; i is the index number of any set sequential position in the video frame, 0≤i <N; Represents the pixel value of the grayscale face image in row 0 and column 0 of the i-th video frame; represents the pixel value of the grayscale face image in the ith video frame at row 0 and column M-1, where M is the size of the matrix; The difference processing of two adjacent grayscale face images satisfies the following relationship: in, Where, ΔFaceGr i A frame difference image representing the grayscale face image of the i-th video frame and the grayscale face image of the adjacent previous video frame; FaceGr i represents the grayscale face image of the i-th video frame; FaceGr i-1 Indicates that it is adjacent to the current FaceGr i The grayscale face image of the previous video frame of the video frame grayscale face image; Each frame difference image ΔFaceGr is cached in the frame difference queue Q2 in a first-in-first-out order. The cache in the frame difference queue Q2 satisfies the following relationship: Q2=[ΔFaceGr0,ΔFaceGr1,...,ΔFaceGr i ,...,ΔFaceGr N-1 ]; Where, Q2 represents the frame difference queue; ΔFaceGr0, ΔFaceGr1, ΔFaceGr i , ΔFaceGr N-1 Indicates the position of the frame difference map ΔFaceGr corresponding to the index order in the video frame; Wherein, i is the index number of any set sequential position in the video frame, 0≤i<N; 0, 1, N-1 are the quantitative assignments for the i index number; N represents the total video frame length.

3. The non-contact vital sign detection method based on deep learning according to claim 1, characterized in that: In step 3, the S_Net architecture separates the extracted color face images Face into a red channel FaceR, a green channel FaceG, and a blue channel FaceB according to the RGB three channels; The S_Net architecture extracts the atlas information in each batch in a multi-channel input manner to satisfy the following relationship: I b =[I b0 ,I b1 ,...,I bj ,...I b,B-1 ]; in, In the formula, I b Represents all inputs of the b-th batch, 0≤b<N1; in, N1 represents the total batch number of video frames; N represents the total length of video frames; B represents the set number of video frames extracted in each batch; I b0 ,I b1 ,I bj ,I b，B-1 Represents the 4-channel stacked feature map of the corresponding sequential input in the b-th batch; the 4-channel stacked feature map is a three-channel face image matrix of the color face image Face The corresponding sequence frame difference map The stacking structure of Represents the red channel matrix of the j-th input face image in the b-th batch; Represents the green channel matrix of the j-th input face image in the b-th batch; Represents the blue channel matrix of the j-th input face image in the b-th batch; represents the frame difference map of the jth input in the bth batch; Wherein, j is the index number of any set sequential position in the bth batch, 0≤j<B; 0, 1, and B-1 are quantitative assignments for the j index number.

4. The non-contact vital sign detection method based on deep learning according to claim 1, characterized in that: In step 4, the S_Conv neural network includes a convolution calculation layer, an activation calculation layer, and a pooling calculation layer that are iterated three times in sequence for each batch of input data, and uses the calculation output of the current layer as the calculation input of the next layer to obtain a spatial feature map; The size of the feature map changes before and after the convolution calculation. The convolution calculation layer calculates the width and height of the feature map size respectively, and the calculation process satisfies the following relationship: Where W outConv Indicates the width / height of the feature map size output by the convolution calculation; W inConv Indicates the width / height of the feature map size of the convolution calculation input; w represents the width / height of the convolution kernel; p represents the number of circles filled with 0; s c Indicates the step size of the convolution kernel sliding; The size of the feature map before and after the activation calculation remains unchanged, and the calculation process of the feature map by the activation calculation layer satisfies the following relationship: f(x)=max(0,x); Where x represents each pixel value in the feature map of the activation calculation input; if x≤0, the value is 0, and if x>0, the value remains unchanged; The size of the feature map changes before and after the pooling calculation. The pooling calculation layer calculates the width and height of the feature map size respectively, and the calculation process satisfies the following relationship: Where W outPool Indicates the width / height of the feature map size output after pooling calculation; W inPool Indicates the width / height of the feature map size of the pooling calculation input; k represents the width / height of the pooling kernel; s p Indicates the step size of the pooling kernel sliding.

5. The non-contact vital sign detection method based on deep learning according to claim 1, characterized in that: In step 5, the spatial self-attention mechanism focuses on the weighted processing of the important spatial features of the spatial feature map X8 extracted by the S_Conv neural network, and the attention calculation satisfies the following relationship: When Key_X8=Value_X8=Query_X8, the calculation of self-attention distribution and the weighting of self-attention satisfy the following relationship: Self_A X =Attention(Query_X8,X8); The self-attention weights of the spatial feature map satisfy the following relationship: X9=X8⊙Self_A X ; Where X9 represents the weighted spatial feature map calculated by self-attention; X8 represents the spatial feature map extracted by the S_Conv neural network; Self_A x Represents the self-attention distribution of the spatial feature map X8; Attention represents the self-attention distribution function of spatial features; Query_X8 represents the query matrix corresponding to the spatial feature map X8, and its value is any element in the spatial feature map X8; Key_X8 represents the key matrix in the key-value pair corresponding to the spatial feature map X8; The association between Query_X8 and Key_X8 is Sim(Query_X8,key_X8)=Query_X8.Key_X8; Value_X8 represents the value matrix of the key-value pairs corresponding to the spatial feature graph X8, which is the weight coefficient; Lx represents the length of the spatial feature map X8.

6. The non-contact vital sign detection method based on deep learning according to claim 1, characterized in that: In step 6, the 1x1 convolution mechanism performs channel transformation fusion on the weighted spatial feature map without changing the width and height of the spatial feature map size; The specific channel transformation fusion process satisfies the following relationship: Where u and v represent the horizontal and vertical coordinates of the pixels in the output fusion feature map X10 respectively; C9 represents the number of channels of the input weighted spatial feature map X9; c represents the channel number of the pixel currently being convolved; K represents the matrix expression of the 1x1 convolution kernel.

7. The non-contact vital sign detection method based on deep learning according to claim 1, characterized in that: In step 7, the shape adjustment mechanism transforms the fusion feature map dimension into a feature map vector, satisfying the following relationship: X11 = flat(X10); Where X11 represents the vector of the output feature map; flat() is a dimension transformation function; X10 represents the input fused feature map; wherein the channel of the fused feature map X10 is 1, the width is W10, and the height is H10.

8. The non-contact vital sign detection method based on deep learning according to claim 1, characterized in that: In step 8, the feature sequence collection mechanism stores the order of each batch of feature graph vectors, satisfying the following relationship: seqB = flat(seq); Where, seq = [seq 0, seq 1,.. seq n,..， seq B-1 ]; seq n =M n ; In the formula, seqB means that the feature sequence seq is transformed into a vector with a length of B*W10*H10; B represents the set number of video frames extracted in each batch; flat() is a dimension transformation function; seq represents the feature sequence of the input video frame. The shape of the feature sequence seq is N rows and W10*H10 columns; seq0, seq1, seq n 、seq B-1 Represents the feature vector obtained by inputting the corresponding order in the feature sequence seq; Where n is the index number of any set sequential position in the feature sequence seq, 0≤n<B; 0, 1, B-1 are quantitative assignments for the n index number; M n Represents the final feature matrix of the face image of the nth video frame in this batch.

9. The non-contact vital sign detection method based on deep learning according to claim 1, characterized in that: In step 9, the rPPG prediction mechanism obtains the rPPG signal for batch prediction from the feature sequence set mechanism through the fully connected FC layer, and the acquisition process satisfies the following relationship: Where, P_rPPG T A vector representing the rPPG signal used for prediction; seq represents the feature sequence of the input video frame; F represents the learnable fully connected matrix; Represents matrix multiplication calculation.

Images