Non-contact physiological and psychological multi-parameter detection method and system

Abstract

The present invention provides a non-contact physiological and psychological multi-parameter detection method and system, which relates to the field of parameter detection. The non-contact physiological and psychological multi-parameter detection method provided by the present invention can detect physiological parameters including heart rate, blood pressure, respiratory rate, facial temperature, heart rate variability, and psychological parameters such as emotions, anxiety, depression, etc. Among them, in terms of physiological parameter detection, the present invention extracts rPPG signals based on a global spatiotemporal attention mechanism. In the signal extraction process, it can make full use of the correlation of rPPG signals at different times and in different facial areas, complement and enhance each other, and improve the effect of rPPG signal extraction. In terms of psychological parameter detection, facial behaviors and physiological features are shared between multiple parameters, giving full play to the role of shared features and saving the time of data processing for each parameter.

Description

Technical Field

[0001] The present invention relates to the technical field of parameter detection, and in particular to a non-contact physiological and psychological multi-parameter detection method and system. Background Art

[0002] With the development of science and technology, the health status of users is often judged by testing physiological and psychological parameters. Among them, the method of testing the physiological and psychological parameters of users based on rPPG (remote PhotoPlethysmoGraphy) signals has a wide range of applications and good development prospects because users do not need to wear any sensors during the testing process.

[0003] However, the extraction of rPPG signals in the prior art is often based only on convolutional neural networks, which have problems such as limited receptive field and inability to model long-range spatiotemporal correlations, resulting in low accuracy of detection results based on rPPG signals in the prior art. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the deficiencies in the prior art, the present invention provides a non-contact physiological and psychological multi-parameter detection method and system, which solves the problem of low detection result accuracy in the existing detection method based on rPPG signals.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] In a first aspect of the present invention, a non-contact physiological and psychological multi-parameter detection method is provided, comprising:

[0009] Acquire a sequence of facial interest region images as a first image sequence;

[0010] Calculating a normalized frame difference between two adjacent frames in the first image sequence to obtain a normalized frame difference sequence;

[0011] Extracting features of the normalized frame difference sequence based on a preset two-dimensional convolutional layer to obtain a first image feature;

[0012] The order of the channel dimension and the time dimension in the first image feature is exchanged, and the width and height dimensions in the spatial dimension are flattened into one dimension to obtain a second image feature;

[0013] Based on a preset two-dimensional convolutional layer, the local spatiotemporal information of the second image feature is integrated to obtain a third image feature;

[0014] Based on a preset global spatiotemporal self-attention layer, the global spatiotemporal information of the third image feature is integrated and feature extraction is performed to obtain a fourth image feature;

[0015] Fusing the spatial dimension features of the fourth image features, and upsampling the time dimension size of the fused fourth image features so that the time dimension size after the upsampling process is the same as the length of the normalized frame difference sequence, and mapping the channel dimension of the fourth image features after the upsampling process to 1, to obtain a mapped signal as an rPPG signal;

[0016] Based on the rPPG signal, the user's physiological parameters are detected; wherein the physiological parameters include blood pressure, respiratory rate and heart rate.

[0017] Optionally, the preset global spatiotemporal self-attention layer includes three identical global spatiotemporal self-attention layers, wherein the processing process of each global spatiotemporal self-attention layer includes:

[0018] Based on the linear mapping operation W in three pixel dimensions Q , W K , W V , respectively for the third image feature F 3 Perform linear mapping to obtain query feature Q, key feature K and value feature V; where Q = W Q ·F 3 , K=W K ·F 3 、V=W V ·F 3 ;

[0019] Q, K, and V are reorganized into four-dimensional features Q′, K′, and V′ respectively. The dimension of the reorganized features is N′×P′×1×C; wherein N′ represents the number of frames of the third image feature, P′ represents the product of the height and width of the third image feature, and C represents the number of channels;

[0020] Traverse the index i of the time dimension and the index j of the space dimension, and according to the feature vector q′ at the i-th time position and the j-th space position in the image feature Q′ ij , respectively obtain P′ feature vectors at the i-th time position and N′ feature vectors at the j-th spatial position in the image features K′ and V′, and concatenate them to obtain a two-dimensional vector k″ ij and v″ ij , and based on the two-dimensional vector k″ of all space-time positions ij and v″ ij, get the feature vectors K″ and V″; reorganize the first and second dimensions of Q′, K″, V″ into part of the batch size; where the feature dimension of Q′ after reorganization is BN′P′×1×C, and the feature dimension of K″ and V″ is BN′P′×(N′+P′)×C, where B represents the batch size;

[0021] Based on the similarity between Q′ and K″ vectors, batch calculate the self-attention score A, and A = Softmax(Q′K″);

[0022] Based on the self-attention score A, V″ is weightedly fused to obtain a fused image feature O, O=AV″;

[0023] The image feature O is linearly mapped to another feature space and reorganized to obtain a fourth image feature; wherein the size of the fourth image feature O is consistent with that of the third image feature.

[0024] Optionally, the method further includes:

[0025] Processing the rPPG signal based on a preset blood pressure detection network to obtain a blood pressure value;

[0026] The rPPG signal is processed based on a preset blood pressure detection network to obtain a blood pressure value; wherein the blood pressure detection network is cascaded with the rPPG signal extraction network; the rPPG signal extraction network uses a normalized frame difference sequence as input data to output the rPPG signal;

[0027] The loss function of the cascade network composed of the rPPG signal extraction network and the blood pressure detection network during the training process includes two parts: the loss function loss_rppg of rPPG signal extraction and the loss function loss_bp of blood pressure prediction. The loss function of rPPG signal extraction is the weighted sum of the time domain loss loss1 and the frequency domain loss loss2 of the rPPG signal, that is, loss_rppg = α·loss1+β·loss2; wherein loss1 is the reverse Pearson correlation coefficient calculated between the rPPG signal and the corresponding PPG signal, and α represents the first preset coefficient; loss2 is the cross entropy loss value calculated between the power spectral density of the rPPG signal and the heart rate label, and β represents the second preset coefficient; the loss function loss_bp of blood pressure prediction is the mean square error between the predicted blood pressure value and the true value.

[0028] Optionally, the detecting of the physiological parameters of the user based on the rPPG signal includes:

[0029] Performing peak detection on the rPPG signal to obtain the interval time length between adjacent peaks;

[0030] Processing the interval time length based on a preset formula to obtain an rPPG time domain signal;

[0031] The rPPG time domain signal is converted into an rPPG frequency domain signal based on Fourier transform, and the power LF in the low frequency range, the power HF in the high frequency range, and the ratio LF / HF of the low frequency to high frequency power in the rPPG frequency domain signal are calculated;

[0032] Performing nonlinear analysis on the rPPG frequency domain signal based on the Poincare scatter plot method, and drawing an elliptical graph of the obtained continuous heartbeat intervals in a two-dimensional plane;

[0033] The semi-major axis SD1 and the semi-minor axis SD2 of the elliptical figure are used as nonlinear indicators of the rPPG frequency domain signal.

[0034] Optionally, before acquiring a sequence of facial region-of-interest images as a first image sequence, the method further includes:

[0035] Collect visible light facial video and thermal infrared video of the user in real time;

[0036] Performing frame processing on the visible light video and the thermal infrared video, using the visible light video after the frame processing as a visible light image, and using the thermal infrared video after the frame processing as a thermal infrared image;

[0037] Acquire key facial feature points in the visible light image based on the openface facial detection algorithm;

[0038] Determine a facial region of interest and a nose region of interest in the visible light image based on the facial feature points;

[0039] The facial region of interest in the visible light image is used as a facial region of interest image.

[0040] Optionally, after determining the facial region of interest and the nose region of interest in the visible light image based on the facial feature points, the method further includes:

[0041] Completing the registration of the visible light image and the thermal infrared image by using affine transformation;

[0042] Based on mapping the visible nose region of interest in the visible light image to the thermal infrared image, an image of the visible nose region of interest in the thermal infrared image is obtained as the nose image of interest;

[0043] Based on the time sequence of the thermal infrared video, a sequence of nose interest images is acquired as a sequence of nose interest images;

[0044] Based on the nose image sequence of interest, the breathing rate is acquired.

[0045] Optionally, after performing frame processing on the visible light video and the thermal infrared video, and using the frame-processed visible light video as a visible light image, and using the frame-processed thermal infrared video as a thermal infrared image, the method includes:

[0046] Based on the openface facial detection algorithm, a facial feature point motion trajectory sequence, a facial motion unit intensity sequence, an eye gaze angle sequence, a head posture coordinate sequence, and each frame of facial region image in the visible light image are obtained;

[0047] Analyze the facial feature point motion trajectory sequence, facial motion unit intensity sequence, eye gaze angle sequence, and head posture coordinate sequence based on the action unit defined by FACS to obtain the user's emotional state;

[0048] Based on the preset deep convolutional neural network DenseNet, each frame of the facial area image is classified and processed to obtain the probability distribution of the emotional state.

[0049] Optionally, after acquiring the facial feature point motion trajectory sequence, facial motion unit intensity sequence, eye gaze angle sequence, head posture coordinate sequence and each frame of facial region image in the visible light image based on the OpenFace facial detection algorithm, the method further comprises:

[0050] The facial feature point motion trajectory sequence, the facial motion unit intensity sequence, the eye gaze angle sequence, the head posture coordinate sequence and the facial spatiotemporal features are serially fused to obtain a processed feature vector; wherein the facial spatiotemporal features are obtained by performing three-dimensional convolution on the facial region image based on a 3DCNN network;

[0051] Performing dimensionality reduction processing on the processed feature vector based on a principal component analysis algorithm to obtain a feature vector after dimensionality reduction processing;

[0052] The feature vector after the dimension reduction process is input into a pre-trained SVM model to obtain the user anxiety probability value output by the SVM model.

[0053] Optionally, after acquiring the facial feature point motion trajectory sequence, facial motion unit intensity sequence, eye gaze angle sequence, head posture coordinate sequence and each frame of facial region image in the visible light image based on the OpenFace facial detection algorithm, the method further comprises:

[0054] Based on the convolutional neural network, the spatial features of the facial motion unit intensity sequence, eye gaze angle sequence, head posture coordinate sequence and physiological data are extracted to obtain the spatial feature sequence S A , S E , S H , S P ;

[0055] The spatial feature sequence S is extracted based on the long short-term memory neural network A , S E , S H , S P The time characteristics are respectively obtained as multi-dimensional spatiotemporal features ST A , ST E , ST H , ST P ;

[0056] The multidimensional spatiotemporal feature ST A , ST E , ST H , ST P Perform multi-channel feature fusion to obtain the feature Y after multi-channel feature fusion;

[0057] The feature Y is subjected to feature dimensionality reduction, and the feature Y after the feature dimensionality reduction is classified based on the multi-classification activation function softmax to obtain classification evaluation data of a preset depression level.

[0058] In a second aspect of the present invention, a non-contact physiological and psychological multi-parameter detection system is provided, the system comprising:

[0059] A first acquisition module is used to acquire a sequence of facial interest region images as a first image sequence;

[0060] A first calculation module, used for calculating a normalized frame difference between two adjacent frames in the first image sequence to obtain a normalized frame difference sequence;

[0061] A first extraction module, used to extract features of the normalized frame difference sequence based on a preset two-dimensional convolutional layer to obtain a first image feature;

[0062] A first exchange module is used to exchange the order of the channel dimension and the time dimension in the first image feature, and flatten the width and height dimensions in the spatial dimension into one dimension to obtain a second image feature;

[0063] A first fusion module, used for fusing the local spatiotemporal information of the second image feature based on a preset two-dimensional convolutional layer to obtain a third image feature;

[0064] A second fusion module, used for fusing the global spatiotemporal information of the third image feature based on a preset global spatiotemporal self-attention layer to obtain a fourth image feature;

[0065] The third fusion module is used to fuse the spatial dimension features of the fourth image feature, and upsample the time dimension size of the fused fourth image feature so that the time dimension size after the upsampling process is the same as the length of the normalized frame difference sequence, and map the channel dimension of the fourth image feature after the upsampling process to 1 to obtain a mapped signal as an rPPG signal; the first detection module is used to detect the user's physiological parameters based on the rPPG signal; wherein the physiological parameters include blood pressure, respiratory rate and heart rate.

[0066] (III) Beneficial effects

[0067] The present invention provides a non-contact physiological and psychological multi-parameter detection method and system. Compared with the prior art, it has the following beneficial effects:

[0068] The present invention provides a non-contact physiological and psychological multi-parameter detection method, which includes: obtaining a sequence of facial interest region images as a first image sequence; calculating the normalized frame difference between two adjacent frames in the first image sequence to obtain a normalized frame difference sequence; extracting features of the normalized frame difference sequence based on a preset two-dimensional convolution layer to obtain a first image feature; exchanging the order of channel dimension and time dimension in the first image feature, and flattening the width and height dimensions in the space dimension into one dimension to obtain a second image feature; fusing the local spatiotemporal information of the second image feature based on a preset two-dimensional convolution layer to obtain a third image feature. image features; based on a preset global spatiotemporal self-attention layer, the global spatiotemporal information of the third image features is fused to obtain a fourth image feature; the spatial dimension features of the fourth image features are fused, and the time dimension size of the fused fourth image features is upsampled so that the time dimension size after the upsampling is the same as the length of the normalized frame difference sequence, and the channel dimension of the upsampled fourth image feature is mapped to 1, and the mapped signal is used as an rPPG signal; based on the rPPG signal, the user's physiological parameters are detected; wherein the physiological parameters include blood pressure, respiratory rate and heart rate.

[0069] Based on the above processing, the rPPG extraction network with global spatiotemporal attention mechanism can make full use of the correlation between rPPG signals at different times, complement and enhance each other, effectively improve the quality of rPPG signal extraction, and thus effectively improve the accuracy of physiological or psychological parameter detection based on rPPG signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 A flowchart of a non-contact physiological and psychological multi-parameter detection method provided by an embodiment of the present invention;

[0072] Figure 2 An architecture diagram of an rPPG signal extraction network provided by an embodiment of the present invention;

[0073] Figure 3 An architecture diagram of a blood pressure detection network provided by an embodiment of the present invention;

[0074] Figure 4 A flowchart of ST-attention depression detection provided by an embodiment of the present invention;

[0075] Figure 5 A hardware architecture diagram of a non-contact multi-parameter detection system provided by an embodiment of the present invention;

[0076] Figure 6 A software architecture diagram of a non-contact multi-parameter detection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] The embodiments of the present application provide a non-contact physiological and psychological multi-parameter detection method and system, which solves the problem of low detection result accuracy in existing detection methods based on rPPG signals. Through the rPPG extraction network with a global spatiotemporal attention mechanism, it is possible to fully utilize the correlation between rPPG signals at different times, complement and enhance each other, effectively improve the extraction quality of rPPG signals, and further effectively improve the accuracy of physiological or psychological parameter detection based on rPPG signals.

[0079] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0080] Most of the existing non-contact physiological signal (i.e., rPPG signal in the present invention) extraction methods are based on convolutional neural networks, which have limited receptive fields and are unable to model long-range spatiotemporal correlations. It is difficult to use the similar characteristics of physiological signals at different times and in different facial regions to complement and enhance each other. At the same time, the existing technology often takes the accuracy of heart rate detection as the goal and measurement standard, ignoring the relationship between physiological signals and blood pressure, and cannot guarantee the accuracy of the extracted physiological signals for blood pressure detection.

[0081] The present invention provides a non-contact physiological and psychological multi-parameter detection method, which includes: obtaining a sequence of facial interest region images as a first image sequence; calculating the normalized frame difference between two adjacent frames in the first image sequence to obtain a normalized frame difference sequence; extracting features of the normalized frame difference sequence based on a preset two-dimensional convolution layer to obtain a first image feature; exchanging the order of channel dimension and time dimension in the first image feature, and flattening the width and height dimensions in the space dimension into one dimension to obtain a second image feature; fusing the local spatiotemporal information of the second image feature based on a preset two-dimensional convolution layer to obtain a second image feature. three image features; based on a preset global spatiotemporal self-attention layer, the global spatiotemporal information of the third image feature is fused to obtain a fourth image feature; the spatial dimension features of the fourth image feature are fused, and the time dimension size of the fused fourth image feature is upsampled so that the time dimension size after the upsampling is the same as the length of the normalized frame difference sequence, and the channel dimension of the upsampled fourth image feature is mapped to 1 to obtain a mapped signal as an rPPG signal; based on the rPPG signal, the user's physiological parameters are detected; wherein the physiological parameters include blood pressure, respiratory rate and heart rate.

[0082] Based on the above processing, the rPPG extraction network with global spatiotemporal attention mechanism can make full use of the correlation between rPPG signals at different times, complement and enhance each other, effectively improve the quality of rPPG signal extraction, and thus effectively improve the accuracy of physiological or psychological parameter detection based on rPPG signals.

[0083] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0084] See also Figure 1 , Figure 1 A flow chart of a non-contact physiological and psychological multi-parameter detection method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0085] S1. Acquire a sequence of facial region-of-interest images as a first image sequence.

[0086] S2. Calculate the normalized frame difference between two adjacent frames in the first image sequence to obtain a normalized frame difference sequence.

[0087] S3. Extract features of the normalized frame difference sequence based on a preset two-dimensional convolutional layer to obtain a first image feature.

[0088] S4. Exchange the order of the channel dimension and the time dimension in the first image feature, and flatten the width and height dimensions in the spatial dimension into one dimension to obtain a second image feature.

[0089] S5. Based on a preset two-dimensional convolutional layer, the local spatiotemporal information of the second image feature is integrated to obtain a third image feature.

[0090] S6. Based on a preset global spatiotemporal self-attention layer, the global spatiotemporal information of the third image feature is integrated to obtain a fourth image feature.

[0091] S7. Fuse the spatial dimension features of the fourth image feature, and upsample the time dimension size of the fused fourth image feature so that the time dimension size after the upsampling process is the same as the length of the normalized frame difference sequence, and map the channel dimension of the fourth image feature after the upsampling process to 1 to obtain a mapped signal as an rPPG signal.

[0092] S8. Detecting physiological parameters of the user based on the rPPG signal; wherein the physiological parameters include blood pressure, respiratory rate and heart rate.

[0093] Based on the above processing, the rPPG extraction network with global spatiotemporal attention mechanism can make full use of the correlation between rPPG signals at different times, complement and enhance each other, effectively improve the quality of rPPG signal extraction, and thus effectively improve the accuracy of physiological or psychological parameter detection based on rPPG signals.

[0094] In addition, the above-mentioned rPPG extraction network based on the global spatiotemporal attention mechanism captures the global spatiotemporal correlation of the constructed frame difference spatiotemporal graph, so that the rPPG signals of different spatiotemporal regions complement and enhance each other, and reduces the amount of computation required for attention by limiting the calculation scope to different times in the same space and different spaces at the same time.

[0095] In some embodiments, before step S1, the method further comprises the following steps:

[0096] S10, collect the visible light facial video and thermal infrared video of the user in real time.

[0097] S11. Perform frame processing on the visible light video and the thermal infrared video, and use the visible light video after the frame processing as a visible light image, and use the thermal infrared video after the frame processing as a thermal infrared image.

[0098] S12. Acquire key facial feature points in the visible light image based on an openface facial detection algorithm.

[0099] S13: Determine a facial region of interest and a nose region of interest in the visible light image based on the facial feature points.

[0100] S14: Use the facial region of interest in the visible light image as a facial region of interest image.

[0101] Specifically, for step S10, based on a visible light camera and a thermal infrared camera 0.5-1.0m away from the user, the user's visible light facial video and thermal infrared facial video are collected in real time. The visible light facial video and thermal infrared facial video can be used as input data for the physiological and psychological parameter detection in the present invention.

[0102] In some embodiments, after step S13, the method further comprises the following steps:

[0103] S15. Complete the registration of the visible light image and the thermal infrared image by using affine transformation.

[0104] S16. Based on the mapping of the visible nose region of interest in the visible light image to the thermal infrared image, an image of the visible nose region of interest in the thermal infrared image is obtained as the nose image of interest.

[0105] S17. Based on the time sequence of the thermal infrared video, a sequence of nose interest images is acquired as a sequence of nose interest images.

[0106] S18. Acquire the respiratory rate based on the nose image sequence of interest.

[0107] In some embodiments, the processing of thermal infrared video and visible light video includes: for the real-time collected thermal infrared video and visible light video, the above video is segmented based on a time window with a sliding window of N+1 frames and a sliding step of S frames, and each video is processed according to the time sequence of the video frames. Then, the openface facial detection algorithm is used to detect the key facial feature points of each frame of the visible light image, and the position of the facial region of interest and the nose region of interest is located based on the coordinates of the feature points, and the facial ROI (region of interest) sequence is cropped as the input data for the subsequent extraction of rPPG signals. Affine transformation is used to realize the registration of visible light images and thermal infrared images, and the facial ROI and nose ROI in the visible light image are mapped to the thermal infrared image. The cropped nose ROI sequence can be used as the input data for breathing detection.

[0108] In some embodiments, the user's facial temperature can be detected based on the above-mentioned thermal infrared facial image, specifically: each frame of the thermal infrared facial image is converted into a temperature matrix, and the highest temperature in the facial ROI area of ​​the thermal infrared image is calculated as the facial temperature corresponding to the image frame.

[0109] In some embodiments, the user's breathing rate can be detected based on the nose ROI sequence, specifically: the nose ROI sequence is converted into a grayscale image, the pixel mean of each frame is calculated, and the preliminary breathing signal is extracted. After the initial breathing signal is detrended and standardized, the Butterworth filter is used to filter out the signal components that are not in the normal breathing frequency range [0.15Hz-0.4Hz], and finally the number of peaks is counted to obtain the breathing rate.

[0110] For step S2, the formula for calculating the normalized frame difference between two adjacent frames in the first image sequence is as follows:

[0111]

[0112] Among them, I′ t It is represented as the tth normalized frame difference in the normalized frame difference sequence I; I t represents the t-th frame image in the time window; t represents the order of the normalized frame difference in the normalized frame difference sequence; N represents the number of images in the time window.

[0113] In some embodiments, a pre-built rPPG signal extraction network can be used to extract rPPG signals. The network architecture can be found in Figure 2 , Figure 2 This is an architecture diagram of an rPPG signal extraction network provided by the present invention, and its network calculation process corresponds to the above steps S3-S7, which specifically includes the following contents:

[0114] First, feature extraction is performed on the input data (i.e., the normalized frame difference sequence I of the present invention) based on three 2D convolutional layers with a size of 3 and a step size of 2, and the feature size is reduced to 1 / 8 of the original size to obtain the first image feature Based on the above processing, the computational complexity of subsequent operations is reduced and the speed of network convergence is accelerated.

[0115] in, Denotes the first image feature F 1 A set of; N×C×H / 8×W / 8 represents the dimension of the first image feature F1, N represents the number of frames, C represents the number of channels, H represents the height of the input data, and W represents the width of the original input of the network.

[0116] The first image feature F 1 The two-dimensional spatial features of each channel at each moment in the image are flattened into a one-dimensional vector to reconstruct the frame difference spatiotemporal graph features as the second image feature F 2 ∈ Based on the above processing, the three-dimensional spatiotemporal feature F 1 Converted into a two-dimensional space-time graph feature F 2 , further reducing the amount of computation for subsequent operations.

[0117] Then, the second image feature F is extracted based on three 2D convolution layers with a kernel size of 3 and a stride of 2. 2 Extract higher-level local spatiotemporal information to obtain local feature maps as the third image feature in,

[0118] Based on three global spatiotemporal self-attention layers, the third image feature F 3 The global spatiotemporal information of 4 , which can be understood as: for the third image feature F 3 The feature vectors at each time and space in the , and the feature vectors at the same space, different time and at the same time, different space are used to calculate the global spatiotemporal self-attention, and the weighted fusion is used to obtain the global feature F 4 .

[0119] Specifically, the specific processing process of each layer of global spatiotemporal self-attention is as follows:

[0120] Based on the linear mapping operation W in three pixel dimensions Q , W K , W V , respectively, the third image feature F 3 Mapped to another feature space, we get the query feature Q = W Q ·F 3 , key feature K = W K ·F3 The sum feature V = W V ·F 3 ,

[0121] Reorganize Q, K, and V into four-dimensional features Q′, K′, and V′ respectively. The dimension of the reorganized features is N′×P′×1×C; where N′ represents the number of frames of the third image feature, P′ represents the product of the height and width of the third image feature, and C represents the number of channels;

[0122] Traverse the index i of the time dimension (1≤i≤N′) and the index j of the space dimension (1≤j≤P′), and find the feature vector q′ of the i-th time and j-th space position of Q′ ij Find the P′ feature vectors at the i-th time position and the N′ feature vectors at the j-th spatial position in K′ and V′ and concatenate them into two-dimensional vectors and The two-dimensional vectors of all space-time positions form a new and

[0123] Then, using the query vector q′ ij and (N′+P′) key vectors k″ ij The similarity between them is used to obtain the weighted attention score of the (N′+P′) value vectors, and the new query vector q″ is obtained by weighted fusion. ij , to achieve the fusion of global spatiotemporal features. Specifically, the first and second dimensions of Q′, K″, and V″ are reorganized into a part of the batch size, where the feature dimension of Q′ after reorganization is BN′P′×1×C, and the feature dimensions of K″ and V″ are BN′P′×(N′+P′)×C, where B represents the batch size (i.e., the number of samples in the training set).

[0124] Then, based on the similarity between the Q′ and K″ vectors, the self-attention score A is calculated in batches, where A = Softmax(Q′K″T)∈BN′P′×1×(N′+P′), and V″ is weightedly fused using the self-attention score A to obtain O = AV″∈BN′P′×1×C. Finally, O is linearly mapped to another feature space and reorganized into a size B×C×N′×P′ that is consistent with the input feature (i.e., the normalized frame difference sequence in the present invention).

[0125] In some embodiments, the processing of each global spatiotemporal self-attention layer includes:

[0126] Based on the linear mapping operation W in three pixel dimensions Q , W K , W V , respectively for the third image feature F 3Perform linear mapping to obtain query feature Q, key feature K and value feature V; where Q = W Q ·F 3 , K=W K ·F 3 、V=W V ·F 3 ;

[0127] Q, K, and V are reorganized into four-dimensional features Q′, K′, and V′ respectively. The dimension of the reorganized features is N′×P′×1×C; wherein N′ represents the number of frames of the third image feature, P′ represents the product of the height and width of the third image feature, and C represents the number of channels.

[0128] Traverse the index i of the time dimension and the index j of the space dimension, and according to the feature vector q′ at the i-th time position and the j-th space position in the image feature Q′ ij , respectively obtain P′ feature vectors at the i-th time position and N′ feature vectors at the j-th spatial position in the image features K′ and V′, and concatenate them to obtain a two-dimensional vector k″ ij and v″ ij , and based on the two-dimensional vector k″ of all space-time positions ij and v″ ij , get the feature vectors K″ and V″; reorganize the first and second dimensions of Q′, K″, and V″ into part of the batch size. Among them, the feature dimension of Q′ after reorganization is BN′P′×1×C, and the feature dimension of K″ and V″ is BN′P′×(N′+P′)×C, where B represents the batch size.

[0129] Based on the similarity between Q′ and K″ vectors, batch calculate the self-attention score A m , and A=Softmax(Q′ m K″ T ).

[0130] Based on the self-attention score A, V″ is weightedly fused to obtain the fused image feature O, O=AV″.

[0131] The image feature O is linearly mapped to another feature space and reorganized to obtain a fourth image feature; wherein the fourth image feature has the same size as the third image feature.

[0132] Among them, the training of the above rPPG signal extraction network uses the weighted sum of the time domain loss loss1 and the frequency domain loss loss2 of the rPPG signal as the loss function, that is, loss_rppg = α·loss1+β·loss2. loss_rppg represents the loss function of the rPPG signal extraction network, loss1 is the reverse Pearson correlation coefficient calculated between the rPPG signal and the corresponding PPG signal, α represents the first preset coefficient; loss2 is the cross entropy loss value calculated between the power spectral density of the rPPG signal and the heart rate label, and β represents the second preset coefficient.

[0133] In some embodiments, when the user's blood pressure is detected based on the rPPG signal, step S8 includes the following steps:

[0134] S81. Process the rPPG signal based on a preset blood pressure detection network to obtain a blood pressure value.

[0135] Among them, the blood pressure detection network is cascaded with the rPPG signal extraction network; the rPPG signal extraction network uses the normalized frame difference sequence as input data to output the rPPG signal.

[0136] The loss function of the rPPG signal extraction network during training is the weighted sum of the time domain loss loss1 and the frequency domain loss loss2 of the rPPG signal, that is, loss_rppg = α·loss1+β·loss2.

[0137] Among them, loss_rppg represents the loss function of the rPPG signal extraction network, loss1 is the reverse Pearson correlation coefficient calculated between the rPPG signal and the corresponding PPG signal, α represents the first preset coefficient; loss2 is the cross entropy loss value calculated between the power spectral density of the rPPG signal and the heart rate label, and β represents the second preset coefficient.

[0138] Since the rPPG signal obtained by the rPPG signal extraction network is not optimal when used directly for blood pressure detection, the trained rPPG extraction network is cascaded with a blood pressure detection network that uses the rPPG signal as input, thereby obtaining a cascade network that can output both rPPG prediction results and blood pressure prediction results. Obviously, the training loss of the cascade network is the mean square error loss loss_bp between the blood pressure prediction value and the true value added to the loss_rppg.

[0139] In one implementation, if the blood pressure detection network is cascaded with the rPPG signal extraction network, the loss function of the rPPG signal extraction network during the training process is the sum of loss_rppg and loss_bp; wherein loss_bp represents the mean square error between the blood pressure prediction value output by the cascaded network and the true blood pressure value.

[0140] In the prior art, technicians often regard rPPG signal extraction and blood pressure detection as two independent tasks, and do not combine the rPPG signal extraction network with the blood pressure detection network. Based on the above processing, the present invention cascades the rPPG signal extraction network with the blood pressure detection network, which can then be used as an overall network for detecting blood pressure based on rPPG signals.

[0141] At the same time, the loss function of the above-mentioned overall network is to add the mean square error loss loss_bp between the predicted blood pressure value and the true value on the basis of loss_rppg, and adjust the parameters of the rPPG signal extraction network and the blood pressure detection network based on this loss function, which can effectively improve the accuracy of the overall network for blood pressure detection.

[0142] For step S81, the blood pressure detection network adopts an encoder-decoder structure. Figure 3 , Figure 3 The present invention provides an architecture diagram of a blood pressure detection network, wherein the output layer includes a convolutional layer and a fully connected layer. Specifically, step S8 may include the following steps:

[0143] S801. Perform feature extraction on the first processed signal based on the coding block in the initial network layer, and input the obtained signal features to the coding block in the next network layer and the decoding block in the same network layer respectively; wherein the coding block in each network layer receives the signal features output by the coding block in the previous network layer, and performs feature extraction on the received signal features, and inputs the extracted signal features to the next network layer and the decoding block in the same network layer respectively; perform one-dimensional convolution processing on the signal features output by the coding block in the last network layer, and input the signal features after the one-dimensional convolution processing to the decoding block in the last network layer.

[0144] S802. Concatenate and extract features of the signal features received in the decoding block in the last network layer, and input the extracted signal features to the decoding block in the previous network layer; wherein the decoding block in each network layer receives the signal features output by the encoding block in the same network layer and the signal features output by the decoding block in the next network layer, concatenates and extracts features of the received signal features, and inputs the extracted signal features to the encoding block in the previous network layer.

[0145] S803: perform concatenation and feature extraction processing on the signal features received by the decoding block in the initial network layer, and use the processed signal features as the second processed signal.

[0146] S804: Input the second processed signal to a preset convolutional layer for feature extraction, and input the second processed signal after feature extraction to a preset fully connected layer for mapping processing, and use the mapped signal as the blood pressure value.

[0147] The preset blood pressure detection network includes multiple network layers; each network layer includes an encoding block and a decoding block; each encoding block includes two convolution layers including batch normalization and activation functions, and a maximum pooling layer with a step size of 2; each decoding block includes a two-fold upsampling operation and two convolution layers including batch normalization and activation functions. Specifically, the encoding block performs one-dimensional convolution and downsampling processing on the input signal; the decoding block performs upsampling and one-dimensional convolution processing on the input signal.

[0148] In some embodiments, the process of processing the rPPG signal based on the above-mentioned blood pressure detection network to obtain the blood pressure value can be as follows: first, the input rPPG signal is downsampled layer by layer using M encoding blocks in the blood pressure detection network and deep features are gradually extracted, and then the encoding features are upsampled layer by layer using M decoding blocks and mapped to features related to blood pressure. Each layer of decoding blocks includes a double upsampling operation, two convolutional layers containing batch normalization and activation functions, and the features output by the previous layer of decoding blocks and the features output by the same layer of encoding blocks are input to obtain the features obtained by concatenating the features. After the decoding block, a convolutional layer and a fully connected layer are used to map the predicted value of blood pressure (including systolic and diastolic pressure).

[0149] In some embodiments, when the user's heart rate is detected based on the rPPG signal, the above step S8 may further include the following steps:

[0150] Step a: performing peak detection on the rPPG signal to obtain the interval time length between adjacent peaks.

[0151] Step b: Processing the interval time length based on a preset formula to obtain an rPPG time domain signal.

[0152] Step c: converting the rPPG time domain signal into an rPPG frequency domain signal based on Fourier transform, and calculating the power LF in the low frequency range, the power HF in the high frequency range, and the ratio LF / HF of the low frequency to high frequency power in the rPPG frequency domain signal.

[0153] Step d: performing nonlinear analysis on the rPPG frequency domain signal based on the Poincare scatter plot method, and drawing an elliptical graph of the obtained continuous heartbeat intervals in a two-dimensional plane.

[0154] Step e: using the semi-major axis SD1 and the semi-minor axis SD2 of the elliptical figure as nonlinear indicators of the frequency domain signal.

[0155] For step a, the peak value of the acquired rPPG signal is detected, and the average RR interval (the interval time length between adjacent peaks) is calculated to obtain the corresponding heart rate HR, where HR = 60 / average RR interval.

[0156] For step b, the preset formula includes:

[0157]

[0158]

[0159]

[0160]

[0161] Where P represents the location of the peak in the time domain signal, PP i represents the length of time between two adjacent peaks, L represents the number of all peaks, LL 50 It indicates the number of peak intervals whose time difference between two adjacent peak intervals is greater than 50ms. TotalLL indicates the number of all peak intervals.

[0162] Based on the above formula, we obtained the mean (MEAN), overall standard deviation (SDNN), mean square standard deviation (RMSSD), and percentage of adjacent interval differences > 50 ms (P 50 ) and other rPPG time domain analysis indicators.

[0163] For step d, the Poincare scatter plot method is used for nonlinear analysis. For a continuous heartbeat interval, the i-th heartbeat interval is used as the horizontal coordinate and the i+1-th heartbeat interval is used as the vertical coordinate. An ellipse-like figure can be obtained in the two-dimensional plane. The semi-major axis SD1 and the semi-minor axis SD2 of the ellipse are the nonlinear indicators of heart rate variability.

[0164] In some embodiments, after step S12, the method further comprises the following steps:

[0165] Step 1: Based on the OpenFace facial detection algorithm, a facial feature point motion trajectory sequence, a facial motion unit intensity sequence, an eye gaze angle sequence, a head posture coordinate sequence, and each frame of facial region image in the visible light image are obtained;

[0166] Step 2: Analyze the facial feature point motion trajectory sequence, facial motion unit intensity sequence, eye gaze angle sequence, and head posture coordinate sequence based on the action unit defined by FACS to obtain the user's emotional state;

[0167] Step 3: Classify each frame of facial area image based on a preset deep convolutional neural network DenseNet to obtain a probability distribution of emotional state.

[0168] For step one, the facial feature point motion trajectory sequence, facial motion unit intensity sequence, eye gaze angle sequence, and head pose coordinate sequence in the visible light image are extracted based on the OpenFace facial detection algorithm.

[0169] Specifically, the facial feature point motion trajectory sequence includes:

[0170] Feature point x-axis coordinate motion trajectory sequence:

[0171]

[0172] Feature point y-axis coordinate motion trajectory sequence:

[0173]

[0174] The x-axis coordinate vector of the feature point in the tth frame:

[0175]

[0176] The y-axis coordinate vector of the feature point in the tth frame:

[0177]

[0178] The facial motor unit intensity sequence includes:

[0179] AU=(AU 1 , AU 2 ,…,AU t ,…,AU T )

[0180] The facial motion unit intensity vector in the tth frame:

[0181]

[0182] The eye gaze angle sequence includes:

[0183] EG=(EG 1 , E.G. 2 , …, EG t , …, EG T )

[0184]

[0185] Among them, EG t represents the eye gaze angle vector in the tth frame, where the components are the x, y, and z components of the left eye gaze direction vector (normalized), the x, y, and z components of the right eye gaze direction vector (normalized), and the average value of the radians of the two eye gaze directions. It indicates the average value of the arc when the eyes look left and right and deviate from the front. It represents the average value of the arc that the eyes deviate from the front when looking up and down (the above coordinates are all in the world coordinate system with the camera as the origin).

[0186] Head pose coordinate sequence:

[0187] HP=(HP 1 , HP 2 , ..., HP t , ..., HP T )

[0188]

[0189] Among them, HP t Represents the eye gaze angle vector in the tth frame. The components are the position (x, y, z) coordinates of the head relative to the camera, and the rotation angle (Yaw, Pitch, Roll) coordinates of the head in three dimensions in space (the above coordinates are all in the world coordinate system with the camera as the origin).

[0190] For step 2, based on the method of combined motion unit feature analysis, on the basis of the facial motion unit intensity sequence obtained through data processing, refer to the detailed explanation of the action unit defined by FACS to analyze the current emotional state of the subject when different facial motion features appear. For example, when AU1 motion unit appears: eyebrows rise, forehead wrinkles appear, the overall shape of eyebrows changes, and the distance between the inner corner of the eyebrow and the eye increases, the emotional state of the subject should be surprise or fear.

[0191] For step three, the facial area image is processed based on the image-level facial feature analysis method, and the deep convolutional neural network DenseNet is used to classify the entire image to obtain the image-level emotional state probability distribution combination.

[0192] The calculation formula of the DenseNet network layer is as follows:

[0193] x l =W l ([x 0 , x 1 , …, x l-1 ])

[0194] Among them, x l is the output of the lth layer, W l is the convolution operation of the lth layer, [x 0 , x 1 , ..., x l-1 ] represents the result of connecting the feature maps from layer 0 to layer l-1. The above connection method makes the transfer of features and gradients more effective, and the training efficiency and accuracy of the network are improved.

[0195] In some embodiments, after step 1, the method further comprises the following steps:

[0196] Step 4: Serially fuse the facial feature point motion trajectory sequence, the facial motion unit intensity sequence, the eye gaze angle sequence, the head posture coordinate sequence and the facial spatiotemporal features to obtain a processed feature vector; wherein the facial spatiotemporal features are obtained by performing 3D convolution on the facial area image based on the 3DCNN network.

[0197] Step 5: Perform dimension reduction processing on the feature vector based on the principal component analysis algorithm to obtain the feature vector after dimension reduction processing.

[0198] Step 6: Input the feature vector after the dimension reduction process into a pre-trained SVM model to obtain the user anxiety probability value output by the SVM model.

[0199] Specifically, for the facial motion trajectory features, eye gaze features, head posture features, non-contact physiological features, and facial spatiotemporal features, feature vectors are constructed based on serial fusion, and dimension reduction is performed using the principal component analysis algorithm. Finally, they are input into the pre-trained SVM model to calculate the user's real-time anxiety.

[0200] In some embodiments, after step 1, the method further comprises the following steps:

[0201] Step 7: Extract the spatial features of the facial motion unit intensity sequence, eye gaze angle sequence, head posture coordinate sequence and physiological data based on the convolutional neural network, and obtain the corresponding spatial feature sequences S A , S E , S H , S P .

[0202] Step 8: Extract the spatial feature sequence S based on the long short-term memory neural network A , S E , S H , S P The time feature of the multidimensional spatiotemporal feature ST A , ST E , STH , ST P Among them, ST A , ST E , ST H , ST P They represent the temporal and spatial characteristics of facial motion unit intensity sequence, eye gaze angle sequence, head pose coordinate sequence and physiological data respectively.

[0203] Step 9: The multidimensional spatiotemporal feature ST A , ST E , ST H , ST P Perform multi-channel feature fusion to obtain the feature Y after multi-channel feature fusion;

[0204] Step 10: Perform feature dimensionality reduction processing on the feature Y, and classify the feature-reduced Y based on the multi-classification activation function softmax to obtain classification evaluation data of a preset depression level.

[0205] For step eight, multidimensional spatiotemporal features ST A , ST E , ST H , ST P It can be expressed as

[0206]

[0207] Where N represents the dimensions of the above four feature vectors; Indicates that at time 1st i The characteristic value of Indicates that at time N, st i The characteristic value of .

[0208] Steps nine and ten can be understood as: capturing multi-dimensional spatiotemporal features ST based on multi-channel attention modules A , ST E , ST H , ST P The relationship between , assigns corresponding weights to each channel in the fusion layer to perform multi-channel feature fusion.

[0209] The specific process includes: first, based on the spatiotemporal feature concatenation method, the four head behavior spatiotemporal feature vectors are combined into an overall head feature F with a dimension of 4×K, and each row of F is encoded as a global feature through a one-dimensional adaptive average pooling function, and the global description feature G with a dimension of 1×4 is output. Then, the sigmoid function is applied to obtain the attention weight vector W=(w A , w E , w P , w P). Finally, the weights of each channel are combined with the corresponding spatiotemporal feature vector ST A ,ST E ,ST H ,ST P Multiply and concatenate to output the feature descriptor Y based on the attention weight, as shown below.

[0210] Y i =w i *ST i (i∈{A,E,H,P})

[0211] Y=(Y A , Y E , Y H , Y P )

[0212] Among them, Y A , Y E , Y H , Y P They represent the feature Y corresponding to the facial motion unit intensity sequence, eye gaze angle sequence, head posture coordinate sequence and physiological data fusion; w i Represents feature weight.

[0213] Finally, the weighted fusion of multi-dimensional head behavior spatiotemporal features was processed for feature dimensionality reduction, and the multi-classification activation function softmax was used to grade depression into five types: no depression, mild depression, moderate depression, moderate to severe depression, and severe depression. Figure 4 , Figure 4 A flowchart of a ST-attention depression detection method provided in an embodiment of the present invention.

[0214] In some embodiments, a non-contact multi-parameter detection system can be constructed based on the non-contact physiological and psychological multi-parameter detection method provided by the present invention. The system is divided into two parts: hardware and software. Figure 5 , 6 , Figure 5 A hardware architecture diagram of a non-contact multi-parameter detection system provided by an embodiment of the present invention, Figure 6 A software architecture diagram of a non-contact multi-parameter detection system provided in an embodiment of the present invention.

[0215] like Figure 5 , 6As shown in the figure, the hardware structure is mainly composed of artificial intelligence motherboard, display screen, buttons, microphone, storage hard disk, wireless module, power supply and visual sensor (including visible light sensor and thermal infrared sensor). The software part is mainly composed of three main layers: hardware driver layer, middle layer and business layer, and the data transmission between each layer is carried out through the EventBus data bus.

[0216] Among them, the hardware driver layer is composed of display driver, sensor driver, network driver, interface driver and basic driver of Android system. The hardware driver layer mainly provides communication between the host system and various hardware, and outputs the basic data stream required by the system; the middle layer is composed of face recognition, computing model, artificial intelligence, file management, database, and image processing modules. This layer is mainly used to process the data generated by the hardware driver layer, generate data that meets the input of each model, and provide a data interface to the business layer. The business layer is composed of user management, experiment management, human-computer interaction, data charts, physiological detection, and psychological detection modules, which are used to process the output results of each model and display the results through a graphical interface.

[0217] It is worth noting that the logical relationship between the various software parts is as follows: the input of the face recognition module is the video data transmitted by the visual sensor, and the output is the video data after the face recognition mark; the input of the image processing module is the video data processed by the face recognition module, and the output is the video data after cropping, alignment and other processing. The model calculation module consists of a physiological parameter detection model and a psychological parameter detection model. Its model input is the video data processed by the image processing module, and the output is the detection value of each physiological and psychological parameter. The file management module is mainly responsible for the management and call of various system data. The database module input is the output data of each module in this layer, mainly including the original video data, the video data processed by the face recognition module and the image processing module, and the physiological and psychological parameters obtained by the calculation model module; the output is mainly the data information called by the file management system.

[0218] The business layer consists of user management, experiment management, human-computer interaction, data charts, physiological detection, and psychological detection modules. The user management module and the experiment management module are mainly used to switch users and experiments to meet the system's needs for different users and experimental scenarios. The human-computer interaction module is mainly composed of the user interface, etc. The humanized user interface and operation mode make the system easy to use. The physiological detection module can realize the effective detection of multiple physiological parameters including heart rate, blood pressure, respiratory rate, and heart rate variability. The input is the detection instruction clicked by the user. The physiological detection module will call the corresponding model calculation module in the middle layer, output the corresponding physiological parameter detection value, and display it to the user through the user interface. The psychological detection module can realize the effective detection of three psychological parameters: expression, anxiety, and depression. The input is the detection instruction clicked by the user. The psychological detection module will call the corresponding model calculation module in the middle layer, output the corresponding psychological parameter detection value, and display it to the user through the user interface. The data chart module inputs the detection values ​​of multiple physiological and psychological parameters over a period of time, and outputs the change curve of physiological and psychological parameters over a period of time. The changes in various physiological and psychological parameters of the user over a period of time can be clearly obtained.

[0219] Based on the above processing, the present invention constructs a non-contact multi-parameter detection system, which includes hardware components such as an artificial intelligence processing module, a visual sensor, a display module, a storage module, and a three-layer software architecture of a hardware driver layer, an intermediate layer, and a business layer, thereby realizing non-contact physiological and psychological multi-parameter detection.

[0220] The technical solution provided by the present invention can be widely used in scenarios such as patient services, family health management, aerospace and navigation, and can provide users with health detection services with high comfort, high integration and high robustness.

[0221] The present invention also provides a non-contact physiological and psychological multi-parameter detection system, which comprises:

[0222] A first acquisition module is used to acquire a sequence of facial interest region images as a first image sequence;

[0223] A first calculation module, used for calculating a normalized frame difference between two adjacent frames in the first image sequence to obtain a normalized frame difference sequence;

[0224] A first extraction module, used to extract features of the normalized frame difference sequence based on a preset two-dimensional convolutional layer to obtain a first image feature;

[0225] A first exchange module is used to exchange the order of the channel dimension and the time dimension in the first image feature, and flatten the width and height dimensions in the spatial dimension into one dimension to obtain a second image feature;

[0226] A first fusion module, used for fusing the local spatiotemporal information of the second image feature based on a preset two-dimensional convolutional layer to obtain a third image feature;

[0227] A second fusion module, used for fusing the global spatiotemporal information of the third image feature based on a preset global spatiotemporal self-attention layer to obtain a fourth image feature;

[0228] The third fusion module is used to fuse the spatial dimension features of the fourth image features, and upsample the time dimension size of the fused fourth image features so that the time dimension size after the upsampling process is the same as the length of the normalized frame difference sequence, and map the channel dimension of the fourth image features after the upsampling process to 1 to obtain a mapped signal as an rPPG signal.

[0229] The first detection module is used to detect the user's physiological parameters based on the rPPG signal; wherein the physiological parameters include blood pressure, respiratory rate and heart rate.

[0230] It can be understood that the non-contact physiological and psychological multi-parameter detection system provided by the embodiment of the present invention corresponds to the above-mentioned non-contact physiological and psychological multi-parameter detection method. The explanations, examples, beneficial effects and other parts of the relevant contents can refer to the corresponding contents in the non-contact physiological and psychological multi-parameter detection method, and will not be repeated here.

[0231] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0232] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0233] In summary, compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0234] 1. The non-contact physiological and psychological multi-parameter detection method provided by the present invention can detect physiological parameters including heart rate, blood pressure, respiratory rate, facial temperature, heart rate variability, and psychological parameters such as emotions, anxiety, depression, etc. In terms of physiological parameter detection, the rPPG extraction network based on the global spatiotemporal attention mechanism can make full use of the correlation between rPPG signals at different times and in different facial areas, complement and enhance each other, and improve the effect of rPPG signal extraction. The joint training of the rPPG extraction network and the blood pressure detection network improves the accuracy of the extracted rPPG signal for blood pressure detection. In terms of psychological parameter detection, facial behaviors and physiological features are shared between multiple parameters, giving full play to the role of shared features and saving the time of data processing for each parameter.

[0235] 2. In the non-contact physiological and psychological multi-parameter detection method, an rPPG extraction network based on the global spatiotemporal attention mechanism is proposed to capture the global spatiotemporal correlation of the constructed frame difference spatiotemporal graph, so that the rPPG signals of different spatiotemporal areas can complement and enhance each other. By limiting the calculation scope to different times in the same space and different spaces at the same time, the amount of calculation required for attention is reduced.

[0236] 3. A blood pressure detection network is cascaded after the rPPG extraction network, and the loss of blood pressure prediction is taken as one of the goals. The two networks are jointly trained to optimize the effect of the rPPG signal extracted by the rPPG extraction network for blood pressure detection tasks.

[0237] 4. Use shared behavioral and physiological information in the detection methods of different psychological parameters, eliminating the need to collect and process data separately for each psychological parameter, thereby improving the detection speed and real-time performance of each psychological parameter.

[0238] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0239] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact physiological and psychological multi-parameter detection method, characterized in that: The method comprises: Acquire a sequence of facial interest region images as a first image sequence; Calculating a normalized frame difference between two adjacent frames in the first image sequence to obtain a normalized frame difference sequence; Extracting features of the normalized frame difference sequence based on a preset two-dimensional convolutional layer to obtain a first image feature; The order of the channel dimension and the time dimension in the first image feature is exchanged, and the width and height dimensions in the spatial dimension are flattened into one dimension to obtain a second image feature; Based on a preset two-dimensional convolutional layer, the local spatiotemporal information of the second image feature is integrated to obtain a third image feature; Based on a preset global spatiotemporal self-attention layer, the global spatiotemporal information of the third image feature is integrated and feature extraction is performed to obtain a fourth image feature; The spatial dimension features of the fourth image feature are fused, and the time dimension size of the fused fourth image feature is upsampled so that the time dimension size after the upsampling process is the same as the length of the normalized frame difference sequence, and the channel dimension of the fourth image feature after the upsampling process is mapped to 1 to obtain a mapped signal as an rPPG signal; Based on the rPPG signal, the user's physiological parameters are detected; wherein the physiological parameters include blood pressure, respiratory rate and heart rate.

2. The method according to claim 1, characterized in that The preset global spatiotemporal self-attention layer includes three identical global spatiotemporal self-attention layers, wherein the processing process of each global spatiotemporal self-attention layer includes: Based on the linear mapping operation W in three pixel dimensions Q , W K , W V , respectively perform linear mapping processing on the third image feature F3 to obtain the query feature Q, key feature K and value feature V; wherein Q = W Q F3, K = W K F3, V = W V F3; Q, K, and V are reorganized into four-dimensional features Q′, K′, and V′ respectively. The dimension of the reorganized features is N′×P′×1×C; wherein N′ represents the number of frames of the third image feature, P′ represents the product of the height and width of the third image feature, and C represents the number of channels; Traverse the index i of the time dimension and the index j of the space dimension, and according to the feature vector q′ at the i-th time position and the j-th space position in the image feature Q′ ij , respectively obtain P′ feature vectors at the i-th time position and N′ feature vectors at the j-th spatial position in the image features K′ and V′, and concatenate them to obtain a two-dimensional vector k″ ij and v″ ij , and based on the two-dimensional vector k″ of all space-time positions ij and v″ ij , get the feature vectors K″ and V″; reorganize the first and second dimensions of Q′, K″, V″ into part of the batch size; where the feature dimension of Q′ after reorganization is BN′P′×1×C, and the feature dimension of K″ and V″ is BN′P′×(N′+P′)×C, where B represents the batch size; Based on the similarity between Q′ and K″ vectors, batch calculate the self-attention score A, and A = Softmax(Q′K″); Based on the self-attention score A, V″ is weightedly fused to obtain a fused image feature O, O=AV″; The image feature O is linearly mapped to another feature space and reorganized to obtain a fourth image feature; wherein the fourth image feature has the same size as the third image feature.

3. The method according to claim 1, characterized in that The detecting of the physiological parameters of the user based on the rPPG signal includes: The rPPG signal is processed based on a preset blood pressure detection network to obtain a blood pressure value; wherein the blood pressure detection network is cascaded with the rPPG signal extraction network; the rPPG signal extraction network uses a normalized frame difference sequence as input data to output the rPPG signal; The loss function of the cascade network composed of the rPPG signal extraction network and the blood pressure detection network during the training process includes two parts: the loss function loss_rppg of rPPG signal extraction and the loss function loss_bp of blood pressure prediction. The loss function of rPPG signal extraction is the weighted sum of the time domain loss loss1 and the frequency domain loss loss2 of the rPPG signal, that is, loss_rppg = α·loss1+β·loss2; wherein loss1 is the reverse Pearson correlation coefficient calculated between the rPPG signal and the corresponding PPG signal, and α represents the first preset coefficient; loss2 is the cross entropy loss value calculated between the power spectral density of the rPPG signal and the heart rate label, and β represents the second preset coefficient; the loss function loss_ of blood pressure prediction is the mean square error between the predicted blood pressure value and the true value.

4. The method according to claim 1, characterized in that The detecting of the physiological parameters of the user based on the rPPG signal includes: Performing peak detection on the rPPG signal to obtain the interval time length between adjacent peaks; Processing the interval time length based on a preset formula to obtain an rPPG time domain signal; The rPPG time domain signal is converted into an rPPG frequency domain signal based on Fourier transform, and the power LF in the low frequency range, the power HF in the high frequency range, and the ratio LF / HF of the low frequency to high frequency power in the rPPG frequency domain signal are calculated; Performing nonlinear analysis on the rPPG frequency domain signal based on the Poincare scatter plot method, and drawing an elliptical graph of the obtained continuous heartbeat intervals in a two-dimensional plane; The semi-major axis SD1 and the semi-minor axis SD2 of the elliptical figure are used as nonlinear indicators of the rPPG frequency domain signal.

5. The method according to claim 1, characterized in that Before acquiring a sequence of facial region-of-interest images as a first image sequence, the method further includes: Collect visible light facial video and thermal infrared video of the user in real time; Performing frame processing on the visible light video and the thermal infrared video, using the visible light video after the frame processing as a visible light image, and using the thermal infrared video after the frame processing as a thermal infrared image; Acquire key facial feature points in the visible light image based on the openface facial detection algorithm; Determine a facial region of interest and a nose region of interest in the visible light image based on the facial feature points; The facial region of interest in the visible light image is used as a facial region of interest image.

6. The method according to claim 5, characterized in that After determining the facial region of interest and the nose region of interest in the visible light image based on the facial feature points, the method further includes: Completing the registration of the visible light image and the thermal infrared image by using affine transformation; Based on mapping the visible nose region of interest in the visible light image to the thermal infrared image, an image of the visible nose region of interest in the thermal infrared image is obtained as the nose image of interest; Based on the time sequence of the thermal infrared video, a sequence of nose interest images is acquired as a sequence of nose interest images; Based on the nose image sequence of interest, a respiratory rate is obtained.

7. The method according to claim 5, characterized in that After the visible light video and the thermal infrared video are subjected to frame processing, and the visible light video subjected to frame processing is used as a visible light image, and the thermal infrared video subjected to frame processing is used as a thermal infrared image, the method includes: Based on the openface facial detection algorithm, a facial feature point motion trajectory sequence, a facial motion unit intensity sequence, an eye gaze angle sequence, a head posture coordinate sequence, and each frame of facial region image in the visible light image are obtained; Analyze the facial feature point motion trajectory sequence, facial motion unit intensity sequence, eye gaze angle sequence, and head posture coordinate sequence based on the action unit defined by FACS to obtain the user's emotional state; Based on the preset deep convolutional neural network DenseNet, each frame of the facial area image is classified and processed to obtain the probability distribution of the emotional state.

8. The method according to claim 7, characterized in that After acquiring the facial feature point motion trajectory sequence, facial motion unit intensity sequence, eye gaze angle sequence, head posture coordinate sequence and each frame of facial region image in the visible light image based on the openface facial detection algorithm, the method further includes: The facial feature point motion trajectory sequence, the facial motion unit intensity sequence, the eye gaze angle sequence, the head posture coordinate sequence and the facial spatiotemporal features are serially fused to obtain a processed feature vector; wherein the facial spatiotemporal features are obtained by performing three-dimensional convolution on the facial region image based on a 3DCNN network; Performing dimensionality reduction processing on the processed feature vector based on a principal component analysis algorithm to obtain a feature vector after dimensionality reduction processing; The feature vector after the dimension reduction process is input into a pre-trained SVM model to obtain the user anxiety probability value output by the SVM model.

9. The method according to claim 7, characterized in that: After acquiring the facial feature point motion trajectory sequence, facial motion unit intensity sequence, eye gaze angle sequence, head posture coordinate sequence and each frame of facial region image in the visible light image based on the OpenFace facial detection algorithm, the method further includes: Based on the convolutional neural network, the spatial features of the facial motion unit intensity sequence, eye gaze angle sequence, head posture coordinate sequence and physiological data are extracted to obtain the spatial feature sequence S A , S E , S H , S P ; The spatial feature sequence S is extracted based on the long short-term memory neural network A , S E , S H , S P The time characteristics are respectively obtained as multi-dimensional spatiotemporal features ST A , ST E , ST H , ST P ; The multidimensional spatiotemporal feature ST A , ST E , ST H , ST P Perform multi-channel feature fusion to obtain the feature Y after multi-channel feature fusion; The feature Y is subjected to feature dimensionality reduction, and the feature Y after the feature dimensionality reduction is classified based on the multi-classification activation function softmax to obtain classification evaluation data of a preset depression level.

10. A non-contact physiological and psychological multi-parameter detection system, characterized in that: The system comprises: A first acquisition module is used to acquire a sequence of facial interest region images as a first image sequence; A first calculation module, used for calculating a normalized frame difference between two adjacent frames in the first image sequence to obtain a normalized frame difference sequence; A first extraction module, used to extract features of the normalized frame difference sequence based on a preset two-dimensional convolutional layer to obtain a first image feature; A first exchange module is used to exchange the order of the channel dimension and the time dimension in the first image feature, and flatten the width and height dimensions in the spatial dimension into one dimension to obtain a second image feature; A first fusion module, used for fusing the local spatiotemporal information of the second image feature based on a preset two-dimensional convolutional layer to obtain a third image feature; A second fusion module, used for fusing the global spatiotemporal information of the third image feature based on a preset global spatiotemporal self-attention layer to obtain a fourth image feature; a third fusion module, configured to fuse the spatial dimension features of the fourth image features, and upsample the time dimension of the fused fourth image features so that the time dimension after the upsampling process is the same as the length of the normalized frame difference sequence, and map the channel dimension of the fourth image features after the upsampling process to 1, to obtain a mapped signal as an rPPG signal; The first detection module is used to detect the user's physiological parameters based on the rPPG signal; wherein the physiological parameters include blood pressure, respiratory rate and heart rate.

Images