Non-contact physiological parameter detection method, system and equipment and storage medium

Abstract

The invention discloses a non-contact physiological parameter detection method, system and device and a storage medium. The method comprises the steps that non-contact PPG signals are extracted from RGB signals of each group of interested areas; inputting the non-contact PPG signal into the trained denoising network to obtain a denoised non-contact PPG signal corresponding to the current group of regions of interest; inputting the denoised non-contact PPG signals corresponding to all groups of regions of interest into the trained signal fusion network to obtain fused non-contact PPG signals; determining body temperature data of the testee based on the region of interest; inputting the RGB signals of all the regions of interest into the trained random forest model to obtain the oxyhemoglobin saturation of the testee; on the basis of the fused non-contact PPG signal, a respiration signal of the testee is confirmed, and then the respiration rate of the testee is confirmed; the pulse wave signal of the testee is confirmed, and the blood pressure and the pulse rate of the testee are confirmed based on the pulse wave signal.

Description

Technical Field

[0001] The present invention relates to the technical field of non-contact physiological parameter detection, and in particular to a non-contact physiological parameter detection method, system, device and storage medium. Background Art

[0002] The detection of physiological signals is the basis for the diagnosis and treatment of many diseases. Although there are many physiological signal detection devices at present, many home detection devices can only detect a few physiological signals and cannot achieve the simultaneous detection and effective integration of multiple physiological parameters. Although professional multi-channel monitors can detect multiple physiological signals, they often require contact sensors such as electrodes to be pasted. The wearing process is cumbersome and requires certain professional knowledge. It is not suitable for physiological signal self-testing in daily life scenarios. At the same time, most contact detection devices must be in close contact with the surface skin of the human body, so when there is an open wound on the skin that needs to be contacted, the contact detection device cannot perform measurements.

[0003] Imaging photoplethysmography technology makes it possible to realize non-contact physiological parameter detection. Imaging photoplethysmography technology is a non-contact detection method based on photoplethysmography technology. This technology is a non-invasive optical detection method that uses a camera as a photosensitive element to obtain photoplethysmographic pulse wave signals. It has the advantages of ease of use, low cost, safety, convenience and the ability to detect a variety of physiological parameters.

[0004] At present, the research on non-contact physiological signal detection based on imaging photoplethysmography technology mainly focuses on 1-2 physiological parameters, while daily health detection and disease prevention often require a combination of multiple physiological parameters for joint judgment. In addition, in daily life scenes, we often face unpredictable environmental interference, such as light and movement. Most of the non-contact physiological signal detection based on cameras uses physical models or deep learning methods, which usually only works well in certain specific scenes where the subjects are in a static state. Because the basic principle of this method is to use a camera to capture the slight changes on the skin surface caused by changes in blood flow, in previous studies, fixed light sources are usually used and subjects are required to remain as still as possible to avoid introducing noise into the image. However, in practical applications, it is difficult to strictly control the detection scene. Due to the particularity of imaging photoplethysmography technology to detect physiological parameters by reflecting light, light changes and motion artifacts will significantly affect the accuracy of the detection results, which is also the difficulty of current research on imaging photoplethysmography technology. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present invention provides a non-contact physiological parameter detection method, system, device and storage medium. Based on imaging photoplethysmography technology, it realizes the detection of four physiological parameters: pulse rate, respiratory rate, blood oxygen saturation and blood pressure. An infrared thermal imaging camera is added to realize the detection of body temperature, so that the physiological parameter detection system can realize the detection of five physiological parameters.

[0006] On the one hand, a non-contact physiological parameter detection method is provided, comprising:

[0007] Obtain the face video data of the subject, extract several regions of interest of the face for each frame of the image, and divide the regions of interest into several groups;

[0008] Extracting non-contact PPG signals from the RGB signals of each group of regions of interest, denoising the non-contact PPG signals, and fusing the denoised non-contact PPG signals to obtain fused non-contact PPG signals;

[0009] Based on the fused non-contact PPG signal, the subject's breathing rate, blood pressure and pulse rate are confirmed.

[0010] In another aspect, a non-contact physiological parameter detection system is provided, comprising:

[0011] An acquisition module is configured to: acquire facial video data of a subject, extract a plurality of regions of interest of the face for each frame of the image, and divide the regions of interest into a plurality of groups;

[0012] A preprocessing module is configured to: extract a non-contact PPG signal from the RGB signals of each group of regions of interest, denoise the non-contact PPG signal, and fuse the denoised non-contact PPG signal to obtain a fused non-contact PPG signal;

[0013] The output module is configured to confirm the breathing rate, blood pressure and pulse rate of the subject based on the fused non-contact PPG signal.

[0014] On the other hand, there is also provided an electronic device, comprising:

[0015] a memory for non-transitory storage of computer-readable instructions; and

[0016] a processor for executing the computer readable instructions,

[0017] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0018] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0019] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] The present invention is based on imaging photoplethysmography technology to detect four physiological parameters: pulse rate, respiratory rate, blood oxygen saturation, and blood pressure, and adds an infrared thermal imaging camera to detect body temperature, so that the physiological parameter detection system can detect five physiological parameters. The present invention also uses an anti-interference algorithm to suppress the interference of light and motion artifacts on the detection results, so that the system can be applied to a variety of detection scenarios, so that residents can obtain accurate physiological parameter detection results through simple operations in daily life, providing support for health detection and disease prevention.

[0022] It can detect multiple physiological parameter values ​​of the user without contacting the user to avoid cross infection. It is suitable for patients with open wounds on their bodies who cannot wear contact detection equipment. It can simultaneously detect multiple physiological parameters of the user, including pulse rate, respiratory rate, blood oxygen saturation, body temperature, and blood pressure. It is easy to operate and suitable for daily health monitoring. Users can obtain their own physiological parameters in real time, reduce waiting time, and understand their physical condition in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is a schematic diagram of the hardware structure of the first embodiment;

[0025] Figure 2 This is a flow chart of the working method of embodiment 1;

[0026] Figure 3 This is a data processing flow chart of Example 1;

[0027] Figure 4 This is a schematic diagram of facial ROI area division in Example 1;

[0028] Figure 5 This is a flow chart of the anti-light interference algorithm of Embodiment 1;

[0029] Figure 6This is a flow chart of anti-interference contactless PPG signal extraction according to Embodiment 1;

[0030] Among them, 1. right frontal area, 2. left frontal area, 3. right temporal area, 4. left temporal area, 5. right infraorbital area, 6. left infraorbital area, 7. right zygomatic area, 8. left zygomatic area, 9. right cheek area, 10. left cheek area, 11. right chin area, 12. left chin area, 13. camera, 14. host computer, 15. data cable, 16. bracket. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0032] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments of the present invention. The terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0034] In this embodiment, all data is obtained in compliance with laws and regulations and based on the user's consent, and is used legally.

[0035] Embodiment 1

[0036] like Figure 3 As shown, this embodiment provides a non-contact physiological parameter detection method, including:

[0037] S101: Obtaining facial video data of a subject, extracting a plurality of regions of interest of the face for each frame of the image, and dividing the regions of interest into a plurality of groups;

[0038] S102: extracting non-contact PPG signals from the RGB signals of each group of regions of interest, denoising the non-contact PPG signals, and fusing the denoised non-contact PPG signals to obtain fused non-contact PPG signals;

[0039] S103: Based on the fused non-contact PPG signal, confirm the subject's breathing rate, blood pressure and pulse rate.

[0040] Furthermore, the S101: acquiring facial video data of the subject, the facial video data including: visible light video and thermal imaging video.

[0041] Among them, the visible light video is obtained by collecting the subject's face through a visible light camera; the thermal imaging video is obtained by collecting the subject's face through a thermal imaging camera.

[0042] It should be understood that the present invention uses a camera as a data acquisition device to receive light reflected from the skin. Since the facial skin contains a large number of capillaries, the heart pumps blood rhythmically, causing the blood perfusion in the skin to also change rhythmically. Hemoglobin in the blood is the main component that absorbs light, and the change in its content causes the light reflected from the skin to contain information about blood perfusion. Based on this, the present invention extracts a variety of physiological parameters from the images taken by the camera. At the same time, in order to obtain body temperature information that the visible light camera cannot obtain, the present invention adds a thermal imaging camera. The visible light camera and the thermal imaging camera transmit the captured facial video of the subject to the host computer.

[0043] In the process of acquiring video data, each frame of image corresponds to a non-contact PPG signal sampling point and is saved in the signal sequence to determine whether the length of the signal sequence is greater than or equal to T seconds. If so, light interference suppression is entered; if not, waiting is continued.

[0044] Furthermore, the step S101: extracting a plurality of regions of interest of a face from each frame of image, further includes:

[0045] Light interference suppression is performed on each frame of the visible light video; and several regions of interest of the face are extracted from each frame after interference suppression.

[0046] Furthermore, if Figure 5 As shown, for each frame of the visible light video, light interference suppression is performed, including:

[0047] First, convert the R, G, and B values ​​of each frame of the face video data to between 0 and 1:

[0048] R=R / 255 (1)

[0049] G=G / 255 (2)

[0050] B=B / 255 (3)

[0051] The three component values ​​of H, S and V are calculated: (4)

[0053] V=max(R,G,B)

[0054]

[0055]

[0056] After converting the image from RGB color space to HSV color space, the V component is obtained, which contains the light and dark information of the face; after setting the V component to 1, the light and dark of the pixel values ​​of the face are balanced;

[0057] The equalized image is then converted from the HSV color model to the RGB color model to achieve noise suppression introduced by light interference.

[0058] It should be understood that the light interference suppression algorithm: Since the light source in the actual physiological parameter detection scene is difficult to predict, uneven and changing light and shadow may occur on the face during the detection process. Because the shadow of the face will increase the local pixel value, if the light and shadow of the face changes, it will be reflected in the change of the mean of the RGB three channels, that is, noise will be introduced into the RGB signal. However, the facial color change caused by light and shadow will cause the color of the three channels to change in the RGB color model, and the light and shadow information cannot be excluded separately. However, in the HSV color model, the brightness, that is, the V component, is included, which can reflect the brightness and darkness of the color. There is a conversion relationship between the RGB color model and the HSV color model.

[0059] Furthermore, the extraction of several regions of interest of the face from each frame of the image after interference suppression includes:

[0060] The facial feature points of the face are recognized, and 12 regions of interest of the face are selected based on the feature points.

[0061] Furthermore, the facial feature point recognition of a face includes:

[0062] Use the 468 facial landmarks provided by MediaPipe, a multimedia machine learning model application framework developed and open sourced by Google, to achieve face recognition and recognition of facial feature points

[0063] Furthermore, the 12 regions of interest of the face are selected based on the feature points, including:

[0064] According to the 468 facial landmark numbers provided by MeidaPipe, the region of interest is circled by connecting multiple landmarks in sequence. The coordinate points of right frontal area 1 are [67,109,10,9,107,66,67] connected in sequence, the coordinate points of left frontal area 2 are [10,338,297,296,336,9,10], the coordinate points of right temporal area 3 are [68,103,67,66,105,63,68], the coordinate points of left temporal area 4 are [297,332,298,293,334,296,297], the coordinate points of right infraorbital area 5 are [228,230,232,126,142,36,216,207,117,228], the coordinate points of left infraorbital area 6 are [448,346,427,436,266,355,452,450,448], and the coordinate points of right zygomatic area 7 are [ The left zygomatic area 8 is [261,265,372,345,352,376,427,346,261], the right cheek area 9 is [147,216,202,170,138,213,147], the left cheek area 10 is [436,376,433,367,395,422,436], the right chin area 11 is [202,83,18,175,171,140,170,202] and the left chin area 12 is [18,313,422,395,369,396,175,18].

[0065] Furthermore, the step S101: dividing the regions of interest into a plurality of groups includes: dividing the regions of interest into a plurality of groups, and removing the blocked regions of interest.

[0066] Furthermore, dividing the regions of interest into several groups and removing the blocked regions of interest includes:

[0067] All regions of interest were divided into six groups, with every two symmetrical regions as one group;

[0068] The area of ​​the left side of the face in each group of regions of interest is set to A. left , the right region of interest is set to A right ;

[0069] T d1 =A right -2×A left ; (7)

[0070] T d2 =A left -2×A right ; (8)

[0071] T d1 and T d2 is the threshold value;

[0072] When T d1 <0 and T d2 <0, it means that the regions of interest on both sides are not blocked and the regions of interest on both sides are retained;

[0073] When T d1 <0 and T d2 > 0, the right ROI is blocked and removed;

[0074] When T d1 >0 and T d2 <0, the left ROI is blocked and removed.

[0075] For each group of regions of interest, the RGB mean of all pixels in the region of interest is calculated to obtain an RGB signal with reduced motion interference.

[0076] The occluded region of interest is removed from each frame.

[0077] It should be understood that since the non-contact PPG signal is easily disturbed by head movement, a region of interest selection and preprocessing algorithm is proposed. Head movement is divided into rigid movement and non-rigid movement. Non-rigid movement refers to the movement of facial muscles. The structure and movement of facial muscles will cause changes in facial light and shadow and the size of the region of interest. Therefore, according to the muscle distribution and three-dimensional structure of the face, a region of interest division method is designed, such as Figure 4 As shown in the figure, avoid the eyes and mouth that are prone to movement, as well as areas that are prone to structural shadows. After obtaining the face video, first identify the facial feature points of the face, and select Figure 4 12 regions of interest are selected. The selection of the regions of interest refers to the three-dimensional structure of the head and the distribution of facial muscles, and avoids the structural shadow areas caused by non-rigid facial movements and the parts such as eyes and mouth that often move as much as possible. The 12 regions of interest are grouped into six groups of symmetrical regions of interest, with every two symmetrical regions. The actual area of ​​each region of interest on the face is calculated separately. When the head of a person deflects left and right, the difference in the area of ​​a symmetrical group of regions of interest increases with the increase of the deflection angle. Therefore, when the area of ​​the region of interest on one side is larger than twice the area of ​​the region of interest on the other side, the present invention believes that the side with a smaller area is blocked due to the left and right deflection movement of the head, which is easy to introduce external noise, so the region of interest on the side with a smaller area is removed, thereby realizing adaptive region of interest selection that resists interference from left and right deflection movement of the head.

[0078] Due to the three-dimensional structure of the face, different skin areas may be in different planes. When the head rotates left and right, the skin in different planes is blocked differently. For example, the area from the cheekbones to the ears is more easily blocked than the cheeks under the eyes. Therefore, when dividing the region of interest, we first make a preliminary distinction based on the three-dimensional structure.

[0079] When people talk and make facial expressions, their muscles contract and relax, which manifests itself on the skin as changes in the local structure of the face. For example, when a person smiles, the zygomatic muscles contract, and structural light and shadows are more likely to appear underneath. Therefore, based on the distribution of muscles, the area covered by a single region of interest should, as much as possible, contain only muscles with the same contraction direction, or areas where contraction movements will not produce excessive changes in the local structure, so that the single region of interest can remain in one plane when there is facial movement.

[0080] Further, the S102: extracting non-contact PPG signals from the RGB signals of each group of regions of interest, denoising the non-contact PPG signals, and fusing the denoised non-contact PPG signals to obtain fused non-contact PPG signals, includes:

[0081] A non-contact PPG signal is extracted from the RGB signal of each group of regions of interest; the non-contact PPG signal is input into the trained denoising network to obtain the denoised non-contact PPG signal corresponding to the current group of regions of interest; the denoised non-contact PPG signals corresponding to all groups of regions of interest are input into the trained signal fusion network to obtain the fused non-contact PPG signal.

[0082] Further, extracting the non-contact PPG signal from the RGB signal of each group of regions of interest specifically includes:

[0083] Based on the Plane Orthogonal to the Skin (POS) algorithm, a plane orthogonal to the skin is found in the RGB space to calculate the non-contact PPG signal.

[0084] First, map the RGB signal to a new plane, where X and Y are two orthogonal signals on the skin color plane:

[0085] X=RB,

[0086] Y=G+B-2R.

[0087] The non-contact PPG signal and the X, Y signal have the following conversion relationship:

[0088] PPG = X-αY,

[0089] α=σ(X) / σ(Y),

[0090] Where σ represents the standard deviation of the calculated signal.

[0091] The POS algorithm is used to extract the PPG signal. The POS algorithm comes from the following literature:

[0092] WANG W,den BRINKER AC,STUIJK S,et al.Algorithmic Principles of RemotePPG[J].IEEE Trans Biomed Eng,2017,64(7):1479-1491.

[0093] Furthermore, if Figure 6 As shown, the non-contact PPG signal is input into the trained denoising network to obtain the denoised non-contact PPG signal corresponding to the current group of regions of interest, wherein the trained denoising network includes:

[0094] The first convolutional layer, the first average pooling layer, the second convolutional layer, the second average pooling layer and the first fully connected layer are connected in sequence.

[0095] Furthermore, the non-contact PPG signal is input into the trained denoising network to obtain the denoised non-contact PPG signal corresponding to the current group of regions of interest, wherein the training process of the trained denoising network includes:

[0096] Constructing a first training set, wherein the first training set is an RGB signal whose true value after denoising is known;

[0097] The first training set is input into the denoising network, and the denoising network is trained. When the loss function value of the denoising network no longer decreases, or the number of iterations exceeds the set number, the training is stopped to obtain the trained denoising network; the loss function of the denoising network is used to calculate the difference between the predicted value of the denoising network output signal and the true value of the known denoising signal.

[0098] Furthermore, the non-contact PPG signal is input into the trained denoising network to obtain the denoised non-contact PPG signal corresponding to the current group of regions of interest, wherein the loss function L of the training process of the trained denoising network is dn include:

[0099]

[0100] It should be understood that the non-contact PPG signal is extracted from the RGB signal using the more robust POS algorithm. A total of 6 non-contact PPG signals are obtained from the six groups of ROIs. The six sub-signals may be subject to different interferences, but they all contain the same PPG signal components, based on which a denoising neural network is designed. The denoising network contains two 1D convolutional networks and a fully connected layer. The convolutional network is used to learn features from the original non-contact PPG signal. Since the pulse wave signal is a relatively simple feature, only a shallow network is used; the final fully connected layer is used to output the denoised non-contact PPG signal; the average pooling layer is selected to further suppress noise by calculating the mean while reducing the number of parameters. The 6 original non-contact PPG signals are taken as input and input into the same denoising network, and the loss function L is designed. dn The loss function is the sum of the absolute values ​​of the differences between the six denoised non-contact PPG signals. The difference between the denoised signals is minimized to extract the same physiological components in the signals.

[0101] Furthermore, the denoised non-contact PPG signals corresponding to all groups of regions of interest are input into the trained signal fusion network to obtain fused non-contact PPG signals; wherein the trained signal fusion network includes:

[0102] Two-dimensional convolution layer, first pooling layer, first spatiotemporal graph convolutional neural network, second pooling layer, second spatiotemporal graph convolutional neural network, third pooling layer and second fully connected layer.

[0103] The first pooling layer, the second pooling layer and the third pooling layer perform pooling operations only in the spatial dimension.

[0104] Furthermore, the denoised non-contact PPG signals corresponding to all groups of regions of interest are input into the trained signal fusion network to obtain fused non-contact PPG signals; wherein the training process of the trained signal fusion network includes:

[0105] Constructing a second training set, wherein the second training set is a plurality of groups of denoised non-contact PPG signals corresponding to regions of interest for which the fused non-contact PPG signals are known;

[0106] The second training set is input into the signal fusion network to train the network. When the loss function value of the signal fusion network no longer decreases, the training is stopped to obtain the trained signal fusion network.

[0107] Furthermore, the denoised non-contact PPG signals corresponding to all groups of regions of interest are input into the trained signal fusion network to obtain fused non-contact PPG signals; wherein the trained signal fusion network, the loss function L used in the training process p include:

[0108]

[0109] Among them, x p It is the contact PPG signal.

[0110] It should be understood that the 6 denoised non-contact PPG signals obtained need to be fused to obtain a final non-contact PPG signal, so the present invention designs a signal fusion network to remove the same noise components that each ROI may be subjected to while fusing the signals. The 6 denoised non-contact PPG signals obtained are combined into a two-dimensional matrix as the input of the signal fusion network.

[0111] The signal fusion network consists of 1 2D convolution layer, 2 spatiotemporal convolution layers and 1 fully connected layer. While learning the signal features, the signal changes in the time dimension are taken into account. The final fully connected layer uses the output to obtain the final non-contact PPG signal; the network only uses spatial average pooling to retain information in the time dimension. The network uses contact PPG signals to train the network. Since contact PPG signals usually use finger clip sensors, and the PPG signals obtained from fingers and faces will have a certain phase offset, the negative Pearson correlation coefficient is used as the loss function in the design of the loss function.

[0112] Through this loss function, the fluctuation trend of the non-contact PPG signal output by the network is made close to the real contact PPG signal, while the interference noise affecting all ROIs is removed.

[0113] Since the PPG signal is a relatively lower-level feature and the system needs to obtain physiological parameters in real time, the present invention reduces the size of the neural network as much as possible to improve the operation speed while increasing the robustness of the network.

[0114] Each frame of the video must be identified by face coordinate points and the skin area from the face is divided as the region of interest. Since the image data volume is large and the pixel value of a local single pixel is easily disturbed, the physiological parameters are usually obtained from the red, green and blue (RGB) three-channel mean extracted from the skin image of a region and the non-contact PPG signal obtained by further processing. For different physiological parameters, the present invention uses corresponding calculation algorithms.

[0115] Furthermore, the step S103: confirming the subject's respiratory rate, blood pressure and pulse rate based on the fused non-contact PPG signal specifically includes:

[0116] Based on the fused non-contact PPG signal, the subject's breathing signal is confirmed, and then the subject's breathing rate is confirmed; based on the fused non-contact PPG signal, the subject's pulse wave signal is confirmed, and based on the pulse wave signal, the subject's blood pressure and pulse rate are confirmed.

[0117] Further, based on the fused non-contact PPG signal, the breathing signal of the subject is confirmed, and then the breathing rate of the subject is confirmed, including: firstly performing band-pass filtering on the non-contact PPG signal with the frequency band of the breathing rate as the passband to obtain the non-contact breathing signal;

[0118] After the non-contact breathing signal is converted into the frequency domain, the value of the abscissa corresponding to the maximum amplitude in the spectrum diagram is the breathing frequency. The breathing frequency is multiplied by 60 seconds to obtain the breathing rate.

[0119] Further, based on the fused non-contact PPG signal, the subject's pulse wave signal is confirmed, and based on the pulse wave signal, the subject's blood pressure and pulse rate are confirmed, including:

[0120] For blood pressure detection, the obtained non-contact photoplethysmography (PPG) PPG signal is input into a trained multi-layer perceptron, and the trained multi-layer perceptron outputs corresponding systolic and diastolic pressures; wherein the training process of the trained multi-layer perceptron is as follows: constructing a fourth training set, wherein the fourth training set is a non-contact PPG signal of known blood pressure; inputting the fourth training set into the multi-layer perceptron, training the multi-layer perceptron, and stopping the training when the loss function value of the multi-layer perceptron no longer decreases, or the number of iterations exceeds the set number, to obtain the trained multi-layer perceptron;

[0121] For pulse rate detection, the fused non-contact PPG signal is filtered, the frequency band of the pulse rate is selected as the passband, and the non-contact PPG signal is bandpass filtered to obtain a non-contact pulse wave signal; the non-contact pulse wave signal is converted to the frequency domain by Fourier transform, and the value of the horizontal axis corresponding to the maximum amplitude in the spectrum diagram is the frequency of the pulse. The pulse frequency is multiplied by 60s to obtain the pulse rate.

[0122] Furthermore, the method also includes: confirming the temperature data of the subject based on the region of interest; inputting the RGB signals of all regions of interest into the trained random forest model to obtain the blood oxygen saturation of the subject.

[0123] Further, based on the region of interest, the subject's body temperature data is confirmed, including:

[0124] An infrared thermal imaging camera is used to photograph the area of ​​interest of the subject, and the reflection coefficient of the skin is set to directly obtain the surface temperature of the human face.

[0125] Furthermore, the RGB signals of all regions of interest are input into the trained random forest model to obtain the blood oxygen saturation of the subjects, including:

[0126] Feature extraction is performed from the RGB signal of the region of interest, and the extracted features are input into the trained random forest model, which outputs the blood oxygen saturation value.

[0127] Since the extraction of percentage blood oxygen saturation is usually based on the different absorption coefficients of deoxygenated hemoglobin and oxyhemoglobin to light of different wavelengths, by selecting two bands of light, the absorption coefficients of oxyhemoglobin and deoxyhemoglobin differ greatly in one band, and the absorption coefficients of oxyhemoglobin and reduced hemoglobin are basically equal in the other band, and the percentage blood oxygen saturation value is indirectly obtained through the AC and DC components of the two bands of light. Since the visible light camera only contains three components, red, green, and blue, the DC and AC related components of the three components are extracted as features and used as the input of the random forest model. First, based on the extracted pulse rate, the RGB signals are respectively subjected to a 5th-order Butterworth bandpass filter with [pulse rate / 60-0.3, pulse rate / 60+0.3] Hz as the passband. The average peak value of the filtered RGB signal is recorded as P, the average valley value is recorded as V, the mean value of the signal is recorded as DC, and the average difference between the peak and valley values ​​of the signal is recorded as AC.

[0128] Furthermore, the feature extraction is performed from the RGB signal of the region of interest, and all the extracted features include:

[0129] F1=(R_P) / (R_V); F2=(G_P) / (G_V); F3=(B_P) / (B_V);

[0130] F4=(R_AC) / (R_DC); F5=(G_AC) / (G_DC); F6=(B_AC) / (B_DC);

[0131] F7=(R_P / R_V) / (G_P / G_V); F8=(R_P / R_V) / (B_P / B_V);

[0132] F9=(G_P / G_V) / (B_P / B_V); F10=(R_AC / R_DC) / (G_AC / G_DC);

[0133] F11=(R_AC / R_DC) / (B_AC / B_DC); F12=(G_AC / G_DC) / (B_AC / B_DC);

[0134] F13=(R_P / R_V) / (G_AC / G_DC); F14=(R_P / R_V) / (B_AC / B_DC);

[0135] F15=(G_P / G_V) / (B_AC / B_DC).

[0136] Furthermore, the training process of the trained random forest model includes: constructing a third training set, wherein the third training set is an RGB signal of known blood oxygen saturation; inputting the third training set into the random forest model, and when the loss function value of the model no longer decreases, stopping the training to obtain the trained random forest model.

[0137] The present invention also builds a real-time detection system. Figure 1 As shown, the system hardware includes: a camera 13 , which is mounted on a bracket 16 , and is connected to a host computer 14 via a data cable 15 .

[0138] The hardware part of the system is a visible light camera and a thermal imaging camera for data acquisition, and a host computer for software algorithm processing. The camera transmits data to the host computer through a cable. Every time the camera takes a frame of image, it is input into the signal extraction module for processing, and the processed data value is stored in the signal sequence. Since human physiological activity is a continuous process, obtaining the physiological parameter value at a certain moment requires using the average physiological parameter value within a short period of time including the current moment as an equivalent replacement. Figure 2 As shown, in the present invention, in order to ensure the accuracy of the data and make the system respond in time, the system reads the signal once per second to calculate the physiological parameters, and each time reads the signal length 10 seconds before the current moment. The calculated physiological parameters will be transmitted to the UI interface of the host computer for display.

[0139] The present invention can realize the collection of multiple physiological parameters. At present, the physiological parameter detection equipment used in daily life can usually only detect a few physiological parameters; and the multi-physiological parameter monitoring equipment is usually used in clinical practice, requires the assistance of professionals to operate, and is expensive and not suitable for daily use. At present, most of the non-contact physiological signal detection research based on imaging photoplethysmography technology only extracts one or two important physiological indicators, and lacks a system for realizing real-time detection of multiple physiological parameters. Based on imaging photoplethysmography technology, the present invention realizes pulse rate, respiratory rate, blood oxygen saturation and blood pressure, a total of 4 important human physiological parameters. For the body temperature parameter that cannot be obtained by imaging photoplethysmography technology, an infrared thermal imaging camera module is used for detection. The present invention combines a visible light camera and an infrared thermal imaging camera to build a non-contact physiological signal detection system, realizes the real-time acquisition of 5 physiological parameters including pulse rate, respiratory rate, blood oxygen saturation, blood pressure and body temperature, and provides support for health detection and disease prevention in daily life.

[0140] The present invention can resist possible light interference in the process of physiological parameter measurement. Since non-contact physiological signal detection uses a camera to capture the tiny color changes on the skin surface caused by heart beats, changes in light conditions will introduce a large number of color changes. When the interference is similar to the heart rate frequency, it is easy to interfere with the detection results, and it is difficult to remove it through conventional denoising methods such as filtering. In daily life scenes, the color, direction and number of light sources are often difficult to determine, and the rotation of the head will also bring about changes in facial light and shadow, and the source of noise is often difficult to control. In the preprocessing stage of the video, the present invention suppresses possible light interference on the face by converting the algorithm of the color model, thereby improving the quality of physiological signals and the accuracy of physiological parameters.

[0141] The present invention can suppress motion artifacts during the detection process. In daily detection scenarios, the relative movement between the user's head and the detection device is usually difficult to avoid, but motion artifacts will affect the quality of the non-contact photoplethysmography (PPG) signal, thereby affecting the accuracy of the detected physiological parameters. The present invention detects multiple regions of interest (ROI) and adaptively selects the region of interest, and extracts non-contact PPG signals through a POS algorithm with a certain robustness. The non-contact PPG signals of each ROI are fused through a neural network, and the light and motion interference in the signal are further suppressed to obtain a non-contact PPG signal close to a real contact PPG signal, thereby improving the accuracy of physiological parameter detection.

[0142] Embodiment 2: This embodiment provides a non-contact physiological parameter detection system, including:

[0143] An acquisition module is configured to: acquire facial video data of a subject, extract a plurality of regions of interest of the face for each frame of the image, and divide the regions of interest into a plurality of groups;

[0144] A preprocessing module is configured to: extract a non-contact PPG signal from the RGB signals of each group of regions of interest, denoise the non-contact PPG signal, and fuse the denoised non-contact PPG signal to obtain a fused non-contact PPG signal;

[0145] The output module is configured to confirm the breathing rate, blood pressure and pulse rate of the subject based on the fused non-contact PPG signal.

[0146] It should be noted that the acquisition module, preprocessing module and output module described above correspond to steps S101 to S103 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules described above as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0147] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0148] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0149] Embodiment 3: This embodiment also provides an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment 1.

[0150] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0151] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0152] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0153] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0154] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0155] Embodiment 4: This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in Embodiment 1 is completed.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A non-contact physiological parameter detection method, characterized in that: include: Obtain the face video data of the subject, extract several regions of interest of the face for each frame of the image, and divide the regions of interest into several groups; Extracting non-contact PPG signals from the RGB signals of each group of regions of interest, denoising the non-contact PPG signals, and fusing the denoised non-contact PPG signals to obtain fused non-contact PPG signals; Based on the fused non-contact PPG signal, the subject's breathing rate, blood pressure and pulse rate are confirmed.

2. A non-contact physiological parameter detection method as claimed in claim 1, characterized in that: Extracting a plurality of regions of interest of a human face from each frame of the image also includes: suppressing light interference from each frame of the visible light video; and extracting a plurality of regions of interest of a human face from each frame of the image after the interference is suppressed.

3. A non-contact physiological parameter detection method as claimed in claim 1, characterized in that: Dividing the regions of interest into several groups includes: symmetrically dividing the regions of interest into several groups, and eliminating the blocked regions of interest.

4. A non-contact physiological parameter detection method as claimed in claim 1, characterized in that: Denoising the non-contact PPG signal, inputting the non-contact PPG signal into a trained denoising network to obtain a denoised non-contact PPG signal corresponding to the current group of regions of interest, wherein the trained denoising network includes: a first convolutional layer, a first average pooling layer, a second convolutional layer, a second average pooling layer and a first fully connected layer connected in sequence; The training process of the trained denoising network includes: constructing a first training set, wherein the first training set is an RGB signal of a known true value of the denoised signal; inputting the first training set into the denoising network to train the denoising network, and when the loss function value of the denoising network no longer decreases, or the number of iterations exceeds a set number, stopping the training to obtain the trained denoising network; the loss function of the denoising network is used to calculate the difference between the predicted value of the denoising network output signal and the known true value of the denoised signal.

5. A non-contact physiological parameter detection method as claimed in claim 1, characterized in that: The denoised non-contact PPG signals are fused by inputting the denoised non-contact PPG signals corresponding to all groups of regions of interest into the trained signal fusion network to obtain fused non-contact PPG signals; wherein the trained signal fusion network includes: Two-dimensional convolutional layer, first pooling layer, first spatiotemporal graph convolutional neural network, second pooling layer, second spatiotemporal graph convolutional neural network, third pooling layer, and second fully connected layer; The training process of the trained signal fusion network includes: constructing a second training set, wherein the second training set is a plurality of groups of denoised non-contact PPG signals corresponding to regions of interest of known fused non-contact PPG signals; inputting the second training set into the signal fusion network, training the network, and stopping the training when the loss function value of the signal fusion network no longer decreases, thereby obtaining the trained signal fusion network.

6. A non-contact physiological parameter detection method as claimed in claim 1, characterized in that: The method further includes: confirming the temperature data of the subject based on the region of interest; determining the blood oxygen saturation of the subject based on the RGB signals of all the regions of interest; the blood oxygen saturation of the subject based on the RGB signals of all the regions of interest is to input the RGB signals of all the regions of interest into the trained random forest model to obtain the blood oxygen saturation of the subject, including: Extract features from the RGB signal of the region of interest, input the extracted features into the trained random forest model, and the trained random forest model outputs the blood oxygen saturation value; the training process of the trained random forest model includes: constructing a third training set, the third training set is the RGB signal of known blood oxygen saturation; inputting the third training set into the random forest model, and stopping the training when the loss function value of the model no longer decreases, so as to obtain the trained random forest model; Confirming the breathing rate of the subject includes: firstly performing bandpass filtering on the non-contact PPG signal with the frequency band of the breathing rate as the passband to obtain the non-contact breathing signal; after converting the non-contact breathing signal to the frequency domain, the value of the abscissa corresponding to the maximum amplitude value in the spectrum diagram is the breathing frequency, and the breathing frequency is multiplied by 60 seconds to obtain the breathing rate.

7. A non-contact physiological parameter detection method as claimed in claim 1, characterized in that: Confirm the subject's blood pressure and pulse rate, including: For blood pressure detection, the obtained PPG signal is input into the trained multilayer perceptron, and the trained multilayer perceptron outputs the corresponding systolic pressure and diastolic pressure; wherein the training process of the trained multilayer perceptron is: constructing a fourth training set, wherein the fourth training set is a non-contact PPG signal of known blood pressure; inputting the fourth training set into the multilayer perceptron, training it, and when the loss function value of the multilayer perceptron no longer decreases, or the number of iterations exceeds the set number, stopping the training to obtain the trained multilayer perceptron; For pulse rate detection, the fused non-contact PPG signal is filtered, the frequency band of the pulse rate is selected as the passband, and the non-contact PPG signal is bandpass filtered to obtain a non-contact pulse wave signal; the non-contact pulse wave signal is converted to the frequency domain by Fourier transform, and the value of the horizontal axis corresponding to the maximum amplitude in the spectrum diagram is the frequency of the pulse. The pulse frequency is multiplied by 60s to obtain the pulse rate.

8. A non-contact physiological parameter detection system, characterized in that: include: An acquisition module is configured to: acquire facial video data of a subject, extract a plurality of regions of interest of the face for each frame of the image, and divide the regions of interest into a plurality of groups; A preprocessing module is configured to: extract a non-contact PPG signal from the RGB signals of each group of regions of interest, denoise the non-contact PPG signal, and fuse the denoised non-contact PPG signal to obtain a fused non-contact PPG signal; The output module is configured to confirm the breathing rate, blood pressure and pulse rate of the subject based on the fused non-contact PPG signal.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 7 are executed.

Images