A non-contact heart rate detection method based on multi-view fusion and device thereof

Abstract

The application discloses a kind of non-contact heart rate detection method and device based on multi-view fusion, comprising: obtaining multi-view original face image;Wherein, multi-view original face image is obtained by imaging device;According to original face image, determine the region of interest, and obtain IPPG signal and yaw angle sequence from the region of interest;The IPPG signal and yaw angle sequence of different view are weighted and fused, to obtain the multi-view signal after fusion and the yaw angle sequence after fusion;The multi-view signal after fusion and the yaw angle sequence after fusion are filtered, remove the influence of noise, obtain multi-view fusion heart rate signal;Multi-view fusion heart rate signal is converted into time-frequency, and according to the frequency domain peak value after conversion, obtain heart rate value.The application can avoid the noise interference and image loss caused by the left and right rotation of user's head, and improve the universality of non-contact system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of non-contact heart rate monitoring, and particularly relates to a non-contact heart rate detection method based on multi-view fusion and a device thereof. BACKGROUND

[0002] Heart rate refers to the number of heartbeats per minute in a normal person in a quiet state, also known as resting heart rate. Generally, the normal heart rate is 60-100 times per minute, and individual differences may occur due to age, gender or other reasons. However, for the same individual, both too fast and too slow heart rates indicate the physical health condition and are closely related to diseases. For example, brain blood vessel diseases and hypothyroidism can cause slow heart rate, and cardiovascular diseases and hyperthyroidism can cause fast heart rate. Long-term monitoring of heart rate on a daily basis helps us to discover and treat diseases in a timely manner.

[0003] At present, heart rate monitoring mainly includes contact and non-contact devices. The contact device needs to be in direct contact with the user during monitoring, and long-term monitoring will bring a sense of restraint to the user and additional pressure. The non-contact device does not need to be in contact with the user and is more suitable for long-term monitoring of the user. The imaging photoplethysmography (IPPG) technology is one of the main implementation manners. The periodic movement of the heart causes the periodic change of blood volume in the blood vessel, and the periodic change of blood volume causes the absorption of incident light to change in accordance with the heartbeat cycle. The change in the intensity of the light reflected by the skin can be captured by a camera to obtain an alternating current signal containing the heartbeat. The signal is amplified and filtered to calculate physiological parameters such as heart rate. The IPPG signal is easily disturbed by motion and ambient light. Through relevant literature, it is found that the user needs to remain still during non-contact monitoring. When the head offset angle is too large, the motion disturbance will cause the signal quality to decrease, and even the camera cannot track the face region, affecting the accuracy of the monitoring result, and making it difficult for the non-contact monitoring device based on the IPPG technology to be applied and popularized in daily life.

[0004] Therefore, it is urgent to improve the defects in the prior art. SUMMARY

[0005] In order to solve the above problems in the prior art, the present application provides a non-contact heart rate detection method based on multi-view fusion and a device thereof. The technical problem to be solved by the present application is solved by the following technical scheme:

[0006] In a first aspect, the present application provides a non-contact heart rate detection method based on multi-view fusion, comprising:

[0007] Obtaining multi-view original facial images; wherein the multi-view original facial images are obtained by an imaging device;

[0008] According to the original facial images, a region of interest is determined, and an IPPG signal and a yaw angle sequence are obtained from the region of interest;

[0009] The IPPG signals and the yaw angle sequences of different views are weighted and fused to obtain a fused multi-view signal and a fused yaw angle sequence;

[0010] The fused multi-view signal and the fused yaw angle sequence are filtered to remove noise effects, and a multi-view fusion heart rate signal is obtained;

[0011] The multi-view fusion heart rate signal is subjected to time-frequency conversion, and a heart rate value is obtained according to the converted frequency domain peak value.

[0012] In a second aspect, the present application also provides a non-contact heart rate detection device based on multi-view fusion, comprising:

[0013] A signal acquisition module is configured to obtain multi-view original facial images; wherein the multi-view original facial images are obtained by an imaging device;

[0014] A signal processing module one is configured to determine a region of interest according to the original facial images, and obtain an IPPG signal and a yaw angle sequence from the region of interest;

[0015] A signal fusion module is configured to weight and fuse IPPG signals and yaw angle sequences of different views to obtain a fused multi-view signal and a fused yaw angle sequence;

[0016] A signal processing module two is configured to filter the fused multi-view signal and the fused yaw angle sequence to remove noise effects, and obtain a multi-view fusion heart rate signal;

[0017] A parameter calculation module is configured to perform time-frequency conversion on the multi-view fusion heart rate signal, and obtain a heart rate value according to the converted frequency domain peak value.

[0018] The present application has the following advantages:

[0019] The application provides a non-contact heart rate detection method and device based on multi-view fusion, which acquires original facial images of a user in a full range and completely through the distribution of imaging devices; the original facial images are preprocessed to obtain an IPPG signal and a yaw angle sequence; the IPPG signal and the yaw angle sequence are fused based on the multi-view fusion method to obtain a fused multi-view signal; the yaw angle in the motion process is analyzed based on wavelet denoising of an adaptive threshold, the yaw angle is regarded as motion noise, the signal and the noise are decomposed respectively, a noise subsegment containing a heart rate frequency subsegment is found, a standard deviation thereof is calculated as a threshold, and filtering is performed; the multi-view fused signal after denoising is subjected to time-frequency conversion, and a heart rate value is obtained according to the converted frequency domain peak value; in this way, the problem that the user's head needs to be restrained during the monitoring process of the imaging device is solved, the user can calculate the heart rate during facial motion, noise interference and image loss caused by facial motion are avoided, the universality of the non-contact system is improved, the application scenario is closer to the actual situation, and the popularization and application of the system for monitoring the heart rate by using the imaging device are promoted.

[0020] The application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of the non-contact heart rate detection method based on multi-view fusion provided by the embodiments of the application;

[0022] Figure 2 is a schematic diagram of the arrangement of the imaging device provided by the embodiments of the application;

[0023] Figure 3 is a schematic diagram of the facial feature points of the user provided by the embodiments of the application;

[0024] Figure 4 is a schematic diagram of the IPPG signal provided by the embodiments of the application;

[0025] Figure 5 is a schematic diagram of the normalized blue channel IPPG signal provided by the embodiments of the application;

[0026] Figure 6 is a schematic diagram of the distribution of the yaw angle in different regions provided by the embodiments of the application;

[0027] Figure 7 is a schematic diagram of the yaw angle sequence provided by the embodiments of the application;

[0028] Figure 8 is a schematic diagram of the fused multi-view signal provided by the embodiments of the application;

[0029] Figure 9 A diagram of the yaw angle sequence after fusion provided by an embodiment of the present application;

[0030] Figure 10 A diagram of the multi-view signal after amplification provided by an embodiment of the present application;

[0031] Figure 11 A diagram of the spectrum of the multi-view signal and the yaw angle sequence provided by an embodiment of the present application;

[0032] Figure 12 A diagram of the multi-view signal after reconstruction provided by an embodiment of the present application;

[0033] Figure 13 A diagram of the spectrum of the multi-view signal after wavelet denoising provided by an embodiment of the present application;

[0034] Figure 14 A diagram of the multi-view signal after band-pass filtering provided by an embodiment of the present application;

[0035] Figure 15 A diagram of the spectrum of the heart rate segment provided by an embodiment of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0037] In the prior art, IPPG technology enters the field of view of researchers, opening up the field of non-contact heart rate monitoring using cameras. In recent years, although the video-based non-contact physiological parameter monitoring method has been continuously developed and improved, the extraction of the region of interest in most detection methods is obtained by recognizing facial feature points. For example, the patent "Non-contact heart rate detection method based on signal fitting" (Application No.: 201810887461.3, Application Date: August 6, 2018) applied by Hefei University of Technology uses the Viola Jones face detector provided by OPENCV to identify the face region, detect the facial feature points, calculate the facial pulse wave, and achieve accurate calculation of the heart rate through pulse wave reconstruction and fitting; the patent "Non-contact heart rate detection method based on photoplethysmography" (Application No.: 202110912250.2, Application Date: August 10, 2021) applied by Beijing University of Aeronautics and Astronautics uses Haar feature to identify the face region to obtain the IPPG signal, and uses fast independent component analysis to calculate the heart rate. As can be seen, the current extraction of pulse wave signals is mainly from the face region, and the face feature points need to be detected first to obtain the image. However, in actual testing, the head may be offset at a large angle, the facial features may not be accurately detected, the signal source may be lost, and head movement may cause motion interference, resulting in a large deviation in the calculation result of the heart rate, which makes it difficult to promote in actual application.

[0038] Therefore, the present application provides a non-contact heart rate detection method and device based on multi-view fusion, which uses multiple imaging devices to obtain videos of the head rotation process from multiple angles, fuses the IPPG signals obtained from multiple angles, uses the face yaw angle, combines wavelet denoising and bandpass filtering to process the original signal, obtains an IPPG signal with high signal-to-noise ratio, and calculates the heart rate.

[0039] Referring to Figure 1 , Figure 1 is a flowchart of the non-contact heart rate detection method based on multi-view fusion provided by the embodiments of the present application. The non-contact heart rate detection method based on multi-view fusion provided by the present application comprises:

[0040] S101, obtaining a multi-view original face image; wherein the multi-view original face image is obtained by an imaging device.

[0041] Specifically, referring to Figure 2 , Figure 2 is a schematic diagram of the arrangement of the imaging device provided by the embodiments of the present application, and before obtaining the original face image, it further comprises:

[0042] The plurality of imaging devices are arranged in an annular array in front of the face of the user, and each imaging device is at the same distance from the face of the user, and optionally, the distance from each imaging device to the face of the user is 0.5 m; wherein the face of the user is taken as the center, and the imaging devices are arranged in front of the face, left front and right front, respectively, and the included angle between adjacent imaging devices is 60°.

[0043] The plurality of imaging devices are corrected for distortion, and the parameters of different imaging devices are set to be the same.

[0044] In the embodiment, before the imaging device acquires the original face image, the imaging device needs to be arranged and preprocessed. Due to production process and other reasons, the imaging device inevitably has distortion, which causes some areas of the original face image to be distorted, thereby causing the IPPG signal to be distorted. In the embodiment, a plurality of imaging devices are used for signal acquisition. In order to ensure the signal quality, the plurality of imaging devices need to be corrected for distortion respectively, to ensure the signal quality and reduce the interference of external factors, and the imaging parameters of the unused imaging devices are set to be the same.

[0045] Please continue to see Figure 2 As shown in the figure, three imaging devices are used to monitor the face of the user from different angles, and the imaging devices are arranged in front of the face of the user, respectively, at positions of 30°, 90° and 150° of the face of the user, to maximize the consistency of the yaw angle range obtained by each viewing angle.

[0046] S102, determining a region of interest (ROI) according to the original face image, and acquiring an IPPG signal and a yaw angle sequence from the region of interest.

[0047] Specifically, please see Figure 3 and Figure 4 As shown in the figure, Figure 3 is a schematic diagram of a user face feature point provided by an embodiment of the application, Figure 4 is a schematic diagram of an IPPG signal provided by an embodiment of the application. In the embodiment, the process of acquiring the IPPG signal includes:

[0048] Detecting feature points in the original face image, determining a region of interest according to the coordinates of each feature point; please continue to see Figure 3 As shown in the figure, for each frame of original face image acquired, the face feature points are detected. Considering the stability of the detection algorithm and the consumption of resources, the Dlib is used to detect 68 feature points of the face of the user in the embodiment, and the region of interest is determined as the nose and the cheek regions on both sides of the nose. Selecting this region can suppress the motion noise caused by angle deviation.

[0049] Obtaining the pixel average value of the RGB channel in the region of interest of the original face image.

[0050] Considering that the IPPG signal is followed by the periodic change of blood volume in blood vessels by detecting the change of received light intensity, the IPPG signal corresponding to the RGB channel is obtained according to the pixel average value of the RGB channel; wherein the IPPG signal y k The expression of (n) is:

[0051]

[0052] Wherein, n is the frame sequence, M is the height of the region of interest, N is the width of the region of interest, k is the different view angle, y is the R, G and B three different color channels, I ky (i,j,n) is the pixel value of the coordinate i,j in the region of interest of the channel y in the nth frame of the original image in the kth view angle, about 150 frames of images are collected in each view angle, the pixel average value of the RGB three channels of the region of interest in each view angle is extracted respectively, a total of 9 signals are extracted from three view angles as IPPG original signals, and the signals are further processed subsequently.

[0053] The acquisition process of the yaw angle sequence includes:

[0054] Detecting feature points in the original face image, using user face posture estimation, mapping the feature points of the original face image with a preset three-dimensional face model, obtaining a rotation matrix, and obtaining the yaw angle of the left and right rotation of the head according to the rotation matrix; wherein

[0055] The expression of the rotation matrix R is:

[0056]

[0057] According to the rotation matrix R, the yaw angle Y k (n) is obtained, and the expression is:

[0058]

[0059] r 02 = sinω;

[0060]

[0061]

[0062] Wherein, Y k(n) is the yaw angle of the n-th frame of original image in the k-th view relative to the imaging device, in addition to the yaw angle, the pitch angle and the roll angle can also be obtained, in daily monitoring, the left and right movement of the user's face will have a greater impact, therefore, the embodiment only focuses on the yaw angle, records the yaw angle of each frame of original face image, and forms a yaw angle sequence.

[0063] According to the yaw angle, the yaw angle sequence is obtained.

[0064] It should be noted that the embodiment also includes limiting the yaw angle corresponding to the original face image under the current view, that is, if the yaw angle corresponding to the original face image under the current view is greater than 35°, the original face image is deleted, and the original face image obtained at the adjacent view is used as an effective frame to detect the yaw angle sequence corresponding to the original image; it can also be understood that, in order to limit the yaw angle of the user's face, the yaw angle of the original face image captured under the current view is limited to be not more than 35°, when the yaw angle exceeds 35° or the image captured under the current view does not detect a face image, the yaw angle is 0°, and is saved in the yaw angle sequence.

[0065] S103, weighting and fusing the IPPG signals of different views and the yaw angle sequence to obtain a fused multi-view signal and a fused yaw angle sequence.

[0066] Specifically, before weighting and fusing the IPPG signals of different views and the yaw angle sequence, the embodiment further includes:

[0067] using segmented polynomial fitting to eliminate the trend of the IPPG signal over time;

[0068] normalizing the IPPG signal after eliminating the trend to obtain a normalized IPPG signal.

[0069] Specifically, before fusing the IPPG signals, in order to unify the fluctuation range and avoid the influence of the trend, improve the accuracy and reliability of the signal, it is necessary to remove the trend and normalize the IPPG signal. The amplitude of the IPPG signal form is affected by factors such as blood vessel contraction and expansion, body position movement, etc. During the slow change of the IPPG signal over time, these factors will cause the IPPG signal to have a trend over time, and this trend may mask important information in the signal. The embodiment uses segmented polynomial fitting to remove the trend, considering that the beating period of the heart rate is 40bpm-240bpm, and the maximum heartbeat period is 1.5s, the embodiment segments the IPPG signal with a window length of 45, ensures that at least one complete heartbeat period is retained in each window, and fits each segment to eliminate the trend.

[0070] The IPPG signal after eliminating the trend also needs to be normalized, and the embodiment utilizes maximum and minimum value normalization to scale the value range of the data to the range of [0, 1], and the calculation formula is as follows:

[0071]

[0072] Wherein, x is the original data to be processed, x min and x max are the maximum and minimum values in the original data, and x norm is the data after normalization; the data range of different perspectives is unified, facilitating subsequent fusion. Please refer to Figure 5 , Figure 5 is a schematic view of the blue channel IPPG signal after normalization provided by the embodiment of the application.

[0073] Please refer to Figure 6 , Figure 6 is a schematic view of the yaw angle distribution in different regions, and the imaging device distribution region is divided into six regions, regions a-f, in an ideal state, the imaging device 1 captures images when the face is oriented towards the regions a and b, the face is oriented towards the region a, and the yaw angle recorded by the current perspective is a negative value, the face is oriented towards the region b, and the yaw angle recorded by the current perspective is a positive value; the imaging device 2 captures images when the face is oriented towards the regions c and d, the face is oriented towards the region c, and the yaw angle recorded by the current perspective is a negative value, the face is oriented towards the region d, and the yaw angle recorded by the current perspective is a positive value; the imaging device 3 captures images when the face is oriented towards the regions e and f, the face is oriented towards the region e, and the yaw angle recorded by the current perspective is a negative value, the face is oriented towards the region f, and the yaw angle recorded by the current perspective is a positive value; the value of the yaw angle obtained by combining the three perspectives can determine the face orientation region and determine the current head position. It should be noted that the imaging device in the embodiment is a video camera or a camera.

[0074] Please continue to refer to Figure 6 , the offset angle of the original face image relative to the current perspective under each perspective during the detection process is recorded, in an ideal state, the yaw angles of the three perspectives will appear six states, as shown in Table 1; state 1 indicates that no effective original face image is detected by the three perspectives, states 2, 3 and 4 indicate that only one perspective detects the face image at present, the IPPG value of the current perspective is added to the IPPG signal sequence, and states 5 and 6 indicate that two perspectives detect the face image at present, when two perspectives simultaneously obtain the original face image, the smaller the offset angle, the smaller the influence of the signal fluctuation introduced on the signal, that is, the data with smaller yaw angle occupies a larger proportion, and the two signals are fused by weighted average. Taking the blue channel as an example.

[0075]

[0076] Wherein, IPPG B (n) is the IPPG signal of the fused blue channel, Bi(n) and Bj(n) are the IPPG signals of the blue channel obtained at different viewing angles.

[0077] Table 1 state classification

[0078] 1 Cam 1 (n) = 0 Cam 2 (n) = 0 Cam 3 (n) = 0 No valid data 2 Cam 1 (n) ≠ 0 Cam 2 (n) = 0 Cam 3 (n) = 0 Add view 1 data 3 Cam 1 (n) = 0 Cam 2 (n) ≠ 0 Cam 3 (n) = 0 Add view 2 data 4 Cam 1 (n) = 0 Cam 2 (n) = 0 Cam 3 (n) ≠ 0 Add view 3 data 5 Cam 1 (n) > 0 Cam 2 (n) < 0 Cam 3 (n) = 0 Weighted average view 1 and view 2 data 6 Cam 1 (n) = 0 Cam 2 (n) > 0 Cam 3 (n) < 0 Weighted average view 2 and view 3 data

[0079] In addition to the above six yaw angle conditions of three viewing angles, due to the influence of external light and distance, the detected yaw angle may have a sudden change, thereby causing the IPPG signal to have a sudden change. For example, the yaw angles calculated by the three viewing angles at the same time, the yaw angles of two viewing angles are both positive or both negative, and the like. For the above cases, the differential method is used for determination and screening. In the process of facial movement, the yaw angle detected by one viewing angle in a very short time is continuous and does not have a large fluctuation. The difference of the yaw angles of the previous ten images and the current image is calculated, the fluctuation of the difference signals between different viewing angles is compared, and it is assumed that the yaw angle obtained at the current time by a certain viewing angle has a large fluctuation. It is considered that the yaw angle obtained at the current time by the viewing angle is caused by interference, and the data of the viewing angle at the current time is abandoned.

[0080] Please refer to Figures 7-9 , Figure 7 is a schematic diagram of the yaw angle sequence provided by the embodiment of the application, Figure 8 is a schematic diagram of the fused multi-view signals provided by the embodiment of the application, Figure 9 is a schematic diagram of the fused yaw angle sequence provided by the embodiment of the application. The embodiment records the fused IPPG signal and the yaw angle. At this time, the yaw angle sequence only needs to pay attention to the relative viewing angle offset degree, and does not need to record the positive and negative directions.

[0081] S104, filtering the fused multi-view signals and the fused yaw angle sequence to remove the noise effect, and obtaining a multi-view fusion heart rate signal.

[0082] Specifically, please refer to Figure 10 , Figure 10 is a schematic diagram of the amplified multi-view signals provided by the embodiment of the application. In the embodiment, the process of removing the noise effect includes:

[0083] Considering that the original face image is often affected by multiple noise signals, and due to the small proportion of blood volume in the blood vessel that periodically beats with the heart, in order to further remove the influence of environmental light and other factors, the signal needs to be amplified, that is, the linear combination of the three channels is amplified, the alternating current component is amplified, the direct current component is suppressed, and the linear combination of the fused R channel multi-view signal R(n), the fused B channel multi-view signal B(n) and the fused G channel multi-view signal G(n) is performed to obtain a combined signal; wherein the expression of the combined signal is:

[0084]

[0085] Wherein, S1(n) and S2(n) are the transmission signals after the combination of different channel signals, IPPG(n) is the extracted pulse wave signal, δ(S i (n)) and i=1,2 are standard deviation operators for dynamically adjusting the sum of multiple transmission signals.

[0086] Please refer to Figure 11 , as shown in the figure, Figure 11 , it is a schematic diagram of the spectrum of the multi-view signal and the yaw angle sequence provided by the embodiment of the application, the fused multi-view signal is observed, at this time the signal is still affected by motion noise, the correlation coefficient of the fused multi-view signal and the fused yaw angle sequence is calculated, and the spectrum between the two signals is observed, it can be obtained that in the same frequency range, the energy distribution of the yaw angle is much larger than that of the IPPG signal, and the two signals have correlation, and the yaw angle signal can be considered as motion noise.

[0087] The combined signal is wavelet decomposed to obtain different first sub-segment signals, the fused yaw angle sequence is wavelet decomposed to obtain second sub-segment signals, the standard deviation of each second sub-segment signal is obtained, and the standard deviation is taken as a threshold value; it is judged whether each first sub-segment signal includes a heart rate band, for the first sub-segment signal not including the heart rate band, the wavelet coefficient thereof is directly set to 0, and for the first sub-segment signal including the heart rate band, the threshold value is used for soft threshold denoising.

[0088] It can be understood that the IPPG signal is the most non-stationary signal, and the wavelet denoising is more suitable for processing of non-stationary signals, the db4 wavelet is selected to perform 5-layer decomposition on the signal and the noise, the signal is decomposed into different sub-segments by wavelet decomposition, the standard deviation of each sub-segment after the decomposition of the yaw angle is calculated, the standard deviation is taken as a threshold value, for the sub-segment not containing the heart rate band, the wavelet coefficient thereof is directly set to 0, and for the sub-segment containing the heart rate band, the threshold value obtained by calculation is selected for soft threshold denoising.

[0089] Please refer to Figure 12 , as shown in the figure, Figure 12An example of the reconstructed multi-view signal provided by the embodiment of the present application is shown in the figure, wherein each sub-segment signal after the de-noising processing is reconstructed, the low-frequency component is retained, and the high-frequency component is removed.

[0090] The reconstructed signal is subjected to band-pass filtering, the interference of the low-frequency noise is removed, and the multi-view fusion signal after the de-noising is obtained.

[0091] It should be noted that the signal obtained after the adaptive threshold wavelet de-noising and the band-pass filtering is the IPPG signal more prominent in the heart rate signal, and can be regarded as the heart rate signal to some extent.

[0092] Referring to Figure 13 and Figure 14 , Figure 13 An example of the multi-view signal after the wavelet de-noising provided by the embodiment of the present application is shown in the figure, Figure 14 An example of the multi-view signal after the band-pass filtering provided by the embodiment of the present application is shown in the figure, by observing the spectrum of the multi-view signal after the adaptive threshold wavelet de-noising, it can be concluded that the signal still contains the low-frequency component, wherein the heart rate is generally 40bpm-240bpm, between 0.7Hz and 4Hz, in order to accurately extract the heart rate band, the signal needs to be subjected to the band-pass filtering to remove the interference of the low-frequency noise. In order to avoid the distortion of the signal, the zero-phase band-pass filtering is selected in the embodiment, the Butterworth band-pass filter with the passband of 0.7Hz to 4Hz is selected to realize the filtering of the signal, and the component synchronized with the heartbeat is extracted.

[0093] In S105, the multi-view fusion heart rate signal is subjected to the time-frequency conversion, and the heart rate value is obtained according to the peak value in the frequency domain after the conversion.

[0094] Specifically, referring to Figure 15 , Figure 15 An example of the spectrum of the heart rate band provided by the embodiment of the present application is shown in the figure, in the embodiment, the multi-view fusion heart rate signal is converted from the time domain space to the frequency domain space, the abscissa corresponding to the peak value in the frequency domain is obtained, and the heart rate value is obtained by multiplying 60.

[0095] The abscissa corresponding to the peak value in the frequency band is 1.39Hz, and the heart rate value at this time is calculated as 84. In this way, by using the sliding window, when 60 frames of effective data are recaptured in one of the views, the heart rate value of the next time is recalculated; the heart rate value basically does not have a large error in a short time, and therefore, it should be noted that when the calculated heart rate value has a large error with the heart rate value of the last time, the error is more than twice, the heart rate value of the last time is output.

[0096] In summary, the embodiment provides a non-contact heart rate detection method based on multi-view fusion, which acquires original facial images of a user in all directions and completely through the distributed positions of imaging devices; pre-processes the original facial images to obtain an IPPG signal and a yaw angle sequence; fuses the IPPG signal and the yaw angle sequence based on the multi-view fusion method provided by the application to obtain a fused multi-view signal; analyzes the total yaw angle in the motion process based on wavelet denoising of an adaptive threshold, regards the yaw angle as motion noise, decomposes the signal and the noise respectively, finds a noise field of a sub-section containing a heart rate frequency band, calculates a standard deviation thereof as a threshold for filtering; performs time-frequency conversion on the denoised multi-view fusion signal, and obtains a heart rate value according to a frequency domain peak value after the conversion; in this way, the problem that the user's head needs to be restrained during the monitoring process of the imaging device is solved, the user can calculate the heart rate during facial motion, noise interference and image loss caused by facial motion are avoided, the universality of the non-contact system is improved, the application scenario is closer to the actual situation, and the popularization and application of the system for monitoring the heart rate by using the imaging device are promoted.

[0097] Based on the same inventive concept, the application further provides a non-contact heart rate detection device based on multi-view fusion, which is applied to the non-contact heart rate detection method based on multi-view fusion provided by the above-mentioned embodiment of the application, and specific embodiments are referred to the above-mentioned embodiment, which will not be described here again. The device comprises:

[0098] A signal acquisition module is configured to acquire multi-view original facial images; wherein the multi-view original facial images are acquired by imaging devices;

[0099] A signal processing module one is configured to determine a region of interest according to the original facial images, and acquire an IPPG signal and a yaw angle sequence from the region of interest;

[0100] A signal fusion module is configured to perform weighted fusion on the IPPG signals and the yaw angle sequences of different views to obtain a fused multi-view signal and a fused yaw angle sequence;

[0101] A signal processing module two is configured to perform filtering processing on the fused multi-view signal and the fused yaw angle sequence to remove noise effects, and obtain a multi-view fusion heart rate signal;

[0102] A parameter calculation module is configured to perform time-frequency conversion on the multi-view fusion heart rate signal, and obtain a heart rate value according to a frequency domain peak value after the conversion.

[0103] It is to be understood that the terminology used herein such as first and second, and the like, is only intended to distinguish between one

[0104] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate different embodiments or examples described in the present specification.

[0105] The above is a further detailed description of the present application in conjunction with specific preferred embodiments, and it cannot be considered that the specific implementation of the present application is limited to these descriptions. For those skilled in the art, a number of simple deductions or replacements can be made without departing from the concept of the present application, which should be considered as falling within the scope of protection of the present application.

Claims

1. A non-contact heart rate detection method based on multi-view fusion, characterized in that, The method comprises the following steps: obtaining a plurality of original facial images from different perspectives; wherein the original facial images are obtained by imaging devices; determining a region of interest from the original facial images, and obtaining an IPPG signal and a yaw angle sequence from the region of interest; further comprising: limiting the yaw angle corresponding to the original facial image under the current perspective, that is, if the yaw angle corresponding to the original facial image under the current perspective is greater than 35°, the original facial image is abandoned, and the original facial image obtained under the adjacent perspective is taken as a valid frame, and the yaw angle sequence corresponding to the original image is detected; weighting and fusing the IPPG signals and the yaw angle sequences from different perspectives to obtain a fused multi-perspective signal and a fused yaw angle sequence; filtering the fused multi-perspective signal and the fused yaw angle sequence to remove noise effects, and obtaining a multi-perspective fusion heart rate signal; the filtering process comprises the following steps: linearly combining the fused R channel multi-view signals , the fused B channel multi-view signals and the fused G channel multi-view signals to obtain a combined signal; wherein the expression of the combined signal is: ； wherein, and are transmission signals after combination of different channel signals, respectively, is an extracted pulse wave signal, , is a standard deviation operator for dynamically adjusting the sum of the plurality of transmission signals. wavelet-decomposing the combined signal to obtain different first sub-segment signals; wavelet-decomposing the fused yaw angle sequence to obtain second sub-segment signals, obtaining the standard deviation of each second sub-segment signal, and taking the standard deviation as a threshold; determining whether each first sub-segment signal includes a heart rate band, directly setting the wavelet coefficient of the first sub-segment signal not including the heart rate band to 0, and using the threshold for soft threshold denoising of the first sub-segment signal including the heart rate band; reconstructing each sub-segment signal after denoising to retain low-frequency components and remove high-frequency components; band-pass filtering the reconstructed signal to remove low-frequency noise interference, and obtaining the denoised multi-perspective fusion heart rate signal; performing time-frequency conversion on the multi-perspective fusion heart rate signal, and obtaining a heart rate value according to the converted frequency domain peak value. 2.The multi-view fusion based non-contact heart rate detection method of claim 1, wherein, Before the step of obtaining the original facial images, the method further comprises the following steps: arranging a plurality of imaging devices in a ring array in front of a user's face, and the distance between each imaging device and the user's face is the same; wherein the imaging devices are arranged in front of the user's face, left front of the user's face, and right front of the user's face, respectively, and the included angle between adjacent imaging devices is 60°; performing distortion correction on the plurality of imaging devices, and setting the parameters of different imaging devices to be the same. 3.The multi-view fusion based non-contact heart rate detection method of claim 1, wherein, The IPPG signal acquisition process comprises the following steps: detecting feature points in the original facial image, and determining a region of interest according to the coordinates of each feature point; obtaining the average value of the RGB channel pixels in the region of interest of the original facial image; According to the pixel average value of the RGB channel, an IPPG signal corresponding to the RGB channel is acquired; wherein the IPPG signal The expression is: ； wherein, is a frame sequence, is a height of a region of interest, is a width of a region of interest, is a different view angle, is a channel of R, G and B three different colors, is a pixel value of a coordinate in a region of interest of a channel in a frame original image of a th view angle, , th view angle. 4.The multi-view fusion based non-contact heart rate detection method of claim 1, wherein, The yaw angle sequence acquisition process comprises the following steps: Detect feature points in the original face image, map the feature points of the original face image to a preset three-dimensional face model, and obtain a rotation matrix The expression is: ； wherein is the rotation angle about the X axis, is the rotation angle about the Y axis, is the rotation angle about the Z axis; According to the rotation matrix, a yaw angle of the head left and right rotation is obtained, and the yaw angle The expression is: ； ； ； ； wherein is the frame original image in the frame original image in the yaw angle of the imaging device, and a yaw angle sequence is obtained according to the yaw angle. 5.The multi-view fusion based non-contact heart rate detection method of claim 1, wherein, Before weighting and fusing the IPPG signals and the yaw angle sequences from different perspectives, the method further comprises the following steps: eliminating the trend of the IPPG signal in time using piecewise polynomial fitting; normalizing the IPPG signal after eliminating the trend to obtain a normalized IPPG signal.

6. The multi-view fusion based non-contact heart rate detection method of claim 5, wherein, The process of weighting and fusing the IPPG signals and the yaw angle sequences from different perspectives to obtain a fused multi-perspective signal comprises the following steps: The normalized IPPG signal is weighted and fused according to the yaw angle sequence to obtain a fused multi-view signal; wherein the expression of the fused multi-view signal is: ； wherein, , are the pixel mean values of different channels of RGB respectively, or , are different view angles respectively, is the yaw angle of the first frame image in the view angle, the fused multi-view signal of R channel is denoted as , the fused multi-view signal of B channel is denoted as , the fused multi-view signal of G channel is denoted as . 7.The multi-view fusion based non-contact heart rate detection method of claim 1, wherein, The process of time-frequency conversion of the multi-view fusion heart rate signal and obtaining a heart rate value according to a converted frequency domain peak value comprises: The denoised multi-view fusion signal is converted from a time domain space to a frequency domain space, a horizontal coordinate corresponding to a frequency domain peak value is obtained, and a heart rate value is obtained by multiplying 60.

8. A non-contact heart rate detection device based on multi-view fusion, characterized in that, Comprise: The signal acquisition module is configured to acquire multi-view original facial images, wherein the multi-view original facial images are acquired by an imaging device. The signal processing module one is configured to determine a region of interest according to the original facial images and acquire an IPPG signal and a yaw angle sequence from the region of interest, and further comprises limiting a yaw angle corresponding to an original facial image under a current view, i.e., if the yaw angle corresponding to the original facial image under the current view is greater than 35°, the original facial image is abandoned, and an original facial image acquired at an adjacent view is used as an effective frame to detect a yaw angle sequence corresponding to the original image. The signal fusion module is configured to perform weighted fusion on the IPPG signals and the yaw angle sequences under different views to acquire a fused multi-view signal and a fused yaw angle sequence. The signal processing module two is configured to perform filtering processing on the fused multi-view signal and the fused yaw angle sequence to remove noise effects and obtain a multi-view fusion heart rate signal. linearly combining the fused R channel multi-view signals , the fused B channel multi-view signals and the fused G channel multi-view signals to obtain a combined signal; wherein the expression of the combined signal is: ； wherein, and are the transmission signals after combination of different channel signals, respectively, is the extracted pulse wave signal, , is a standard deviation operator for dynamically adjusting the sum of the plurality of transmission signals. The process of filtering processing on the fused multi-view signal to remove noise effects comprises: Wavelet decomposition is performed on the combined signal to obtain different first subsegment signals, wavelet decomposition is performed on the fused yaw angle sequence to obtain second subsegment signals, a standard deviation of each second subsegment signal is obtained and used as a threshold value, it is determined whether each first subsegment signal includes a heart rate frequency band, for a first subsegment signal not including the heart rate frequency band, a wavelet coefficient thereof is directly set to 0, and for a first subsegment signal including the heart rate frequency band, soft threshold denoising is performed on the first subsegment signal using the threshold value. Each subsegment signal after denoising processing is reconstructed to retain low-frequency components and remove high-frequency components. Band-pass filtering is performed on the reconstructed signal to remove interference of low-frequency noise and obtain the denoised multi-view fusion heart rate signal. The parameter calculation module is configured to perform time-frequency conversion on the multi-view fusion heart rate signal and obtain a heart rate value according to a converted frequency domain peak value.

Images