Non-contact heart rate monitoring method, device and medium based on NIR video stream

Abstract

The present invention relates to a non-contact heart rate monitoring method, device, and storage medium based on NIR video streams. The method comprises: acquiring a near-infrared image sequence and extracting facial key points based on the acquired near-infrared image sequence; determining one or more target detection areas on the face based on the extracted facial key points; cropping the near-infrared image sequence to obtain image sequences of each target detection area and performing rotation correction; fusing the image sequences of all the rotation-corrected target detection areas to obtain a one-dimensional rPPG signal; preprocessing the obtained one-dimensional rPPG signal to obtain a target rPPG signal; and obtaining the heart rate of the target rPPG signal by selecting frequency points based on wavelet transform correlation and removing singular values. Compared with existing technologies, the present invention has the advantages of high speed and high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of non-contact heart rate monitoring, and in particular to a non-contact heart rate monitoring method, device and medium based on NIR video stream. Background Art

[0002] Heart rate, as one of the most important vital signs, reflects a person's health status in real time. Accurate heart rate monitoring is crucial for diagnosis, treatment, and care. For disabled patients, declining physical function often prevents them from promptly identifying any health issues. Therefore, continuous heart rate monitoring can help them identify these issues promptly and take appropriate intervention measures. Currently, the most advanced high-precision heart rate measurement methods include electrocardiography, pulse oximetry, and photoplethysmography. Although non-invasive, these methods are inconvenient and can cause discomfort because they acquire ECG and PPG signals through electrodes or contact sensors. Remote photoplethysmography (rPPG) allows for remote heart rate measurement. This non-contact monitoring technology uses real-time image analysis of blood volume pulse (BVP) signals to determine heart rate (HR) and heart rate variability. Therefore, rPPG plays a crucial role in assessing a person's psychological and physiological state and holds great promise for medical applications.

[0003] For example, Chinese patent CN116758066A discloses a non-contact heart rate measurement method, device, and medium. The method uses a visible light camera to obtain a viewer's ROI video stream, obtains a first frame image and initializes m tracking points, determines the RGB three-channel mean within a preset range of the m tracking points, and accumulates the three-channel mean values ​​of the m tracking points. When the accumulation time t reaches a preset time threshold, the accumulated RGB three-channel mean sequence of the m tracking points is converted into m ppg signals, the m ppg signals are formed into a ppg matrix P, an autocorrelation matrix is ​​determined based on the ppg matrix P, and heart rate characteristic coefficients of the m tracking points are determined based on the autocorrelation matrix. The ppg signal of the tracking point with the largest heart rate characteristic coefficient among the m heart rate characteristic coefficients within a time period t is extracted. The maximum value is determined using the ppg signal within the time period t, and the viewer's heart rate is then calculated as the maximum value*60, in an attempt to significantly reduce the cost of visually measuring heart rate.

[0004] However, the above method first obtains the ROI, performs m-point tracking based on the first frame image, obtains m PPG signals to construct the autocorrelation matrix, and then calculates the heart rate value. This method still has some defects. First, the heart rate signal based on the video stream is relatively weak. Performing m-point tracking based only on the first frame image can easily introduce other noise and even drown out the heart rate signal. Second, the m-point tracking is highly complex and consumes a lot of resources. Finally, the power spectrum or Fourier transform is prone to produce a "double peak phenomenon", which in turn causes phenomena such as heart rate value jumps. Summary of the Invention

[0005] The purpose of the present invention is to provide a non-contact heart rate monitoring method, device and medium based on NIR video stream. The near-infrared image sequence is used as input. It has the characteristics of all-weather, no radiation, strong penetration, good biocompatibility and good night observation effect, and can adapt to a variety of complex scenes, such as insufficient light or dark scenes. In addition, the facial key point detection is used to obtain one or more target detection areas on the face, and then the rPPG signal is extracted, which can improve adaptability and accuracy.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A non-contact heart rate monitoring method based on NIR video stream, comprising:

[0008] Acquire a near-infrared image sequence, and extract facial key points based on the acquired near-infrared image sequence;

[0009] Based on the extracted facial key points, determine one or more target detection areas on the face;

[0010] Based on the near-infrared image sequence, the image sequence of each target detection area is cropped and rotated and corrected;

[0011] Based on the image sequences of all target detection areas after rotation correction, the one-dimensional rPPG signal is fused;

[0012] Preprocessing the obtained one-dimensional rPPG signal to obtain a target rPPG signal;

[0013] For the target rPPG signal, the heart rate is obtained by selecting the frequency points based on wavelet transform correlation and removing the singular values.

[0014] There are four target detection areas, namely the forehead area, the left cheek area, the right cheek area and the jaw area.

[0015] The process of determining the forehead area includes:

[0016] Determine the roll angle of the face based on the two key points between the eyebrows, as well as the distance between the two key points between the eyebrows:

[0017] i =arctan[( y 38 - y 37 ) / ( x 38 - x 37 )]

[0018]

[0019] in: i For human face roll angle, x 38 is the horizontal coordinate of the key point on the left eyebrow, y 38 is the vertical coordinate of the key point on the left eyebrow, x 38 is the horizontal coordinate of the key point on the right eyebrow, y 38 is the vertical coordinate of the key point on the right eyebrow, fhw is the distance between the two key points on the eyebrow,

[0020] Based on the key point on the left eyebrow, the distance between the two key points on the eyebrow is collected to obtain the coordinates of the third corner of the forehead:

[0021] x p1-3 = x 38 + fhw×sin i

[0022] y p1-3 = y 38 -fhw×cos i

[0023] in: x p1-3 is the horizontal coordinate of the third corner of the forehead, y p1-3 is the vertical coordinate of the third corner of the forehead,

[0024] Based on the key point on the right eyebrow, the coordinates of the fourth corner of the forehead are obtained by combining the distance between the two key points on the eyebrow:

[0025] x p1-4 = x 37 + fhw×sin i

[0026] y p1-4 =y 37 - fhw×cos i

[0027] in: x p1-4 is the horizontal coordinate of the fourth corner of the forehead, y p1-4 is the vertical coordinate of the fourth corner of the forehead,

[0028] The forehead area is defined as the area consisting of the two upper glabellar key points, the third corner point of the forehead, and the fourth corner point of the forehead as four vertices;

[0029] The process of determining the right cheek area includes:

[0030] Determine the width of the right cheek area based on the right eyebrow lower key point and the right eyebrow corner key point:

[0031]

[0032] Where: cwk1 is the width of the right cheek area, x 67 is the horizontal coordinate of the key point below the right eyebrow, y 67 is the vertical coordinate of the key point below the right eyebrow, x 33 is the horizontal coordinate of the key point of the right eyebrow corner, y 33 is the vertical coordinate of the key point of the right eyebrow corner,

[0033] Determine the height of the cheek area based on the nose bridge feature points and nose tip key points:

[0034]

[0035] Among them: cwh1 is the height of the cheek area, x 46 is the horizontal coordinate of the nose tip key point, y 46 is the vertical coordinate of the nose tip key point, x 44 is the horizontal coordinate of the nose bridge feature point, y 44 is the vertical coordinate of the nose bridge feature point, which is the intersection of the nose bridge axis and the line connecting the bottoms of the two eye bags;

[0036] Based on the obtained width and height of the right cheek area and the face roll angle, the four corner points of the right cheek area are determined;

[0037] The area formed by the four corner points of the right cheek area as four vertices is taken as the right cheek area;

[0038] The process of determining the left cheek area includes:

[0039] Determine the width of the left cheek area based on the left eyebrow lower keypoint and the left eyebrow corner keypoint:

[0040]

[0041] Where: cwk2 is the width of the left cheek area, x 68 is the horizontal coordinate of the key point below the left eyebrow, y 68 is the vertical coordinate of the key point below the left eyebrow, x 42 is the horizontal coordinate of the left eyebrow corner key point, y 42 is the vertical coordinate of the left eyebrow corner key point,

[0042] Based on the obtained width and height of the left cheek area and the face roll angle, the four corner points of the left cheek area are determined;

[0043] The area formed by the four corner points of the left cheek area as the four vertices is taken as the left cheek area;

[0044] The process of determining the mandibular area includes:

[0045] Get the coordinates of the left and right chin feature points, where the left and right chin feature points are the intersection points of the lower edges of the two mouth corners and the facial contour;

[0046] Determine the width of the mandibular region based on the coordinates of the left chin feature point and the right chin feature point;

[0047] Based on the coordinates of the midpoint of the lower lip, combined with the roll angle of the face and the width of the mandibular area, the third and fourth corners of the mandibular area are determined:

[0048] x p4-3 = x 93 -cnw / 2×cos i

[0049] y p4-3 = y 93 -cnw / 2×sin i

[0050] x p4-4 = x 93 +cnw / 2×cos i

[0051] y p4-4 = y 93 +cnw / 2×sin i

[0052] in: x p4-3 is the horizontal coordinate of the third corner of the mandible, y p4-3 is the vertical coordinate of the third corner of the mandible, cnw is the width of the mandibular area, x p4-4 is the horizontal coordinate of the fourth corner of the mandible, y p4-4 is the vertical coordinate of the fourth corner of the mandible.

[0053] The image sequences of all target detection areas after rotation correction are fused to obtain a one-dimensional rPPG signal, including:

[0054] Obtain image sequences of all target detection areas after rotation correction;

[0055] The arithmetic mean of all pixels in each target detection area is calculated in time sequence to obtain the rPPG component signal corresponding to each target area;

[0056] After detrending and standardizing the rPPG component signals corresponding to each target area, the one-dimensional rPPG signal was obtained by weighted averaging combined with the weight of each target area.

[0057] The preprocessing of the obtained one-dimensional rPPG signal includes:

[0058] The obtained one-dimensional rPPG signal is denoised based on the anomaly detection module;

[0059] The denoised one-dimensional rPPG signal is band-pass filtered to obtain the target rPPG signal.

[0060] The anomaly detection module is composed of a rolling standard deviation detector, a value range detector and a difference monitor connected in series.

[0061] The heart rate of the target rPPG signal is obtained by selecting the frequency points based on the wavelet transform correlation and removing the singular values, including:

[0062] Set multiple scale factors, and generate wavelets corresponding to each scale factor based on each scale factor combined with the mother wave function, where each scale factor corresponds to a frequency point;

[0063] The target rPPG signal is cropped to obtain multiple unit signals arranged in time sequence, and the multiple unit signals are spliced ​​in time sequence to obtain multiple target signals, wherein, of any two adjacent target signals, the former target signal is the front part of the latter target signal;

[0064] Convolve each target signal with each wavelet respectively, and set the convolution value with the largest amplitude among all convolution results of the same target signal to 1, and set the other convolution values ​​to 0;

[0065] After accumulating the convolution values ​​corresponding to the same wavelet to obtain the accumulated values ​​of each wavelet, the frequency point corresponding to the wavelet with the largest accumulated value is taken as the heart rate frequency point;

[0066] Based on the obtained heart rate frequency points, the heart rate is obtained after removing the singular values.

[0067] The process of removing the singular values ​​is specifically implemented through a Euro filter.

[0068] A non-contact heart rate monitoring device based on NIR video stream includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the method described above is implemented.

[0069] A storage medium stores a program, which implements the above method when executed.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] 1. Near-infrared image sequences are used as input. They are all-weather, radiation-free, have strong penetration, good biocompatibility, and have good nighttime observation effects. They can adapt to a variety of complex scenarios, such as low-light or dark scenes. In addition, facial key point detection is used to obtain one or more target detection areas on the face, and then the rPPG signal is extracted, which can improve adaptability and accuracy.

[0072] 2. The denoising method used can effectively remove the outliers generated by non-rigid and rigid instantaneous motion, so that the heart rate values ​​in these complex scenarios can be accurately calculated.

[0073] 3. Based on multi-scale time-incremental wavelet transform and determining the frequency points by accumulation, the double peak problem caused by small disturbances can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 Schematic diagram of the main process of the method of the present invention. DETAILED DESCRIPTION

[0075] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0076] A non-contact heart rate monitoring method based on NIR video stream, such as Figure 1 Shown, including:

[0077] (1) obtaining a near-infrared image sequence, and extracting facial key points based on the obtained near-infrared image sequence;

[0078] Specifically, for near-infrared face detection and key point related technologies, the mainstream detection models YOLOv5 and MobileNetv2 algorithms are fine-tuned. For the target scenario of contactless care, the face detection algorithm must meet the requirements of lightweight design suitable for system terminal deployment and meet the requirements of face target detection at any posture angle. To this end, during the neural network training process, multiple data processing projects such as image scaling, padding, translation, rotation, multi-image splicing and noise interference are added. At the same time, the network model is cropped to reduce the computing pressure of the terminal. In some embodiments, after the face target is successfully detected, 108 key points of the face at any posture angle are added to provide key information for downstream tasks. Of course, in other embodiments, other key point detection models can also be used.

[0079] (2) determining one or more target detection areas on the face based on the extracted facial key points;

[0080] There are four target detection areas, namely the forehead area, the left cheek area, the right cheek area and the jaw area.

[0081] The process of determining the forehead area includes:

[0082] Determine the roll angle of the face based on the two key points between the eyebrows, as well as the distance between the two key points between the eyebrows:

[0083] i =arctan[( y 38 - y 37 ) / ( x 38 - x 37 )]

[0084]

[0085] in: i For human face roll angle, x 38is the horizontal coordinate of the key point on the left eyebrow, y 38 is the vertical coordinate of the key point on the left eyebrow, x 38 is the horizontal coordinate of the key point on the right eyebrow, y 38 is the vertical coordinate of the key point on the right eyebrow, fhw is the distance between the two key points on the eyebrow,

[0086] Based on the key point on the left eyebrow, the distance between the two key points on the eyebrow is collected to obtain the coordinates of the third corner of the forehead:

[0087] x p1-3 = x 38 + fhw×sin i

[0088] y p1-3 = y 38 -fhw×cos i

[0089] in: x p1-3 is the horizontal coordinate of the third corner of the forehead, y p1-3 is the vertical coordinate of the third corner of the forehead,

[0090] Based on the key point on the right eyebrow, the coordinates of the fourth corner of the forehead are obtained by combining the distance between the two key points on the eyebrow:

[0091] x p1-4 = x 37 + fhw×sin i

[0092] y p1-4 = y 37 - fhw×cos i

[0093] in: x p1-4 is the horizontal coordinate of the fourth corner of the forehead, y p1-4 is the vertical coordinate of the fourth corner of the forehead,

[0094] The forehead area is defined as the area consisting of the two upper glabellar key points, the third corner point of the forehead, and the fourth corner point of the forehead as four vertices;

[0095] The process of determining the right cheek area includes:

[0096] Determine the width of the right cheek area based on the right eyebrow lower key point and the right eyebrow corner key point:

[0097]

[0098] Where: cwk1 is the width of the right cheek area, x 67 is the horizontal coordinate of the key point below the right eyebrow, y 67 is the vertical coordinate of the key point below the right eyebrow, x 33 is the horizontal coordinate of the key point of the right eyebrow corner, y 33 is the vertical coordinate of the key point of the right eyebrow corner,

[0099] Determine the height of the cheek area based on the nose bridge feature points and nose tip key points:

[0100]

[0101] Among them: cwh1 is the height of the cheek area, x 46 is the horizontal coordinate of the nose tip key point, y 46 is the vertical coordinate of the nose tip key point, x 44 is the horizontal coordinate of the nose bridge feature point, y 44 is the vertical coordinate of the nose bridge feature point, which is the intersection of the nose bridge axis and the line connecting the bottoms of the two eye bags;

[0102] Based on the obtained width and height of the right cheek area, the four corner points of the right cheek area are determined in combination with the roll angle of the face. In some embodiments, this process can specifically include drawing parallel lines through the right lower eyebrow key point and the right eyebrow corner key point, connecting the nose bridge feature point and the nose tip key point, and drawing parallel lines through the bridge feature point and the nose tip key point, connecting the right lower eyebrow key point and the right eyebrow corner key point, and using the four intersection points as the four corner points of the right cheek area.

[0103] Alternatively, in other embodiments, the intersection point between the outer side of the right nostril contour and the cheek is used as one of the intersection points, and the width and height of the right cheek area are combined to obtain the remaining three corner points using the face roll angle.

[0104] The area formed by the four corner points of the right cheek area as four vertices is taken as the right cheek area;

[0105] The process of determining the left cheek area is similar to that of the right cheek area, including:

[0106] Determine the width of the left cheek area based on the left eyebrow lower keypoint and the left eyebrow corner keypoint:

[0107]

[0108] Where: cwk2 is the width of the left cheek area, x 68 is the horizontal coordinate of the key point below the left eyebrow, y 68 is the vertical coordinate of the key point below the left eyebrow, x 42 is the horizontal coordinate of the left eyebrow corner key point, y 42 is the vertical coordinate of the left eyebrow corner key point,

[0109] Based on the obtained width and height of the left cheek area and the face roll angle, the four corner points of the left cheek area are determined;

[0110] The area formed by the four corner points of the left cheek area as the four vertices is taken as the left cheek area;

[0111] The process of determining the mandibular area includes:

[0112] Get the coordinates of the left and right chin feature points, where the left and right chin feature points are the intersection points of the lower edges of the two mouth corners and the facial contour;

[0113] Determine the width of the mandibular region based on the coordinates of the left chin feature point and the right chin feature point;

[0114] Based on the coordinates of the midpoint of the lower lip, combined with the roll angle of the face and the width of the mandibular area, the third and fourth corners of the mandibular area are determined:

[0115] x p4-3 = x 93 -cnw / 2×cos i

[0116] y p4-3 = y 93 -cnw / 2×sin i

[0117] x p4-4 = x 93 +cnw / 2×cos i

[0118] y p4-4 = y93 +cnw / 2×sin i

[0119] in: x p4-3 is the horizontal coordinate of the third corner of the mandible, y p4-3 is the vertical coordinate of the third corner of the mandible, cnw is the width of the mandibular area, x p4-4 is the horizontal coordinate of the fourth corner of the mandible, y p4-4 is the vertical coordinate of the fourth corner of the mandible.

[0120] In this way, the obtained target detection area has stronger anti-distortion ability.

[0121] (3) Based on the near-infrared image sequence, crop the image sequence of each target detection area and perform rotation correction;

[0122] (IV) Based on the image sequences of all target detection areas after rotation correction, a one-dimensional rPPG signal is obtained by fusing, specifically, including:

[0123] Obtain image sequences of all target detection areas after rotation correction;

[0124] The arithmetic mean of all pixels in each target detection area is calculated in time sequence to obtain the rPPG component signal corresponding to each target area;

[0125] After detrending and standardizing the rPPG component signals corresponding to each target area, a one-dimensional rPPG signal is obtained by weighted averaging in combination with the weight of each target area. In some embodiments, a larger weight can be assigned to the left cheek area and the right cheek area. Of course, in some embodiments, the weight of each target area can be equal.

[0126] This one-dimensional rPPG signal often contains spikes caused by small body movements, other motions, and lighting changes. Using an FFT to directly determine the frequency components can produce a double peak, meaning two or more main frequency components. Over time, the heart rate frequency can jump between these main frequency components, resulting in a continuous jump in the heart rate value.

[0127] (5) Preprocessing the obtained one-dimensional rPPG signal to obtain a target rPPG signal, specifically including:

[0128] The obtained one-dimensional rPPG signal is denoised based on the anomaly detection module;

[0129] The denoised one-dimensional rPPG signal is band-pass filtered to obtain the target rPPG signal.

[0130] The anomaly detection module consists of a rolling standard deviation detector, a value range detector and a differential monitor connected in series.

[0131] (6) For the target rPPG signal, the heart rate is obtained by selecting the frequency points based on the wavelet transform correlation and removing the singular values, including:

[0132] Set multiple scale factors, and generate wavelets corresponding to each scale factor based on each scale factor combined with the mother wave function, where each scale factor corresponds to a frequency point;

[0133] The target rPPG signal is cropped to obtain multiple unit signals arranged in time sequence, and the multiple unit signals are spliced ​​in time sequence to obtain multiple target signals, wherein, of any two adjacent target signals, the former target signal is the front part of the latter target signal;

[0134] Convolve each target signal with each wavelet respectively, and set the convolution value with the largest amplitude among all convolution results of the same target signal to 1, and set the other convolution values ​​to 0;

[0135] After accumulating the convolution values ​​corresponding to the same wavelet to obtain the accumulated values ​​of each wavelet, the frequency point corresponding to the wavelet with the largest accumulated value is taken as the heart rate frequency point;

[0136] Based on the obtained heart rate frequency points, the heart rate is obtained after removing the singular values.

[0137] The process of removing singular values ​​is specifically implemented through a Euro filter.

[0138] Specifically, in this embodiment, "morl" is used as the mother wave function, and the frequency range of the mother wave is 0.67 ~ 0.0, which includes a total of 140 frequency values, that is, the corresponding mother wave has 140 scale factors.

[0139] Through experimental verification, the heart rate test set contains 195 videos, each video is 1 minute long, and the test results are shown in Table 1.

[0140] Table 1

[0141]

[0142] As can be seen from Table 1, the heart rate monitoring effect of this application is excellent, the error is very small, and the test of this solution can be measured in real time, and the CPU test takes about 10ms.

[0143] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

Claims

1. A non-contact heart rate monitoring method based on NIR video stream, characterized in that: include: Acquire a near-infrared image sequence, and extract facial key points based on the acquired near-infrared image sequence; Based on the extracted facial key points, determine one or more target detection areas on the face; Based on the near-infrared image sequence, the image sequence of each target detection area is cropped and rotated and corrected; Based on the image sequences of all target detection areas after rotation correction, the one-dimensional rPPG signal is fused; Preprocessing the obtained one-dimensional rPPG signal to obtain a target rPPG signal; For the target rPPG signal, the heart rate is obtained by selecting the frequency points based on the wavelet transform correlation and removing the singular values; There are four target detection areas, namely the forehead area, the left cheek area, the right cheek area and the jaw area; The process of determining the forehead area includes: Determine the roll angle of the face based on the two key points between the eyebrows, as well as the distance between the two key points between the eyebrows: θ =arctan[( y 38 - y 37 ) / ( x 38 - x 37 )] in: θ For human face roll angle, x 38 is the horizontal coordinate of the key point on the left eyebrow, y 38 is the vertical coordinate of the key point on the left eyebrow, x 37 is the horizontal coordinate of the key point on the right eyebrow, y 37 is the vertical coordinate of the key point on the right eyebrow, fhw is the distance between the two key points between the eyebrows, Based on the key point on the left eyebrow, the distance between the two key points on the eyebrow is collected to obtain the coordinates of the third corner of the forehead: x p1-3 = x 38 + fhw×sin θ y p1-3 = y 38 -fhw×cos θ in: x p1-3 is the horizontal coordinate of the third corner of the forehead, y p1-3 is the vertical coordinate of the third corner of the forehead, Based on the key point on the right eyebrow, the coordinates of the fourth corner of the forehead are obtained by combining the distance between the two key points on the eyebrow: x p1-4 = x 37 + fhw×sin θ y p1-4 = y 37 - fhw×cos θ in: x p1-4 is the horizontal coordinate of the fourth corner of the forehead, y p1-4 is the vertical coordinate of the fourth corner of the forehead, The forehead area is defined as the area consisting of the two upper glabellar key points, the third corner point of the forehead, and the fourth corner point of the forehead as four vertices; The process of determining the right cheek area includes: Determine the width of the right cheek area based on the right eyebrow lower key point and the right eyebrow corner key point: in: cwk 1 is the width of the right cheek area, x 67 is the horizontal coordinate of the key point below the right eyebrow, y 67 is the vertical coordinate of the key point below the right eyebrow, x 33 is the horizontal coordinate of the key point of the right eyebrow corner, y 33 is the vertical coordinate of the key point of the right eyebrow corner, Determine the height of the cheek area based on the nose bridge feature points and nose tip key points: Among them: cwh1 is the height of the cheek area, x 46 is the horizontal coordinate of the nose tip key point, y 46 is the vertical coordinate of the nose tip key point, x 44 is the horizontal coordinate of the nose bridge feature point, y 44 is the vertical coordinate of the nose bridge feature point, which is the intersection of the nose bridge axis and the line connecting the bottoms of the two eye bags; Based on the obtained width and height of the right cheek area and the face roll angle, the four corner points of the right cheek area are determined; The area formed by the four corner points of the right cheek area as four vertices is taken as the right cheek area; The process of determining the left cheek area includes: Determine the width of the left cheek area based on the left eyebrow lower keypoint and the left eyebrow corner keypoint: Where: cwk2 is the width of the left cheek area, x 68 is the horizontal coordinate of the key point below the left eyebrow, y 68 is the vertical coordinate of the key point below the left eyebrow, x 42 is the horizontal coordinate of the left eyebrow corner key point, y 42 is the vertical coordinate of the left eyebrow corner key point, Based on the obtained width and height of the left cheek area and the face roll angle, the four corner points of the left cheek area are determined; The area formed by the four corner points of the left cheek area as the four vertices is taken as the left cheek area; The process of determining the mandibular area includes: Get the coordinates of the left and right chin feature points, where the left and right chin feature points are the intersection points of the lower edges of the two mouth corners and the facial contour; Determine the width of the mandibular region based on the coordinates of the left chin feature point and the right chin feature point; Based on the coordinates of the midpoint of the lower lip, combined with the roll angle of the face and the width of the mandibular area, the third and fourth corners of the mandibular area are determined: x p4-3 = x 93 -cnw / 2×cos θ y p4-3 = y 93 -cnw / 2×sin θ x p4-4 = x 93 +cnw / 2×cos θ y p4-4 = y 93 +cnw / 2×sin θ in: x p4-3 is the horizontal coordinate of the third corner of the mandible, y p4-3 is the vertical coordinate of the third corner of the mandible, cnw is the width of the mandibular area, x p4-4 is the horizontal coordinate of the fourth corner of the mandible, y p4-4 is the ordinate of the fourth corner of the mandible, x 93 is the horizontal coordinate of the midpoint of the lower lip, y 93 is the ordinate of the midpoint of the lower lip; The area consisting of the left chin feature point, the right chin feature point, the third corner point and the fourth corner point of the mandible as the four vertices is defined as the mandible area; The image sequences of all target detection areas after rotation correction are fused to obtain a one-dimensional rPPG signal, including: Obtain image sequences of all target detection areas after rotation correction; The arithmetic mean of all pixels in each target detection area is calculated in time sequence to obtain the rPPG component signal corresponding to each target area; After detrending and standardizing the rPPG component signals corresponding to each target area, the weighted average is combined with the weight of each target area to obtain a one-dimensional rPPG signal; The preprocessing of the obtained one-dimensional rPPG signal includes: The obtained one-dimensional rPPG signal is denoised based on the anomaly detection module; The denoised one-dimensional rPPG signal is band-pass filtered to obtain the target rPPG signal; The heart rate of the target rPPG signal is obtained by selecting the frequency points based on the wavelet transform correlation and removing the singular values, including: Set multiple scale factors, and generate wavelets corresponding to each scale factor based on each scale factor combined with the mother wave function, where each scale factor corresponds to a frequency point; The target rPPG signal is cropped to obtain multiple unit signals arranged in time sequence, and the multiple unit signals are spliced ​​in time sequence to obtain multiple target signals, wherein, of any two adjacent target signals, the former target signal is the front part of the latter target signal; Convolve each target signal with each wavelet respectively, and set the convolution value with the largest amplitude among all convolution results of the same target signal to 1, and set the other convolution values ​​to 0; After accumulating the convolution values ​​corresponding to the same wavelet to obtain the accumulated values ​​of each wavelet, the frequency point corresponding to the wavelet with the largest accumulated value is taken as the heart rate frequency point; Based on the obtained heart rate frequency points, the heart rate is obtained after removing the singular values.

2. The non-contact heart rate monitoring method based on NIR video stream according to claim 1, characterized in that: The anomaly detection module is composed of a rolling standard deviation detector, a value range detector and a difference monitor connected in series.

3. The non-contact heart rate monitoring method based on NIR video stream according to claim 1, characterized in that: The process of removing the singular values ​​is specifically implemented through a Euro filter.

4. A non-contact heart rate monitoring device based on NIR video stream, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

5. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 3 is implemented.

Images