Non-contact respiratory rate monitoring method, device and storage medium based on NIR video stream in complex scenes

Abstract

The present invention relates to a non-contact respiratory rate monitoring method, device, and storage medium based on NIR video streams in complex scenarios, wherein the method comprises: acquiring a near-infrared image sequence; extracting facial key points based on the acquired near-infrared image sequence, and determining a face detection frame area and facial posture based on the extracted facial key points; determining multiple chest cavity candidate regions based on the face detection frame area and facial posture, combined with pre-configured bias information; acquiring the optical flow intensity of each chest cavity candidate region, and determining the chest cavity region based on the optical flow intensity of each chest cavity candidate region; obtaining an optical flow signal of the chest cavity region based on the near-infrared image sequence; and obtaining the respiratory rate based on the obtained optical flow signal of the chest cavity region through filtering and wavelet transform correlation frequency selection. Compared with the existing technology, the present invention has the advantages of wide application range and high accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of non-contact respiratory rate monitoring, and in particular to a non-contact respiratory rate monitoring method, device and storage medium based on NIR video stream in complex scenarios. Background Art

[0002] Respiration is the most fundamental physiological activity of the human body, exchanging gases with the environment to ensure normal function. Respiration directly reflects a person's metabolic activity and cardiopulmonary function. Respiratory rate is a key parameter in respiratory physiology and a sensitive indicator of respiratory disease, including acute functional respiratory disorders. Regular monitoring of respiratory rate can help detect and prevent respiratory, cardiovascular, and cerebrovascular diseases early. Therefore, it has been widely used in pulmonary function testing, cardiopulmonary coupling assessment, and intensive care, possessing significant clinical value. Traditional respiratory rate measurement devices are mostly contact-based devices, which estimate respiration by detecting physical changes caused by respiration. This method requires attaching sensors to the subject, causing physical and psychological discomfort. In recent years, computer-assisted video image processing technology has been applied in the medical field, playing an important role in clinical diagnosis and health monitoring. Video-based respiratory rate measurement is a typical application of video image processing in medicine, and is of great significance in the prevention of respiratory diseases. The video-based respiratory rate detection method can provide an all-weather, automated respiratory rate detection mode, which has more advantages than the manual detection method of medical staff and is one of the important methods for home respiratory rate detection in the future.

[0003] For example, Chinese patent CN114170201A discloses a non-contact respiratory rate detection method and system based on edge optical flow information. It obtains edge information of each area in the image through edge algorithm and algebraic masking method, and performs sparse optical flow encoding on the information; converts the image into a binary image to remove unnecessary color interference in the image, and determines the region of interest by tracking lines and mask trajectories; cuts out invalid areas by differentiating the background image, and obtains the region of interest of chest and abdominal breathing, thereby determining the human respiratory rate. Specifically, the method includes the following steps: first, smoothing and sharpening the video image, then edge extraction of the acquired video, and representation of the edges of objects in the image to obtain edge information; segmenting the edge information to obtain edge lines; using the optical flow method to calculate the optical flow value of each point on the edge line, deriving the optical flow of each point, performing threshold segmentation on the optical flow, distinguishing the foreground from the background, and obtaining the moving target area; according to the inter-frame optical flow change roadmap of the moving target area, using the inter-frame difference method, determining that the set area in the image is the respiratory movement interest area; judging the change in the size of the respiratory movement interest area, drawing the respiratory waveform based on the change in the distance between the expanded area outside the respiratory movement interest area and the center point of the area, and calculating the respiratory rate based on the peak detection of the respiratory waveform.

[0004] However, on the one hand, the region of interest for respiratory movement in this scheme should be the chest or abdomen. However, if fluffy clothing is worn or the chest and other areas are blocked, it is difficult for the edge algorithm to determine the specific location of the chest or abdomen, and thus it is impossible to calculate the respiratory rate based on the change in the distance between the outer expansion of the region of interest and the center point. In addition, in low-light or dark scenes, the edge contour information tends to be blurred, causing the performance of this method to degrade and the accuracy of respiratory rate measurement to drop seriously. Summary of the Invention

[0005] The purpose of the present invention is to provide a non-contact respiratory rate monitoring method, device and storage medium based on NIR video stream in complex scenes. It uses near-infrared image sequence as input. It has the characteristics of all-weather, no radiation, strong penetration, good biocompatibility and good night observation effect, and can adapt to a variety of complex scenes, such as insufficient light or dark scenes. In addition, the face detection frame area and face posture are used to determine multiple chest candidate areas, and the chest area can still be accurately located in complex scenes with cover or other complex scenes that affect the edge optical flow, with high accuracy.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A non-contact respiratory rate monitoring method based on NIR video stream in complex scenarios, comprising:

[0008] Acquire a near-infrared image sequence;

[0009] Based on the acquired near-infrared image sequence, facial key points are extracted, and based on the extracted facial key points, the face detection frame area and facial pose are determined;

[0010] Based on the face detection frame area and face pose, combined with pre-configured bias information, multiple chest cavity candidate regions are determined;

[0011] Obtaining the optical flow intensity of each chest cavity candidate region, and determining the chest cavity region based on the optical flow intensity of each chest cavity candidate region;

[0012] Obtain the optical flow signal of the chest area based on the near-infrared image sequence;

[0013] Based on the optical flow signal of the chest area, the respiratory rate is obtained by filtering and wavelet transform correlation frequency selection.

[0014] The facial key points include at least the midpoint of the chin and the intersection of the eyes and nose, wherein the intersection of the eyes and nose is the intersection of the line connecting the corners of the two eyes and the midline of the nose bridge;

[0015] The face pose is the face pose roll angle, specifically:

[0016] θ=arctan[(x 16 -x 51 ) / (y 16 -y 51 )]

[0017] Where: θ is the face roll angle, (x 16 ,y 16 ) is the coordinate of the midpoint of the chin, (x 51 ,y 51 ) are the coordinates of the intersection of the eyes and nose;

[0018] The bias signal includes a plurality of bias angles.

[0019] The candidate chest regions are specifically:

[0020] ch=h×α

[0021] cw=w×β

[0022] cx=x+L×sin(θ+Δθ)

[0023] cy=y+L×cos(θ+Δθ)

[0024] Where ch is the height of the chest candidate area, cw is the width of the chest candidate area, h is the height of the face frame, w is the width of the face frame, α and β are scale factors, cx is the x-coordinate value of the center point of the chest candidate area, cy is the y-coordinate value of the center point of the chest candidate area, x is the x-coordinate value of the center point of the face frame, y is the y-coordinate value of the center point of the face frame, L is the distance from the center point of the face to the center point of the chest area, and Δθ is the offset angle.

[0025] The obtaining of the optical flow intensity of each chest candidate region and determining the chest region based on the optical flow intensity of each chest candidate region is as follows: obtaining the optical flow intensity of each chest candidate region and taking the chest candidate region with the largest optical flow intensity as the chest region.

[0026] The optical flow intensity of each chest candidate region is obtained, and the chest region is determined based on the optical flow intensity of each chest candidate region, as follows:

[0027] Obtain the optical flow intensity of each chest candidate region and determine whether the optical flow intensity of any chest candidate region exceeds the pre-configured threshold intensity. If so, exit the detection and reacquire the near-infrared image sequence. Otherwise, the chest candidate region with the largest optical flow intensity is used as the chest region.

[0028] The method of obtaining an optical flow signal of the chest area based on the near-infrared image sequence includes:

[0029] Cropping the near-infrared image sequence to obtain an image sequence of the chest area;

[0030] The image sequence of the chest area is rotationally corrected and then downsampled to obtain the optical flow signal of the chest area.

[0031] The respiratory rate is obtained by wavelet transforming and filtering the optical flow signal of the chest area, including:

[0032] Based on the obtained optical flow signal of the chest area, preprocessing is performed to obtain a denoised optical flow signal;

[0033] The respiration rate frequency is obtained from the denoised optical flow signal by using the wavelet transform correlation frequency selection method.

[0034] Based on the obtained respiratory rate frequency, the respiratory rate is obtained after filtering through a Euro filter.

[0035] The method of obtaining the respiratory rate frequency point from the denoised optical flow signal by a wavelet transform correlation frequency point selection method includes:

[0036] Set multiple scale factors, and generate wavelets corresponding to each scale factor based on each scale factor combined with the mother wave function, where each scale factor corresponds to a frequency point;

[0037] The denoised optical flow signal is cropped to obtain multiple unit signals arranged in time sequence, and the multiple unit signals are spliced in time sequence to obtain multiple target signals, where the former target signal of any two adjacent target signals is the front part of the latter target signal;

[0038] Convolve each target signal with each wavelet respectively, and set the convolution value with the largest amplitude among all convolution results of the same target signal to 1, and set the other convolution values to 0;

[0039] After accumulating the convolution values corresponding to the same wavelet to obtain the accumulated values of each wavelet, the frequency point corresponding to the wavelet with the largest accumulated value is taken as the respiratory rate frequency point.

[0040] A non-contact respiratory rate monitoring device based on NIR video stream in complex scenarios includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the method as described above is implemented.

[0041] A storage medium stores a program, which implements the above method when executed.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. Using near-infrared image sequences as input, it has the characteristics of all-weather, radiation-free, strong penetration, good biocompatibility and good night observation effect, which can adapt to a variety of complex scenes, such as low-light or dark scenes. In addition, using the face detection frame area and face posture to determine multiple chest cavity candidate areas, it can accurately locate the chest cavity area with high accuracy even in complex scenes with cover or other factors that affect the edge optical flow.

[0044] 2. Added a mechanism to eliminate interference with body movement, making the positioning of the chest area more accurate.

[0045] 3. Based on dense optical flow, the corresponding respiratory optical flow signal is directly extracted in the region of interest, rather than calculating the respiratory rate indirectly through the outer expansion of the region of interest and the change in the distance between the center points. This invention is more direct and has better efficiency and robustness.

[0046] 4. Based on multi-scale time-incremental wavelet transform and determining the frequency points by accumulation, the double peak problem caused by small disturbances can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the main steps of the method of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0049] A non-contact respiratory rate monitoring method based on NIR video stream in complex scenarios, such as Figure 1 As shown, including:

[0050] Acquire a near-infrared image sequence;

[0051] Based on the acquired near-infrared image sequence, facial key points are extracted, and based on the extracted facial key points, the face detection frame area and facial pose are determined;

[0052] Based on the face detection frame area and face pose, combined with pre-configured bias information, multiple chest cavity candidate regions are determined;

[0053] Obtaining the optical flow intensity of each chest cavity candidate region, and determining the chest cavity region based on the optical flow intensity of each chest cavity candidate region;

[0054] Obtain the optical flow signal of the chest area based on the near-infrared image sequence;

[0055] Based on the optical flow signal of the chest area, the respiratory rate is obtained by filtering and wavelet transform correlation frequency selection.

[0056] In most embodiments, facial key points include at least the midpoint of the chin and the intersection of the eyes and nose. The intersection of the eyes and nose is the intersection of the line connecting the corners of the two eyes and the midline of the nose bridge, so as to correctly calculate the roll angle of the facial posture. In this embodiment, the extraction of facial key points is implemented using this module in the mainstream detection models YOLOv5 and MobileNetv2. The face detection algorithm must meet the requirements of lightweight design suitable for system terminal deployment and meet the requirements of facial target detection at arbitrary posture angles. To this end, during the neural network training process, multiple data processing processes such as image scaling, padding, translation, rotation, multi-image splicing, and noise interference are added. At the same time, the network model is cropped to reduce the computing pressure of the terminal. At the same time, after the successful detection of the facial target, 108 key points of the face at arbitrary posture angles are added to provide key information for downstream tasks.

[0057] The face pose is the face pose roll angle, specifically:

[0058] θ=arctan[(x 16 -x 51 ) / (y 16 -y 51 )]

[0059] Where: θ is the face roll angle, (x 16,y 16 ) is the coordinate of the midpoint of the chin, (x 51 ,y 51 ) is the coordinate of the intersection of the eyes and nose.

[0060] The face detection frame area is represented by a four-dimensional vector (cx, cy, w, h), and the chest candidate area is specifically:

[0061] ch=h×α

[0062] cw=w×β

[0063] cx=x+L×sin(θ+Δθ)

[0064] cy=y+L×cos(θ+Δθ)

[0065] Where ch is the height of the chest candidate area, cw is the width of the chest candidate area, h is the height of the face frame, w is the width of the face frame, α and β are scale factors, cx is the x-coordinate value of the center point of the chest candidate area, cy is the y-coordinate value of the center point of the chest candidate area, x is the x-coordinate value of the center point of the face frame, y is the y-coordinate value of the center point of the face frame, L is the distance from the center point of the face to the center point of the chest area, and Δθ is the offset angle.

[0066] In this embodiment, there are three offset angles, namely -15 degrees, 0 degrees and 15 degrees.

[0067] In addition, considering that the application scenarios of this application are mostly the care of ALS patients or patients, they are in bed in most cases, so the torso moves less, and most of the position movement is in the chest cavity. Therefore, in some embodiments, the optical flow intensity of each chest cavity candidate area is obtained, and the chest cavity area is determined based on the optical flow intensity of each chest cavity candidate area, which is: obtaining the optical flow intensity of each chest cavity candidate area, and taking the chest cavity candidate area with the largest optical flow intensity as the chest cavity area. Generally, the optical flow intensity of each chest cavity candidate area within 30 seconds can be obtained.

[0068] Of course, in some embodiments, some measures can also be implemented to address the issue of the patient's body movement. In such embodiments, the optical flow intensity of each chest candidate region is obtained, and the chest region is determined based on the optical flow intensity of each chest candidate region. The following steps are performed: the optical flow intensity of each chest candidate region is obtained, and it is determined whether the optical flow intensity of any chest candidate region exceeds a pre-configured threshold intensity. If so, the detection is exited and the near-infrared image sequence is re-acquired. Otherwise, the chest candidate region with the largest optical flow intensity is used as the chest region. In this way, when the body moves, the optical flow intensity will be particularly large, and in this case, re-detection is required.

[0069] Furthermore, in most embodiments, obtaining an optical flow signal of the chest region based on the near-infrared image sequence includes:

[0070] Cropping the near-infrared image sequence to obtain an image sequence of the chest area;

[0071] The image sequence of the chest area is rotationally corrected and then downsampled to obtain the optical flow signal of the chest area.

[0072] Downsampling can effectively solve the problem of insufficient computing power of edge computing devices.

[0073] In addition, in most embodiments, the respiratory rate is obtained based on the optical flow signal of the chest area through wavelet transformation and filtering, including:

[0074] Based on the obtained optical flow signal of the chest area, preprocessing is performed to obtain a denoised optical flow signal;

[0075] The respiration rate frequency is obtained from the denoised optical flow signal by using the wavelet transform correlation frequency selection method.

[0076] Based on the obtained respiratory rate frequency, the respiratory rate is obtained after filtering through a Euro filter.

[0077] Based on a Euro filter, it can effectively improve the followability and make the breathing rate more stable and reliable.

[0078] In this embodiment, since the denoised optical flow signal usually contains small body movements or spurs caused by other movements, lighting changes, etc., if the frequency point is directly obtained using FFT, a double peak phenomenon will occur. Therefore, the respiration rate frequency point is obtained by the denoised optical flow signal through the wavelet transform correlation frequency point selection method, including:

[0079] Set multiple scale factors, and generate wavelets corresponding to each scale factor based on each scale factor combined with the mother wave function, where each scale factor corresponds to a frequency point;

[0080] The denoised optical flow signal is cropped to obtain multiple unit signals arranged in time sequence, and the multiple unit signals are spliced in time sequence to obtain multiple target signals, where the former target signal of any two adjacent target signals is the front part of the latter target signal;

[0081] Convolve each target signal with each wavelet respectively, and set the convolution value with the largest amplitude among all convolution results of the same target signal to 1, and set the other convolution values to 0;

[0082] After accumulating the convolution values corresponding to the same wavelet to obtain the accumulated values of each wavelet, the frequency point corresponding to the wavelet with the largest accumulated value is taken as the respiratory rate frequency point.

[0083] Specifically, this embodiment uses the "Mexican hat function (mexh)" as the mother wave function, with a total of 70 frequency points, and the frequency point range is 0.05 to 0.67.

[0084] The proposed solution was verified through experiments. In addition, the proposed solution was tested on a self-collected 563IDs dataset. The test results showed a mean error (ME) of -1.21 BPM and a mean absolute error (MAE) of 1.85 BPM, demonstrating the robustness and accuracy of the proposed breathing rate solution. The proposed solution was deployed on two real devices for testing. The results are shown in Tables 1 and 2, respectively.

[0085] Table 1

[0086]

[0087]

[0088] Table 2

[0089]

[0090] It can be seen from Tables 1 and 2 that the real machine test tested the solution of this application from different angles. The results show that the respiratory rate error at different head angles is small, which proves the superiority of this application.

[0091] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A non-contact respiratory rate monitoring method based on NIR video stream in complex scenes, characterized in that: include: Acquire a near-infrared image sequence; Based on the acquired near-infrared image sequence, facial key points are extracted, and based on the extracted facial key points, the face detection frame area and facial pose are determined; Based on the face detection frame area and face pose, combined with pre-configured bias information, multiple chest cavity candidate regions are determined; Obtaining the optical flow intensity of each chest cavity candidate region, and determining the chest cavity region based on the optical flow intensity of each chest cavity candidate region; Obtain the optical flow signal of the chest area based on the near-infrared image sequence; Based on the optical flow signal of the chest area, the respiratory rate is obtained by filtering and wavelet transform correlation frequency selection; The facial key points include at least the midpoint of the chin and the intersection of the eyes and nose, wherein the intersection of the eyes and nose is the intersection of the line connecting the corners of the two eyes and the midline of the nose bridge; The face pose is the face pose roll angle, specifically: θ =arctan[( x 16 - x 51 ) / ( y 16 - y 51 )] in: θ is the face pose roll angle, ( x 16 , y 16 ) is the coordinate of the midpoint of the chin, ( x 51 , y 51 ) are the coordinates of the intersection of the eyes and nose; The bias signal includes a plurality of bias angles; The candidate chest regions are specifically: ch = h × α cw = w × β cx = x + L ×sin( θ +D θ ) cy = y + L ×cos( θ +D θ ) in, ch is the height of the thoracic candidate area, cw is the width of the chest candidate area, h is the face frame height, w is the face frame width, α and β is the scale factor, cx is the x-coordinate value of the center point of the chest candidate area, cy is the y coordinate value of the center point of the chest candidate area, x is the x-coordinate value of the center point of the face frame, y is the y coordinate of the center point of the face frame, L is the distance from the center of the face to the center of the chest area, Δ θ is the offset angle; The respiratory rate is obtained by filtering and wavelet transforming the optical flow signal of the chest area and selecting the correlation frequency point, including: Based on the obtained optical flow signal of the chest area, preprocessing is performed to obtain a denoised optical flow signal; The respiration rate frequency is obtained from the denoised optical flow signal by using the wavelet transform correlation frequency selection method. Based on the obtained respiratory rate frequency, the respiratory rate is obtained after filtering through a Euro filter.

2. The non-contact respiratory rate monitoring method based on NIR video stream in complex scenes according to claim 1 is characterized in that: The obtaining of the optical flow intensity of each chest candidate region and determining the chest region based on the optical flow intensity of each chest candidate region is as follows: obtaining the optical flow intensity of each chest candidate region and taking the chest candidate region with the largest optical flow intensity as the chest region.

3. The non-contact respiratory rate monitoring method based on NIR video stream in a complex scene according to claim 1 is characterized in that: The optical flow intensity of each chest candidate region is obtained, and the chest region is determined based on the optical flow intensity of each chest candidate region, as follows: Obtain the optical flow intensity of each chest candidate region and determine whether the optical flow intensity of any chest candidate region exceeds the pre-configured threshold intensity. If so, exit the detection and reacquire the near-infrared image sequence. Otherwise, the chest candidate region with the largest optical flow intensity is used as the chest region.

4. The non-contact respiratory rate monitoring method based on NIR video stream in complex scenes according to claim 1 is characterized in that: The method of obtaining an optical flow signal of the chest area based on the near-infrared image sequence includes: Cropping the near-infrared image sequence to obtain an image sequence of the chest area; The image sequence of the chest area is rotationally corrected and then downsampled to obtain the optical flow signal of the chest area.

5. The non-contact respiratory rate monitoring method based on NIR video stream in complex scenes according to claim 1 is characterized in that: The method of obtaining the respiratory rate frequency point from the denoised optical flow signal by a wavelet transform correlation frequency point selection method includes: Set multiple scale factors, and generate wavelets corresponding to each scale factor based on each scale factor combined with the mother wave function, where each scale factor corresponds to a frequency point; The denoised optical flow signal is cropped to obtain multiple unit signals arranged in time sequence, and the multiple unit signals are spliced in time sequence to obtain multiple target signals, where the former target signal of any two adjacent target signals is the front part of the latter target signal; Convolve each target signal with each wavelet respectively, and set the convolution value with the largest amplitude among all convolution results of the same target signal to 1, and set the other convolution values to 0; After accumulating the convolution values corresponding to the same wavelet to obtain the accumulated values of each wavelet, the frequency point corresponding to the wavelet with the largest accumulated value is taken as the respiratory rate frequency point.

6. A non-contact respiratory rate monitoring device based on NIR video stream in complex scenarios, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

7. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 5 is implemented.

Images