A non-contact respiratory signal extraction method based on infrared thermal imaging video data

Abstract

The application provides a non-contact respiratory signal extraction method based on infrared thermal imaging video data, comprising the following steps: step 1, using a face detection and face key point positioning model on the original infrared video frame, and combining a sparse optical flow method to obtain the key point coordinates of each frame; step 2, obtaining a final region of interest; step 3, mapping the final region of interest back to the original infrared video frame without enhancement to obtain an original signal sequence; step 4, performing empirical mode decomposition on the original signal sequence to obtain an intrinsic mode function; according to the respiratory frequency characteristics, selecting the intrinsic mode function obtained by decomposition to reconstruct a reconstructed signal, performing band-pass filtering on the reconstructed signal to obtain a final respiratory signal; analyzing the power spectrum of the final respiratory signal, identifying the peak frequency point with the maximum amplitude, and calculating the final respiratory rate. The application has high motion robustness, and solves the common loss of lock problem in non-contact measurement through the cascade of multiple tracking mechanisms.

Description

Technical Field

[0001] This invention relates to the fields of medical signal processing and computer vision technology, and in particular to a non-contact respiratory signal extraction method based on infrared thermal imaging video data. Background Technology

[0002] Respiratory rate (RR) is one of the important indicators for monitoring clinical vital signs and has significant application value in scenarios such as intensive care, sleep monitoring, and telemedicine.

[0003] Traditional respiratory monitoring typically uses contact sensors, such as breathing straps, chest and abdominal motion sensors, or nasal cannulas. However, these methods have drawbacks, including requiring direct contact with the human body, which may cause patient discomfort; prolonged wear may lead to skin irritation or damage; and inconvenience in sleep or remote monitoring scenarios.

[0004] To address these issues, various non-contact respiratory monitoring technologies have emerged in recent years, including visible light video monitoring, depth camera monitoring, and infrared thermal imaging monitoring. Among these, infrared thermal imaging technology is considered a non-contact respiratory monitoring method with high application potential because it can directly sense the temperature distribution on the human body surface. During human respiration, the inhalation of cold air and the exhalation of warm air create periodic temperature changes near the nostrils. Therefore, respiratory signals can be extracted by analyzing the thermal infrared changes in the nostril area.

[0005] However, existing technologies still have the following problems:

[0006] 1. The target area is easily lost: Subjects may turn their heads or move slightly in a natural state, making it difficult to maintain a stable position in the nostril area.

[0007] 2. Severe background temperature interference: Changes in facial skin temperature, fluctuations in ambient temperature, and sensor drift can all cause signal baseline drift.

[0008] 3. Weak respiratory signal: The temperature change caused by respiratory airflow is small and easily drowned out by the noise of facial skin thermal radiation.

[0009] Therefore, how to achieve stable and high signal-to-noise ratio respiratory signal extraction under the presence of head movement and ambient temperature interference has become a technical problem that urgently needs to be solved in the field of infrared respiratory monitoring. Summary of the Invention

[0010] Objective: The technical problem this invention aims to solve is to address the shortcomings of existing technologies by providing a non-contact respiratory signal extraction method based on infrared thermal imaging video data. This method utilizes facial key point localization and sparse optical flow to achieve continuous respiratory signal extraction even with head movement, demonstrating robustness. Furthermore, it employs a low-temperature percentile extraction strategy and dynamic environmental benchmark compensation to achieve high-precision extraction of weak respiratory signals.

[0011] This method includes the following steps:

[0012] Step 1: Limit contrast adaptive histogram equalization (CLAHE) is applied to the original infrared video frames to increase the contrast of the infrared images. Face detection and face key point localization models are used in combination with the Lucas-Kanade sparse optical flow method to obtain the key point coordinates of each frame.

[0013] Step 2: Based on anatomical knowledge and facial key points, dynamically draw the initial region of interest (ROI) rectangle, perform CLAHE enhancement on the region within the rectangle, calculate the multimodal candidate mask, and take the intersection of the candidate masks to obtain the final ROI.

[0014] Step 3: Map the final region of interest (ROI) back to the original unenhanced infrared video frame. Within the final masked area, extract the average value of the pixels with the lowest brightness values ​​within a certain range as the signal points of the current frame. Subtract the environmental thermal compensation from the signal points to obtain the original signal sequence.

[0015] Step 4: Perform empirical mode decomposition on the original signal sequence to obtain intrinsic mode functions; select the fourth intrinsic mode function obtained from the decomposition based on the respiratory frequency characteristics. The fifth intrinsic modulus function obtained by decomposition and the sixth intrinsic modulus function obtained by decomposition The reconstructed signal is obtained by reconstructing the signal, and then bandpass filtering is applied to the reconstructed signal to obtain the final respiratory signal. The power spectrum of the final respiratory signal is analyzed by applying Fast Fourier Transform to identify the peak frequency point with the largest amplitude, and the final respiratory rate is calculated.

[0016] Step 1 includes the following steps:

[0017] Step 1-1: Use an infrared camera to collect facial thermal radiation data of the subject and output raw infrared video frames. ;

[0018] Steps 1-2: Maintain two image streams: a computation stream and a mapping stream; in the computation stream, the original infrared video frames are... Expand to the specified width to obtain a new video frame. ,right CLAHE enhancement is performed to improve image contrast; the original infrared video frames are preserved in the mapped stream. ;

[0019] Steps 1-3, for An ensemble model combining histogram of oriented gradients, support vector machines, and regression trees was used to locate 54 key physiological points on the face. Histogram of Oriented Gradients (HOG) is used as a feature extractor for face detection. The image is first divided into small cells, the gradient direction of each pixel within the cell is calculated, and a histogram of gradient directions is generated. Finally, the histograms of two or more cells are normalized to form feature vectors. Support Vector Machines (SVMs) are used as classifiers for face detection. During training, labeled face and non-face samples are used to learn a linear hyperplane to separate face and non-face samples. During detection, the HOG features extracted from the sliding window are used to determine whether a face is present. An ensemble regression tree model is used for facial keypoint localization. Keypoint positions are progressively refined through multi-level regression trees. Each level of the regression tree extracts features based on the current estimated position and predicts the positional shift. Finally, the trees are ensembled, with two or more trees voting. If a face is not detected in the current frame, resulting in keypoint detection failure, the Lucas-Kanade sparse optical flow method is used, based on the grayscale information of the previous frame. Predict the location of key points in the current frame.

[0020] In steps 1-3, the specific process of predicting key point locations using the Lucas-Kanade sparse optical flow method includes:

[0021] The formula for assuming that the brightness of the same object point in an image remains constant between adjacent frames is as follows:

[0022] ,

[0023] in, At time t, the position is pixel brightness, It is a displacement vector. It is a time interval;

[0024] Define optical flow vector The optical flow constraint equation is obtained as follows:

[0025] ,

[0026] in Let be the gradient of the image in the x-direction. It is the gradient of the image in the y-direction. It is the gradient of the image in the time direction. It is the velocity component in the x-direction. It represents the velocity component in the y-direction; d denotes the differential sign. Indicates the partial differential symbol;

[0027] Set all pixels within a local window on the image to have the same optical flow vector, set at the point There is one around A window containing n pixels. For each pixel within the window, there is an optical flow constraint equation:

[0028] ,

[0029] in Indicates the first The coordinates of each pixel The width of the window. The coordinates on the image are The pixels in gradient of direction, The coordinates on the image are The pixels in Gradient of direction;

[0030] The pixel corresponding to the point within the window can be obtained by solving the following formula. :

[0031] ,

[0032] ,

[0033] ,

[0034] Where det represents the determinant value of the matrix.

[0035] Step 2 includes the following steps:

[0036] Step 2-1, obtain the key point locations Mapping back to the original frame ,get 54 key points on the top Next, a rectangular region of interest that closely follows the nostril area is dynamically generated using the keypoint coordinates. ;

[0037] Calculate the width of the rectangle Rectangle height x-coordinate of the center point of the rectangle The ordinate of the center point of the moment The calculation formula is:

[0038] ,

[0039] ,

[0040] ,

[0041] ,

[0042] in, These are the x-coordinates of key points 32 and 36, and the y-coordinate of key point 31, respectively. This refers to the ratio of the rectangle's height to its width. The offset is the ratio of the rectangle's width;

[0043] Step 2-2, for Local CLAHE enhancement is performed on the internal pixels to obtain a new region. The Otsu method is applied to automatically find the temperature gradient threshold, separating the low-temperature region corresponding to the nostrils from the skin background and generating a static segmentation mask. ;

[0044] The specific operation of Ostu's method includes: assuming the image has L gray levels, the gray levels... The number of pixels is , The total number of pixels N in the image is:

[0045] ,

[0046] grayscale probability for:

[0047] ,

[0048] Let the threshold be The image is divided into two categories: background: grayscale values. Foreground category: Grayscale value ;

[0049] Probability of the background class for:

[0050] ,

[0051] Probability of foreground class for:

[0052] ,

[0053] Mean of background class for:

[0054] ,

[0055] Mean of foreground class for:

[0056] ,

[0057] The overall mean of the image Defined as:

[0058] ,

[0059] Between-class variance Defined as:

[0060] ,

[0061] Simplified to:

[0062] ,

[0063] Ostu's method selects the optimal threshold by maximizing the inter-class variance. :

[0064] ,

[0065] Define cumulative probability :

[0066] ,

[0067] Define cumulative mean :

[0068] ,

[0069] Between-class variance The final calculation formula is:

[0070] ;

[0071] Ostu's method applies to each possible threshold. , can calculate , choose to Maximum threshold Then, the foreground and background are binarized and segmented, and the segmented background region is taken as the static segmentation mask. ;

[0072] Steps 2-3, for , will the current frame Compared to the previous frame Perform alignment, and then calculate the current frame. Analyze the optical flow vector magnitude of each pixel and find those whose optical flow vector magnitude is greater than a threshold value. The pixels; after binarization, the magnitude of the extracted optical flow vector is greater than Pixels are used to generate a dynamic active mask. ;

[0073] Steps 2-4: Using keypoint 32 as seed point A and keypoint 35 as seed point B, use the region growing method to grow from the inside out to obtain two connected regions, forming an airflow connected region mask, i.e., an anatomical mask. ;

[0074] Steps 2-5: Take the intersection of the masks to obtain the final valid ROI mask. :

[0075] .

[0076] In steps 2-4, the region growing method specifically includes: Let the image be... ,in Represents pixel coordinates, Represents the grayscale value of a pixel, let... To represent a region, ,in It is the entire domain of the image; the definition of neighborhood uses eight neighborhoods, i.e., pixels. neighborhood for:

[0077] ;

[0078] For a candidate pixel ,like grayscale satisfy: Then it will be incorporated into the region; among which, Refers to the reference grayscale value. finger tolerance, Tolerance.

[0079] In steps 2-5, if Area less than area threshold If the airflow signal is determined to be weak, the prediction mechanism is invoked: the effective mask from the previous frame is projected onto the current frame using a sparse optical flow field; if the projection... The area is still smaller than the area threshold. If so, the final valid ROI mask from the previous frame will be used directly.

[0080] Step 3 includes the following steps:

[0081] Step 3-1, obtain the final current frame Map back to the original, unenhanced frame Above; a low-temperature percentile extraction strategy was adopted: through the analysis of... Sort all pixels in the image by grayscale values, then select the pixels with the lowest grayscale values ​​within a certain range and calculate their average value. ;

[0082] Setting the final There are M pixels. Let the gray value of the Mth pixel be... ;

[0083] Sort by grayscale value from smallest to largest:

[0084] ,

[0085] in This refers to the grayscale value in the Mth position after arranging grayscale values ​​from smallest to largest.

[0086] Extract the K pixels with the smallest grayscale values ​​to obtain the pixels with the lowest grayscale values ​​within the region:

[0087] ,

[0088] in To sort the grayscale values ​​from smallest to largest, and then... The grayscale value of bits, where ;

[0089] Calculate the mean :

[0090] ,

[0091] in The mean of the smallest 10% of gray values;

[0092] Step 3-2: Introduce dynamic compensation based on environmental benchmarks. First, select key point 30 on the bridge of the nose as the benchmark. Using key point 30 as the center, extract... The median grayscale value of the pixel block is calculated as a representation of the ambient temperature. , The side length of the pixel block;

[0093] First, obtain the center point of the region. expression:

[0094] ,

[0095] in, The center point of the pixel block used to calculate ambient heat. The x-coordinate of key point number 30 The ordinate of key point number 30;

[0096] use Represents a pixel block. The pixels within a pixel block are represented by the following relationship:

[0097] ,

[0098] use Represents the grayscale value of a pixel within a pixel block:

[0099] ,

[0100] Extract the median of the grayscale values ​​and record it as the ambient temperature. :

[0101] ,

[0102] in This represents the median;

[0103] Step 3-3: Subtract the ambient temperature from the mean to obtain the original signal. :

[0104] .

[0105] Step 4 includes:

[0106] Step 4-1: Use empirical mode decomposition to transform the original non-stationary signal. Decomposed into several intrinsic modulo functions And a residual trend term;

[0107] Let the original signal be Through empirical mode decomposition get:

[0108] ,

[0109] in, It is the first One eigenmode function It is the nth residual term. The intrinsic mode functions obtained from the decomposition quantity;

[0110] Empirical Mode Decomposition Follow the following recursive rules:

[0111] ,

[0112] The original signal is represented as:

[0113] ,

[0114] The extraction of IMF components depends on the sieving process. Let the original signal be... The screening process includes:

[0115] Find the signal All local maxima and local minimum points Record function values:

[0116] ,

[0117] in represent The A maximum value, for The The function values ​​corresponding to the local maxima;

[0118] ,

[0119] in represent The A minimum value, for The The function values ​​corresponding to each local minimum point;

[0120] Connect the coordinates of all the maxima using interpolation to obtain the upper envelope. :

[0121] ,

[0122] in Indicates interpolation;

[0123] Connect the coordinates of all local minima using interpolation to obtain the lower envelope. :

[0124] ,

[0125] Take the average value to obtain the mean envelope. :

[0126] ,

[0127] The mean envelope here That is, the first intrinsic modulus function The new input signal is obtained by subtracting the mean envelope from the current signal. :

[0128] ,

[0129] Final iteration screening: As the new input, we get:

[0130] ,

[0131] in For the first The mean envelope calculated in the next iteration. The number of iterations performed to satisfy the stopping criterion, when hour, Original signal ;

[0132] Until the stopping criterion is met: if If the function is monotonic or has only one extreme point, stop the decomposition.

[0133] Step 4-2: Select based on the physiological characteristics of respiratory rate. , , Signal reconstruction is performed to filter out low-frequency trends and high-frequency interference, resulting in a reconstructed signal. A fourth-order Butterworth bandpass filter is then applied to the reconstructed signal to further filter out low-frequency and high-frequency noise, yielding the final breathing signal. ;

[0134] Step 4-3, analyze the final respiratory signal Perform a Fast Fourier Transform to obtain the amplitude-frequency response curve, and find the frequency of maximum amplitude peak value in the interval [0.1Hz, 0.8Hz]. According to the formula respiratory rate .

[0135] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0136] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0137] In summary, this method, through the processing of thermal infrared video data, including infrared image preprocessing and key point localization, multimodal fusion for nostril ROI extraction, anti-interference original signal extraction, signal reconstruction, and respiratory rate estimation, achieves effective and accurate extraction of respiratory signals and respiratory rate, contributing to the development of contactless medical care.

[0138] Beneficial effects:

[0139] (1) The present invention has extremely high motion robustness and solves the common loss of lock problem in non-contact measurement by cascading multiple tracking mechanisms.

[0140] (2) This invention achieves precise region locking through a multimodal mask fusion strategy when extracting the ROI of the nostrils. This invention adopts a "dynamic and static combination" mask extraction strategy. By taking the intersection of the static segmentation extracted by Otsu's method and the dynamic active mask calculated by the optical flow difference method, the interference of environmental background and static skin on the subject's face is effectively eliminated. Region growth is performed using seed points, and the resulting anatomical mask is then intersected with the previous results to ensure that the mask always closely follows the core airflow area of ​​the nostrils, thereby improving the original signal-to-noise ratio of the signal from the source.

[0141] (3) This invention achieves sensitive capture of weak breathing signals. This invention uses a low-temperature percentile extraction strategy. Instead of using the full mean of the ROI region, the algorithm calculates the mean of the top 10% of cold pixels within the ROI. This design greatly amplifies the weak temperature difference characteristics caused by the inhalation of cold air during breathing, making the recognition accuracy of shallow or rapid breathing significantly higher than traditional methods. Simultaneously, dynamic environmental baseline compensation is performed, by subtracting environmental heat to offset environmental temperature fluctuations and the baseline drift of the sensor itself. Attached Figure Description

[0142] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0143] Figure 1 This is a flowchart of the method of the present invention.

[0144] Figure 2 This is a schematic diagram of the original infrared video frame.

[0145] Figure 3 It involves face detection and 54-point key point localization on infrared video frames. A schematic diagram of the extracted results.

[0146] Figure 4 This is a schematic diagram of the results from the optical flow difference method.

[0147] Figure 5 This is a schematic diagram of the results of the region growing method.

[0148] Figure 6 This is a schematic diagram of the background segmentation results using the Otsu method.

[0149] Figure 7 This is a diagram illustrating the intersection result.

[0150] Figure 8 This is a schematic diagram of the final respiratory signal FFT result.

[0151] Figure 9 This is a schematic diagram of the actual FFT results of the collected respiratory signals. Detailed Implementation

[0152] like Figure 1 As shown, this invention provides a non-contact respiratory signal extraction method based on infrared thermal imaging video data, comprising the following steps:

[0153] Step 1: Perform CLAHE (Contrast Limited Adaptive Histogram Equalization) enhancement on the original infrared video frames to increase the contrast of the infrared images. Using a face detection and face landmark localization model, combined with the Lucas-Kanade sparse optical flow method, the landmark coordinates of each frame are obtained.

[0154] Step 2: Based on anatomical knowledge and facial key points, dynamically draw the initial Region of Interest (ROI) bounding box. Then, perform CLAHE enhancement on the region within the bounding box, calculate the multimodal candidate mask, and take the intersection of the three candidate masks to obtain the final ROI.

[0155] Step 3: The final masked region is mapped back to the unenhanced original infrared video frame to ensure the accuracy of signal extraction. Within the final masked region, the system does not calculate the average pixel value of all Regions of Interest (ROIs). Instead, it extracts the average value of the top 10% of pixels with the lowest grayscale values ​​(temperature values) as the signal points of the current frame. Environmental thermal compensation is subtracted from the signal points to offset the ambient background temperature drift, resulting in the original signal sequence.

[0156] Step 4: Perform empirical mode decomposition on the extracted original signal sequence (EMD). ), yielding several intrinsic modulus functions ( Based on respiratory rate characteristics, select , and The reconstructed signal is obtained through reconstruction. This step effectively filters out high-frequency noise and low-frequency drift caused by body temperature changes. The reconstructed signal is then bandpass filtered to obtain the final respiratory signal. Finally, a Fast Fourier Transform (FFT) is applied to analyze the power spectrum of the final respiratory signal, identifying the peak frequency point with the largest amplitude, and calculating the final respiratory rate.

[0157] Step 1 includes:

[0158] Step 1-1: Use an infrared camera to collect facial thermal radiation data of the subject and output raw infrared video frames. .

[0159] Steps 1-2, Entering the system. The system refers to the entire code flow in this invention. To balance processing speed and signal accuracy, the system maintains two image streams: a computation stream and a mapping stream. In the computation stream, to improve processing speed, the system scales the original frame to a specified width, obtaining... The image is then enhanced with CLAHE to improve contrast. In the mapped stream, the system retains the original resolution without enhancement. This is used for subsequent high-accuracy signal extraction. In this system, the specified width for scaling is 640.

[0160] Steps 1-3, for An ensemble model combining directional gradient histograms, linear support vector machines, and regression trees was used to locate 54 core physiological key points on the face. The Histogram of Oriented Gradients (HOG) is used as a feature extractor for face detection in this system. Its principle is to first divide the image into small cells, calculate the gradient direction of each pixel within the cell, compile a histogram of gradient directions, and finally normalize multiple cells to form a feature vector. The Support Vector Machine (SVM) is used as a classifier for face detection in this system. Its principle is that during the training phase, labeled face and non-face samples are used to learn a linear hyperplane to separate the two classes. During the detection phase, the HOG features extracted by the sliding window are used with the SVM to determine whether it is a face. The Regression Tree Ensemble (ERT) is used for facial keypoint localization in this project. Its principle is to progressively refine the keypoint positions through multi-level regression trees; each level of regression tree extracts features based on the current estimated position and predicts the positional shift; finally, the trees are ensembled, with multiple trees voting to improve robustness. In addition, to ensure the continuity and smoothness of the extracted signal, if the face is not detected in the current frame, resulting in key point detection failure, the Lucas-Kanade sparse optical flow method is used, based on the grayscale information of the previous frame. Predict the location of key points in the current frame to ensure signal continuity.

[0161] The specific principle of Lucas-Kanade sparse optical flow method for predicting key point locations is as follows:

[0162] The basic assumption of Lucas-Kanade optical flow is that the brightness of the same object point in an image remains constant between adjacent frames, as expressed by the formula:

[0163] ,

[0164] in, At time t, the position is pixel brightness, It is a displacement vector. It is a time interval.

[0165] Define optical flow vector The optical flow constraint equation is obtained as follows:

[0166] ,

[0167] in Let be the gradient of the image in the x-direction. It is the gradient of the image in the y-direction. It is the gradient of the image in the time direction. It is the velocity component in the x-direction. It is the velocity component in the y-direction.

[0168] The Lucas-Kanade sparse optical flow method assumes that all pixels within a local window of an image have the same optical flow vector. This is set at a point... There is one around windows (e.g.) The window contains n pixels. For each pixel within the window, there is an optical flow constraint equation:

[0169] ,

[0170] Finally, the corresponding pixel within the window can be obtained using the following formula. :

[0171] ,

[0172] ,

[0173] ,

[0174] Therefore, given the coordinates of key points in the previous frame, the corresponding optical flow vector can be calculated based on the window containing the key points in the previous frame. This allows us to obtain the optical flow vectors of key points and predict the coordinates of key points in the next frame.

[0175] Step 2 includes the following steps:

[0176] Step 2-1, obtain the key point locations Mapping back to the original frame ,get 54 key points on the top Next, a rectangular region of interest that closely follows the nostril area is dynamically generated using keypoint coordinates. .

[0177] Calculate the width of the rectangle Rectangle height x-coordinate of the center point of the rectangle The ordinate of the center point of the moment The calculation formula is as follows:

[0178] ,

[0179] ,

[0180] ,

[0181] ,

[0182] in, The x-coordinates of key points 32 and 36 are respectively, and the y-coordinate of key point 31 is respectively. This represents the ratio of the rectangle's height to its width; in this method, it is taken as 0.6. The offset is the ratio of the offset to the width of the rectangle; in this method, it is set to 0.15.

[0183] Step 2-2, for the obtained Local CLAHE enhancement is performed on the internal pixels to obtain... The Otsu method is applied to automatically find the temperature gradient threshold, separating the low-temperature zone (or airflow zone) corresponding to the nostrils from the skin background and generating a static segmentation mask. .

[0184] The specific principle of Otsu's method is as follows: Otsu's method is an adaptive threshold selection method based on maximizing the inter-class variance, used for image binarization. This method assumes the image consists of two classes of pixels: foreground and background, and finds the optimal threshold that maximizes the variance between the two classes.

[0185] Suppose the image has L gray levels (usually) ), grayscale value grayscale The number of pixels is The total number of pixels N in the image is:

[0186] ,

[0187] grayscale probability for:

[0188] ,

[0189] Let the threshold be The image is divided into two categories: background: grayscale values. Foreground category: Grayscale value .

[0190] Probability of the background class for:

[0191] ,

[0192] Probability of foreground class for:

[0193] ,

[0194] Mean of background class for:

[0195] ,

[0196] Mean of foreground class for:

[0197] ,

[0198] The overall mean of the image Defined as:

[0199] ,

[0200] Between-class variance Defined as:

[0201] ,

[0202] Simplified to:

[0203] ,

[0204] The Otsu method selects the optimal threshold by maximizing the inter-class variance. :

[0205] ,

[0206] Define cumulative probability :

[0207] ,

[0208] Define cumulative mean :

[0209] ,

[0210] Between-class variance The final calculation formula is:

[0211] ;

[0212] Ostu's method is that for each possible threshold... , can calculate , choose to Maximum threshold Then, the foreground and background are segmented. In this invention, the nostril area has a significant temperature difference with the surrounding skin area due to the presence of cold respiratory airflow, resulting in a large difference in grayscale values ​​in the infrared video frame. Therefore, the CLAHE-enhanced... Binarization segmentation using Ostu's method can initially extract the background region. Since airflow has a lower temperature than the human body and a smaller grayscale value, the background region is extracted after binarization and used as a static segmentation mask. .

[0213] Steps 2-3, for , will the current frame Compared to the previous frame Perform alignment, and then calculate the current frame. Analyze the optical flow vector magnitude of each pixel and find those whose optical flow vector magnitude is greater than a threshold value. The pixels. Because the airflow region to be captured has large temperature variations, resulting in large variations in grayscale values ​​between frames, the optical flow vector amplitude is large. Therefore, the optical flow vector amplitude extracted after binarization is greater than [value missing]. Pixels are used to generate a dynamic active mask. Here The system takes 20.

[0214] Steps 2-4: Using keypoint 32 as seed point A and keypoint 35 as seed point B, region growing is used from the inside out to obtain two connected components. Since keypoints 32 and 35 are the closest to the right nostril and the left nostril respectively among the 54 keypoints, and the gray values ​​of the nostril airflow region are relatively similar to those of the surrounding nostril airflow regions but differ significantly from the gray values ​​of other skin regions, it can be assumed that the two connected components respectively contain the right and left nostrils. Binarization segmentation is performed on the results to extract the two connected components, forming an airflow connected component mask, i.e., an anatomical mask. .

[0215] The region growing method is explained in detail below: Let the image be... ,in Represents pixel coordinates, This represents the grayscale value of the pixel. To represent a region, ,in It refers to the entire domain of the image. The definition of a neighborhood uses an eight-neighborhood, which is the number of pixels. neighborhood for:

[0216] ;

[0217] The core of region growing is defining similarity conditions between pixels. In this method, the similarity condition is that the grayscale of the pixels in the region to be added must fall within the tolerance range of the reference grayscale. That is, for a given candidate pixel... If its grayscale satisfy: Then it can be incorporated into the region; among which, The reference grayscale value is taken here as the grayscale value of the pixel in the region to be added, which is already within the region's neighboring pixels. The tolerance is 2 in this case. The upper tolerance is 2 in this case.

[0218] Steps 2-5: Take the intersection of the three masks above to obtain the final effective ROI mask. The formula is:

[0219] ,

[0220] At the same time, its area is verified. Area smaller If the system determines that the airflow signal is not obvious, it will invoke the prediction mechanism: projecting the effective mask of the previous frame onto the current frame using a sparse optical flow field; if the projection... If the area is still too small, then the final valid ROI mask from the previous frame will be used directly. This refers to the area threshold, which is set to 90 here.

[0221] Step 3 includes the following steps:

[0222] Step 3-1, obtain the final current frame Map back to the original, unenhanced frame This is to ensure the accuracy of the extracted signal. Considering the significant interference from skin heat in infrared images, this system does not... Instead of taking the average of all pixels, it innovatively adopts a low-temperature percentile extraction strategy: by... Sort all pixels by their grayscale values, then select the 10% of pixels with the lowest grayscale values ​​(corresponding to temperature values) within the region and calculate their average value. .

[0223] This strategy can accurately capture the core temperature drop characteristics during the exchange of gases between the nostrils and the outside world during breathing, precisely locate the core airflow, greatly reduce the interference from pixel values ​​in other irrelevant skin areas, significantly improve signal accuracy, and reduce noise.

[0224] Setting the final It contains M pixels, whose grayscale values ​​are respectively ;

[0225] Sort them according to their grayscale values ​​from smallest to largest:

[0226] ,

[0227] Pick Extract the K pixels with the smallest grayscale values ​​to obtain the top 10% of pixels with the lowest grayscale values ​​(corresponding to temperature values) within the region:

[0228] ,

[0229] Calculate its mean, and obtain :

[0230] ,

[0231] Step 3-2: To address the issue of fluctuating ambient temperature, this invention innovatively introduces dynamic compensation for the environmental baseline to eliminate interference from ambient temperature on the original signal. First, key point 30 on the bridge of the nose is selected as the baseline because its temperature is relatively stable and almost unaffected by respiratory airflow. Then, using this point as the center, the signal is extracted... The median gray value of the pixel block is calculated as a representation of the ambient temperature. . The side length of the pixel block is in this system. Take 21.

[0232] First, obtain the center point of the region. expression:

[0233] ,

[0234] in, The center point of the pixel block used to calculate ambient heat. The x-coordinate of key point number 30 The vertical coordinate of key point number 30.

[0235] Therefore, using Represents a pixel block. The pixels within a pixel block are represented by the following relationship:

[0236] ,

[0237] use Represents the grayscale value of a pixel within a pixel block:

[0238] ,

[0239] Extract the median of the grayscale values ​​and record it as the ambient temperature. :

[0240] ,

[0241] Step 3-3: Subtract the ambient temperature from the mean to obtain the original signal.

[0242] ,

[0243] This step can effectively eliminate baseline drift caused by sensor heating or periodic temperature changes in the indoor environment.

[0244] Step 4 includes:

[0245] Step 4-1, using empirical mode decomposition (EMD) The original non-stationary signal Decomposed into several intrinsic modulo functions And a residual trend term.

[0246] Specifically, the following is an example: Empirical Mode Decomposition (EMD) EMD is an adaptive signal decomposition method. EMD can decompose arbitrarily complex signals into a finite number of eigenmode functions (EMFs). The method is a sum of a term and a residual term. It does not require pre-defined basis functions and can adaptively handle nonlinear and non-stationary signals.

[0247] Let the original signal be , Decompose it into:

[0248] ,

[0249] in, It is the first eigenmode functions ( ), It is the residual term, representing the trend or mean of the signal. It is obtained by decomposition quantity.

[0250] Decomposition is a recursive process, following the recursive rules below:

[0251] ,

[0252] Therefore, the original signal is represented as:

[0253] ,

[0254] The extraction of IMF components depends on the sieving process. Let the original signal be... The screening process is as follows:

[0255] Find the signal All local maxima and local minimum points Record function values:

[0256] ,

[0257] ,

[0258] Connect the coordinates of all the maxima using interpolation to obtain the upper envelope. :

[0259] ,

[0260] Connect the coordinates of all local minima using interpolation to obtain the lower envelope. :

[0261] ,

[0262] Take the average value to obtain the mean envelope:

[0263] ,

[0264] Mean envelope here That is The current signal minus the mean envelope:

[0265] ,

[0266] Final iteration screening: Repeat the above steps as new input:

[0267] ,

[0268] in For the first The mean envelope calculated in the next iteration. The number of iterations performed to satisfy the stopping criterion, when hour, Original signal ;

[0269] Until the stopping criterion is met: if If the function is monotonic or has only one extreme point, stop the decomposition.

[0270] Step 4-2: Based on the physiological characteristics of respiratory rate (generally between 0.1-0.8 Hz), select... , , Signal reconstruction is performed to filter out low-frequency trends (body temperature changes) and high-frequency interference, yielding the reconstructed signal. A fourth-order Butterworth bandpass filter is then applied to the reconstructed signal to further filter low-frequency and high-frequency noise, resulting in the final respiratory signal. .

[0271] Step 4-3: Perform a Fast Fourier Transform (FFT) on the final respiratory signal to obtain the amplitude-frequency response curve, and find the frequency of the maximum peak amplitude in the interval [0.1Hz, 0.8Hz]. According to the formula (Unit: bpm), respiratory rate (output) ).

[0272] Figure 2 This is a frame from an infrared video, showing the subject's face.

[0273] Figure 3 To perform face detection on this infrared video frame, 54 key points were located. The extraction results. The bright green box in the image represents the face rectangle, which contains the complete subject's face. The dark blue dots represent 54 facial landmarks, which are accurately located and precisely depict the subject's key facial contours. The bright yellow rectangle represents... The frame shows the area including the subject's nostrils.

[0274] Figure 4 The result of optical flow differential segmentation of this frame is shown in the image, where the yellow-green outline is... The boundary can be seen, compared to Its area is further reduced, allowing for more precise positioning of the airflow area in the nostrils.

[0275] Figure 5 The result of region growing segmentation of this frame is shown in the image; the bright blue outline represents... The boundary can be seen, compared to Its area is further reduced, while it can accurately outline the upper edge of the nostrils, thereby improving the accuracy of ROI positioning.

[0276] Figure 6 The red area in the image represents the result of Ostu's method segmentation of this frame. The masked area shows that, compared to Its area has further shrunk. With In contrast, when the subject's head rotates, the keypoints on which the two seed points of the region growing method are based may shift to some extent, thus leading to... There is some deviation, and Unaffected by critical points. Meanwhile, It may include part of the tip of the nose, and This problem will not occur. Therefore, using both the region growing method and Ostu's method simultaneously, and then taking the intersection of the two results, can play a very good complementary role.

[0277] Figure 7 The result is the intersection, where the blue part is... The result is that As a result, we can see that The addition of [something] effectively reduced The static skin area portion of the image was improved, which improved the positioning accuracy. Although it still includes a certain part of the nose, the subsequent use of a low-temperature percentile extraction strategy, which only uses the average value of the 10% of pixels with the lowest temperature, effectively eliminates interference and improves accuracy.

[0278] Figure 8 The result of FFT on the final respiratory signal. Figure 9 The results of FFT on the synchronously acquired actual respiratory signals show that the two signals have high spectral similarity and the respiratory rates extracted from the two signals are the same, which effectively proves the accuracy of the present invention.

[0279] This invention provides a non-contact respiratory signal extraction method based on infrared thermal imaging video data. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A non-contact respiratory signal extraction method based on infrared thermal imaging video data, characterized in that, Includes the following steps: Step 1: Perform contrast-limited adaptive histogram equalization enhancement on the original infrared video frames to increase the contrast of the infrared images. Use face detection and face key point localization models, combined with the Lucas-Kanade sparse optical flow method, to obtain the key point coordinates of each frame. Step 2: Based on anatomical knowledge and facial key points, dynamically draw the initial region of interest (ROI) rectangle, perform CLAHE enhancement on the region within the rectangle, calculate the multimodal candidate mask, and take the intersection of the candidate masks to obtain the final ROI. Step 3: Map the final region of interest (ROI) back to the original unenhanced infrared video frame. Within the final masked area, extract the average value of the pixels with the lowest brightness values ​​within a certain range as the signal points of the current frame. Subtract the environmental thermal compensation from the signal points to obtain the original signal sequence. Step 4: Perform empirical mode decomposition on the original signal sequence to obtain the intrinsic mode functions; Based on the respiratory rate characteristics, the fourth intrinsic mode function obtained from the decomposition is selected. The fifth intrinsic modulus function obtained by decomposition and the sixth intrinsic modulus function obtained by decomposition The reconstructed signal is obtained by reconstructing the signal, and then bandpass filtering is applied to the reconstructed signal to obtain the final respiratory signal. The power spectrum of the final respiratory signal is analyzed by applying Fast Fourier Transform to identify the peak frequency point with the largest amplitude, and the final respiratory rate is calculated.

2. The method according to claim 1, characterized in that, Step 1 includes the following steps: Step 1-1: Use an infrared camera to collect facial thermal radiation data of the subject and output raw infrared video frames. ; Steps 1-2: Maintain two image streams: a computation stream and a mapping stream; in the computation stream, the original infrared video frames are... Expand to the specified width to obtain a new video frame. ,right Perform CLAHE enhancement to improve image contrast; In the mapped stream, the original infrared video frames are preserved. ; Steps 1-3, for An ensemble model combining histogram of oriented gradients, support vector machines, and regression trees was used to locate 54 key physiological points on the face. Histogram of Oriented Gradients (HGP) is used as a feature extractor for face detection. First, the image is divided into small cells, the gradient direction of each pixel in the cell is calculated, the histogram of gradient directions is calculated, and finally, two or more cells are normalized to form a feature vector. Support Vector Machines are used as classifiers for face detection. During the training phase, labeled face samples and non-face samples are used to learn a linear hyperplane that separates face samples and non-face samples. During the detection phase, the Histogram of Oriented Gradients (HOG) features extracted from the sliding window are used to determine whether a face is present. A regression tree ensemble model is used for facial keypoint localization, refining the keypoint positions through multi-level regression trees. Each level of the regression tree extracts features based on the currently estimated position and predicts the positional shift. Finally, the trees are integrated, with two or more trees voting. If a face is not detected in the current frame, resulting in keypoint detection failure, the Lucas-Kanade sparse optical flow method is used, based on the grayscale information from the previous frame. Predict the location of key points in the current frame.

3. The method according to claim 2, characterized in that, In steps 1-3, the specific process of predicting key point locations using the Lucas-Kanade sparse optical flow method includes: The formula for assuming that the brightness of the same object point in an image remains constant between adjacent frames is as follows: ， in, At time t, the position is pixel brightness, It is a displacement vector. It is a time interval; Define optical flow vector The optical flow constraint equation is obtained as follows: ， in Let be the gradient of the image in the x-direction. It is the gradient of the image in the y-direction. It is the gradient of the image in the time direction. It is the velocity component in the x-direction. It represents the velocity component in the y-direction; d denotes the differential sign. Indicates the partial differential symbol; Set all pixels within a local window on the image to have the same optical flow vector, set at the point There is one around A window containing n pixels. For each pixel within the window, there is an optical flow constraint equation: ， in Indicates the first The coordinates of each pixel The width of the window. The coordinates on the image are The pixels in gradient of direction, The coordinates on the image are The pixels in Gradient of direction; The pixel corresponding to the point within the window can be obtained by solving the following formula. : ， ， ， Where det represents the determinant value of the matrix.

4. The method according to claim 3, characterized in that, Step 2 includes the following steps: Step 2-1, obtain the key point locations Mapping back to the original frame ,get 54 key points on the top Next, a rectangular region of interest that closely follows the nostril area is dynamically generated using the keypoint coordinates. ; Calculate the width of the rectangle Rectangle height x-coordinate of the center point of the rectangle The ordinate of the center point of the moment The calculation formula is: ， ， ， ， in, These are the x-coordinates of key points 32 and 36, and the y-coordinate of key point 31, respectively. This refers to the ratio of the rectangle's height to its width. The offset is the ratio of the rectangle's width; Step 2-2, for Local CLAHE enhancement is performed on the internal pixels to obtain a new region. The Otsu method is applied to automatically find the temperature gradient threshold, separating the low-temperature region corresponding to the nostrils from the skin background and generating a static segmentation mask. ; The specific operation of Ostu's method includes: assuming the image has L gray levels, the gray levels... The number of pixels is , The total number of pixels N in the image is: ， grayscale probability for: ， Let the threshold be The image is divided into two categories: background: grayscale values. Foreground category: Grayscale value ; Probability of the background class for: ， Probability of foreground class for: ， Mean of background class for: ， Mean of foreground class for: ， The overall mean of the image Defined as: ， Between-class variance Defined as: ， Simplified to: ， Ostu's method selects the optimal threshold by maximizing the inter-class variance. : ， Define cumulative probability : ， Define cumulative mean : ， Between-class variance The final calculation formula is: ； Ostu's method applies to each possible threshold. , can calculate , choose to Maximum threshold Then, the foreground and background are binarized and segmented, and the segmented background region is taken as the static segmentation mask. ; Steps 2-3, for , will the current frame Compared to the previous frame Perform alignment, and then calculate the current frame. Analyze the optical flow vector magnitude of each pixel and find those whose optical flow vector magnitude is greater than a threshold value. The pixels; after binarization, the magnitude of the extracted optical flow vector is greater than Pixels are used to generate a dynamic active mask. ; Steps 2-4: Using keypoint 32 as seed point A and keypoint 35 as seed point B, use the region growing method to grow from the inside out to obtain two connected regions, forming an airflow connected region mask, i.e., an anatomical mask. ; Steps 2-5: Take the intersection of the masks to obtain the final valid ROI mask. : 。 5. The method according to claim 4, characterized in that, In steps 2-4, the region growing method specifically includes: Let the image be... ,in Represents pixel coordinates, Represents the grayscale value of a pixel, let... To represent a region, ,in It is the entire domain of the image; the definition of neighborhood uses eight neighborhoods, i.e., pixels. neighborhood for: ； For a candidate pixel ,like grayscale satisfy: Then it will be incorporated into the region; among which, Refers to the reference grayscale value. finger tolerance, Tolerance.

6. The method according to claim 5, characterized in that, In steps 2-5, if Area less than area threshold If the airflow signal is determined to be weak, the prediction mechanism is invoked: the effective mask from the previous frame is projected onto the current frame using a sparse optical flow field; if the projection... The area is still smaller than the area threshold. If so, the final valid ROI mask from the previous frame will be used directly.

7. The method according to claim 6, characterized in that, Step 3 includes the following steps: Step 3-1, obtain the final current frame Map back to the original, unenhanced frame Above; adopting a low-temperature percentile extraction strategy: through the analysis of... Sort all pixels in the image by grayscale values, then filter out the pixels within a certain range of the lowest grayscale values ​​in the region, and calculate the average value. ; Setting the final There are M pixels. Let the gray value of the Mth pixel be... ; Sort by grayscale value from smallest to largest: ， in This refers to the grayscale value in the Mth position after arranging grayscale values ​​from smallest to largest. Extract the K pixels with the smallest grayscale values ​​to obtain the pixels with the lowest grayscale values ​​within the region: ， in To sort the grayscale values ​​from smallest to largest, and then position them as the [number]th [item]... The grayscale value of bits, where ; Calculate the mean : ， in The mean of the smallest 10% of gray values; Step 3-2: Introduce dynamic compensation based on environmental benchmarks. First, select key point 30 on the bridge of the nose as the benchmark. Using key point 30 as the center, extract... The median grayscale value of the pixel block is calculated as a representation of the ambient temperature. , The side length of the pixel block; First, obtain the center point of the region. expression: ， in, The center point of the pixel block used to calculate ambient heat. The x-coordinate of key point number 30 The ordinate of key point number 30; use Represents a pixel block. The pixels within a pixel block are represented by the following relationship: ， use Represents the grayscale value of a pixel within a pixel block: ， Extract the median of the grayscale values ​​and record it as the ambient temperature. : ， in This represents the median; Step 3-3: Subtract the ambient temperature from the mean to obtain the original signal. : 。 8. The method according to claim 7, characterized in that, Step 4 includes: Step 4-1: Use empirical mode decomposition to transform the original non-stationary signal. Decomposed into several intrinsic modulo functions And a residual trend term; Let the original signal be Through empirical mode decomposition get: ， in, It is the first One eigenmode function It is the nth residual term. The intrinsic mode functions obtained from the decomposition quantity; Empirical Mode Decomposition Follow the following recursive rules: ， The original signal is represented as: ， The extraction of IMF components depends on the sieving process. Let the original signal be... The screening process includes: Find the signal All local maxima and local minimum points Record function values: ， in represent The A maximum value, for The The function values ​​corresponding to the local maxima; ， in represent The A minimum value, for The The function values ​​corresponding to the local minimum points; Connect the coordinates of all the maxima using interpolation to obtain the upper envelope. : ， in Indicates interpolation; Connect the coordinates of all local minima using interpolation to obtain the lower envelope. : ， Take the average value to obtain the mean envelope. : ， The mean envelope here That is, the first intrinsic modulus function The new input signal is obtained by subtracting the mean envelope from the current signal. : ， Final iteration screening: As the new input, we get: ， in For the first The mean envelope calculated in the next iteration. The number of iterations performed to satisfy the stopping criterion, when hour, Original signal ; Until the stopping criterion is met: if If the function is monotonic or has only one extreme point, stop the decomposition. Step 4-2: Select based on the physiological characteristics of respiratory rate. , , Signal reconstruction is performed to filter out low-frequency trends and high-frequency interference, resulting in a reconstructed signal. A fourth-order Butterworth bandpass filter is then applied to the reconstructed signal to further filter out low-frequency and high-frequency noise, yielding the final breathing signal. ; Step 4-3, analyze the final respiratory signal Perform a Fast Fourier Transform to obtain the amplitude-frequency response curve, and find the frequency of maximum amplitude peak value in the interval [0.1Hz, 0.8Hz]. According to the formula respiratory rate .

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, It stores a computer program or instructions that, when run on a computer, perform the steps of the method as described in any one of claims 1 to 8.

Images