Thermal infrared video respiration monitoring method and device based on mask auto-encoder

Abstract

The invention relates to the technical field of crossing of physiological signal monitoring and computer vision, in particular to a thermal infrared video respiration monitoring method and equipment based on a mask auto-encoder. According to the method, the affine transformation matrix is constructed through a small amount of RGB-thermal infrared face key points, and the key point tracking result of the RGB video is mapped to the thermal infrared video, so that mouth and nose region-of-interest tracking without large-scale thermal infrared video labeling is realized, a corresponding mouth and nose thermal infrared sub-graph sequence is obtained, and the sample acquisition cost is reduced; spatial partitioning and time stacking are carried out on the mouth and nose thermal infrared subgraph sequence, a thermal infrared space-time diagram facing the breathing task is constructed, and therefore space-time information of the sample is fully utilized; according to the method, the space-time diagram is subjected to self-supervision pre-training after being subjected to mask processing, secondary fine-tuning training is carried out through annotation data, and a mask auto-encoder after training is output, so that the accuracy and robustness of an output result are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of physiological signal monitoring and computer vision interdisciplinary technology, and in particular to a thermal infrared video respiratory monitoring method and device based on a mask autoencoder. Background Technology

[0002] Respiratory rate, respiratory depth, and changes in gas intake and output during the respiratory cycle are key vital signs reflecting the body's metabolic state, respiratory system function, and emotional stress levels. Long-term, continuous, and accurate respiratory monitoring helps to: identify respiratory disease risks early, such as COPD and bullae; assess psychological states such as anxiety, stress, and mood swings; and support applications such as intensive care, sleep monitoring, and telemedicine.

[0003] Traditional respiratory monitoring relies heavily on contact-based devices such as chest / abdominal strap sensors, flow meters, and mask-type respiratory sensors. These methods have the following problems: 1. They require close contact with the body, interfering with the subject's natural breathing and daily activities; 2. They are uncomfortable to wear for extended periods, making it difficult to achieve truly "unnoticeable" long-term monitoring.

[0004] To overcome the aforementioned problems, non-contact respiratory monitoring methods based on millimeter-wave radar, ultra-wideband radar, and ordinary visible light video, as well as nasal temperature monitoring based on thermal infrared imaging, have emerged in recent years. However, radar systems are costly and complex to install, ordinary visible light video is sensitive to factors such as lighting and obstruction, and subtle displacements of the chest and abdomen or changes in texture in the mouth and nose area caused by breathing are difficult to capture stably in complex environments.

[0005] Thermal infrared imaging offers several advantages: it requires no external light source and can operate in low-light conditions; it directly reflects facial temperature distribution and can capture temperature changes in the mouth and nose area caused by respiration; and it is completely non-contact, making it suitable for long-distance, long-term monitoring. However, existing thermal infrared-based video respiration monitoring solutions still face several key challenges:

[0006] 1. ROI tracking relies on a specialized model in the thermal infrared domain. Current practices typically involve directly training detection and tracking models for the nostrils or mouth and nose regions on thermal infrared images. Due to the lack of large-scale thermal infrared facial keypoint datasets, a large amount of manual annotation is often required, which is costly and has limited generalization ability for complex head movements.

[0007] 2. Insufficient utilization of spatiotemporal information Most methods only average the pixel values ​​within the nasal or oral ROI to obtain a single-channel temperature time series, ignoring the differences in temperature changes and their temporal structure in different local areas, making it difficult to characterize fine-grained breathing patterns.

[0008] 3. Lack of labeled respiratory waveform data Collecting synchronous respiratory zone reference signals is costly and strict synchronization is difficult. The size of labeled samples available for supervised training is limited. Deep models are prone to overfitting when trained directly end-to-end and lack robustness to complex environments.

[0009] 4. Single output metric Many studies only provide respiratory rate or whether there is apnea, lacking modeling of complete respiratory waveforms and structural indicators that are closer to physiological mechanisms, such as inspiratory and expiratory duration and air intake and exhaust volume.

[0010] Therefore, there is a need for a non-contact, lower-cost, more accurate, and more robust thermal infrared video breathing monitoring method based on a mask autoencoder. Summary of the Invention

[0011] The purpose of this invention is to overcome the problems of existing technologies that require a large number of manually labeled samples, are costly, and cannot fully utilize the spatiotemporal information of the samples, resulting in insufficient accuracy and robustness. This invention provides a thermal infrared video respiration monitoring method based on a mask autoencoder.

[0012] To achieve the above-mentioned objectives, the present invention provides the following technical solution: A thermal infrared video respiration monitoring method based on a mask autoencoder includes the following steps: The spatiotemporal map of the thermal infrared respiration of the person to be monitored is input into the pre-trained mask autoencoder; The mask autoencoder outputs the breathing waveform of the person being monitored; The pre-training of the mask autoencoder includes the following steps: S1: Input sample data of several subjects under the set head movement; the sample data includes RGB video, thermal infrared video and contact breathing reference signal synchronously collected by the subject under the current head movement; S2: Mark facial key points in the first frame of the RGB video and thermal infrared video in each of the sample data respectively, and generate several mouth and nose thermal infrared sub-image sequences by constructing affine transformation and facial key point tracking algorithm. S3: Divide each of the oral and nasal thermal infrared sub-image sequences into grids according to set parameters, calculate the average temperature value of each sub-block in each frame image, stack them in spatiotemporal order to form a two-dimensional matrix, and output as several spatiotemporal images of thermal infrared breathing. S4: Divide each of the spatiotemporal graphs into several image blocks of a set size, and randomly mask the image blocks of each spatiotemporal graph according to a set ratio; then train the pre-constructed encoder and decoder using the image blocks of each spatiotemporal graph after masking. S5: After training is complete, remove the decoder and connect a linear prediction head after the encoder. Train the model using several spatiotemporal graphs and corresponding contact breathing reference signals. Once the model converges, output a mask autoencoder.

[0013] As a preferred embodiment of the present invention, the generation of the oral and nasal thermal infrared sub-image sequence for each sample data in S2 includes the following steps: S21: Mark facial landmarks in the first frame of the RGB video and thermal infrared video in the sample data respectively; S22: Based on the coordinate information of the facial key points, establish the affine transformation relationship between the RGB video and the thermal infrared video; S23: Obtain the coordinate trajectory of facial key points in RGB video through a facial key point tracking algorithm; S24: Map the coordinate trajectory of the facial key points to the thermal infrared video through the affine transformation relationship to obtain the coordinate trajectory of the facial key points in the thermal infrared video; S25: Connect the facial key points to form a region of interest, and output the mouth and nose thermal infrared sub-image sequence corresponding to the current thermal infrared video based on the coordinate trajectory of the facial key points.

[0014] As a preferred embodiment of the present invention, S3 further includes standardizing the output spatiotemporal graph, the standardization process including: The various spatiotemporal diagrams are unified in the time dimension through interpolation or truncation. The temperature values ​​in the unified spatiotemporal graph are normalized.

[0015] As a preferred embodiment of the present invention, the pre-built encoder and decoder in S4 are composed of several stacked Multi-HeadSelf-Attention modules; The Multi-Head Self-Attention module includes a multi-head self-attention mechanism and a feedforward neural network.

[0016] As a preferred embodiment of the present invention, the loss function trained in S4 adopts mean squared error.

[0017] As a preferred embodiment of the present invention, the masking process includes one or more of random masking, block masking, and grid masking.

[0018] As a preferred embodiment of the present invention, the loss function trained in S5 adopts waveform mean square error and correlation loss.

[0019] As a preferred embodiment of the present invention, the pre-training further includes S6; S6: The prediction results of the mask autoencoder are evaluated by frequency domain evaluation metrics and time domain structure evaluation metrics to generate a comprehensive model score. When the comprehensive model score is lower than a preset threshold, the process proceeds to S1 for pre-training again. The frequency domain evaluation metrics are used to assess the overall fitting accuracy of the respiratory waveform, including mean absolute error, root mean square error, and Pearson correlation coefficient. The time-domain structure evaluation indicators include the ratio of air intake to air output and the mean absolute error of inhalation and exhalation duration. The inhalation-exhalation volume ratio is estimated by integrating the respiratory waveform to measure the inspiratory and expiratory volumes for each respiratory cycle; the mean absolute error of the inhalation and exhalation duration is used to assess the temporal accuracy of the respiratory rhythm.

[0020] As a preferred embodiment of the present invention, the model comprehensive score adopts weighted comprehensive score, hierarchical evaluation or principal component analysis.

[0021] A thermal infrared video respiration monitoring device based on a mask autoencoder includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform any of the above-described thermal infrared video respiration monitoring methods based on a mask autoencoder.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs an affine transformation matrix using a small number of RGB-thermal infrared facial key points, mapping the key point tracking results of RGB video onto thermal infrared video. This enables the tracking of the mouth and nose regions of interest without large-scale thermal infrared video annotation, obtaining the corresponding mouth and nose thermal infrared sub-image sequence and reducing the cost of sample acquisition. Furthermore, by spatially partitioning and temporally stacking the mouth and nose thermal infrared sub-image sequence, a thermal infrared spatiotemporal map oriented towards breathing tasks is constructed, thus fully utilizing the spatiotemporal information of the samples. Then, after masking the spatiotemporal map, self-supervised pre-training is performed, followed by secondary fine-tuning training using labeled data, outputting a trained masked autoencoder, which effectively improves the accuracy and robustness of the output results. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a thermal infrared video respiration monitoring method based on a mask autoencoder as described in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the pre-training process of the mask autoencoder in a thermal infrared video respiration monitoring method based on a mask autoencoder as described in Embodiment 1 of the present invention. Figure 3This is a schematic diagram of the output result of the mask autoencoder in the thermal infrared video respiration monitoring method based on the mask autoencoder described in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the structure of a thermal infrared video respiration monitoring device based on a mask autoencoder, which utilizes the thermal infrared video respiration monitoring method based on a mask autoencoder described in the foregoing embodiments, as described in Embodiment 4 of the present invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0025] Example 1 like Figure 1 As shown, a thermal infrared video respiration monitoring method based on a mask autoencoder includes the following steps: The spatiotemporal map of the thermal infrared respiration of the person to be monitored is input into the pre-trained mask autoencoder; The mask autoencoder outputs the breathing waveform of the person being monitored; Among them, such as Figure 2 As shown, the pre-training of the mask autoencoder includes the following steps: S1: Input sample data of several subjects under the set head movement; the sample data includes RGB video, thermal infrared video and contact breathing reference signal synchronously collected by the subject under the current head movement.

[0026] S2: Mark facial landmarks in the first frame of the RGB video and thermal infrared video in each of the sample data respectively. Then, generate several mouth and nose thermal infrared sub-image sequences by constructing affine transformation and facial landmark tracking algorithm.

[0027] S3: Divide each of the oral and nasal thermal infrared sub-image sequences into a grid according to the set parameters, calculate the average temperature value of each sub-block in each frame image, stack them in spatiotemporal order to form a two-dimensional matrix, and output as several spatiotemporal images of thermal infrared breathing.

[0028] S4: Divide each of the spatiotemporal graphs into several image blocks of a set size, and randomly mask the image blocks of each spatiotemporal graph according to a set ratio; then train the pre-constructed encoder and decoder using the image blocks of each spatiotemporal graph after masking.

[0029] S5: After training is complete, remove the decoder and connect a linear prediction head after the encoder. Train the model using several spatiotemporal graphs and corresponding contact breathing reference signals. Once the model converges, output a mask autoencoder.

[0030] Example 2 This embodiment is a specific pre-training implementation scheme of the masked autoencoder in the thermal infrared video respiration monitoring method based on a masked autoencoder described in Embodiment 1, including the following steps: S1: Input sample data of several subjects under the set head movement; the sample data includes RGB video, thermal infrared video and contact breathing reference signal synchronously collected by the subject under the current head movement.

[0031] Furthermore, the sample data described in this embodiment includes the following collection requirements: ① Subjects and the setting: Twenty-seven participants aged 20–40 years were selected and data was collected in an indoor environment. Participants sat facing the display screen, and an RGB camera and a thermal infrared camera were fixedly mounted above or below the screen, with their field of view covering the participants' heads.

[0032] ②Collection content and parameters: RGB camera: Frame rate approximately 30 frames per second; Thermal infrared camera: Frame rate approximately 25 frames per second; Contact breathing signal: breathing band sampling rate of approximately 2000 Hz; synchronized with video time via a synchronization module.

[0033] ③Task design: The experiment included several pre-defined head movement phases, each lasting approximately 30 seconds. After data collection, each data segment was divided into 6-second windows, and after removing segments with severe artifacts, a sample dataset was formed. This dataset was then divided into training and validation sets based on the participants. In this embodiment, the pre-defined head movements included sitting still, giving a speech, slow left and right head shaking, rapid left and right head shaking, rapid nodding, and head shaking in random directions.

[0034] S2: Mark facial landmarks in the first frame of the RGB video and thermal infrared video in each of the sample data respectively. Then, generate several mouth and nose thermal infrared sub-image sequences by constructing affine transformation and facial landmark tracking algorithm.

[0035] The generation of the oral and nasal thermal infrared sub-image sequence for each sample data includes the following steps: S21: Perform facial landmark annotation on the first frame of the RGB video and thermal infrared video in the sample data respectively.

[0036] In the first frame of each video, several facial key points that can be simultaneously identified in both RGB and thermal infrared images are manually selected. In this embodiment, the center of the eyebrows, left and right cheeks, tip of the nose, and chin are selected. The coordinates of the key points in both the RGB and IR (thermal infrared) domains are recorded.

[0037] S22: Based on the coordinate information of the facial key points, establish the affine transformation relationship between the RGB video and the thermal infrared video.

[0038] Furthermore, the mathematical form of the affine transformation matrix is:

[0039] Among them, (x IR ,y IR (x) represents the coordinates in the thermal infrared video. RGB ,y RGB Let ) represent the coordinates in the RGB video, where a, b, c, and d are linear transformation parameters, and t x t y Let be the translation parameters. For N anchor point pairs, the transformation parameters that minimize the mapping error are determined using the least squares method.

[0040] S23: Obtain the coordinate trajectory of facial key points in RGB video through a facial key point tracking algorithm.

[0041] In an RGB sequence, existing facial landmark tracking algorithms are used to obtain the position of each landmark in each frame. This invention preferably uses the Mediapipe framework developed by Google for facial landmark tracking. This framework can detect 468 3D facial landmarks in real time, exhibiting high accuracy and robustness. Besides Mediapipe, other mature facial landmark tracking algorithms can also be used, such as the Dlib library (which can detect 68 landmarks) based on HOG features and an SVM classifier, or the OpenFace framework based on deep learning. These algorithms have all been trained on large-scale datasets on RGB videos, enabling stable tracking of facial landmarks and avoiding the high cost of retraining a dedicated model in the thermal infrared domain.

[0042] S24: The coordinate trajectory of the facial key points is mapped to the thermal infrared video through the affine transformation relationship to obtain the coordinate trajectory of the facial key points in the thermal infrared video.

[0043] For each time step, the RGB keypoint coordinates are converted into IR coordinates using the affine matrix described above, thereby obtaining the corresponding keypoint position in the thermal infrared image and realizing cross-modal keypoint tracking.

[0044] S25: Connect the facial key points to form a region of interest, and output the mouth and nose thermal infrared sub-image sequence corresponding to the current thermal infrared video based on the coordinate trajectory of the facial key points.

[0045] In the IR domain, using key points such as the tip of the nose, chin, and left and right corners of the mouth as references, a bounding rectangle containing the nostrils and mouth area is constructed as the ROI (Region of Interest) for the mouth and nose. As time progresses, the ROI position is updated based on the mapped key point trajectories, resulting in a series of nasal and oral thermal infrared sub-image sequences that change with head movement.

[0046] In this embodiment, the operator directly annotates corresponding facial key points (such as the center of the forehead, left and right cheeks, nose tip, and chin) on both the RGB and thermal infrared images in the first frame of the video, forming RGB-IR anchor point pairs. Starting from the second frame, the system automatically acquires the key point trajectories in the RGB video sequence using a mature facial key point tracking algorithm. Finally, the affine transformation matrix described above maps the key point coordinates in the RGB domain to the IR domain in real time, achieving automatic tracking of the mouth and nose ROI. This embodiment avoids training a separate key point detection model in the IR domain, limits the annotation to a small number of anchor points, and eliminates the need for large-scale thermal infrared facial key point annotation data. It enables stable tracking of the mouth and nose ROI in thermal infrared videos at a lower cost, significantly reducing the difficulty of constructing thermal infrared datasets.

[0047] This invention relies on only a small number of RGB–IR anchor points and achieves two-camera viewpoint alignment through an affine transformation matrix. It uses a mature facial landmark tracking method on the RGB side to map the landmark trajectories to the IR domain, thereby obtaining the nose and mouth ROI in thermal infrared. This eliminates the need to train a separate landmark model and build large-scale labeled data in the IR domain, significantly reducing data annotation and model development costs. Furthermore, it maintains relatively stable ROI tracking even under complex poses such as head movements.

[0048] S3: Divide each of the oral and nasal thermal infrared sub-image sequences into a grid according to the set parameters, calculate the average temperature value of each sub-block in each frame image, stack them in spatiotemporal order to form a two-dimensional matrix, and output as several spatiotemporal images of thermal infrared breathing.

[0049] S31: Spatial partitioning: For each frame, the ROI of the mouth and nose is divided into N sub-blocks according to a set fixed grid, for example, into several rows and columns of small squares.

[0050] S32: Time stacking: For each sub-block, the average temperature value of the pixels within that sub-block is calculated frame by frame on the entire 6-second thermal infrared video, forming a time series of length T.

[0051] S33: Formation of the spacetime map: Arrange the time series of all sub-blocks in spatial order to form a two-dimensional array of size N×T, which is the spatiotemporal diagram of the respiration corresponding to the segment.

[0052] Furthermore, this step also includes standardizing the output spatiotemporal map to reduce the impact of differences in absolute temperature between individuals; the standardization process includes: The various spatiotemporal diagrams are unified in the time dimension through interpolation or truncation. The temperature values ​​in the unified spatiotemporal graph are normalized.

[0053] This invention performs spatial segmentation and temporal stacking on the oral and nasal regions of interest (ROIs) to construct a spatiotemporal map of respiration specifically designed to characterize thermal infrared temperature trends. This spatiotemporal map simultaneously encodes local spatial temperature differences and temporal periodic variations caused by respiration. Compared to traditional one-dimensional temperature sequences, the spatiotemporal map obtained through this step preserves both the differences in temperature changes at different spatial locations and the periodic fluctuations over time, providing a richer and more structured representation of respiratory features for subsequent deep models.

[0054] S4: Divide each of the spatiotemporal graphs into several image blocks of a set size, and randomly mask the image blocks of each spatiotemporal graph according to a set ratio; then train the pre-constructed encoder and decoder using the image blocks of each spatiotemporal graph after masking.

[0055] Furthermore, this embodiment includes the following steps: S41: Divide the spatiotemporal graph into several image blocks of fixed size, and map them into a sequence of feature vectors through an embedding layer.

[0056] S42: Randomly select a portion of image blocks at a relatively high ratio (e.g., around 75%) for masking, retaining only a small number of image blocks as encoder input.

[0057] Masking strategy is a key design element in the pre-training of the MAE encoder. This invention preferably employs a random masking strategy, where approximately 75% of the image patches obtained from the spatiotemporal graph are uniformly and randomly selected for masking, with only the remaining 25% of visible image patches input to the encoder. This high masking ratio forces the model to learn the global structural information of the spatiotemporal graph. Besides random masking, other masking strategies can be used depending on the specific application scenario, such as block-wise masking (masking continuous rectangular regions) or grid masking (masking at fixed intervals using a pattern). Different masking strategies may affect the model's learning preferences for local or global features.

[0058] S43: The pre-constructed encoder and decoder are trained by using image blocks of each spatiotemporal graph after masking.

[0059] Furthermore, the encoder in this embodiment is constructed using an 8-layer stacked Multi-Head Self-Attention (MHSA) module, with each layer containing a multi-head self-attention mechanism and a feedforward neural network. Specifically, for the input visible image patch embedding sequence, spatiotemporal features are extracted through the following transformation: ; Where Attention() is the attention mechanism, Q, K, and V are the query, key, and value matrices, respectively, T is the transpose symbol, and d k is the dimension of the key. The encoder outputs a high-level feature representation of the visible image patch.

[0060] The decoder uses a stack of 6 layers of MHSA modules. After concatenating the encoder output with learnable mask tokens, it reconstructs the complete spatiotemporal graph through a self-attention mechanism.

[0061] The loss function in the pre-training phase uses the mean square error of the masked region. for:

[0062] Where M is the set of indices of the masked image patches, X i and These are the original image patch and the reconstructed image patch, respectively. The loss function is computed only on the masked image patch, encouraging the model to reconstruct the complete spatiotemporal structure from limited visible information.

[0063] S5: After training is complete, remove the decoder and connect a linear prediction head after the encoder. Train the model using several spatiotemporal graphs and corresponding contact breathing reference signals. Once the model converges, output a mask autoencoder.

[0064] Furthermore, this embodiment includes the following steps: After pre-training is complete, remove the MAE decoder and keep only the encoder.

[0065] A linear prediction head is connected after the encoder output to map the features extracted by the encoder into a one-dimensional breathing waveform of length T. Using synchronously acquired respiratory zone signals as supervisory labels, the encoder and prediction head are jointly optimized using waveform mean square error, correlation and other losses; Specifically, the loss function during the fine-tuning phase It consists of two parts: waveform mean square error and correlation loss. Its mathematical form is:

[0066] Among them, the mean square error (MSE) is used to calculate the predicted waveform. Compared with the actual waveform The pointwise difference is expressed as:

[0067] in, Let be the predicted waveform at time t. Let t be the actual waveform at time t, and T be the total duration of the waveform.

[0068] The Pearson correlation coefficient (PCC) measures the morphological similarity between two waveforms, and its expression is:

[0069] Where Cov represents covariance. These represent the standard deviations of the predicted and actual waveforms, respectively. The weighting parameter λ is used to balance the contributions of the two losses, typically ranging from 0.1 to 1.0. This joint loss function ensures both point-to-point accuracy of the waveform and maintains the consistency of the overall shape.

[0070] The fine-tuning phase uses a different learning rate and training rounds than the pre-training phase to ensure that the advantages of the pre-training features are further adapted to the specific task.

[0071] This invention pre-trains the encoder on a large number of unlabeled spatiotemporal maps using mask reconstruction, enabling the encoder to automatically learn the spatiotemporal structure of the oral and nasal temperature fields and respiratory cycle information. Fine-tuning is then performed on a small amount of respiratory-labeled data, effectively alleviating the overfitting problem caused by insufficient labeling and improving the model's generalization ability to complex scenes. The thermal infrared STMap is input into the mask autoencoder trained in this embodiment, yielding the output waveform as shown below. Figure 3 As shown, orange and blue represent the actual label results measured by the respiratory band and the model prediction results, respectively.

[0072] Example 3 The difference between this embodiment and the above embodiment is that the pre-training further includes S6; S6: The prediction results of the mask autoencoder are evaluated by frequency domain evaluation metrics and time domain structure evaluation metrics to generate a comprehensive model score. When the comprehensive model score is lower than a preset threshold, the process proceeds to S1 for pre-training again.

[0073] In order to comprehensively evaluate the consistency between the predicted respiratory waveform and the reference waveform from multiple perspectives, this embodiment designs a comprehensive evaluation index system from both the frequency domain and time domain dimensions.

[0074] Specifically: (1) Frequency domain evaluation indicators: The frequency domain evaluation metrics are used to assess the overall fitting accuracy of the respiratory waveform, including mean absolute error (MAE) and root mean square error (RMSE), and their expressions are as follows:

[0075]

[0076] Where B is the batch size, A pred,i and A gt,i These are the predicted and actual values ​​of the peak spectrum of the i-th sample, respectively; the smaller the values ​​of MAE and RMSE, the higher the overall fitting accuracy of the current model.

[0077] (2) Temporal structure evaluation index: The time-domain structure evaluation indicators include Pearson correlation coefficient (PCC), air intake-to-exhaust ratio, and mean absolute error of inhalation and exhalation duration. Pearson correlation coefficient (PCC):

[0078] Among them, Y pred,i and Y gt,i Let be the predicted value and the actual value of the waveform of the i-th sample, respectively; Cov represents the covariance. represents the standard deviation of the predicted waveform and the standard deviation of the actual waveform for the i-th sample, respectively; the higher the PCC value, the higher the similarity between the predicted waveform and the actual waveform of the current model.

[0079] The air intake / exhaust ratio (R) in / R ex The expression for estimating the inspiratory and expiratory volumes for each respiratory cycle by integrating the respiratory waveform is as follows:

[0080]

[0081] Among them, t in,start to t in,end For the inhalation period, t ex,start to t ex,end The time interval of the exhalation phase is automatically determined by a waveform peak and trough detection algorithm; Let be the predicted waveform at time t. This represents the actual waveform at time t. The integral value approximately represents the amount of gas exchanged during this stage. R in and R ex The smaller the value, the more accurate the current model's estimation of the intake and exhaust volume of air.

[0082] The mean absolute error of the inhalation and exhalation duration (D) in / D ex The expression for assessing the timing accuracy of respiratory rhythm is:

[0083]

[0084] Where T in and T ex These represent the duration of inhalation and exhalation, respectively. and These represent the predicted and actual values ​​of the inhalation duration, respectively. and D represents the predicted and actual values ​​of the expiratory duration, respectively; in and D ex The smaller the value, the more accurate the respiratory rhythm assessed by the current model. These indicators are of significant clinical importance for assessing respiratory diseases (such as COPD and bullae) and emotional stress.

[0085] (3) Model comprehensive score: Depending on the specific application requirements, methods such as weighted comprehensive scoring, hierarchical evaluation, or principal component analysis can be used to comprehensively evaluate multi-dimensional indicators. The experimental verification of this invention uses a parallel presentation of each indicator to fully demonstrate the model's performance across all dimensions. The following are some comprehensive evaluation schemes for reference: ① Weighted composite scoring method:

[0086] The normalization function is:

[0087]

[0088] Among them, f ** For the calculated indicator score, the weight w i Configure according to the application scenario, for example: Respiratory rate monitoring: w1=w2=w3=0.2, w4=w5=w6=w7=0.05 Respiratory disease assessment: w1=w2=w3=0.1, w4=w5=0.25, w6=w7=0.15 Emotional state analysis: all weights are equal, w i =1 / 7.

[0089] ②Stratified evaluation method: First, assess whether the frequency domain metrics meet the basic requirements, and then evaluate the time domain structure metrics. The specific evaluation criteria are as follows:

[0090] ③ Principal component analysis: Principal component analysis (PCA) was performed on all indicators, and the first principal component was extracted as the composite score.

[0091] Where, α i The loading coefficients of the first principal component for each evaluation index are automatically determined from the training set data.

[0092] As shown in Table 1 below, in comparative experiments with several traditional thermal infrared respiratory rate detection methods and general deep models (such as ResNet18, ConvNeXt, ViT, etc.), the method of this invention performs better overall in terms of MAE, RMSE, PCC, as well as the ratio of air intake to exhaust volume and duration of inhalation and exhalation. This indicates that spatiotemporal graph representation and MAE pre-training can effectively improve the accuracy and stability of thermal infrared respiratory waveform prediction under limited annotation conditions.

[0093] Table 1 Comparative Experiment Results Experimental model Mean Absolute Error Root mean square error Pearson correlation coefficient intake volume air output Average error of inhalation and exhalation duration Absolute error of inhalation and exhalation duration ResNet 18 0.07±0.01 0.68±0.05 0.04±0.04 0.33±0.07 0.30±0.16 0.66±0.10 0.63±0.06 ConvNeXt 0.07±0.01 0.70±0.01 0.01±0.04 0.34±0.13 0.28±0.09 0.62±0.22 0.79±0.36 ViT 0.07±0.02 0.75±0.10 0.07±0.07 0.40±0.08 0.29±0.09 0.66±0.14 0.93±0.27 This invention 0.07±0.02 0.69±0.11 0.15±0.04 0.40±0.12 0.26±0.05 0.79±0.19 0.79±0.10 In addition to conventional waveform error indicators such as MAE, RMSE, and PCC, this invention designs indicators that are closer to the respiratory physiological mechanism, such as the intake-to-output ratio and the mean absolute error of inspiratory / expiratory duration. This allows the model performance evaluation to go beyond frequency or point-to-point error, and to comprehensively evaluate the prediction quality from both waveform morphology and respiratory rhythm structure, providing more interpretable quantitative evidence for subsequent clinical and health management applications.

[0094] Example 4 like Figure 4 As shown, a thermal infrared video respiration monitoring device based on a mask autoencoder includes at least one processor, a memory communicatively connected to the at least one processor, and at least one input / output interface communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enables the at least one processor to perform the thermal infrared video respiration monitoring method based on a mask autoencoder described in the foregoing embodiments. The input / output interface may include a display, keyboard, mouse, and USB interface for inputting and outputting data.

[0095] Furthermore, the thermal infrared video respiration monitoring device based on a mask autoencoder can be a desktop computer, mobile phone, tablet computer, wearable thermal infrared video respiration monitoring device based on a mask autoencoder, or any other device capable of deep information recognition.

[0096] Furthermore, the processor may include one or more processing cores. The processor connects various parts within the mask-autoencoder-based thermal infrared video respiration monitoring device using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also be implemented separately as a communication chip, without being integrated into the processor.

[0097] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets, such as instructions or code sets used to implement the thermal infrared video respiratory monitoring method based on a mask autoencoder provided in this application. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created during the use of the thermal infrared video respiratory monitoring device based on the mask autoencoder (such as a mapping table of modulation sequences and depths, image data, spectrogram data, etc.).

[0098] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0099] When the integrated units of the present invention are implemented as software functional units and sold or used as independent products, they can also be stored in a computer-readable storage medium. The computer-readable storage medium stores program code, which can be called by a processor to execute the methods described in the above method embodiments. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes electronic memories such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that executes any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0100] This invention is used in a reasonable manner.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A thermal infrared video respiration monitoring method based on a mask autoencoder, characterized in that, Includes the following steps: The spatiotemporal map of the thermal infrared respiration of the person to be monitored is input into the pre-trained mask autoencoder; The mask autoencoder outputs the breathing waveform of the person being monitored; The pre-training of the mask autoencoder includes the following steps: S1: Input sample data of several subjects under the set head movement; the sample data includes RGB video, thermal infrared video and contact breathing reference signal synchronously collected by the subject under the current head movement; S2: Mark facial key points in the first frame of the RGB video and thermal infrared video in each of the sample data respectively, and generate several mouth and nose thermal infrared sub-image sequences by constructing affine transformation and facial key point tracking algorithm. S3: Divide each of the oral and nasal thermal infrared sub-image sequences into grids according to set parameters, calculate the average temperature value of each sub-block in each frame image, stack them in spatiotemporal order to form a two-dimensional matrix, and output as several spatiotemporal images of thermal infrared breathing. S4: Divide each of the spatiotemporal graphs into several image blocks of a set size, and randomly mask the image blocks of each spatiotemporal graph according to a set ratio; then train the pre-constructed encoder and decoder using the image blocks of each spatiotemporal graph after masking. S5: After training is complete, remove the decoder and connect a linear prediction head after the encoder. Train the model using several spatiotemporal graphs and corresponding contact breathing reference signals. Once the model converges, output a mask autoencoder.

2. The thermal infrared video respiration monitoring method based on a mask autoencoder according to claim 1, characterized in that, The generation of the oral and nasal thermal infrared sub-image sequence for each sample data in S2 includes the following steps: S21: Mark facial landmarks in the first frame of the RGB video and thermal infrared video in the sample data respectively; S22: Based on the coordinate information of the facial key points, establish the affine transformation relationship between the RGB video and the thermal infrared video; S23: Obtain the coordinate trajectory of facial key points in RGB video through a facial key point tracking algorithm; S24: Map the coordinate trajectory of the facial key points to the thermal infrared video through the affine transformation relationship to obtain the coordinate trajectory of the facial key points in the thermal infrared video; S25: Connect the facial key points to form a region of interest, and output the mouth and nose thermal infrared sub-image sequence corresponding to the current thermal infrared video based on the coordinate trajectory of the facial key points.

3. The thermal infrared video respiration monitoring method based on a mask autoencoder according to claim 1, characterized in that, S3 further includes standardizing the output spatiotemporal graph, the standardization process including: The various spatiotemporal diagrams are unified in the time dimension through interpolation or truncation. The temperature values ​​in the unified spatiotemporal graph are normalized.

4. A thermal infrared video respiration monitoring method based on a mask autoencoder according to claim 1, characterized in that, The pre-built encoder and decoder in S4 are composed of several stacked Multi-Head Self-Attention modules; The Multi-Head Self-Attention module includes a multi-head self-attention mechanism and a feedforward neural network.

5. The thermal infrared video respiration monitoring method based on a mask autoencoder according to claim 4, characterized in that, The loss function used in S4 is the mean squared error.

6. The thermal infrared video respiration monitoring method based on a mask autoencoder according to claim 4, characterized in that, The masking process includes one or more of random masks, block masks, and grid masks.

7. The thermal infrared video respiration monitoring method based on a mask autoencoder according to claim 1, characterized in that, The loss function used in S5 for training employs waveform mean square error and correlation loss.

8. The thermal infrared video respiration monitoring method based on a mask autoencoder according to claim 1, characterized in that, The pre-training also includes S6; S6: The prediction results of the mask autoencoder are evaluated by frequency domain evaluation metrics and time domain structure evaluation metrics to generate a comprehensive model score. When the comprehensive model score is lower than a preset threshold, the process proceeds to S1 for pre-training again. The frequency domain evaluation metrics are used to assess the overall fitting accuracy of the respiratory waveform, including mean absolute error, root mean square error, and Pearson correlation coefficient. The time-domain structure evaluation indicators include the ratio of air intake to air output and the mean absolute error of inhalation and exhalation duration. The inhalation-exhalation volume ratio is estimated by integrating the respiratory waveform to measure the inspiratory and expiratory volumes for each respiratory cycle; the mean absolute error of the inhalation and exhalation duration is used to assess the temporal accuracy of the respiratory rhythm.

9. A thermal infrared video respiration monitoring method based on a mask autoencoder according to claim 8, characterized in that, The model's overall score employs weighted comprehensive scoring, hierarchical evaluation, or principal component analysis.

10. A thermal infrared video respiration monitoring device based on a mask autoencoder, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform a thermal infrared video respiration monitoring method based on a mask autoencoder as described in any one of claims 1 to 9.

Images