Facial thermal imaging-based non-contact respiratory rate detection method and system

Abstract

The invention provides a non-contact respiratory rate detection method and system based on face thermal imaging, a storage medium and electronic equipment, and relates to the field of non-contact respiratory rate detection. According to the method, a thermal infrared video is utilized, and a deep learning technology is combined to accurately and stably extract a high-quality respiration signal from a thermal infrared spectrum. Meanwhile, the self-adaptive respiratory signal noise reduction network based on the attention mechanism is designed to carry out self-adaptive automatic noise reduction processing on the initial respiratory wave signals, local details and global features in the respiratory signals can be deeply analyzed, noise can be automatically distinguished and effectively suppressed, key physiological information is reserved, and the accuracy of noise reduction is improved. And the quality of respiratory signals and the precision of respiratory rate estimation are obviously improved. Through training of the deep learning model, the method can adapt to diversified breathing characteristics of different crowds and environments, the generalization ability in various practical application scenes is enhanced, and the stability and high accuracy of breathing rate detection are ensured.

Description

Technical Field

[0001] The present invention relates to the field of non-contact respiratory rate detection, and in particular to a non-contact respiratory rate detection method, system, storage medium and electronic device based on facial thermal imaging. Background Art

[0002] Breathing is crucial in the body's metabolism. Regularly monitoring breathing can help detect and prevent respiratory, cardiovascular and cerebrovascular diseases as early as possible.

[0003] In the related art, the respiratory rate detection technology is mainly divided into two categories: contact and non-contact. Among them, non-contact respiratory rate detection does not require the use of traditional contact equipment. It can detect human body movement and physiological signals by using cameras, microwave radars or other sensors to analyze the respiratory rate. For example, patent CN117357093A discloses a non-contact respiratory rate monitoring method and system based on dual-spectrum facial video, including: simultaneously acquiring thermal infrared video and visible light video; extracting a first thermal infrared image; extracting the respiratory signal corresponding to the first thermal infrared image as the first respiratory signal; and selecting a first visible light image frame; using an affine transformation method to map the nose area of ​​interest of the user under test in the first visible light image frame to the first thermal infrared image frame, and extracting the image of the mapped area in the first thermal infrared image frame as the second thermal infrared image; extracting the respiratory signal corresponding to the second thermal infrared image as the second respiratory signal; performing signal fusion on the first respiratory signal and the second respiratory signal, and using the fused signal as the third respiratory signal; obtaining a respiratory signal curve; based on the respiratory signal curve, obtaining the respiratory frequency of the user under test.

[0004] However, the above solutions have shortcomings such as limited applicable scenarios and limited accuracy of respiratory rate estimation in variable actual scenarios. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the deficiencies in the prior art, the present invention provides a non-contact respiratory rate detection method, system, storage medium and electronic device based on facial thermal imaging, which solves the technical problems of insufficient robustness and generalization in a variety of practical application scenarios.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A non-contact respiratory rate detection method based on facial thermal imaging is based on a robust non-contact respiratory rate detection framework, wherein the robust non-contact respiratory rate detection framework includes a thermal infrared video signal processing network and an adaptive respiratory signal denoising network; the method includes:

[0010] Obtain real-time thermal infrared video of the subject;

[0011] For the image frame sequence of the real-time thermal infrared video, a fully convolutional single-stage target detection neural network of the thermal infrared video signal processing network is used to track the facial ROI, and an initial respiratory wave signal is obtained based on the change in the average pixel value of the tracked facial nose area;

[0012] For the initial respiratory wave signal, based on the one-dimensional convolution layer and the attention mechanism layer of the adaptive respiratory signal denoising network, the key features of the signal are captured to obtain a denoised respiratory wave signal;

[0013] The peak of the noise-reduced respiratory wave signal is detected, and the respiratory rate of the subject is calculated.

[0014] Preferably, the fully convolutional single-stage target detection neural network includes a ResNet50 backbone network, a feature pyramid, and a shared detection head; the fully convolutional single-stage target detection neural network based on the thermal infrared video signal processing network performs facial ROI tracking for the image frame sequence of the real-time thermal infrared video; including:

[0015] In the ResNet50 backbone network: the current image frame is continuously subjected to five layers of convolution, wherein the last three layers of convolution respectively extract image features C3, C4, and C5 of three different scales;

[0016] In the feature pyramid: C3, C4, and C5 are respectively processed by a 2DCNN with a convolution kernel of 1×1; the processed C5 is used as D5, and the processed C5 is upsampled and concatenated with the processed C4 to obtain D4, and the processed C4 is upsampled and concatenated with the processed C3 to obtain D3; D5, D4, and D3 are input into the feature extraction layer, and are respectively processed by a 2DCNN with a convolution kernel of 3×3 to form P3, P4, and P5, and P5 is separately processed by a 2DCNN with a convolution kernel of 3×3 twice to form P6 and P7 respectively;

[0017] In the detection head: P3, P4, P5, P6, and P7 are passed through four 2DCNN-GN-ReLU layers on different branches, and then through corresponding convolution layers with a convolution kernel of 3×3, and the facial ROI category, ROI coordinates, and confidence on the current image frame are output respectively.

[0018] Preferably, the method of obtaining the initial respiratory wave signal based on the change in the average pixel value of the tracked facial nose area comprises:

[0019] Calculate the average pixel value of the nose area of ​​the face in the current image frame t:

[0020]

[0021] Among them, pixel ij is the coordinate (x i ,y j ), (x i ,y j ) is the plane coordinate of the pixel located at the i-th row and j-th column;

[0022] After traversing and calculating the average pixel value of the nose area in each image frame, the initial breathing wave signal is formed:

[0023]

[0024] Where m is the total number of frames of the real-time thermal infrared video.

[0025] Preferably, the adaptive breathing signal denoising network specifically includes five 1DCNNs, two MaxPooling layers, two UPSAMPLE layers and one Attention layer; for the initial breathing wave signal, based on the one-dimensional convolution layer and the attention mechanism layer of the adaptive breathing signal denoising network, the key features of the signal are captured to obtain the denoised breathing wave signal; including:

[0026] At the encoder, the initial respiratory wave signal is input into the first 1DCNN for convolution calculation to obtain the first local feature, and then the first local feature is input into the first MaxPooling layer for maximum pooling operation to obtain the first down-sampling feature;

[0027] And inputting the first down-sampled feature into a second 1DCNN for convolution calculation to obtain a second local feature, and then inputting the second local feature into a second MaxPooling layer for maximum pooling operation to obtain a second down-sampled feature;

[0028] Input the second downsampled features into the attention mechanism layer to focus on the key information of the input signal by assigning different weights;

[0029] At the decoder, the output features of the attention mechanism layer are input into the third 1DCNN for convolution calculation to obtain the third local features, and then the third local features are input into the first UPSAMPLE layer for upsampling operation to obtain the first upsampling features with the same dimension as the input signal;

[0030] Input the first up-sampled feature into the fourth 1DCNN for convolution calculation to obtain a fourth local feature, and then input the fourth local feature into the second UPSAMPLE layer for up-sampling operation to obtain a second up-sampled feature with the same dimension as the input signal;

[0031] And input the second up-sampled feature into the fifth 1DCNN for convolution calculation, obtain and output the fifth local feature, and use the fifth local feature as the denoised respiratory wave signal.

[0032] Preferably, the respiratory rate is expressed as:

[0033]

[0034] Among them, RR is the respiration rate, PN is the number of detected peaks, the indexes of the first peak and the last peak are marked as FP and LP respectively, the average distance between two adjacent peaks is ADP, and TF is the total number of frames of real-time thermal infrared video acquired within one minute.

[0035] Preferably, the robust contactless breathing rate detection framework is trained using a diverse dataset during a training phase.

[0036] Preferably, the robust non-contact respiratory rate detection framework selects the following loss function during the training phase:

[0037]

[0038] Among them, x and y are the predicted respiratory wave signal and the real respiratory band signal respectively, and T is the signal length.

[0039] A non-contact respiratory rate detection system based on facial thermal imaging is based on a robust non-contact respiratory rate detection framework, which includes a thermal infrared video signal processing network and an adaptive respiratory signal denoising network; the system includes:

[0040] An acquisition module is used to acquire real-time thermal infrared video of the subject;

[0041] A processing module, for performing facial ROI tracking based on a fully convolutional single-stage target detection neural network of the thermal infrared video signal processing network for an image frame sequence of the real-time thermal infrared video, and obtaining an initial respiratory wave signal based on a change in an average pixel value of a nose region of the tracked face;

[0042] A denoising module, for capturing key features of the signal based on the one-dimensional convolution layer and the attention mechanism layer of the adaptive respiratory signal denoising network for the initial respiratory wave signal, so as to obtain a denoised respiratory wave signal;

[0043] The detection module is used to detect the peak of the noise-reduced respiratory wave signal and calculate the respiratory rate of the subject.

[0044] A storage medium stores a computer program for non-contact breathing rate detection based on facial thermal imaging, wherein the computer program enables a computer to execute the non-contact breathing rate detection method as described above.

[0045] An electronic device, comprising:

[0046] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including methods for executing the non-contact breathing rate detection method as described above.

[0047] (III) Beneficial effects

[0048] The present invention provides a non-contact respiratory rate detection method, system, storage medium and electronic device based on facial thermal imaging. Compared with the prior art, it has the following beneficial effects:

[0049] The present invention uses thermal infrared video and combines deep learning technology to accurately and stably extract high-quality respiratory signals from thermal infrared spectra. At the same time, by designing an adaptive respiratory signal denoising network based on the attention mechanism to perform adaptive automatic denoising on the initial respiratory wave signal, it is possible to deeply analyze the local details and global features in the respiratory signal, automatically distinguish and effectively suppress noise, retain key physiological information, and significantly improve the quality of the respiratory signal and the accuracy of respiratory rate estimation. Through the training of the deep learning model, this method can adapt to the diverse respiratory characteristics of different populations and environments, enhance the generalization ability in a variety of practical application scenarios, and ensure the stability and high accuracy of respiratory rate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A block diagram of a non-contact respiratory rate detection method based on facial thermal imaging provided by an embodiment of the present invention;

[0052] Figure 2 A robust non-contact respiratory rate detection framework diagram provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the structure of a thermal infrared video signal processing network provided by an embodiment of the present invention;

[0054] Figure 4 A structural schematic diagram of a feature extraction layer provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the structure of an adaptive breathing signal denoising network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The embodiments of the present application solve the technical problems of insufficient robustness and generalization in various practical application scenarios by providing a non-contact respiratory rate detection method, system, storage medium and electronic device based on facial thermal imaging.

[0058] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0059] The embodiment of the present invention innovatively proposes a non-contact respiratory rate detection method and system for monitoring respiratory rate physiological parameters. The method uses thermal infrared information and combines deep learning technology to accurately and stably extract high-quality respiratory signals from the thermal infrared spectrum. At the same time, by designing an adaptive respiratory signal denoising network based on the attention mechanism, the extracted initial respiratory wave signal is adaptively and automatically denoised, and the correlation between the local and global information of the respiratory signal is mined, so as to achieve the generalization processing capability of the respiratory signals extracted from different scenes and populations and the accurate estimation of the respiratory rate, so as to realize real-time analysis of individual physiological health status and provide objective and accurate physiological perception information for respiratory disease screening.

[0060] In the signal acquisition stage, the embodiment of the present invention abandons the traditional tracking method that relies on visible light facial key point detection, and instead adopts thermal infrared video facial ROI tracking technology based on FCOS neural network. This technology is trained with multi-scene facial data and can quickly, accurately and stably locate the facial ROI area in the thermal infrared video. Compared with traditional methods, the tracking technology of the present invention is not affected by scene changes such as changes in the distance of the detector and movement, and shows higher robustness, and can cope with interference in a static or moving state.

[0061] In the signal processing stage, traditional signal denoising methods rely on manually adjusting algorithm parameters through signal preprocessing operations such as detrending, normalization, and fast Fourier transform, and are unable to robustly and adaptively process respiratory signals generated by different breathing patterns. The adaptive signal denoising method proposed in the embodiment of the present invention, by designing a neural network automatic denoising network, can deeply analyze the local details and global features in the respiratory signal, automatically distinguish and effectively suppress noise, retain key physiological information, and significantly improve the quality of the respiratory signal and the accuracy of respiratory rate estimation. Through the training of the deep learning model, the method can adapt to the diverse respiratory characteristics of different populations and environments, enhance the generalization ability in a variety of practical application scenarios, and ensure the stability and high accuracy of respiratory rate detection.

[0062] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0063] Embodiment 1:

[0064] like Figure 1 As shown, the embodiment of the present invention provides a non-contact respiratory rate detection method based on facial thermal imaging. Figure 2 The robust non-contact breathing rate detection framework shown in the embodiment includes a thermal infrared video signal processing network and an adaptive breathing signal denoising network; the method includes:

[0065] S1, obtaining real-time thermal infrared video of the subject;

[0066] S2. For the image frame sequence of the real-time thermal infrared video, a fully convolutional single-stage target detection neural network based on the thermal infrared video signal processing network is used to track the facial ROI, and the initial respiratory wave signal is obtained based on the change in the average pixel value of the tracked facial nose area;

[0067] S3. For the initial respiratory wave signal, based on the one-dimensional convolution layer and attention mechanism layer of the adaptive respiratory signal denoising network, the key features of the signal are captured to obtain the denoised respiratory wave signal;

[0068] S4. Detect the peak of the noise-reduced respiratory wave signal and calculate the respiratory rate of the subject.

[0069] The embodiment of the present invention proposes a facial region of interest (ROI) tracking technology based on deep learning to achieve fast, accurate and stable tracking of the facial ROI of thermal infrared video, thereby improving the quality and consistency of the extracted initial respiratory wave signal. At the same time, based on the adaptive respiratory signal denoising network, it is possible to analyze the local and global features in the initial respiratory wave signal, automatically distinguish and suppress noise, retain important physiological information, and restore a better quality respiratory wave signal, thereby significantly improving the accuracy of respiratory rate estimation.

[0070] Next, we will introduce the steps of the above scheme in detail:

[0071] In step S1, a real-time thermal infrared video of the subject is obtained.

[0072] To ensure that the facial video of the subject can be captured, the thermal infrared camera can be kept 0.5m to 0.8m away from the subject. After clicking the start acquisition button, the acquisition device records the real-time thermal infrared video of the subject.

[0073] In step S2, for the image frame sequence of the real-time thermal infrared video, the fully convolutional single-stage target detection neural network based on the thermal infrared video signal processing network performs facial ROI tracking, and based on the change in the average pixel value of the tracked facial nose area, the initial respiratory wave signal is obtained.

[0074] This step is based on Figure 3 The thermal infrared video signal processing network shown performs a respiratory wave signal extraction operation, involving facial region of interest tracking and respiratory wave signal extraction, specifically including:

[0075] (1) Facial ROI tracking based on thermal infrared video

[0076] Accurate tracking of facial regions of interest (ROI) in thermal infrared videos is one of the key steps in obtaining breathing signals. Traditional ROI tracking methods that rely on affine transformation and feature point matching techniques are mainly based on RGB camera imaging and thermal infrared camera dual-spectrum matching. However, in complex scenes with insufficient or excessive lighting conditions, traditional face detection methods are difficult to effectively track the ROI area, resulting in the failure of affine transformation and feature point matching techniques, and unable to achieve stable tracking of the facial ROI area. In contrast, methods based on thermal infrared videos can capture thermal images generated by human activities and are less dependent on ambient light, making them more reliable.

[0077] To this end, an embodiment of the present invention designs a thermal infrared video ROI tracking method based on a fully convolutional one-stage object detection (FCOS) neural network to robustly acquire the coordinate position of the facial ROI area in the thermal infrared video, thereby ensuring that a stable and reliable respiratory wave signal is obtained from the thermal infrared video in complex and changeable scenes.

[0078] like Figure 3 As shown, the FCOS neural network includes a ResNet50 backbone network, a feature pyramid, and a shared detection head. Correspondingly, the specific implementation steps of facial ROI tracking based on thermal infrared video are:

[0079] 1) In the ResNet50 backbone network: the current image frame is continuously subjected to five layers of convolution, wherein the last three layers of convolution respectively extract image features C3, C4, and C5 of three different scales, and these feature information will be subsequently used for ROI classification and positioning.

[0080] 2) In the feature pyramid: C3, C4, and C5 are respectively subjected to the corresponding 2DCNN with a convolution kernel of 1×1; the processed C5 is used as D5, the processed C5 is upsampled and concatenated with the processed C4 to obtain D4, and the processed C4 is upsampled and concatenated with the processed C3 to obtain D3; D5, D4, and D3 are input as follows Figure 4 The feature extraction layer shown in the figure is formed by passing through the corresponding 2DCNN with a convolution kernel of 3×3 to form P3, P4, and P5, and P5 is passed through the 2DCNN with a convolution kernel of 3×3 twice to form P6 and P7 respectively. Finally, the feature extraction layer extracts features of five scales, P3, P4, P5, P6, and P7.

[0081] 3) In the detection head: The purpose of the detection head is to classify the features extracted from the feature pyramid, regress the bounding box and predict the center of the target, and the features of the above five scales share one detection head. P3, P4, P5, P6, and P7 pass through four 2DCNN-GN-ReLU layers on different branches, and then pass through the corresponding convolution layer with a convolution kernel of 3×3, and output the facial ROI category, ROI coordinates and confidence on the current image frame respectively.

[0082] It can be understood that the above-mentioned FCOS network demonstrates good robustness and generalization through methods such as anchor-free frame design and multi-scale feature fusion; its fully convolutional structure and end-to-end training process further enhance the model's adaptability to different data sets and tasks, enabling it to maintain efficient measurement performance in various complex scenarios.

[0083] (2) Absorption signal extraction

[0084] Since the temperature change in the nose area is the most obvious during the human breathing process, the embodiment of the present invention is set to extract the initial respiratory wave signal based on the image sequence of the nose area of ​​the thermal infrared facial video. During normal breathing, the gas exhaled from the mouth and nose is mainly carbon dioxide, which is produced by the metabolism in the exhaled gas from the nose. The temperature of the exhaled gas is significantly higher than the temperature of the air around the nose, while the inhaled air is significantly lower than the temperature of the exhaled gas. This causes a slight change in the temperature of the air around the nose, which is manifested as a change in the average pixel value in the thermal infrared face area of ​​interest.

[0085] Correspondingly, the specific implementation steps of absorbing signal extraction are:

[0086] Calculate the average pixel value (AP) of the nose area of ​​the face in the current image frame t:

[0087]

[0088] Among them, pixel ij is the coordinate (x i ,y j ), (x i ,y j ) is the plane coordinate of the pixel located at the i-th row and j-th column.

[0089] After traversing and calculating the average pixel value of the nose area in each image frame, the initial breathing wave signal is formed:

[0090]

[0091] Where m is the total number of frames of the real-time thermal infrared video.

[0092] In step S3, for the initial respiratory wave signal, based on the one-dimensional convolution layer and attention mechanism layer of the adaptive respiratory signal denoising network, the key features of the signal are captured to obtain the denoised respiratory wave signal.

[0093] Since the initially processed respiratory wave signal has some noise, it cannot be directly applied to the accurate estimation of respiratory rate, so it is necessary to perform noise reduction on the initial respiratory wave signal. However, the traditional respiratory wave signal is basically realized by normalization, detrending, bandpass filtering, etc., which relies on experience or pre-set frequency range for manual configuration. The fixed setting range cannot cope with the changing respiratory patterns and scenarios, and lacks adaptability and intelligent processing capabilities.

[0094] Therefore, in order to improve the anti-interference robustness of respiratory signal processing, the embodiment of the present invention designs the following Figure 5The "1DCNN+ATTENTION" adaptive denoising network shown in the figure is designed to effectively capture the key features of the signal and extract the respiratory features in the physiological information to achieve automatic denoising of the initial respiratory wave signal.

[0095] In terms of the overall network architecture, Figure 5 As shown, the adaptive breathing signal denoising network mainly includes three structures: encoder, attention layer and decoder. Among them:

[0096] The encoder converts the input data from high-dimensional to low-dimensional representation, and extracts features at different levels by superimposing multiple layers of neural networks. The attention layer assigns different weights to the features of different parts obtained, so that the model can dynamically focus on the key parts of the data and help the model identify important fragments in the signal. The decoder gradually generates the target output from the low-dimensional features generated by the encoder, restores the features compressed by the encoder to the same spatial dimension as the input signal, and finally generates the target output.

[0097] The neural network aims to significantly improve the signal noise reduction effect by efficiently capturing the key features of the signal locally, globally and in context. At the same time, the network can accurately extract the respiratory features in the physiological signal, thereby effectively restoring the real respiratory signal with higher quality.

[0098] Specifically, if Figure 5 As shown, the adaptive breathing signal denoising network includes five 1DCNNs, two MaxPooling layers, two UPSAMPLE layers and one Attention layer. This structural design is intended to improve the adaptability and practicality of the network.

[0099] It can be understood that the adaptive signal denoising module based on the "1DCNN+ATTENTION" neural network is a deep learning model, and these network layers can effectively perform adaptive denoising on respiratory signals. Since the network processes one-dimensional respiratory signals, 1DCNN is considered for all five convolutional layers, and the one-dimensional convolutional layer is responsible for performing convolution operations on all respiratory signals. Through convolution operations, the convolutional layer can extract local features in the signal. These local features can reflect key features of the respiratory signal, such as frequency and waveform, and are very important for understanding the characteristics of the respiratory signal. Specifically, the first two 1DCNN layers are mainly used to extract relevant features such as the input initial respiratory wave signal, and the last three 1DCNN layers are mainly used to restore the extracted high-dimensional features to signals with the same dimension as the input signal, thereby achieving the purpose of adaptive signal denoising.

[0100] The MaxPooling layer divides the features extracted by the convolution layer into several blocks of the same size and selects the maximum value of each block for output. This operation can reduce the spatial dimension of the data while retaining the most important features, thereby reducing the size of the network and making network calculations more efficient. The Dropout layer randomly selects a part of neurons and their connections during the training process, which helps reduce the network's dependence on training samples, reduces the risk of network overfitting, and improves the network's generalization ability. The Attention layer is a technology used to enhance the neural network's ability to focus on key information. The attention mechanism can more effectively reduce the noise of the signal by assigning different weights to different parts of the input signal. In addition, the attention mechanism allows the network to dynamically adjust its focus according to changes in the input signal, thereby achieving adaptive learning, which can help the network identify more signal categories and perform more effective signal noise reduction.

[0101] Correspondingly, the specific steps for automatically reducing the noise of the initial respiratory wave signal are as follows:

[0102] At the encoder, the initial respiratory wave signal is input into the first 1DCNN for convolution calculation to obtain the first local feature, and then the first local feature is input into the first MaxPooling layer for maximum pooling operation to obtain the first down-sampling feature to retain the most important features and reduce the spatial dimension of the data, thereby improving computational efficiency.

[0103] And input the first down-sampled feature into the second 1DCNN for convolution calculation to obtain the second local feature, and then input the second local feature into the second MaxPooling layer for maximum pooling operation to obtain the second down-sampled feature to further compress the feature data and improve calculation efficiency.

[0104] The second down-sampled feature is input into the attention mechanism layer, and different weights are assigned to focus on the key information of the input signal to improve the effect of signal noise reduction and realize adaptive learning.

[0105] At the decoder, the output features of the attention mechanism layer are input into the third 1DCNN for convolution calculation to obtain the third local features, and then the third local features are input into the first UPSAMPLE layer for upsampling operation to obtain the first upsampling features with the same dimension as the input signal.

[0106] The first up-sampled feature is input into the fourth 1DCNN for convolution calculation to obtain the fourth local feature, and then the fourth local feature is input into the second UPSAMPLE layer for up-sampling operation to obtain the second up-sampled feature with the same dimension as the input signal.

[0107] And input the second up-sampled feature into the fifth 1DCNN for convolution calculation, obtain and output the fifth local feature, and use the fifth local feature as the denoised respiratory wave signal.

[0108] After the adaptive signal denoising network, the output form of the denoised respiratory wave signal is as follows:

[0109]

[0110] In step S4, the peak of the noise-reduced respiratory wave signal is detected and the respiratory rate of the subject is calculated.

[0111] Detection of noise-reduced respiratory wave signals The peak of the signal, assuming that the number of detected peaks is called PN, the indexes of the first peak and the last peak are marked as FP and LP respectively, and the average distance between two adjacent peaks is ADP. In addition, since the information of [0, FP] and [LP, TF] is ignored in the calculation of RR, the number of breaths in this part needs to be considered. The total number of frames of real-time thermal infrared video acquired within one minute is TF, and the calculation formula of the respiratory rate RR is as follows:

[0112]

[0113] So far, the embodiment of the present invention has completed the entire process of non-contact respiratory rate detection.

[0114] In particular, in order to improve the prediction effect, during the training stage of the robust non-contact respiratory rate detection framework, the embodiment of the present invention collects facial videos of different scenes and test subjects and records the real respiratory wave signals of the test subjects through a breathing belt to obtain a diverse data set for training, so that it can learn more scenes and relevant features of the test subjects.

[0115] In addition, the robust non-contact respiratory rate detection framework specifically selects the following loss function during the training phase:

[0116]

[0117] Among them, x and y are the predicted respiratory wave signal and the real respiratory band signal respectively, and T is the signal length.

[0118] Embodiment 2:

[0119] The embodiment of the present invention provides a non-contact respiratory rate detection system based on facial thermal imaging, based on a robust non-contact respiratory rate detection framework, the robust non-contact respiratory rate detection framework includes a thermal infrared video signal processing network and an adaptive respiratory signal denoising network; the system includes:

[0120] An acquisition module is used to acquire real-time thermal infrared video of the subject;

[0121] A processing module, for performing facial ROI tracking based on a fully convolutional single-stage target detection neural network of the thermal infrared video signal processing network for an image frame sequence of the real-time thermal infrared video, and obtaining an initial respiratory wave signal based on a change in an average pixel value of a nose region of the tracked face;

[0122] A denoising module, for capturing key features of the signal based on the one-dimensional convolution layer and the attention mechanism layer of the adaptive respiratory signal denoising network for the initial respiratory wave signal, so as to obtain a denoised respiratory wave signal;

[0123] The detection module is used to detect the peak of the noise-reduced respiratory wave signal and calculate the respiratory rate of the subject.

[0124] Embodiment 3:

[0125] An embodiment of the present invention provides a storage medium, characterized in that it stores a computer program for non-contact respiratory rate detection based on facial thermal imaging, wherein the computer program enables a computer to execute the non-contact respiratory rate detection method as described in Example 1.

[0126] Embodiment 4:

[0127] An embodiment of the present invention provides an electronic device, characterized in that it includes:

[0128] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including means for executing the non-contact respiratory rate detection method as described in Example 1.

[0129] It can be understood that the non-contact respiratory rate detection system, storage medium and electronic device based on facial thermal imaging provided in the embodiments of the present invention correspond to the non-contact respiratory rate detection method based on facial thermal imaging provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the non-contact respiratory rate detection method, which will not be repeated here.

[0130] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0131] 1. The embodiment of the present invention designs a thermal infrared video facial ROI area tracking method based on FCOS neural network. Through multi-scene facial data training, the proposed method can quickly, accurately and stably locate the facial region of interest of thermal infrared video to improve the extraction quality of respiratory signals, laying a solid foundation for subsequent signal processing.

[0132] 2. The embodiment of the present invention proposes an adaptive respiratory signal denoising method based on an attention mechanism. Through an end-to-end learning method, the intrinsic connection between local and global information in the respiratory signal is deeply explored, and the denoising strategy is intelligently adjusted, which effectively overcomes the problem of decreased accuracy of physiological parameter estimation in abnormal ranges in traditional signal preprocessing methods.

[0133] 3. The deep learning network of the embodiment of the present invention is designed with full consideration of the diversity of different individuals and environments. Through a large amount of data training, the adaptability and generalization of the model to new situations are enhanced, ensuring stability and reliability in different application scenarios. At the same time, the embodiment of the present invention has also made innovations in network structure design, especially in the introduction of the attention mechanism, which enables the network to focus more on the key information in the signal, improving the pertinence and effectiveness of signal processing.

[0134] 4. The embodiments of the present invention achieve significant improvements in comprehensive performance, including the accuracy of signal extraction, the robustness of noise reduction processing, and the generalization ability of the model.

[0135] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0136] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact respiratory rate detection method based on facial thermal imaging, characterized in that: Based on a robust non-contact breathing rate detection framework, the robust non-contact breathing rate detection framework includes a thermal infrared video signal processing network and an adaptive breathing signal denoising network; The method includes: Obtain real-time thermal infrared video of the subject; For the image frame sequence of the real-time thermal infrared video, a fully convolutional single-stage target detection neural network of the thermal infrared video signal processing network is used to track the facial ROI, and an initial respiratory wave signal is obtained based on the change in the average pixel value of the tracked facial nose area; For the initial respiratory wave signal, based on the one-dimensional convolution layer and the attention mechanism layer of the adaptive respiratory signal denoising network, the key features of the signal are captured to obtain a denoised respiratory wave signal; The peak of the noise-reduced respiratory wave signal is detected, and the respiratory rate of the subject is calculated.

2. The non-contact breathing rate detection method according to claim 1, characterized in that: The fully convolutional single-stage target detection neural network includes a ResNet50 backbone network, a feature pyramid, and a shared detection head; the fully convolutional single-stage target detection neural network based on the thermal infrared video signal processing network performs facial ROI tracking for the image frame sequence of the real-time thermal infrared video; including: In the ResNet50 backbone network: the current image frame is continuously subjected to five layers of convolution, wherein the last three layers of convolution respectively extract image features C3, C4, and C5 of three different scales; In the feature pyramid: C3, C4, and C5 are respectively processed by a 2DCNN with a convolution kernel of 1×1; the processed C5 is used as D5, and the processed C5 is upsampled and concatenated with the processed C4 to obtain D4, and the processed C4 is upsampled and concatenated with the processed C3 to obtain D3; D5, D4, and D3 are input into the feature extraction layer, and are respectively processed by a 2DCNN with a convolution kernel of 3×3 to form P3, P4, and P5, and P5 is separately processed by a 2DCNN with a convolution kernel of 3×3 twice to form P6 and P7 respectively; In the detection head: P3, P4, P5, P6, and P7 are passed through four 2DCNN-GN-ReLU layers on different branches, and then through corresponding convolution layers with a convolution kernel of 3×3, and the facial ROI category, ROI coordinates, and confidence on the current image frame are output respectively.

3. The non-contact breathing rate detection method according to claim 1, characterized in that: The method of obtaining an initial respiratory wave signal based on the change in the average pixel value of the nose area of ​​the tracked face comprises: Calculate the average pixel value of the nose area of ​​the face in the current image frame t: Among them, pixel ij is the coordinate (x i ,y j ), (x i ,y j ) is the plane coordinate of the pixel located at the i-th row and j-th column; After traversing and calculating the average pixel value of the nose area in each image frame, the initial breathing wave signal is formed: Where m is the total number of frames of the real-time thermal infrared video.

4. The non-contact breathing rate detection method according to claim 1, characterized in that: The adaptive breathing signal denoising network specifically includes five 1DCNNs, two MaxPooling layers, two UPSAMPLE layers and one Attention layer; for the initial breathing wave signal, based on the one-dimensional convolution layer and the attention mechanism layer of the adaptive breathing signal denoising network, the key features of the signal are captured to obtain the denoised breathing wave signal; including: At the encoder, the initial respiratory wave signal is input into the first 1DCNN for convolution calculation to obtain the first local feature, and then the first local feature is input into the first MaxPooling layer for maximum pooling operation to obtain the first down-sampling feature; And inputting the first down-sampled feature into a second 1DCNN for convolution calculation to obtain a second local feature, and then inputting the second local feature into a second MaxPooling layer for maximum pooling operation to obtain a second down-sampled feature; Input the second downsampled features into the attention mechanism layer to focus on the key information of the input signal by assigning different weights; At the decoder, the output features of the attention mechanism layer are input into the third 1DCNN for convolution calculation to obtain the third local features, and then the third local features are input into the first UPSAMPLE layer for upsampling operation to obtain the first upsampling features with the same dimension as the input signal; Input the first up-sampled feature into the fourth 1DCNN for convolution calculation to obtain a fourth local feature, and then input the fourth local feature into the second UPSAMPLE layer for up-sampling operation to obtain a second up-sampled feature with the same dimension as the input signal; And input the second up-sampled feature into the fifth 1DCNN for convolution calculation, obtain and output the fifth local feature, and use the fifth local feature as the denoised respiratory wave signal.

5. The non-contact breathing rate detection method according to claim 1, characterized in that: The respiration rate is expressed as: Among them, RR is the respiration rate, PN is the number of detected peaks, the indexes of the first peak and the last peak are marked as FP and LP respectively, the average distance between two adjacent peaks is ADP, and TF is the total number of frames of real-time thermal infrared video acquired within one minute.

6. The non-contact breathing rate detection method according to any one of claims 1 to 5, characterized in that: The robust contactless respiration rate detection framework is trained using a diverse dataset during the training phase.

7. The non-contact breathing rate detection method according to any one of claims 1 to 5, characterized in that: The robust contactless respiration rate detection framework selects the following loss function during the training phase: Among them, x and y are the predicted respiratory wave signal and the real respiratory band signal respectively, and T is the signal length.

8. A non-contact respiratory rate detection system based on facial thermal imaging, characterized in that: Based on a robust non-contact breathing rate detection framework, the robust non-contact breathing rate detection framework includes a thermal infrared video signal processing network and an adaptive breathing signal denoising network; The system includes: An acquisition module is used to acquire real-time thermal infrared video of the subject; A processing module, for performing facial ROI tracking based on a fully convolutional single-stage target detection neural network of the thermal infrared video signal processing network for an image frame sequence of the real-time thermal infrared video, and obtaining an initial respiratory wave signal based on a change in an average pixel value of a nose region of the tracked face; A denoising module, for capturing key features of the signal based on the one-dimensional convolution layer and the attention mechanism layer of the adaptive respiratory signal denoising network for the initial respiratory wave signal, so as to obtain a denoised respiratory wave signal; The detection module is used to detect the peak of the noise-reduced respiratory wave signal and calculate the respiratory rate of the subject.

9. A storage medium, characterized in that: The computer program for non-contact respiratory rate detection based on facial thermal imaging is stored therein, wherein the computer program enables the computer to execute the non-contact respiratory rate detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the non-contact breathing rate detection method according to any one of claims 1 to 7.

Images