Sleep breathing state monitoring method and device based on audio and thermal imaging video collaboration

Abstract

The present invention discloses a method and device for monitoring sleep respiratory status by synergizing audio with thermal imaging video. The method comprises the following steps: synchronously collecting sleep audio and facial thermal imaging video, proposing a new feature of the audio segment, wavelet information entropy, and combining it with the Mel-frequency cepstral coefficient as a feature for distinguishing snoring segments from non-snoring segments; framing the facial thermal imaging video to obtain multiple frames of images, proposing an adaptive measurement region positioning algorithm based on centroid mapping, locating the image measurement region, acquiring the respiratory signal from this region, and denoising it; extracting the time-frequency features of the denoised respiratory signal, combining it with the audio features, training an SVM classifier, monitoring the respiratory status, and distinguishing between normal and abnormal breathing. By integrating the different representations of sleep respiratory status in audio and thermal imaging video, the present invention provides a detailed, accurate, and comfortable monitoring method for sleep respiratory status.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent data monitoring, and in particular to a method and device for monitoring sleep respiratory status by coordinating audio and thermal imaging video. Background Art

[0002] Sleep breathing is closely related to human health, and monitoring sleep breathing status has practical significance.

[0003] Current methods for monitoring sleep breathing status are mainly contact methods. This type of method uses direct contact between the corresponding sensor and the user to detect the corresponding physiological signals and then monitor sleep breathing. Its comfort level is low, and the monitoring results are easily disturbed by the user's body movements, which is not conducive to popularization. Therefore, non-contact methods are considered. Snoring, apnea or hypopnea are different manifestations of sleep breathing status. Therefore, the audio signal of the sleeping user can be obtained and analyzed to identify snoring events, and the breathing signal can be obtained and analyzed to identify apnea and hypopnea events. The two can be combined to comprehensively monitor the sleep breathing status. Infrared thermal imaging captures the changes in nasal cavity temperature during breathing and converts them into changes in imaging grayscale to obtain respiratory signals. As a non-contact, non-invasive, and light-free detection method, it is not only simple to measure, but also does not cause radiation to the body. It is suitable for obtaining respiratory signals in a sleeping environment.

[0004] Existing snoring recognition methods extract features from the time and frequency domains of snoring. However, snoring signals are typically nonlinear and nonstationary, making them difficult to characterize with standard time-frequency features, resulting in low detection accuracy. Existing thermal imaging-based respiration detection methods use temperature information combined with facial shape information to locate the measurement area on the face. This positioning method lacks individual and environmental adaptability and suffers from poor robustness. Inaccurate measurement area positioning can affect the final respiration signal detection results. Summary of the Invention

[0005] The purpose of the present invention is to solve the above-mentioned defects in the prior art and provide a sleep breathing state monitoring method and device that cooperates with audio and thermal imaging video. The method collects audio signals and facial thermal imaging video signals in the sleep state, proposes a new feature of audio signals, wavelet information entropy, and uses it together with Mel-frequency cepstral coefficients as features to distinguish snoring segments from non-snoring segments; uses a deep learning target detection algorithm combined with a newly proposed measurement area positioning algorithm based on centroid mapping to accurately locate the measurement area from the facial thermal imaging video, extracts the respiratory signal, and extracts time-frequency features based on the respiratory signal. The time-frequency features can identify apnea and hypopnea events; combines audio features with time-frequency features of respiratory signals to monitor sleep breathing state and classify sleep breathing state into normal breathing and abnormal breathing.

[0006] The first object of the present invention can be achieved by adopting the following technical solutions:

[0007] A sleep breathing state monitoring method using audio and thermal imaging video collaboration includes the following steps:

[0008] S1. Synchronously collect audio signals in the sleeping environment and thermal imaging video signals of the subject's face; the audio signals are collected using a recording device placed next to the sleeping user, and the thermal imaging video signals are collected using an infrared thermal imager placed directly opposite the sleeping user;

[0009] S2. Performing sound event endpoint detection on the collected audio signal using a double threshold method based on short-time energy and zero-crossing rate to separate the voiced segment and the silent segment signals, thereby obtaining the voiced segment signal x(n);

[0010] S3, extracting a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC from the voiced segment signal x(n);

[0011] S4, extracting a 5-dimensional wavelet information entropy feature vector H from the voiced segment signal x(n);

[0012] S5. Using a deep learning target detection algorithm to locate a region of interest (ROI) in the thermal imaging video signal of the sleeping user's face, using an adaptive measurement region positioning algorithm based on centroid mapping to locate a measurement region based on the ROI, obtaining the grayscale mean of pixels in the measurement region, and normalizing the result to obtain an original respiratory signal y; wherein the ROI is the nose region of the face, and the measurement region is the nasal cavity region;

[0013] S6. De-noising the original respiratory signal y by using an empirical wavelet transform algorithm to remove high-frequency noise, and fitting a baseline using a cubic spline interpolation method to remove baseline drift, thereby obtaining a de-noised respiratory signal y';

[0014] S7. Extract a time domain feature vector E and a frequency domain feature vector F from the denoised respiratory signal y'; wherein the time domain feature vector E includes three time domain features: the number of upper envelope peaks N, the variance of the upper envelope peaks V, and the variance of the normalized short-time energy VE, denoted as E = [N, V, VE]; the frequency domain feature vector F includes the minimum value M of the instantaneous frequency, the variance VF of the instantaneous frequency, and the range G of the instantaneous frequency, denoted as F = [M, VF, G];

[0015] S8. Concatenate the Mel-frequency cepstral coefficient eigenvector MFCC, the information entropy eigenvector H, and the time-domain eigenvector E and frequency-domain eigenvector F of the thermal imaging respiratory signal into a final eigenvector P, denoted as P = [MFCC, H, E, F]. Use the eigenvector P of the training samples with known respiratory states to train an SVM classifier to obtain a respiratory state binary classification model.

[0016] The information in the training samples used to train the SVM classifier was collected in a sleep laboratory. While collecting audio signals and thermal imaging video signals, polysomnography was used to simultaneously monitor breathing and record the physiological data obtained by the polysomnography for subsequent classification of respiratory states.

[0017] S9. Using the breathing state binary classification model, the breathing state of the sleeping user is judged based on the audio signal and the breathing signal extracted from the thermal imaging video, and the breathing state is determined to be normal breathing or abnormal breathing.

[0018] Furthermore, the step S2 uses a double threshold method based on short-time energy and zero-crossing rate to perform sound event endpoint detection;

[0019] Among them, short-time energy is the energy of each frame signal. For noisy signals, the energy of the voiced segment includes the energy of the signal and noise, while the silent segment only includes the energy of the noise. Therefore, there is a significant difference in energy between the two. This feature can be used to distinguish between voiced and silent segments. The zero-crossing rate refers to the number of times the amplitude of the signal sampling point in each frame changes from positive to negative, or from negative to positive. It can roughly describe the frequency of the signal. Combining it with the short-time energy can achieve accurate detection of the endpoint of the sound event.

[0020] Furthermore, the process of step S3 is as follows:

[0021] S301, pre-emphasize, frame, and window the voiced segment signal x(n), assuming the frame length is N1 and the frame shift is Frame division to obtain M1 frame;

[0022] Pre-emphasis refers to the use of a first-order high-pass filter to process the signal. Its function is to improve the signal-to-noise ratio of high-frequency signals so that they will not be severely attenuated during transmission and affect the signal quality. In addition, pre-emphasis processing also helps to offset the influence of the vocal cords and lips on pronunciation, thereby compensating for the defect of the high-frequency part of the speech signal being suppressed and making the formant detection results more accurate. Framing is to divide several sampling points of the signal into a frame to ensure that the signal within a frame is stable and can be subjected to Fourier analysis. The role of framing is to reduce the complexity of signal processing. Windowing is to multiply each frame of the signal by a specific window function. Its function is to make the signal transition smooth in the frequency domain and reduce the impact of spectral leakage.

[0023] S302, calculate the short-time energy of the pre-processed M1 frame signal frame by frame, and perform discrete Fourier transform, Mel filtering, logarithm, and inverse Fourier transform to obtain the 13-dimensional Mel-frequency cepstral coefficient features of each frame, forming a 13-dimensional Mel-frequency cepstral coefficient feature vector Y j(i),i=1,2,...,M1,j=1,2,...,13;

[0024] Mel-frequency cepstral coefficients are commonly used features in acoustic signal processing. They are cepstral parameters extracted in the Mel-scale frequency domain. The Mel scale describes the nonlinear characteristics of the human ear frequency and is more consistent with human auditory characteristics. They still have good recognition performance when the signal-to-noise ratio is low, and can be used to reflect the difference between snoring and non-snoring segments.

[0025] S303, add the time variation to the 13-dimensional Mel cepstral coefficient feature vector Yj(i), that is, calculate ΔY j (i) and ΔΔY j (i), ΔY j (i) represents the cepstral coefficient Y between frames j (i) Change, ΔΔY j (i) represents the ΔY between frames j The change of (i) is expressed as:

[0026]

[0027] Among them, the dynamic changes of Mel-frequency cepstral coefficients between frames can also represent acoustic features. Combining them with Mel-frequency cepstral coefficients can more accurately depict the difference between snoring and non-snoring segments.

[0028] Based on the i-th frame, we average Yj(i), ΔYj(i), and ΔΔYj(i) to obtain AVG(Yj), AVG(ΔYj), and AVG(ΔΔYj). The formulas are as follows:

[0029]

[0030] The above three 13-dimensional feature vectors are concatenated into a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC, which is recorded as MFCC = [AVG (Y j ),AVG(ΔY j ),AVG(ΔΔY j )].

[0031] Furthermore, the process of step S4 is as follows:

[0032] S401, the voiced segment signal x(n) is frame-shifted by a frame length of N2. Divide into M2 frames and get M2 frame signal x u (n),u=1,2,...,M2;

[0033] S402, M2 frame signal x u Each frame signal in (n) is decomposed into four layers of discrete wavelet to obtain the fourth layer low-frequency signal xu,4L (n);

[0034] Discrete wavelet decomposition discretizes the scale and translation of the basic wavelet. The principle is to pass the original signal through a set of high-pass and low-pass filters to obtain a low-frequency component and a high-frequency component. The low-frequency component represents the characteristics of the signal and is also called the approximation coefficient. The high-frequency component represents the details or differences of the signal and is therefore also called the detail coefficient. Since the signal characteristics of snoring and non-snoring segments are significantly different, their low-frequency components can be analyzed to distinguish them.

[0035] Among them, the four-layer discrete wavelet decomposition is to convert x u (n) After passing through the fourth discrete wavelet low-pass filter, formula (3) is the response function of the discrete wavelet low-pass filter, and the low-frequency signal x is obtained by formula (4) u,L (n):

[0036]

[0037]

[0038] Where τ is time, x u (n) is the original signal, x u (2n) is x u (n) The signal obtained after two times downsampling; the obtained low-frequency signal x u,L (n) Pass through the low-pass filter again, repeat 4 times, and get the fourth layer of low-frequency signal x u,4L (n);

[0039] S403, first calculate the fourth layer low frequency component x u,4L (n),u=1,2,...,the information entropy h of M2 u , then find M2 information entropy h u The mean AVG(h u )、VAR(h u ), median MID(h u ), maximum value MAX(h u )、Minimum value MIN(h u ), and obtain the five-dimensional wavelet information entropy feature vector H, which is recorded as H = [AVG (hu), VAR (hu), MID (hu), MAX (hu), MIN (hu)];

[0040] Among them, information entropy reflects the complexity of the signal. The more complex the signal is, the more different situations there are, and the greater its information entropy. The information entropy of the low-frequency component of each frame of snoring fluctuates less and is relatively stable. The information entropy of the low-frequency component of each frame of non-snoring fluctuates frequently and with a large amplitude. There are also significant differences in the distribution range and amplitude of the two. Therefore, the low-frequency component information entropy is used to accurately distinguish snoring and non-snoring segments.

[0041] Among them, x u,4L The information entropy h of (n) u The calculation method is: Let x u,4L The number of sampling points (n) is Q, and the minimum value of the sampling point is X min , the maximum value is X max , the interval [X min ,X max ] is divided into D (D<Q) subintervals, and the length of each subinterval is Let c=1,2,...,D, and record x u,4L (n) The median is greater than or equal to X min +(c-1)×SPB, less than X min +c×SPB The number of sampling points is n c , use formula (5) to calculate x u,4L (n) D equally divided information entropy h u :

[0042]

[0043] Furthermore, the process of step S5 is as follows:

[0044] S501, dividing the collected facial thermal imaging video into frames to obtain multiple frames of single-channel grayscale images arranged in time;

[0045] S502: Fusing publicly available online data with self-collected facial thermal imaging data to create a target detection dataset. Labelme, a labeling tool, is used to annotate the nose region of each image in the dataset as a region of interest to be identified. A deep learning target detection model is trained using the target detection dataset. The deep learning target detection model is used to locate the region of interest in each frame of the multiple single-channel grayscale images.

[0046] Among them, the deep learning target detection model consists of a convolutional layer, a region of interest pooling layer, a fully connected layer and a bounding box regression layer. First, the entire image to be detected is input into the convolutional layer to obtain a convolution feature map containing the final features; then the region of interest pooling layer is used to pool the regions in the convolution feature map, mapping regions of interest of different sizes into feature maps of the same size, thereby facilitating subsequent processing; next, two fully connected layers are used to generate a feature vector of the region of interest, and the category probability of the target object is output based on the feature vector; finally, the bounding box regression layer is used, which outputs the bounding box position of the target object to obtain the target detection result; the deep learning target detection model can accurately locate the region of interest in both indoor and outdoor scenes, at different brightness levels, at different distances, and in multi-target scenes, and has strong robustness;

[0047] S503, selecting a frame with the largest pixel mean in the region of interest as the clearest frame;

[0048] S504, first calculate the coordinates of the center of mass of the region of interest of the clearest frame (P x ,P y ); then, a threshold segmentation method is used to obtain a measurement area based on the region of interest of the clearest frame; finally, a position mapping relationship between all pixels in the measurement area and the centroid pixel of the region of interest is established, and the position mapping relationship is used to subsequently locate the measurement area of ​​other frames;

[0049] The centroid of the connected area of ​​the image refers to the average position of the grayscale value distribution of all pixels. Its position is only related to the grayscale value of each pixel and its relative position in the area. In the region of interest, the relative position of the centroid and the measurement area remains basically unchanged. Therefore, the measurement area of ​​each frame of image can be tracked by locating the centroid of each frame of image.

[0050] Among them, the formula for obtaining the coordinates of the centroid of the region of interest is:

[0051]

[0052] Where w represents the number of pixels in the region of interest, Indicates the X-direction coordinate position of the s-th pixel point in the region of interest, Indicates the Y coordinate position of the sth pixel point in the region of interest, v s Represents the gray value of the s-th pixel in the region of interest;

[0053] The threshold segmentation method resets the grayscale of pixels in the region of interest with a grayscale less than a threshold of 128 to 0, while the grayscale of pixels greater than or equal to the threshold of 128 remains unchanged. Since the grayscale value of the pixels in the measurement area of ​​this frame is greater than 128, and the grayscale value of pixels in other areas is less than 128, the threshold segmentation method can accurately obtain the measurement area. The final measurement area is the area composed of pixels with non-zero grayscale.

[0054] The process of obtaining the position mapping relationship is as follows: suppose there are R pixels in the measurement area, then the coordinate vector composed of all the pixels in the measurement area is (P xo ,P yo ),o=1,2,...,R, the centroid coordinates of the region of interest are (P x ,P y ), then the position mapping relationship is:

[0055]

[0056] Where, Indicates the distance between the oth pixel point and the centroid in the X direction. Indicates the distance between the oth pixel point and the centroid in the Y direction. After locating the centroid coordinates of other frames, the coordinates of all pixels in the measurement area of ​​the frame can be obtained through this mapping relationship.

[0057] S505. Locate the centroid of the region of interest of the other frames using formula (7), then use the position mapping relationship to obtain the coordinate vectors of the pixel points in the measurement area of ​​the other frames, set the grayscale of the other pixel points outside the coordinate vector to 0, and obtain the measurement area of ​​each frame;

[0058] S506 , obtaining the grayscale mean value of the measurement area of ​​each frame, performing fitting with time as the horizontal axis, and obtaining the original respiratory signal y after normalization.

[0059] Among them, since people exchange gases with the outside world through their noses during breathing, this process causes the nasal cavity temperature to change periodically over time. This change process is captured by the thermal imager and converted into changes in the imaging grayscale. Therefore, the grayscale mean of the nasal cavity in the image measurement area can be regarded as the amplitude of the respiratory signal at that moment, and the curve obtained by fitting it according to time can be regarded as the original respiratory signal.

[0060] Furthermore, the process of step S6 is as follows:

[0061] S601, after removing the DC component, decompose the original respiratory signal y into three components IMF1, IMF2, and IMF3 using an empirical wavelet transform, and select the one with the same spectral peak as the peak of the signal before decomposition as the low-noise respiratory signal with high-frequency noise removed;

[0062] The principle of empirical wavelet decomposition is to segment the signal spectrum, construct a suitable wavelet filter bank, and decompose the signal into components of different frequencies. Component screening involves performing fast Fourier transforms on the signal before decomposition and the three components, obtaining the spectrum and the peak value of the spectrum. The frequency corresponding to the peak value can be regarded as the frequency corresponding to the main part of the signal. The one with the same peak frequency as the signal before decomposition is the desired low-noise breathing signal.

[0063] S602, for the low-noise respiratory signal, locate all trough points, use cubic spline interpolation to fit the positions of all trough points to obtain a baseline drift curve, subtract the baseline drift curve from the low-noise respiratory signal to obtain a denoised respiratory signal y';

[0064] Among them, cubic spline interpolation refers to constructing a cubic spline interpolation function between every two adjacent trough points, so that the function value and differential value at these two points are the same. In this way, the data between every two adjacent trough points can be calculated by the cubic spline interpolation function, and then all trough points can be fitted to obtain the baseline drift curve. After removing the baseline drift by the cubic spline interpolation method, the trough points of the respiratory signal are basically located near the value of 0, which effectively removes the baseline drift noise in the respiratory signal.

[0065] Furthermore, the time-frequency features of the respiratory signal extracted in step S7 can characterize the differences in respiratory depth, frequency, and rhythm between normal breathing and apnea or hypopnea events, reflecting the physiological characteristics of apnea and hypopnea. In this way, the feature vector is extracted to identify apnea and hypopnea events.

[0066] Furthermore, the process of step S8 is as follows:

[0067] S801, concatenate the Mel-frequency cepstral coefficient feature vector MFCC of the voiced audio segment, the wavelet information entropy feature vector H, and the time domain feature vector E and frequency domain feature vector F of the respiratory signal into a final feature vector P;

[0068] The two feature vectors of the audio segment can distinguish snoring from non-snoring segments, and the time-frequency feature vector of the respiratory signal can identify apnea and hypopnea events. Snoring and apnea-hypopnea events are important indicators of respiratory abnormalities, so the combined feature vector can identify the respiratory state.

[0069] S802. Extract the feature vectors P of all training samples to form a feature vector set x. Use the respiratory state in the analysis results of the data obtained from the polysomnography of the same period as the classification standard to divide the training samples into normal breathing and abnormal breathing, and set a label k for each sample, k∈{0,1}, where 0 represents normal breathing and 1 represents abnormal breathing. Input the sample label and the feature vector x into the SVM classifier and train the SVM classifier. Through training the SVM classifier, establish a mapping relationship between the feature vector and the respiratory state, and obtain a two-classification model of the respiratory state.

[0070] The second object of the present invention can be achieved by adopting the following technical solutions:

[0071] A sleep breathing state monitoring device that cooperates with audio and thermal imaging video, the sleep breathing state monitoring device comprising:

[0072] A signal acquisition module is used to synchronously collect audio signals in the sleeping environment and thermal imaging video signals of the subject's face; the audio signals are collected using a recording device placed next to the sleeping user, and the thermal imaging video signals are collected using an infrared thermal imager placed directly opposite the sleeping user;

[0073] The signal separation module is used to perform sound event endpoint detection on the collected audio signal using a double threshold method based on short-time energy and zero-crossing rate, separate the voiced segment and the silent segment signals, and obtain the voiced segment signal x(n);

[0074] A first extraction module is configured to extract a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC from the voiced segment signal x(n);

[0075] A second extraction module is used to extract a 5-dimensional wavelet information entropy feature vector H from the voiced segment signal x(n);

[0076] a feature processing module that uses a deep learning target detection algorithm to locate a region of interest (ROI) in the thermal imaging video signal of the sleeping user's face, uses an adaptive measurement region positioning algorithm based on centroid mapping to locate a measurement region based on the ROI, obtains the grayscale mean of pixels in the measurement region, and normalizes the result to obtain the original respiratory signal y; wherein the ROI refers to the nose region of the face, and the measurement region refers to the nasal cavity region;

[0077] a denoising processing module, configured to perform denoising on the original respiratory signal y, remove high-frequency noise using an empirical wavelet transform algorithm, and fit a baseline using a cubic spline interpolation method to remove baseline drift, thereby obtaining a denoised respiratory signal y';

[0078] A feature extraction module is used to extract a time domain feature vector E and a frequency domain feature vector F from the denoised respiratory signal; wherein the time domain feature vector E includes the number of upper envelope peaks N, the variance of the upper envelope peaks V, and the variance of the normalized short-time energy VE, denoted as E = [N, V, VE]; the frequency domain feature vector F includes the minimum value M of the instantaneous frequency, the variance VF of the instantaneous frequency, and the range G of the instantaneous frequency, denoted as F = [M, VF, G];

[0079] The feature concatenation module concatenates the Mel-frequency cepstral coefficient feature vector MFCC, the information entropy feature vector H, and the time domain feature vector E and frequency domain feature vector F of the thermal imaging respiratory signal into the final feature vector P, denoted as P = [MFCC, H, E, F]. The feature vector P of the training samples with known respiratory states is used to train the SVM classifier to obtain a two-class respiratory state classification model.

[0080] The information in the training samples used to train the SVM classifier was collected in a sleep laboratory. While collecting audio signals and thermal imaging video signals, polysomnography was used to simultaneously monitor breathing and record the physiological data obtained by the polysomnography for subsequent classification of respiratory states.

[0081] The state judgment module uses the respiratory state binary classification model to judge the respiratory state of the sleeping user based on the audio signal and the respiratory signal extracted from the thermal imaging video, and determines the respiratory state as normal breathing or abnormal breathing.

[0082] The present invention has the following advantages and effects compared to the prior art:

[0083] 1) This invention achieves comprehensive, non-contact monitoring of sleep breathing status by integrating the distinct representations of sleep breathing through audio and thermal imaging video. Compared to other contact-based methods, this method avoids the issues of inconvenient equipment use, poor user experience, and low measurement accuracy. Furthermore, compared to single-parameter monitoring methods, the multi-parameter fusion monitoring method produces more accurate and robust results.

[0084] 2) The audio feature wavelet information entropy extracted by this invention uses the nonlinearity of the audio signal to more accurately describe the difference between snoring and other nighttime noises. Existing methods only use traditional Mel-frequency cepstral coefficient features to distinguish snoring segments from noise segments, which leads to inaccurate feature descriptions and low snoring recognition accuracy. Combining wavelet signal entropy with Mel-frequency cepstral coefficients can effectively enhance the accuracy of distinguishing snoring from noise, thereby more accurately determining respiratory status.

[0085] 3) The present invention proposes a new measurement area positioning method for acquiring respiratory signals based on infrared thermal imaging, which makes the acquired respiratory signals more accurate. The present invention uses a deep learning target detection algorithm to locate the region of interest, and combines it with a newly proposed measurement area positioning algorithm based on centroid mapping to further reduce the region of interest to the measurement area, eliminating the noise caused by irrelevant areas. Compared with the facial measurement area located by the existing traditional morphological processing method, the positioning accuracy and robustness of this method are significantly improved, and thus the accuracy of the acquired respiratory signal is also significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0087] Figure 1 This is a flowchart of a non-contact sleep respiratory status monitoring method using audio and thermal imaging video collaboration disclosed in Example 1 of the present invention;

[0088] Figure 2 1 is a diagram showing the result of endpoint detection of a sound signal in Example 1 of the present invention;

[0089] Figure 3 Flowchart of extracting the wavelet information entropy feature vector of the voiced segment signal in embodiment 1 of the present invention;

[0090] Figure 4 This is a flowchart of the four-layer discrete wavelet decomposition in Example 1 of the present invention;

[0091] Figure 5 This is a flow chart of obtaining an original respiratory signal from a facial thermal imaging video in Example 1 of the present invention;

[0092] Figure 6 is a result diagram of each process of locating the measurement area from the thermal imaging video in Example 1 of the present invention;

[0093] Figure 6 (a) Figure 6 (b) Figure 6 (c) Figure 6 (d) is a graph showing the results of the first, second, third, and fourth processes of locating the measurement area from the thermal imaging video in Example 1 of the present invention;

[0094] Figure 7 1 is a diagram showing the respiratory signal detection result of the subject to be tested in Example 2 of the present invention;

[0095] Figure 8 2 is a structural diagram of a sleep respiratory status monitoring device in Example 3 of the present invention. DETAILED DESCRIPTION

[0096] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0097] Example 1

[0098] This embodiment discloses a sleep breathing state monitoring method using audio and thermal imaging video. Figure 1 The specific steps are as follows:

[0099] S1. Synchronously collect audio signals in a sleeping environment and facial thermal imaging video signals of the subject to be tested.

[0100] The audio signal is collected using a recording device placed 30 cm away from the side of the object to be measured, and the thermal imaging video signal is collected using an infrared thermal imager placed 1 m away from the object to be measured.

[0101] In this embodiment, 30 seconds of synchronous audio and thermal imaging video signals are extracted and analyzed at a time to obtain the breathing state of the subject to be measured during this time period.

[0102] S2. Perform sound event endpoint detection on the audio signal collected by S1 using a double threshold method based on short-time energy and zero-crossing rate, separate the voiced segment and silent segment signals, and obtain the voiced segment signal x(n).

[0103] The dual-threshold method based on short-time energy and zero-crossing rate is used to obtain voiced signals as follows: First, the signal is framed and the short-time energy and short-time zero-crossing rate are calculated for each frame. Then, a threshold T1 is set for the short-time energy and a threshold T2 is set for the short-time zero-crossing rate. Signals with short-time energy greater than T1 and short-time zero-crossing rate greater than T2 are selected as voiced signals. Figure 2 This is the result diagram of the voiced signal obtained by the double threshold method, where the black solid line and dotted line correspond to the starting point and end point of the voiced signal, respectively.

[0104] The short-time energy refers to the energy of each frame signal, and the short-time zero-crossing rate refers to the number of times the sampling point of each frame signal changes from positive to negative, or from negative to positive.

[0105] S3. Extract a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC from the voiced segment signal x(n).

[0106] Mel-frequency cepstral coefficients are commonly used features in the field of acoustic signal processing. They are cepstral parameters extracted in the Mel-scale frequency domain. The Mel-scale describes the nonlinear characteristics of the human ear frequency, which is more consistent with human auditory characteristics. It still has good recognition performance when the signal-to-noise ratio is low, and can be used to reflect the difference between snoring and non-snoring segments. The specific process of extracting the Mel-frequency cepstral coefficient feature vector is as follows:

[0107] S301 : Pre-process the voiced segment signal x(n) by pre-emphasis, framing, and windowing.

[0108] In this embodiment, the frame length of the divided frame is 1024 sampling points, the frame shift is 512 sampling points, and the divided frame obtains 856 frames.

[0109] Among them, since the main lobe width of the Hamming window is large and the first side lobe attenuation is significant, the spectrum leakage problem can be well avoided. Therefore, in this embodiment, a Hamming window is added to the signal.

[0110] S302, calculate the short-time energy of the 856 frames of preprocessed signals, and perform discrete Fourier transform, Mel filtering, logarithm, and inverse Fourier transform to obtain the 13-dimensional Mel-frequency cepstral coefficient features of each frame, forming a 13-dimensional Mel-frequency cepstral coefficient vector Y j (i), i=1,2,...,856, j=1,2,...,13.

[0111] S303, 13-dimensional Mel-frequency cepstral coefficient eigenvector Y j (i) Extract the 13-dimensional first-order difference vector ΔY according to formula (1) j (i) and 13-dimensional second-order difference vector ΔΔY j (i). According to formula (2), j (i) ΔY j (i), ΔΔY j (i) Based on the average of the i-th frame, we get AVG(Y j )、AVG(ΔY j )、AVG(ΔΔY j ). These three feature vectors are concatenated into a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC, recorded as MFCC = [AVG (Y j ),AVG(ΔY j ),AVG(ΔΔY j )].

[0112] S4. Extract a 5-dimensional wavelet information entropy feature vector H from the voiced segment signal x(n).

[0113] Among them, wavelet information entropy reflects the complexity of the sound signal, accurately describes the nonlinear characteristics of snoring, and can accurately distinguish snoring from non-snoring segments.

[0114] Figure 3 This is the flow chart of wavelet information entropy extraction. The specific process is as follows:

[0115] S401: Divide the voiced segment signal x(n) into frames.

[0116] In this embodiment, the frame length of the frame is 1024 sampling points, the frame shift is 512 sampling points, and 856 frames of signal are obtained after framing, which is recorded as x u (n),u=1,2,...,856.

[0117] S402: the signal x u Each frame signal in (n) is decomposed into four layers of discrete wavelet according to formula (4), and the fourth layer low-frequency signal x is obtained u,4L (n). Figure 4 This is the flowchart of the four-layer discrete wavelet decomposition of the signal.

[0118] S403, using formula (5) to calculate the fourth layer low-frequency signal x u,4L (n) D equally divided information entropy h u In this embodiment, D=6, that is, the calculated signal x u,4L The information entropy of the 6 equal parts of (n). Find h u The mean AVG(h u )、VAR(h u ), median MID(h u ), maximum value MAX(h u )、Minimum value MIN(h u ), we get the 5-dimensional wavelet information entropy feature vector H, which is recorded as H = [AVG (h u ),VAR(h u ),MID(h u ),MAX(h u ),MIN(h u )].

[0119] S5. Extracting an original breathing signal y from the facial thermal imaging video signal of the subject to be measured.

[0120] Among them, because people exchange gases with the outside world through their noses during breathing, this process causes the nasal cavity temperature to change periodically over time. This change process is captured by the thermal imager and converted into changes in the imaging grayscale. With time as the horizontal axis value and grayscale as the vertical axis value, a curve is fitted, which can be regarded as the original respiratory signal. Therefore, to extract the respiratory signal, the video must first be divided into multiple frames of grayscale images sorted by time, and then the measurement area of ​​each frame of the image - the nasal cavity - is located. The grayscale mean of the area is obtained, and the grayscale mean amplitude is normalized to obtain the original respiratory signal by fitting. Therefore, how to accurately locate the measurement area of ​​each frame of the image is the key. This process is completed using a deep learning target detection algorithm combined with a measurement area positioning algorithm based on centroid mapping.

[0121] Figure 5 The flowchart for extracting the original respiratory signal is as follows:

[0122] S501, the collected facial thermal imaging video is divided into frames to obtain multiple frames of single-channel grayscale images arranged in time, such as Figure 6 As shown in (a).

[0123] S502. Fuse publicly available online data and self-collected facial thermal imaging data to establish a target detection dataset. Use the labeling tool Labelme to label the nose area of ​​each image in the dataset as the region of interest to be identified. Use the target detection dataset to train a deep learning target detection model. Use the deep learning target detection model to locate the region of interest in each frame of the above-mentioned multi-frame single-channel grayscale images. Figure 6 (b) is the obtained result map of the region of interest.

[0124] The deep learning target detection model consists of a convolutional layer, a region of interest pooling layer, a fully connected layer, and a bounding box regression layer. First, the entire image to be detected is input into the convolutional layer to obtain a convolutional feature map containing the final features. The region of interest pooling layer then performs a pooling operation on the regions in the convolutional feature map, mapping regions of interest of different sizes into feature maps of the same size, thereby facilitating subsequent processing. Next, two fully connected layers generate a feature vector for the region of interest, and based on this feature vector, the class probability of the target object is output. Finally, the bounding box regression layer outputs the bounding box position of the target object, resulting in the target detection result.

[0125] S503 , selecting a frame with the largest pixel mean value in the region of interest as the clearest frame.

[0126] S504, first use formula (6) to obtain the centroid coordinates (P x ,P y ),like Figure 6As shown on the left side of (c), the bright spot in the image is the centroid. Then the threshold segmentation method is used to obtain the measurement area based on the region of interest in the clearest frame, as shown in Figure 6 As shown on the right side of (c), the measurement area is the area outside the black area. Finally, according to formula (7), the position mapping relationship between all pixels in the measurement area and the centroid pixel of the region of interest is obtained, so that the measurement area of ​​other frames can be located through this mapping relationship.

[0127] Among them, the threshold segmentation method resets the grayscale of pixels in the region of interest with a grayscale less than a threshold of 128 to 0, and the pixels with a grayscale of 0 appear as pure black. The grayscale of pixels greater than or equal to the threshold of 128 remains unchanged. The measurement area is the set of pixels with non-zero grayscale values.

[0128] S505: Obtain the measurement area of ​​each frame through the position mapping relationship, such as Figure 6 (d) First, the centroid of the region of interest of other frames is located using formula (7), and then the coordinate vectors of the pixel points in the measurement area of ​​other frames are obtained using the position mapping relationship. The grayscale of other pixels outside the coordinate vector is set to 0 to obtain the measurement area of ​​each frame.

[0129] S506 , obtaining the grayscale mean value of the measurement area of ​​each frame, performing fitting with time as the horizontal axis, and obtaining the original respiratory signal y after normalization.

[0130] The time is the current frame number divided by the sampling rate per second Fs of the infrared thermal imager. In this embodiment, the sampling rate per second Fs of the infrared thermal imager is 25.

[0131] S6. Perform denoising on the original respiratory signal y to obtain a denoised respiratory signal y′.

[0132] The high-frequency noise and baseline drift in the original respiratory signal are mainly caused by environmental influences and the power frequency noise of the hardware equipment itself. The denoising process is as follows:

[0133] S601: Apply empirical wavelet transform to the original respiratory signal y to remove high-frequency noise, thereby obtaining a low-noise respiratory signal. First, the DC component of the original signal is removed. Then, empirical wavelet decomposition is used to decompose the DC-depleted signal into three components: IMF1, IMF2, and IMF3. Finally, component screening is performed to identify the desired low-noise respiratory signal.

[0134] Among them, the principle of empirical wavelet decomposition is to divide the spectrum of the signal, construct a suitable wavelet filter bank, and decompose the signal.

[0135] Among them, component screening is to perform fast Fourier transform on the signal before decomposition and the three components respectively, obtain the spectrum, and get the spectrum peak. The frequency corresponding to the peak can be regarded as the frequency corresponding to the main part of the signal. The one with the same peak frequency as the signal before decomposition among the three components is the required low-noise breathing signal.

[0136] S602 : For the low-noise respiratory signal, locate all trough points, fit the positions of all trough points using cubic spline interpolation to obtain a baseline drift curve, and subtract the baseline drift curve from the low-noise respiratory signal to obtain a denoised respiratory signal y′.

[0137] The trough points are the minimum points on the respiratory signal. If the value of a point is smaller than the values ​​of the two adjacent points, it is determined to be a minimum point.

[0138] S7. Extract three time domain features and three frequency domain features from the denoised respiratory signal y' to obtain a three-dimensional time domain feature vector E and a three-dimensional frequency domain feature vector F;

[0139] The time domain features include: the number of upper envelope peaks N, the variance of the upper envelope peaks V, and the variance of the normalized short-time energy VE. The frequency domain features include: the minimum instantaneous frequency M, the variance of the instantaneous frequency VF, and the range of the instantaneous frequency G. Therefore, the eigenvector E is recorded as E = [N, V, VE], and the eigenvector F is recorded as F = [M, VF, G].

[0140] S8. Concatenate MFCC, H, E, and F into the final feature vector P. Use the feature vector P of the training sample with known respiratory state to train the SVM classifier to obtain a respiratory state binary classification model.

[0141] Among them, the signal samples used to train the classifier are collected in a sleep laboratory. When collecting audio signals and thermal imaging video signals, a polysomnography is used to synchronously monitor breathing and record the physiological data obtained by the polysomnography for subsequent classification of respiratory status categories.

[0142] The process is as follows:

[0143] S801 , concatenate the Mel-frequency cepstral coefficient feature vector MFCC of the voiced audio segment, the wavelet information entropy feature vector H, and the time domain feature vector E and frequency domain feature vector F of the respiratory signal into a final feature vector P, denoted as P=[MFCC, H, E, F].

[0144] The two feature vectors of the spoken audio segment can distinguish snoring from non-snoring segments, and the time-frequency feature vector of the respiratory signal can identify apnea and hypopnea events. Snoring and apnea-hypopnea events are important indicators of respiratory abnormalities, so the combined feature vector can identify respiratory status.

[0145] S802. Extract the feature vectors P of all training samples to form a feature vector set x. Analyze the data obtained from the polysomnography during the same period and determine the respiratory state from this data. Use this result as a classification criterion to divide the training samples into normal breathing and abnormal breathing. Assign a label k to each sample, k∈{0,1}, where 0 indicates normal breathing and 1 indicates abnormal breathing. Input the sample label and feature vector x into a support vector machine (SVM) classifier and train the SVM classifier. Through training the SVM classifier, a mapping relationship between the feature vector and the respiratory state is established, resulting in a two-class respiratory state classification model.

[0146] S9. Using the binary classification model, the breathing state of the subject to be tested is determined based on the audio and the breathing signal extracted from the thermal imaging video, and the breathing state is determined to be normal breathing or abnormal breathing.

[0147] This example compares the model prediction results with the monitoring results of polysomnography, and the two results are consistent, which proves the reliability of this method for respiratory status monitoring.

[0148] Example 2

[0149] This embodiment further discloses a sleep respiratory status monitoring method using audio and thermal imaging video collaboration, including the following steps:

[0150] S1. Referring to the corresponding steps in Example 1, collect a sleep audio signal and a facial thermal imaging video signal of a subject to be tested.

[0151] In this embodiment, 30 seconds of synchronous audio and video signals are extracted for analysis to obtain the breathing state of the subject during this period.

[0152] S2. Referring to the corresponding steps in Example 1, obtain the voiced segment signal x(n) in the audio signal.

[0153] S3. Referring to the corresponding steps in Example 1, feature extraction is performed on the voiced segment signal x(n) to obtain its Mel-frequency cepstral coefficient feature vector MFCC and wavelet information entropy feature vector H.

[0154] In this example, x(n) is divided into 30ms frames with a 15ms frame shift, resulting in 452 frames. Next, the 39-dimensional Mel-frequency cepstral coefficients of x(n) are extracted frame by frame and averaged to obtain its Mel-frequency cepstral coefficient feature vector. Finally, x(n) is subjected to a four-layer discrete wavelet decomposition frame by frame, and the information entropy of the D equal-divisions is calculated. In this example, D = 8. The mean, variance, median, maximum, and minimum values ​​of the information entropy vectors of the eight equal-divisions are taken to form the wavelet information entropy feature vector.

[0155] S4. Refer to the corresponding steps in Example 1 and do not repeat them here.

[0156] S5. Based on Example 1, the respiratory signal y' of the subject to be measured is obtained as follows: Figure 7 The time domain feature vector E and frequency domain feature vector F of the breathing signal are extracted and concatenated with the audio feature vectors MFCC and H obtained in step S3 to form the final feature vector P. The specific values ​​of each feature in P are shown in Table 1:

[0157] Table 1. Corresponding characteristic value table of the objects to be tested in Example 2

[0158] feature <![CDATA[AVG(Y1)]]> <![CDATA[AVG(Y2)]]> <![CDATA[AVG(Y3)]]> <![CDATA[AVG(Y4)]]> <![CDATA[AVG(Y5)]]> Eigenvalue -4.6590 -6.3991 -4.7347 3.4635 1.3010 feature <![CDATA[AVG(Y6)]]> <![CDATA[AVG(Y7)]]> <![CDATA[AVG(Y8)]]> <![CDATA[AVG(Y9)]]> <![CDATA[AVG(Y 10 )]]> Eigenvalue 10.6615 0.7099 7.4241 -1.1899 -1.1028 feature <![CDATA[AVG(Y 11 )]]> <![CDATA[AVG(Y 12 )]]> <![CDATA[AVG(Y 13 )]]> <![CDATA[AVG(ΔY1)]]> <![CDATA[AVG(ΔY2)]]> Eigenvalue 0.3205 0.2024 3.2593 -0.3194 -0.8500 feature <![CDATA[AVG(ΔY3)]]> <![CDATA[AVG(ΔY4)]]> <![CDATA[AVG(ΔY5)]]> <![CDATA[AVG(ΔY6)]]> <![CDATA[AVG(ΔY7)]]> Eigenvalue -0.5906 -0.4383 -0.0086 0.0293 0.2131 feature <![CDATA[AVG(ΔY8)]]> <![CDATA[AVG(ΔY9)]]> <![CDATA[AVG(ΔY 10 )]]> <![CDATA[AVG(ΔY 11 )]]> <![CDATA[AVG(ΔY 12 )]]> Eigenvalue 0.2439 0.1882 0.1406 0.0153 0.0132 feature <![CDATA[AVG(ΔY 13 )]]> <![CDATA[AVG(ΔΔY1)]]> <![CDATA[AVG(ΔΔY2)]]> <![CDATA[AVG(ΔΔY3)]]> <![CDATA[AVG(ΔΔY4)]]> Eigenvalue 0.3231 0.0287 0.0336 0.0359 0.0020 feature <![CDATA[AVG(ΔΔY5)]]> <![CDATA[AVG(ΔΔY6)]]> <![CDATA[AVG(ΔΔY7)]]> <![CDATA[AVG(ΔΔY8)]]> <![CDATA[AVG(ΔΔY9)]]> Eigenvalue -0.0106 -0.0773 0.0832 0.0240 0.0833 feature <![CDATA[AVG(ΔΔY 10 )]]> <![CDATA[AVG(ΔΔY 11 )]]> <![CDATA[AVG(ΔΔY 12 )]]> <![CDATA[AVG(ΔΔY 13 )]]> <![CDATA[AVG(h u )]]> Eigenvalue 0.0464 0.0116 0.0108 0.0611 2.5599 feature <![CDATA[VAR(h u )]]> <![CDATA[MID(h u )]]> <![CDATA[MAX(h u )]]> <![CDATA[MIN(h u )]]> N Eigenvalue 0.2211 2.6751 2.9011 0 4 feature V VE M VF G Eigenvalue 0.0298 0.0012 0.0006 0.9279 12.4989

[0159] S6. Based on Example 1, the trained SVM classification model is used to identify the breathing state based on the feature vector P, and the identification result is abnormal breathing.

[0160] Example 3

[0161] like Figure 8 This embodiment is based on a sleep breathing state monitoring method using audio and thermal imaging video in collaboration disclosed in Example 1, and correspondingly discloses a sleep breathing state monitoring device using audio and thermal imaging video in collaboration corresponding to the sleep state monitoring method. The sleep breathing state monitoring device includes a signal acquisition module 801, a signal separation module 802, a first extraction module 803, a second extraction module 804, a feature processing module 805, a denoising processing module 806, a feature extraction module 807, a feature splicing module 808, and a state judgment module 809, which are connected in sequence.

[0162] Signal acquisition module 801, for synchronously collecting audio signals in the sleeping environment and thermal imaging video signals of the subject's face; wherein the audio signals are collected using a recording device placed next to the sleeping user, and the thermal imaging video signals are collected using an infrared thermal imager placed directly opposite the sleeping user;

[0163] The signal separation module 802 is used to perform sound event endpoint detection on the collected audio signal using a double threshold method based on short-time energy and zero-crossing rate, separate the voiced segment and the silent segment signals, and obtain the voiced segment signal x(n);

[0164] A first extraction module 803 is configured to extract a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC from the voiced segment signal x(n);

[0165] A second extraction module 804 is configured to extract a 5-dimensional wavelet information entropy feature vector H from the voiced segment signal x(n);

[0166] Feature processing module 805 uses a deep learning target detection algorithm to locate a region of interest (ROI) in the thermal imaging video signal of the sleeping user's face, uses an adaptive measurement region positioning algorithm based on centroid mapping to locate a measurement region based on the ROI, obtains the grayscale mean of pixels in the measurement region, and normalizes the result to obtain the original respiratory signal y; wherein the ROI is the nose region of the face, and the measurement region is the nasal cavity region;

[0167] De-noising module 806 is used to perform denoising on the original respiratory signal y by using an empirical wavelet transform algorithm to remove high-frequency noise and a cubic spline interpolation method to fit the baseline and remove baseline drift, thereby obtaining a denoised respiratory signal y';

[0168] The feature extraction module 807 is configured to extract a time domain feature vector E and a frequency domain feature vector F from the denoised respiratory signal. The time domain feature vector E includes the number of upper envelope peaks N, the variance of the upper envelope peaks V, and the variance of the normalized short-time energy VE, denoted as E = [N, V, VE]. The frequency domain feature vector F includes the minimum value M of the instantaneous frequency, the variance VF of the instantaneous frequency, and the range G of the instantaneous frequency, denoted as F = [M, VF, G].

[0169] The feature concatenation module 808 concatenates the Mel-frequency cepstral coefficient feature vector MFCC, the information entropy feature vector H, and the time domain feature vector E and frequency domain feature vector F of the thermal imaging respiratory signal into a final feature vector P, denoted as P = [MFCC, H, E, F]. The feature vector P of the training sample with known respiratory state is used to train an SVM classifier to obtain a respiratory state binary classification model.

[0170] The information in the training samples used to train the SVM classifier was collected in a sleep laboratory. While collecting audio signals and thermal imaging video signals, polysomnography was used to simultaneously monitor breathing and record the physiological data obtained by the polysomnography for subsequent classification of respiratory states.

[0171] The state judgment module 809 uses the respiratory state binary classification model to judge the respiratory state of the sleeping user based on the audio signal and the respiratory signal extracted from the thermal imaging video, and determines the respiratory state as normal breathing or abnormal breathing.

[0172] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A sleep breathing state monitoring method using audio and thermal imaging video, characterized in that: The monitoring method comprises the following steps: S1. Synchronously collect audio signals in the sleeping environment and thermal imaging video signals of the subject's face; the audio signals are collected using a recording device placed next to the sleeping user, and the thermal imaging video signals are collected using an infrared thermal imager placed directly opposite the sleeping user; S2. Performing sound event endpoint detection on the collected audio signal using a double threshold method based on short-time energy and zero-crossing rate to separate the voiced segment and the silent segment signals, thereby obtaining the voiced segment signal x(n); S3, extracting a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC from the voiced segment signal x(n); S4, extracting a 5-dimensional wavelet information entropy feature vector H from the voiced segment signal x(n); S5. Using a deep learning target detection algorithm to locate a region of interest (ROI) in the thermal imaging video signal of the sleeping user's face, using an adaptive measurement region positioning algorithm based on centroid mapping to locate a measurement region based on the ROI, obtaining the grayscale mean of pixels in the measurement region, and normalizing the result to obtain an original respiratory signal y; wherein the ROI is the nose region of the face, and the measurement region is the nasal cavity region; S6. De-noising the original respiratory signal y by using an empirical wavelet transform algorithm to remove high-frequency noise, and fitting a baseline using a cubic spline interpolation method to remove baseline drift, thereby obtaining a de-noised respiratory signal y'; S7. Extracting a time domain eigenvector E and a frequency domain eigenvector F from the denoised respiratory signal; wherein the time domain eigenvector E includes the number of upper envelope peaks N, the variance of the upper envelope peaks V, and the variance of the normalized short-time energy VE, denoted as E = [N, V, VE]; the frequency domain eigenvector F includes the minimum value M of the instantaneous frequency, the variance VF of the instantaneous frequency, and the range G of the instantaneous frequency, denoted as F = [M, VF, G]; S8. Concatenate the Mel-frequency cepstral coefficient eigenvector MFCC, the information entropy eigenvector H, and the time-domain eigenvector E and frequency-domain eigenvector F of the thermal imaging respiratory signal into a final eigenvector P, denoted as P = [MFCC, H, E, F]. Use the eigenvector P of the training samples with known respiratory states to train an SVM classifier to obtain a respiratory state binary classification model. The information in the training samples used to train the SVM classifier was collected in a sleep laboratory. While collecting audio signals and thermal imaging video signals, polysomnography was used to simultaneously monitor breathing and record the physiological data obtained by the polysomnography for subsequent classification of respiratory states. S9. Using the breathing state binary classification model, the breathing state of the sleeping user is judged based on the audio signal and the breathing signal extracted from the thermal imaging video, and the breathing state is determined to be normal breathing or abnormal breathing.

2. The sleep state monitoring method based on audio and thermal imaging video collaboration according to claim 1, characterized in that: The process of step S3 is as follows: S301, pre-emphasize, frame, and window the voiced segment signal x(n), assuming the frame length is N1 and the frame shift is Frame division to obtain M1 frame; S302, calculate the short-time energy of the pre-processed M1 frame signal frame by frame, and perform discrete Fourier transform, Mel filtering, logarithm, and inverse Fourier transform to obtain the 13-dimensional Mel-frequency cepstral coefficient features of each frame, forming a 13-dimensional Mel-frequency cepstral coefficient feature vector Y j (i),i=1,2,...,M1,j=1,2,...,13; S303, add the time variation to the 13-dimensional Mel cepstral coefficient feature vector Yj(i), that is, calculate ΔY j (i) and ΔΔY j (i), ΔY j (i) represents the cepstral coefficient Y between frames j (i) Change, ΔΔY j (i) represents the ΔY between frames j The change of (i) is expressed as: Based on the i-th frame, we average Yj(i), ΔYj(i), and ΔΔYj(i) to obtain AVG(Yj), AVG(ΔYj), and AVG(ΔΔYj). The formulas are as follows: The above three 13-dimensional feature vectors are concatenated into a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC, which is recorded as MFCC = [AVG (Y j ),AVG(ΔY j ),AVG(ΔΔY j )].

3. The sleep breathing monitoring method using audio and thermal imaging video collaboration according to claim 1, characterized in that: The process of step S4 is as follows: S401, the voiced segment signal x(n) is frame-shifted by a frame length of N2. Divide into M2 frames and get M2 frame signal x u (n),u=1,2,...,M2; S402, M2 frame signal x u Each frame signal in (n) is decomposed into four layers of discrete wavelet to obtain the fourth layer low-frequency signal x u,4L (n); where the discrete wavelet four-layer decomposition is to convert x u (n) After passing through the fourth discrete wavelet low-pass filter, formula (3) is the response function of the discrete wavelet low-pass filter, and the low-frequency signal x is obtained by formula (4) u,L (n): Where τ is time, x u (n) is the original signal, x u (2n) is x u (n) The signal obtained after two times downsampling; the obtained low-frequency signal x u,L (n) Pass through the low-pass filter again, repeat 4 times, and get the fourth layer of low-frequency signal x u,4L (n); S403, first calculate the fourth layer low frequency component x u,4L (n),u=1,2,...,the information entropy h of M2 u , then find M2 information entropy h u The mean AVG(h u )、VAR(h u ), median MID(h u ), maximum value MAX(h u )、Minimum value MIN(h u ), we get the five-dimensional wavelet information entropy feature vector H, which is recorded as H = [AVG (h u ),VAR(h u ),MID(h u ),MAX(h u ),MIN(h u )]; Among them, x u,4L The information entropy h of (n) u The calculation method is: Let x u,4L The number of sampling points (n) is Q, and the minimum value of the sampling point is X min , the maximum value is X max , the interval [X min ,X max ] is divided into D (D<Q) subintervals, and the length of each subinterval is Let c=1,2,...,D, and record x u,4L (n) The median is greater than or equal to X min +(c-1)×SPB, less than X min +c×SPB The number of sampling points is n c , use formula (5) to calculate x u,4L (n) D equally divided information entropy h u :

4. The sleep breathing monitoring method using audio and thermal imaging video collaboration according to claim 1, characterized in that: The process of step S5 is as follows: S501, dividing the collected facial thermal imaging video into frames to obtain multiple frames of single-channel grayscale images arranged in time; S502: Fusing publicly available online data with self-collected facial thermal imaging data to create a target detection dataset. Labelme, a labeling tool, is used to annotate the nose region of each image in the dataset as a region of interest to be identified. A deep learning target detection model is trained using the target detection dataset. The deep learning target detection model is used to locate the region of interest in each frame of the multiple single-channel grayscale images. The deep learning target detection model consists of a convolutional layer, a region of interest pooling layer, a fully connected layer, and a bounding box regression layer. First, the entire image to be detected is input into the convolutional layer to obtain a convolutional feature map containing the final features. Then, the region of interest pooling layer is used to pool the regions in the convolutional feature map, mapping regions of interest of different sizes into feature maps of the same size, thereby facilitating subsequent processing. Next, two fully connected layers are used to generate a feature vector of the region of interest, and the category probability of the target object is output based on the feature vector. Finally, the bounding box regression layer outputs the bounding box position of the target object to obtain the target detection result. S503, selecting a frame with the largest pixel mean in the region of interest as the clearest frame; S504, first calculate the coordinates of the center of mass of the region of interest of the clearest frame (P x ,P y ); then, a threshold segmentation method is used to obtain a measurement area based on the region of interest of the clearest frame; finally, a position mapping relationship between all pixels in the measurement area and the centroid pixel of the region of interest is established, and the position mapping relationship is used to subsequently locate the measurement area of ​​other frames; Among them, the formula for obtaining the coordinates of the centroid of the region of interest is: Where w represents the number of pixels in the region of interest, P xs Indicates the X-direction coordinate position of the s-th pixel point in the region of interest, P ys Indicates the Y coordinate position of the sth pixel point in the region of interest, v s Represents the grayscale value of the s-th pixel in the region of interest; The threshold segmentation method resets the grayscale of pixels in the region of interest whose grayscale is less than a threshold of 128 to 0, and the grayscale of pixels whose grayscale is greater than or equal to the threshold of 128 remains unchanged. The measurement area is the area consisting of pixels with non-zero grayscale. The process of obtaining the position mapping relationship is as follows: suppose there are R pixels in the measurement area, then the coordinate vector composed of all the pixels in the measurement area is (P xo ,P yo ),o=1,2,...,R, the centroid coordinates of the region of interest are (P x ,P y ), then the position mapping relationship is: Where, Indicates the distance between the oth pixel point and the centroid in the X direction. Indicates the distance between the oth pixel point and the centroid in the Y direction. After locating the centroid coordinates of other frames, the coordinates of all pixels in the measurement area of ​​the frame can be obtained through this mapping relationship. S505. Locate the centroid of the region of interest of the other frames using formula (7), then use the position mapping relationship to obtain the coordinate vectors of the pixel points in the measurement area of ​​the other frames, set the grayscale of the other pixel points outside the coordinate vector to 0, and obtain the measurement area of ​​each frame; S506 , obtaining the grayscale mean value of the measurement area of ​​each frame, performing fitting with time as the horizontal axis, and obtaining the original respiratory signal y after normalization.

5. The sleep breathing monitoring method using audio and thermal imaging video collaboration according to claim 1, characterized in that: The process of step S6 is as follows: S601, after removing the DC component, decompose the original respiratory signal y into three components IMF1, IMF2, and IMF3 using an empirical wavelet transform, and select the one with the same spectral peak as the peak of the signal before decomposition as the low-noise respiratory signal with high-frequency noise removed; S602 , locate all trough points of the low-noise respiratory signal, fit the positions of all trough points using cubic spline interpolation to obtain a baseline drift curve, and subtract the baseline drift curve from the low-noise respiratory signal to obtain a denoised respiratory signal y′.

6. The sleep breathing monitoring method using audio and thermal imaging video collaboration according to claim 1, characterized in that: The process of step S8 is as follows: S801, concatenate the Mel-frequency cepstral coefficient feature vector MFCC of the voiced audio segment, the wavelet information entropy feature vector H, and the time domain feature vector E and frequency domain feature vector F of the respiratory signal into a final feature vector P; S802. Extract the feature vectors P of all training samples to form a feature vector set x. Use the respiratory state in the analysis results of the data obtained from the polysomnography of the same period as the classification standard to divide the training samples into normal breathing and abnormal breathing, and set a label k for each sample, k∈{0,1}, where 0 represents normal breathing and 1 represents abnormal breathing. Input the sample label and the feature vector x into the SVM classifier and train the SVM classifier. Through training the SVM classifier, establish a mapping relationship between the feature vector and the respiratory state, and obtain a two-classification model of the respiratory state.

7. A sleep respiratory status monitoring device based on the sleep respiratory status monitoring method of audio and thermal imaging video collaboration according to any one of claims 1 to 6, characterized in that: The sleep breathing state monitoring device comprises: A signal acquisition module is used to synchronously collect audio signals in the sleeping environment and thermal imaging video signals of the subject's face; the audio signals are collected using a recording device placed next to the sleeping user, and the thermal imaging video signals are collected using an infrared thermal imager placed directly opposite the sleeping user; The signal separation module is used to perform sound event endpoint detection on the collected audio signal using a double threshold method based on short-time energy and zero-crossing rate, separate the voiced segment and the silent segment signals, and obtain the voiced segment signal x(n); A first extraction module is configured to extract a 39-dimensional Mel-frequency cepstral coefficient feature vector MFCC from the voiced segment signal x(n); A second extraction module is used to extract a 5-dimensional wavelet information entropy feature vector H from the voiced segment signal x(n); a feature processing module that uses a deep learning target detection algorithm to locate a region of interest (ROI) in the thermal imaging video signal of the sleeping user's face, uses an adaptive measurement region positioning algorithm based on centroid mapping to locate a measurement region based on the ROI, obtains the grayscale mean of pixels in the measurement region, and normalizes the result to obtain the original respiratory signal y; wherein the ROI refers to the nose region of the face, and the measurement region refers to the nasal cavity region; a denoising processing module, configured to perform denoising on the original respiratory signal y, remove high-frequency noise using an empirical wavelet transform algorithm, and fit a baseline using a cubic spline interpolation method to remove baseline drift, thereby obtaining a denoised respiratory signal y'; A feature extraction module is used to extract a time domain feature vector E and a frequency domain feature vector F from the denoised respiratory signal; wherein the time domain feature vector E includes the number of upper envelope peaks N, the variance of the upper envelope peaks V, and the variance of the normalized short-time energy VE, denoted as E = [N, V, VE]; the frequency domain feature vector F includes the minimum value M of the instantaneous frequency, the variance VF of the instantaneous frequency, and the range G of the instantaneous frequency, denoted as F = [M, VF, G]; The feature concatenation module concatenates the Mel-frequency cepstral coefficient feature vector MFCC, the information entropy feature vector H, and the time domain feature vector E and frequency domain feature vector F of the thermal imaging respiratory signal into the final feature vector P, denoted as P = [MFCC, H, E, F]. The feature vector P of the training samples with known respiratory states is used to train the SVM classifier to obtain a two-class respiratory state classification model. The information in the training samples used to train the SVM classifier was collected in a sleep laboratory. While collecting audio signals and thermal imaging video signals, polysomnography was used to simultaneously monitor breathing and record the physiological data obtained by the polysomnography for subsequent classification of respiratory states. The state judgment module uses the respiratory state binary classification model to judge the respiratory state of the sleeping user based on the audio signal and the respiratory signal extracted from the thermal imaging video, and determines the respiratory state as normal breathing or abnormal breathing.

Images