Sleep Apnea Detection Methods and Devices

Abstract

This application discloses a method and apparatus for detecting sleep apnea. The method includes: acquiring a thermal imaging video of a target subject during sleep; determining a first respiratory airflow signal and a first respiratory motion signal of the target subject in the video; segmenting the first respiratory airflow signal and the first respiratory motion signal using a preset sliding window to obtain second respiratory airflow signals and second respiratory motion signals under multiple time windows; within each time window, determining multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort difference features based on the second respiratory airflow signal and the second respiratory motion signal, and combining these features as a feature vector corresponding to the time window; clustering the multiple feature vectors corresponding to the multiple time windows to obtain a classification result corresponding to each time window. This application solves the technical problem in related technologies of the difficulty in accurately detecting different types of sleep apnea.

Description

Technical Field

[0001] This application relates to the field of sleep apnea detection technology, and more specifically, to a method and apparatus for sleep apnea detection. Background Technology

[0002] Sleep apnea syndrome is a common and potentially dangerous sleep disorder. Its main symptom is repeated sleep apnea (SA) during sleep. Based on different pathogenesis, sleep apnea can be mainly divided into central sleep apnea (CSA) and obstructive sleep apnea (OSA).

[0003] Currently, camera-based sleep apnea detection systems are commonly used to monitor breathing apnea during sleep. However, this approach has certain drawbacks: camera-based sleep apnea monitoring systems are relatively limited in function, only capable of detecting a single type of sleep apnea and unable to classify and detect OSA and CSA. Based on this, researchers have proposed using deep learning models or adding information sources to achieve OSA and CSA classification and detection. However, deep learning models require a large amount of sleep data to train the neural network, which is often difficult to meet in practical applications. Furthermore, the enormous computational demands limit deep learning-based systems to cloud-based deployments, preventing local deployment. Adding information sources requires multiple sensors for sleep apnea detection, but multi-sensor systems increase system complexity and instability. Therefore, neither deep learning models nor adding information sources can simultaneously achieve comprehensive apnea detection and system cost-effectiveness.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method and apparatus for detecting sleep apnea, which at least solves the technical problem in the related art of accurately detecting different types of sleep apnea.

[0006] According to one aspect of the embodiments of this application, a method for detecting sleep apnea is provided, comprising: acquiring a thermal imaging video of a target object during sleep, and determining a first respiratory airflow signal characterizing the respiratory airflow state of the target object and a first respiratory motion signal characterizing the chest and abdominal movement state of the target object in the thermal imaging video; segmenting the first respiratory airflow signal and the first respiratory motion signal using a preset sliding window to obtain a second respiratory airflow signal and a second respiratory motion signal under multiple time windows; within each time window, determining multiple respiratory airflow features of different dimensions based on the second respiratory airflow signal, and determining a respiratory effort intensity feature characterizing the amplitude of chest and abdominal movement and a respiratory effort difference feature characterizing the phase of chest and abdominal movement based on the second respiratory motion signal; combining the multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort difference features as a feature vector corresponding to the time window; clustering the multiple feature vectors corresponding to the multiple time windows to obtain a classification result corresponding to each time window, wherein the classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing.

[0007] Optionally, determining a first respiratory airflow signal for characterizing the respiratory airflow state of the target object and a first respiratory motion signal for characterizing the chest and abdominal motion state of the target object in the thermal imaging video includes: for each frame of thermal image in the thermal imaging video, determining a head region in the thermal image, determining a nostril region from the head region, and determining a chest and abdominal region based on the head region; determining a first respiratory airflow signal based on the pixels of the nostril region in each frame of thermal image; and determining a first respiratory motion signal based on the pixels of the chest and abdominal region in each frame of thermal image.

[0008] Optionally, determining the head region in the thermal image includes: using an image segmentation algorithm to determine at least one high-pixel value region in the thermal image, wherein the pixel value in the high-pixel value region is higher than a first preset threshold; and selecting the head region from the at least one high-pixel value region based on the morphology of each high-pixel value region.

[0009] Optionally, determining the nostril region from the head region includes: determining multiple preset directions; in each preset direction, using a linear filter of the preset direction to perform edge and texture enhancement processing on the head region in each frame of the thermal image in the thermal imaging video to obtain multiple corresponding head images; for each head image, using the inter-frame difference method to determine the first region in the head image where motion occurs; finding the intersection of the first regions corresponding to each head image within the target time window to obtain the second region corresponding to the preset direction; finding the intersection of the second regions corresponding to each preset direction to obtain the nostril region in the thermal image, wherein the thermal image is the first frame of the thermal image in the target time window, and the target time window is the time window corresponding to the sliding window.

[0010] Optionally, the chest and abdomen region is determined based on the head region, including: determining a region of a preset size directly below the head region in the thermal image as the chest and abdomen region.

[0011] Optionally, determining the first respiratory airflow signal based on the pixels in the nostril region of each frame of thermal image includes: for each frame of thermal image, determining the average value of all pixels in the nostril region of the thermal image as the respiratory airflow signal value corresponding to the thermal image; and arranging the respiratory airflow signal values ​​corresponding to all thermal images in chronological order to obtain the first respiratory airflow signal.

[0012] Optionally, determining the first respiratory motion signal based on the pixels in the chest and abdomen region of each frame of thermal image includes: for each frame of thermal image, determining the displacement of each pixel in the chest and abdomen region of the thermal image using dense optical flow method, and obtaining the time interval between two consecutive frames in the thermal imaging video, and determining the motion velocity of each pixel based on the displacement and time interval; dividing the chest and abdomen region into a first number of grids on an average basis; for each grid, determining the average velocity of the motion velocity of all pixels in the grid as the respiratory motion signal value corresponding to the grid, arranging the respiratory motion signal values ​​corresponding to the grids in all thermal images in time sequence to obtain the third respiratory motion signal corresponding to the grid; and using the third respiratory motion signal corresponding to each grid as a channel of the first respiratory motion signal.

[0013] Optionally, the respiratory airflow characteristics include at least one of the following: variance, waveform area, 90th percentile, and instantaneous respiratory interval. Multiple respiratory airflow characteristics of different dimensions are determined based on the second respiratory airflow signal, including: performing noise filtering on the second respiratory airflow signal to obtain a third respiratory airflow signal; determining the variance of all respiratory airflow signal values ​​in the third respiratory airflow signal; determining the area enclosed by the waveform of the third respiratory airflow signal and a preset baseline as the waveform area; sorting all respiratory airflow signal values ​​in the third respiratory airflow signal from smallest to largest, and determining the 90th percentile of all sorted respiratory airflow signal values; determining all first respiratory airflow signal peaks in the first respiratory airflow signal that are greater than a second preset threshold; traversing all first respiratory airflow signal peaks; if there are multiple consecutive first respiratory airflow signal peaks with time intervals less than a third preset threshold, retaining only the maximum value among the multiple first respiratory airflow signal peaks; determining the remaining first respiratory airflow signal peaks after traversal as second respiratory airflow signal peaks; determining whether the third respiratory airflow signal includes second respiratory airflow signal peaks; if it includes them, determining the instantaneous respiratory interval as 1; if it does not include them, determining the instantaneous respiratory interval as 0.

[0014] Optionally, the respiratory effort intensity feature for characterizing the amplitude of chest and abdominal movements and the respiratory effort difference feature for characterizing the phase of chest and abdominal movements are determined based on the second respiratory motion signal, including: for each grid, determining the power spectrum corresponding to the third respiratory motion signal of the grid by using autocorrelation and Fourier transform; if the peak frequency of the power spectrum is within a preset normal breathing frequency band, determining the total power corresponding to a preset width centered on the peak frequency within the power spectrum as the normal breathing power; if the peak frequency of the power spectrum is not within the normal breathing frequency band, determining the total power corresponding to the normal breathing frequency band as the normal breathing power; calculating the power difference between the total power of the power spectrum and the normal breathing power, and determining the ratio of the normal breathing power to the power difference as the relative power index corresponding to the grid; determining the respiratory effort intensity feature based on the relative power index of each grid; and determining the respiratory effort difference feature based on the third respiratory motion signal and the relative power index of each grid.

[0015] Optionally, the breathing effort intensity feature is determined based on the relative power index of each grid, including: determining a second number of grids with a relative power index greater than a fourth preset threshold; and determining the ratio of the second number to the first number as the breathing effort intensity feature.

[0016] Optionally, the respiratory effort difference characteristics are determined based on the third respiratory motion signal and relative power index of each grid, including: determining the maximum value of the relative power index among all grids as the reference signal; for each grid, performing Butterworth low-pass filtering on the third respiratory motion signal corresponding to the grid to obtain the fourth respiratory motion signal, determining the cross-correlation function between the fourth respiratory motion signal and the reference signal, determining the ratio of the signal shift corresponding to the maximum value of the cross-correlation function to the period of the reference signal, binarizing the cross-correlation value by 0 / 180° to obtain the signal phase difference corresponding to the grid; determining the third number of grids with a signal phase difference of 180°; and determining the ratio of the third number to the first number as the respiratory effort difference characteristics.

[0017] Optionally, multiple feature vectors corresponding to multiple time windows are clustered to obtain the classification result for each time window, including: three initial cluster centers corresponding to the three states of central sleep apnea, obstructive sleep apnea, and normal sleep breathing; K-means clustering is performed on multiple feature vectors based on the three initial cluster centers to obtain three clusters, wherein the state corresponding to the cluster to which the feature vector of each time window belongs is the classification result for the time window.

[0018] According to another aspect of the embodiments of this application, a sleep apnea detection device is also provided, comprising: an acquisition module, configured to acquire a thermal imaging video of a target object during sleep, and determine a first respiratory airflow signal characterizing the respiratory airflow state of the target object and a first respiratory motion signal characterizing the chest and abdominal movement state of the target object in the thermal imaging video; a segmentation module, configured to segment the first respiratory airflow signal and the first respiratory motion signal using a preset sliding window to obtain a second respiratory airflow signal and a second respiratory motion signal under multiple time windows; a feature extraction module, configured to determine multiple respiratory airflow features of different dimensions based on the second respiratory airflow signal within each time window, and determine a respiratory effort intensity feature characterizing the amplitude of chest and abdominal movement and a respiratory effort difference feature characterizing the phase of chest and abdominal movement based on the second respiratory motion signal, and combine the multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort difference features as a feature vector corresponding to the time window; and a classification module, configured to cluster the multiple feature vectors corresponding to the multiple time windows to obtain a classification result corresponding to each time window, wherein the classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing.

[0019] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described sleep apnea detection method by running the computer program.

[0020] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described sleep apnea detection method through the computer program.

[0021] In this embodiment, a thermal imaging video of the target object during sleep is acquired, and a first respiratory airflow signal characterizing the respiratory airflow state of the target object and a first respiratory motion signal characterizing the chest and abdominal movement state of the target object are determined from the thermal imaging video. The first respiratory airflow signal and the first respiratory motion signal are segmented using a preset sliding window to obtain a second respiratory airflow signal and a second respiratory motion signal under multiple time windows. Within each time window, multiple respiratory airflow features of different dimensions are determined based on the second respiratory airflow signal, and respiratory effort intensity features characterizing the amplitude of chest and abdominal movement and respiratory effort difference features characterizing the phase of chest and abdominal movement are determined based on the second respiratory motion signal. The multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort difference features are combined as the feature vector corresponding to the time window. The multiple feature vectors corresponding to the multiple time windows are clustered to obtain the classification result corresponding to each time window. The classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing. This method can acquire sleep thermal imaging videos through a single camera system, and then use algorithms to mine respiratory airflow characteristics, respiratory effort intensity characteristics, and respiratory effort similarity and difference characteristics from the thermal imaging videos to characterize the symptom characteristics of OSA and CSA in sleep apnea, thereby achieving the classification and detection of OSA and CSA, effectively solving the technical problem of accurately detecting different types of sleep apnea in related technologies. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a schematic diagram of the structure of an optional computer terminal according to an embodiment of this application;

[0024] Figure 2 This is a schematic flowchart of an optional sleep apnea detection method according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram illustrating the location of an optional chest and abdominal region according to an embodiment of this application;

[0026] Figure 4 This is a waveform diagram illustrating the respiratory effort intensity characteristics under an optional OSA and CSA event according to an embodiment of this application;

[0027] Figure 5 This is a waveform diagram illustrating the differences in respiratory effort characteristics under optional OSA and CSA events according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of an optional sleep apnea detection device according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0032] Otsu's method is an automatic thresholding method that is adaptive to bimodal cases. It divides the image into two parts, background and target, according to the grayscale characteristics of the image. The larger the inter-class variance between the background and the target, the greater the difference between the two parts that make up the image.

[0033] Optical flow is a method used to describe the motion of an observed target, surface, or edge caused by motion relative to an observer.

[0034] Inter-frame difference method: This is a method to obtain the contour of a moving target by performing a difference operation on two adjacent frames in a video image sequence.

[0035] Dense Optical Flow (DOF) is an image registration method that performs point-by-point matching on an image.

[0036] K-means clustering algorithm (K-means) is an iterative clustering analysis algorithm that divides the data into K groups, randomly selects K objects as initial cluster centers, calculates the distance between each object and each sub-cluster center, and assigns each object to the cluster center closest to it.

[0037] Time domain analysis is a method that analyzes the stability, transient, and stable performance of a system based on the time-domain expression of the output under a given input.

[0038] Frequency Domain Analysis (FDMA) is an engineering method that uses graphical analysis to evaluate system performance in the frequency domain.

[0039] Example 1

[0040] According to an embodiment of this application, a method for detecting sleep apnea is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0041] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a sleep apnea detection method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0042] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0043] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the sleep apnea detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned application vulnerability detection method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0044] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0045] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0046] Under the above operating environment, this application provides a method for detecting sleep apnea, such as... Figure 2 As shown, the method includes the following steps:

[0047] Step S202: Acquire a thermal imaging video of the target object during sleep, and determine the first respiratory airflow signal used to characterize the respiratory airflow state of the target object and the first respiratory motion signal used to characterize the chest and abdominal motion state of the target object in the thermal imaging video.

[0048] Step S204: The first respiratory airflow signal and the first respiratory motion signal are segmented using a preset sliding window to obtain the second respiratory airflow signal and the second respiratory motion signal under multiple time windows;

[0049] Step S206: Within each time window, determine multiple respiratory airflow features of different dimensions based on the second respiratory airflow signal, determine the respiratory effort intensity feature for characterizing the amplitude of chest and abdominal movement and the respiratory effort difference feature for characterizing the phase of chest and abdominal movement based on the second respiratory motion signal, and combine the multiple respiratory airflow features, respiratory effort intensity features and respiratory effort difference features as the feature vector corresponding to the time window.

[0050] Step S208: Cluster the multiple feature vectors corresponding to multiple time windows to obtain the classification result corresponding to each time window. The classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing.

[0051] Since the medical criterion for sleep apnea events is a pause lasting more than 10 seconds, the preset sliding window can be set to 10 seconds.

[0052] The following describes each step of the sleep apnea detection method in conjunction with the specific implementation process.

[0053] As an optional implementation, the first respiratory airflow signal and the first respiratory motion signal in the thermal imaging video can be determined as follows: for each frame of thermal image in the thermal imaging video, the head region in the thermal image is determined, the nostril region is determined from the head region, and the chest and abdomen region is determined based on the head region; the first respiratory airflow signal is determined based on the pixels of the nostril region in each frame of thermal image; and the first respiratory motion signal is determined based on the pixels of the chest and abdomen region in each frame of thermal image.

[0054] Considering the significant temperature differences between different areas of the human body and the background, head screening can be performed using temperature in thermal imaging. Specifically, the head region in the thermal image can be determined as follows: at least one high-pixel value region in the thermal image is determined using an image segmentation algorithm, wherein the pixel value in the high-pixel value region is higher than a first preset threshold; the head region is screened out from the at least one high-pixel value region based on the morphology of each high-pixel value region.

[0055] Among them, the image segmentation algorithm can use the OTSU algorithm. Considering that there may be other high-temperature areas besides the head in thermal imaging, when filtering out multiple high-pixel value areas, the high-pixel values ​​can be further filtered by the approximately circular shape of the two-dimensional projection of the head to obtain the head region.

[0056] Considering that interference may move in the head region of thermal imaging video, and these interferences usually appear in the image at locations with large gradient changes and extend along the edge in a certain direction, the first respiratory airflow signal will be approximately concentrated in a circular area. It is understandable that the movement of interference and the first respiratory airflow signal will have different responses to the edge operator. Therefore, the pixel intensity variation pattern can be used to filter pixels and thus determine the nostril region in the head region.

[0057] Specifically, the nostril region can be determined from the head region as follows: Multiple preset directions are determined; in each preset direction, a linear filter is used to perform edge and texture enhancement processing on the head region in each frame of the thermal imaging video to obtain multiple corresponding head images; for each head image, the first region in the head image where motion occurs is determined using the inter-frame difference method; the intersection of the first regions corresponding to each head image within the target time window is obtained to obtain the second region corresponding to the preset direction; the intersection of the second regions corresponding to each preset direction is obtained to obtain the nostril region in the thermal image, where the thermal image is the first frame of the thermal image in the target time window, and the target time window is the time window corresponding to the sliding window.

[0058] For example, when the preset directions are θ = 0°, 45°, 95°, 135° and the target time window is t, the Gabor filter G(θ) is used to convolve each frame of thermal image I(x,y,t) in the thermal imaging video to enhance the image edges and textures. The specific formula is as follows:

[0059]

[0060] Then its corresponding first region M(x,y,t,θ) is:

[0061]

[0062] In this formula, τ = 2s is the time interval between two adjacent frames, and γ is the threshold, which can be set according to the actual situation.

[0063] Then, all pixel regions that exhibit significant motion within the target time window are accumulated, and the pixel region RFM(x,y) that responds in all four preset directions is calculated. The specific formula is as follows:

[0064]

[0065] It should be noted that the calculation result of RFM(x,y) is the ROI of the nostril region. RF If no first respiratory airflow signal is detected in the thermal image of the current frame, the thermal image of the previous frame is used for calculation.

[0066] Since the chest and abdomen lack obvious recognizable features and contain less texture detail in thermal images, the general geometric dependencies between the head and chest and abdomen are used for localization. Specifically, the chest and abdomen region can be determined based on the head region as follows: the area of ​​a preset size directly below the head region in the thermal image is determined as the chest and abdomen region.

[0067] Assuming the length of the segmented head region is taken as the head length l, then a rectangular region with length l·β and width l·α directly below the head region can be taken as the ROI of the chest and abdomen. RM The values ​​of α and β are determined by anatomical principles and the camera's shooting angle. Figure 3 A schematic diagram of an optional chest and abdominal region is shown.

[0068] The first respiratory airflow signal can then be determined as follows: for each frame of thermal image, the average value of all pixels in the nostril region of the thermal image is determined as the respiratory airflow signal value corresponding to the thermal image; the first respiratory airflow signal is obtained by arranging the respiratory airflow signal values ​​corresponding to all thermal images in chronological order.

[0069] For example, the ROI of the nostrils in the thermal image I(x,y,t) within the target time window t. RF The average value of all pixels (assuming the number is K) within the image yields the pixel temperature change trajectory, which is the respiratory airflow signal value RF(t) corresponding to the thermal image. The specific formula is as follows:

[0070]

[0071] Optionally, the first respiratory motion signal can be determined as follows: For each frame of thermal image, the displacement of each pixel in the chest and abdomen region of the thermal image is determined using dense optical flow method, and the time interval between two consecutive frames in the thermal imaging video is obtained. The motion velocity of each pixel is determined based on the displacement and the time interval. The chest and abdomen region is divided into a first number of grids. For each grid, the average velocity of the motion velocity of all pixels in the grid is determined as the respiratory motion signal value corresponding to the grid. The respiratory motion signal values ​​corresponding to the grids in all thermal images are arranged in time sequence to obtain the third respiratory motion signal corresponding to the grid. The third respiratory motion signal corresponding to each grid is used as a channel of the first respiratory motion signal.

[0072] Specifically, due to the ROI in the chest and abdomen region RMThe motion at different locations varies significantly and contains considerable heterogeneity, therefore multi-channel processing is required to transform the ROI. RM Dividing the space into M×N equal-area grids, the resulting first respiratory motion signal can be represented as RM. i.j (t), where i = 1, 2, ..., M, j = 1, 2, ..., N, M = l·α / r, N = l·β / r, and the selection of the value of r should take into account the computational complexity, the size of the human body in the image, and the magnitude of the chest and abdominal movements.

[0073] In order to describe the typical characteristics of the waveform morphology of the respiratory airflow signal from different perspectives and to detect the presence of respiratory airflow, respiratory airflow characteristics are introduced to describe the respiratory airflow signal. The respiratory airflow characteristics include at least one of the following: variance, waveform area, 90th percentile, and instantaneous respiratory interval.

[0074] Accordingly, multiple respiratory airflow characteristics of different dimensions can be determined as follows: noise filtering is applied to the second respiratory airflow signal to obtain the third respiratory airflow signal; the variance of all respiratory airflow signal values ​​in the third respiratory airflow signal is determined; the area enclosed by the waveform of the third respiratory airflow signal and the preset baseline is determined as the waveform area; all respiratory airflow signal values ​​in the third respiratory airflow signal are sorted from smallest to largest, and the 90th percentile of all sorted respiratory airflow signal values ​​is determined; all first respiratory airflow signal peaks in the first respiratory airflow signal that are greater than the second preset threshold are determined; all first respiratory airflow signal peaks are traversed; if there are multiple consecutive first respiratory airflow signal peaks with time intervals less than the third preset threshold, only the maximum value among the multiple first respiratory airflow signal peaks is retained; the remaining first respiratory airflow signal peaks after traversal are determined as the second respiratory airflow signal peaks; whether the third respiratory airflow signal includes the second respiratory airflow signal peaks is determined; if it includes them, the instantaneous breathing interval is determined to be 1; if it does not include them, the instantaneous breathing interval is determined to be 0.

[0075] Specifically, a 6th-order FIR bandpass filter (Finite Impulse Response) with a passband frequency of 0.125-0.425Hz can be used to filter out noise interference in the signal to obtain the third respiratory airflow signal RF. f (t); the variance Var can reflect the dispersion of the amplitude of the third respiratory airflow signal, and its specific formula is: Where μ is the mean of the third respiratory airflow signal; the waveform area Area reflects the average level of the third respiratory airflow signal amplitude, and its specific formula is: Among them, f sThe sampling frequency of the third respiratory airflow signal is denoted by b, which is a preset baseline determined by the local average value of the third respiratory airflow signal over one minute; the 90th percentile R... r It can reflect the distribution of the high-amplitude portion of the sampling points. If all respiratory airflow signal values ​​can be represented as RF... f (t)(n=1,2,...,N), sorting them in ascending order gives us {R1,R2,…R} N}, then the 90th percentile is R. r Where r = 0.5 + N·90 / 100; the second preset threshold can be obtained by multiplying the average of all effective peak points of the first respiratory airflow signal by a fixed ratio of 0.6; the third preset threshold can be set to 1.5s, and when the instantaneous breathing interval is 0, it can be considered that the target object has experienced an interruption of expiratory airflow.

[0076] Because the movement of the thermal pattern is weaker than the thermal changes at the nostrils and is subject to more noise interference, the amplitude characteristics of the first respiratory motion signal are not stable. If we continue to analyze the amplitude characteristics of the signal, we cannot accurately determine the amplitude of the respiratory motion. Therefore, we introduce respiratory effort intensity characteristics and respiratory effort similarity and difference characteristics to determine the amplitude of the respiratory motion.

[0077] Specifically, the respiratory effort intensity characteristics and respiratory effort difference characteristics can be determined in the following ways: For each grid, the power spectrum corresponding to the third respiratory motion signal of the grid is determined by autocorrelation and Fourier transform; if the peak frequency of the power spectrum is within a preset normal breathing frequency band, the total power corresponding to the preset width centered on the peak frequency within the power spectrum is determined as the normal breathing power; if the peak frequency of the power spectrum is not within the normal breathing frequency band, the total power corresponding to the normal breathing frequency band is determined as the normal breathing power; the power difference between the total power of the power spectrum and the normal breathing power is calculated, and the ratio of the normal breathing power to the power difference is determined as the relative power index corresponding to the grid; the respiratory effort intensity characteristics are determined based on the relative power index of each grid; and the respiratory effort difference characteristics are determined based on the third respiratory motion signal and the relative power index of each grid.

[0078] For example, for the third respiratory motor signal RM i.j The power spectrum P corresponding to (t) i,j (f)(f=0,1,...,f s / 2), its corresponding normal respiratory power rep i,j The specific formula is:

[0079]

[0080] Among them, f s =5Hz is the sampling rate of the third respiratory motion signal, fpeak(i,j) It is the power spectrum P i,j (f) is the peak frequency, b = 4 samples is the width of the peak proximity region set empirically, and f1 and f2 are the boundaries of the respiratory band. It is known that the average respiratory rate of an adult at rest is in the range of 12-20 bpm. To improve the robustness of the feature, the respiratory band should be appropriately widened based on the average respiratory rate range (12-20 bpm, i.e., 0.2-0.33 Hz). However, if the lower boundary f1 is too small, it is easy to misjudge the noise frequency component as the respiratory peak frequency. Taking all factors into consideration, f1 and f2 are set to 0.13 Hz and 0.50 Hz, respectively.

[0081] During normal breathing, respiratory components dominate, while during apnea, only noise components are present. Therefore, the relative power index can reflect the signal-to-noise ratio corresponding to the third respiratory motion signal. The signal-to-noise ratio can be used to measure the respiratory effort intensity characteristics of chest and abdominal movements. The stability of the respiratory effort intensity characteristics can be further improved through the above method.

[0082] To further improve the anti-interference ability of the feature and reduce the differences between different individuals, the breathing effort intensity feature can be determined based on the relative power index of each grid. This can be done as follows: determine the second number of grids whose relative power index is greater than a fourth preset threshold; determine the ratio of the second number to the first number as the breathing effort intensity feature.

[0083] For example, in the ROI of the chest and abdomen. RM The specific formula for the Respiratory Effort Intensity (REI) characteristic can be expressed as:

[0084]

[0085] In this formula, #X represents the number of elements in the fourth preset threshold set whose relative power index is greater than the number of elements in the fourth preset threshold set. The fourth preset threshold η is the minimum relative power requirement for the signal to contain effective respiratory information, which can be set to 0.4 in this embodiment. Figure 4 A waveform diagram illustrating the respiratory effort intensity characteristics under optional OSA and CSA events is shown.

[0086] Optionally, the respiratory effort difference feature can be determined as follows: The maximum value among the relative power indices of all grids is determined as the reference signal; for each grid, the third respiratory motion signal corresponding to the grid is subjected to Butterworth low-pass filtering to obtain the fourth respiratory motion signal; the cross-correlation function between the fourth respiratory motion signal and the reference signal is determined; the ratio of the signal shift corresponding to the maximum value of the cross-correlation function to the period of the reference signal is determined; the cross-correlation value is binarized at 0 / 180° to obtain the signal phase difference corresponding to the grid; the third number of grids with a signal phase difference of 180° is determined; the ratio of the third number to the first number is determined as the respiratory effort difference feature.

[0087] Specifically, the Butterworth low-pass filter can use a third-order filter with a cutoff frequency set to 1.5Hz, when the reference signal is represented as RM. ref (t), cross-correlation function R i,j The specific formula for (k) is:

[0088]

[0089] Where P is the total length of the fourth respiratory motion signal.

[0090] Correspondingly, if the signal shift corresponding to the maximum value of the cross-correlation function is K... i,j The period of the reference signal is T. ref The signal phase difference is pha i,j The specific formula for the respiratory effort difference feature REA is:

[0091]

[0092] During an OSA event, the abdominal region exhibits a 180° phase due to the opposite movements of the chest and abdomen. During normal breathing, the chest and abdomen move in the same direction, resulting in almost zero phase across the entire chest and abdomen region. In contrast, during a CSA event, the initial respiratory motion signals are all noise, leading to a chaotic and irregular phase distribution. Based on this, OSA, CSA, and normal breathing events can be distinguished by the difference in respiratory effort. Figure 5 A waveform diagram illustrating the differences in respiratory effort characteristics under optional OSA and CSA events is shown.

[0093] Optionally, multiple feature vectors corresponding to multiple time windows can be clustered in the following way to obtain the classification result for each time window: three initial cluster centers are preset, corresponding to the three states of central sleep apnea, obstructive sleep apnea, and normal sleep breathing; K-means clustering is performed on multiple feature vectors based on the three initial cluster centers to obtain three clusters, where the state corresponding to the cluster to which the feature vector of each time window belongs is the classification result for the time window.

[0094] Then, the corresponding respiratory state can be determined based on the classification results of each time window. For example, if multiple consecutive frames of thermal imaging video show CSA results within a certain time window, then a CSA event is considered to have occurred within that time window.

[0095] In this embodiment, a thermal imaging video of the target object during sleep is acquired, and a first respiratory airflow signal characterizing the respiratory airflow state of the target object and a first respiratory motion signal characterizing the chest and abdominal movement state of the target object are determined from the thermal imaging video. The first respiratory airflow signal and the first respiratory motion signal are segmented using a preset sliding window to obtain a second respiratory airflow signal and a second respiratory motion signal under multiple time windows. Within each time window, multiple respiratory airflow features of different dimensions are determined based on the second respiratory airflow signal, and respiratory effort intensity features characterizing the amplitude of chest and abdominal movement and respiratory effort difference features characterizing the phase of chest and abdominal movement are determined based on the second respiratory motion signal. The multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort difference features are combined as the feature vector corresponding to the time window. The multiple feature vectors corresponding to the multiple time windows are clustered to obtain the classification result corresponding to each time window. The classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing. This technology can acquire sleep thermal imaging videos using a single camera system, and then use algorithms to mine respiratory airflow characteristics, respiratory effort intensity characteristics, and respiratory effort similarity and difference characteristics from the thermal imaging videos to characterize the symptom characteristics of OSA and CSA in sleep apnea, thereby achieving the classification and detection of OSA and CSA. This effectively solves the technical problem of accurately detecting different types of sleep apnea in related technologies.

[0096] Example 2

[0097] According to an embodiment of this application, a sleep apnea detection device for implementing the sleep apnea detection method in Embodiment 1 is also provided, such as... Figure 6 As shown, the sleep apnea detection device includes at least: an acquisition module 61, a segmentation module 62, a feature extraction module 63, and a classification module 64, wherein:

[0098] The acquisition module 61 can acquire thermal imaging video of the target object during sleep, and determine the first respiratory airflow signal used to characterize the respiratory airflow state of the target object and the first respiratory motion signal used to characterize the chest and abdominal motion state of the target object in the thermal imaging video.

[0099] The segmentation module 62 can use a preset sliding window to segment the first respiratory airflow signal and the first respiratory motion signal to obtain the second respiratory airflow signal and the second respiratory motion signal under multiple time windows.

[0100] The feature extraction module 63 can determine multiple respiratory airflow features of different dimensions based on the second respiratory airflow signal within each time window, and determine the respiratory effort intensity feature for characterizing the amplitude of chest and abdominal movement and the respiratory effort difference feature for characterizing the phase of chest and abdominal movement based on the second respiratory motion signal. The multiple respiratory airflow features, respiratory effort intensity features and respiratory effort difference features are combined as the feature vector corresponding to the time window.

[0101] The classification module 64 can cluster multiple feature vectors corresponding to multiple time windows to obtain the classification result for each time window. The classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing.

[0102] The functions of each module of the sleep apnea detection device are explained below in conjunction with the specific implementation process.

[0103] As an optional implementation, the acquisition module can determine the first respiratory airflow signal and the first respiratory motion signal in the thermal imaging video in the following manner: for each frame of thermal image in the thermal imaging video, determine the head region in the thermal image, determine the nostril region from the head region, and determine the chest and abdomen region based on the head region; determine the first respiratory airflow signal based on the pixels of the nostril region in each frame of thermal image; determine the first respiratory motion signal based on the pixels of the chest and abdomen region in each frame of thermal image.

[0104] Considering the significant temperature differences between different areas of the human body and the background, head screening can be performed using temperature in thermal imaging. Specifically, the head region in the thermal image can be determined as follows: at least one high-pixel value region in the thermal image is determined using an image segmentation algorithm, wherein the pixel value in the high-pixel value region is higher than a first preset threshold; the head region is screened out from the at least one high-pixel value region based on the morphology of each high-pixel value region.

[0105] Among them, the image segmentation algorithm can use the OTSU algorithm. Considering that there may be other high-temperature areas besides the head in thermal imaging, when filtering out multiple high-pixel value areas, the high-pixel values ​​can be further filtered by the approximately circular shape of the two-dimensional projection of the head to obtain the head region.

[0106] Considering that interference may move in the head region of thermal imaging video, and these interferences usually appear in the image at locations with large gradient changes and extend along the edge in a certain direction, the first respiratory airflow signal will be approximately concentrated in a circular area. It is understandable that the movement of interference and the first respiratory airflow signal will have different responses to the edge operator. Therefore, the pixel intensity variation pattern can be used to filter pixels and thus determine the nostril region in the head region.

[0107] Specifically, the nostril region can be determined from the head region as follows: Multiple preset directions are determined; in each preset direction, a linear filter is used to perform edge and texture enhancement processing on the head region in each frame of the thermal imaging video to obtain multiple corresponding head images; for each head image, the first region in the head image where motion occurs is determined using the inter-frame difference method; the intersection of the first regions corresponding to each head image within the target time window is obtained to obtain the second region corresponding to the preset direction; the intersection of the second regions corresponding to each preset direction is obtained to obtain the nostril region in the thermal image, where the thermal image is the first frame of the thermal image in the target time window, and the target time window is the time window corresponding to the sliding window.

[0108] Since the chest and abdomen lack obvious recognizable features and contain less texture detail in thermal images, the general geometric dependencies between the head and chest and abdomen are used for localization. Specifically, the chest and abdomen region can be determined based on the head region as follows: the area of ​​a preset size directly below the head region in the thermal image is determined as the chest and abdomen region.

[0109] Assuming the length of the segmented head region is taken as the head length l, then a rectangular region with length l·β and width l·α directly below the head region can be taken as the ROI of the chest and abdomen. RM The values ​​of α and β are determined by anatomical principles and the camera's shooting angle. Figure 3 A schematic diagram of an optional chest and abdominal region is shown.

[0110] The acquisition module can then determine the first respiratory airflow signal as follows: for each frame of thermal image, the average value of all pixels in the nostril region of the thermal image is determined as the respiratory airflow signal value corresponding to the thermal image; the first respiratory airflow signal is obtained by arranging the respiratory airflow signal values ​​corresponding to all thermal images in chronological order.

[0111] Optionally, the acquisition module can determine the first respiratory motion signal in the following way: for each frame of thermal image, the displacement of each pixel in the chest and abdomen region of the thermal image is determined using dense optical flow method, and the time interval between two consecutive frames in the thermal imaging video is obtained. The motion velocity of each pixel is determined based on the displacement and the time interval. The chest and abdomen region is divided into a first number of grids on an average basis. For each grid, the average velocity of the motion velocity of all pixels in the grid is determined as the respiratory motion signal value corresponding to the grid. The respiratory motion signal values ​​corresponding to the grids in all thermal images are arranged in time sequence to obtain the third respiratory motion signal corresponding to the grid. The third respiratory motion signal corresponding to each grid is used as a channel of the first respiratory motion signal.

[0112] Specifically, due to the ROI in the chest and abdomen region RM The motion at different locations varies significantly and contains considerable heterogeneity, therefore multi-channel processing is required to transform the ROI. RM Dividing the space into M×N equal-area grids, the resulting first respiratory motion signal can be represented as RM. i.j (t), where i = 1, 2, ..., M, j = 1, 2, ..., N, M = l·α / r, N = l·β / r, and the selection of the value of r should take into account the computational complexity, the size of the human body in the image, and the magnitude of the chest and abdominal movements.

[0113] In order to describe the typical characteristics of the waveform morphology of the respiratory airflow signal from different perspectives and to detect the presence of respiratory airflow, respiratory airflow characteristics are introduced to describe the respiratory airflow signal. The respiratory airflow characteristics include at least one of the following: variance, waveform area, 90th percentile, and instantaneous respiratory interval.

[0114] Accordingly, multiple respiratory airflow characteristics of different dimensions can be determined as follows: noise filtering is applied to the second respiratory airflow signal to obtain the third respiratory airflow signal; the variance of all respiratory airflow signal values ​​in the third respiratory airflow signal is determined; the area enclosed by the waveform of the third respiratory airflow signal and the preset baseline is determined as the waveform area; all respiratory airflow signal values ​​in the third respiratory airflow signal are sorted from smallest to largest, and the 90th percentile of all sorted respiratory airflow signal values ​​is determined; all first respiratory airflow signal peaks in the first respiratory airflow signal that are greater than the second preset threshold are determined; all first respiratory airflow signal peaks are traversed; if there are multiple consecutive first respiratory airflow signal peaks with time intervals less than the third preset threshold, only the maximum value among the multiple first respiratory airflow signal peaks is retained; the remaining first respiratory airflow signal peaks after traversal are determined as the second respiratory airflow signal peaks; whether the third respiratory airflow signal includes the second respiratory airflow signal peaks is determined; if it includes them, the instantaneous breathing interval is determined to be 1; if it does not include them, the instantaneous breathing interval is determined to be 0.

[0115] Because the movement of the thermal pattern is weaker than the thermal changes at the nostrils and is subject to more noise interference, the amplitude characteristics of the first respiratory motion signal are not stable. If we continue to analyze the amplitude characteristics of the signal, we cannot accurately determine the amplitude of the respiratory motion. Therefore, we introduce respiratory effort intensity characteristics and respiratory effort similarity and difference characteristics to determine the amplitude of the respiratory motion.

[0116] Specifically, the respiratory effort intensity characteristics and respiratory effort difference characteristics can be determined in the following ways: For each grid, the power spectrum corresponding to the third respiratory motion signal of the grid is determined by autocorrelation and Fourier transform; if the peak frequency of the power spectrum is within a preset normal breathing frequency band, the total power corresponding to the preset width centered on the peak frequency within the power spectrum is determined as the normal breathing power; if the peak frequency of the power spectrum is not within the normal breathing frequency band, the total power corresponding to the normal breathing frequency band is determined as the normal breathing power; the power difference between the total power of the power spectrum and the normal breathing power is calculated, and the ratio of the normal breathing power to the power difference is determined as the relative power index corresponding to the grid; the respiratory effort intensity characteristics are determined based on the relative power index of each grid; and the respiratory effort difference characteristics are determined based on the third respiratory motion signal and the relative power index of each grid.

[0117] During normal breathing, respiratory components dominate, while during apnea, only noise components are present. Therefore, the relative power index can reflect the signal-to-noise ratio corresponding to the third respiratory motion signal. The signal-to-noise ratio can be used to measure the respiratory effort intensity characteristics of chest and abdominal movements. The stability of the respiratory effort intensity characteristics can be further improved through the above method.

[0118] To further improve the anti-interference ability of the feature and reduce the differences between different individuals, the breathing effort intensity feature can be determined based on the relative power index of each grid. This can be done as follows: determine the second number of grids whose relative power index is greater than a fourth preset threshold; determine the ratio of the second number to the first number as the breathing effort intensity feature.

[0119] Optionally, the respiratory effort difference feature can be determined as follows: The maximum value among the relative power indices of all grids is determined as the reference signal; for each grid, the third respiratory motion signal corresponding to the grid is subjected to Butterworth low-pass filtering to obtain the fourth respiratory motion signal; the cross-correlation function between the fourth respiratory motion signal and the reference signal is determined; the ratio of the signal shift corresponding to the maximum value of the cross-correlation function to the period of the reference signal is determined; the cross-correlation value is binarized at 0 / 180° to obtain the signal phase difference corresponding to the grid; the third number of grids with a signal phase difference of 180° is determined; the ratio of the third number to the first number is determined as the respiratory effort difference feature.

[0120] During an OSA event, the abdominal region exhibits a 180° phase due to the opposite movements of the chest and abdomen. During normal breathing, the chest and abdomen move in the same direction, resulting in almost zero phase across the entire chest and abdomen region. In contrast, during a CSA event, the initial respiratory motion signals are all noise, leading to a chaotic and irregular phase distribution. Based on this, OSA, CSA, and normal breathing events can be distinguished by the difference in respiratory effort. Figure 5 A waveform diagram illustrating the differences in respiratory effort characteristics under optional OSA and CSA events is shown.

[0121] Optionally, the classification module can cluster multiple feature vectors corresponding to multiple time windows in the following way to obtain the classification result for each time window: preset three initial cluster centers corresponding to the three states of central sleep apnea, obstructive sleep apnea, and normal sleep breathing; perform K-means clustering on multiple feature vectors based on the three initial cluster centers to obtain three clusters, wherein the state corresponding to the cluster to which the feature vector of each time window belongs is the classification result for the time window.

[0122] Then, the corresponding respiratory state can be determined based on the classification results of each time window. For example, if multiple consecutive frames of thermal imaging video show CSA results within a certain time window, then a CSA event is considered to have occurred within that time window.

[0123] It should be noted that each module in the sleep apnea detection device in this application embodiment corresponds one-to-one with each implementation step of the sleep apnea detection method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0124] Example 3

[0125] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the sleep apnea detection method in Embodiment 1 by running the computer program.

[0126] Specifically, the device containing the non-volatile storage medium executes the following steps by running the computer program: acquiring thermal imaging video of the target object during sleep, and determining a first respiratory airflow signal characterizing the respiratory airflow state of the target object and a first respiratory motion signal characterizing the chest and abdominal movement state of the target object in the thermal imaging video; segmenting the first respiratory airflow signal and the first respiratory motion signal using a preset sliding window to obtain a second respiratory airflow signal and a second respiratory motion signal under multiple time windows; within each time window, determining multiple respiratory airflow features of different dimensions based on the second respiratory airflow signal, and determining respiratory effort intensity features characterizing the amplitude of chest and abdominal movement and respiratory effort similarity / similarity features characterizing the phase of chest and abdominal movement based on the second respiratory motion signal; combining multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort similarity / similarity features as feature vectors corresponding to the time window; clustering multiple feature vectors corresponding to multiple time windows to obtain a classification result corresponding to each time window, wherein the classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing.

[0127] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the sleep apnea detection method in Embodiment 1 when it runs.

[0128] Specifically, the computer program executes the following steps during runtime: acquiring thermal imaging video of the target object during sleep, and determining a first respiratory airflow signal characterizing the target object's respiratory airflow state and a first respiratory motion signal characterizing the target object's chest and abdominal movement state in the thermal imaging video; segmenting the first respiratory airflow signal and the first respiratory motion signal using a preset sliding window to obtain second respiratory airflow signals and second respiratory motion signals under multiple time windows; within each time window, determining multiple respiratory airflow features of different dimensions based on the second respiratory airflow signal, and determining respiratory effort intensity features characterizing the amplitude of chest and abdominal movement and respiratory effort similarity / similarity features characterizing the phase of chest and abdominal movement based on the second respiratory motion signal; combining multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort similarity / similarity features as feature vectors corresponding to the time window; clustering multiple feature vectors corresponding to multiple time windows to obtain classification results corresponding to each time window, wherein the classification results are used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing.

[0129] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the sleep apnea detection method of Embodiment 1 through the computer program.

[0130] Specifically, the computer program executes the following steps during runtime: acquiring thermal imaging video of the target object during sleep, and determining a first respiratory airflow signal characterizing the target object's respiratory airflow state and a first respiratory motion signal characterizing the target object's chest and abdominal movement state in the thermal imaging video; segmenting the first respiratory airflow signal and the first respiratory motion signal using a preset sliding window to obtain second respiratory airflow signals and second respiratory motion signals under multiple time windows; within each time window, determining multiple respiratory airflow features of different dimensions based on the second respiratory airflow signal, and determining respiratory effort intensity features characterizing the amplitude of chest and abdominal movement and respiratory effort similarity / similarity features characterizing the phase of chest and abdominal movement based on the second respiratory motion signal; combining multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort similarity / similarity features as feature vectors corresponding to the time window; clustering the multiple feature vectors corresponding to multiple time windows to obtain a classification result corresponding to each time window, wherein the classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing. The above embodiment numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0131] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0136] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting sleep apnea, characterized in that, include: Acquire thermal imaging video of the target object during sleep, and determine a first respiratory airflow signal characterizing the respiratory airflow state of the target object and a first respiratory motion signal characterizing the chest and abdominal motion state of the target object in the thermal imaging video; The first respiratory airflow signal and the first respiratory motion signal are segmented using a preset sliding window to obtain the second respiratory airflow signal and the second respiratory motion signal under multiple time windows. Within each time window, multiple respiratory airflow features of different dimensions are determined based on the second respiratory airflow signal, and respiratory effort intensity features for characterizing the amplitude of chest and abdominal movement and respiratory effort similarity and dissimilarity features for characterizing the phase of chest and abdominal movement are determined based on the second respiratory motion signal. The multiple respiratory airflow features, respiratory effort intensity features and respiratory effort similarity and dissimilarity features are combined as the feature vector corresponding to the time window. Clustering is performed on multiple feature vectors corresponding to multiple time windows to obtain a classification result for each time window, wherein the classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing; The method for determining the first respiratory motion signal includes: for each frame of thermal image in the thermal imaging video, determining the head region in the thermal image, and determining the chest and abdomen region based on the head region; for each frame of thermal image, determining the displacement of each pixel in the chest and abdomen region of the thermal image using dense optical flow method, and obtaining the time interval between two consecutive frames in the thermal imaging video, determining the motion speed of each pixel based on the displacement and the time interval; dividing the chest and abdomen region into a first number of grids on an average basis; for each grid, determining the average speed of the motion speed of all pixels in the grid as the respiratory motion signal value corresponding to the grid, arranging the respiratory motion signal values ​​corresponding to the grid in all thermal images in time sequence to obtain the third respiratory motion signal corresponding to the grid; and using the third respiratory motion signal corresponding to each grid as one channel of the first respiratory motion signal. The method for determining the respiratory effort difference feature includes: for each grid, determining the power spectrum corresponding to the third respiratory motion signal of the grid using autocorrelation and Fourier transform; if the peak frequency of the power spectrum is within a preset normal respiratory frequency band, determining the total power within the power spectrum corresponding to a preset width centered on the peak frequency as the normal respiratory power; if the peak frequency of the power spectrum is not within the normal respiratory frequency band, determining the total power corresponding to the normal respiratory frequency band as the normal respiratory power; calculating the power difference between the total power of the power spectrum and the normal respiratory power, and determining the ratio of the normal respiratory power to the power difference as the value of the grid. The corresponding relative power index is determined; the maximum value of the relative power index among all grids is determined as the reference signal; for each grid, the third respiratory motion signal corresponding to the grid is subjected to Butterworth low-pass filtering to obtain the fourth respiratory motion signal, the cross-correlation function between the fourth respiratory motion signal and the reference signal is determined, the ratio of the signal shift corresponding to the maximum value of the cross-correlation function to the period of the reference signal is determined, the ratio is binarized to 0 / 180° to obtain the signal phase difference corresponding to the grid; the third number of grids with a signal phase difference of 180° is determined; the ratio of the third number to the first number is determined as the respiratory effort difference feature.

2. The method according to claim 1, characterized in that, The method for determining the first respiratory airflow signal includes: For each frame of thermal image in the thermal imaging video, the nostril region is determined from the head region in the thermal image; The first respiratory airflow signal is determined based on the pixels of the nostril region in each frame of the thermal image.

3. The method according to claim 1, characterized in that, Determining the head region in the thermal image includes: At least one high pixel value region in the thermal image is determined using an image segmentation algorithm, wherein the pixel value in the high pixel value region is higher than a first preset threshold. The head region is selected from the at least one high pixel value region based on the shape of each high pixel value region.

4. The method according to claim 2, characterized in that, Determining the nostril region from the head region in the thermal image includes: Determine multiple preset directions; In each preset direction, the head region in each frame of the thermal imaging video is subjected to edge and texture enhancement processing using a linear filter in the preset direction to obtain multiple corresponding head images. For each head image, the first region in the head image where motion occurs is determined using the inter-frame difference method. The intersection of the first regions corresponding to each head image within the target time window is obtained to obtain the second region corresponding to the preset direction. The intersection of the second regions corresponding to each preset direction is obtained to obtain the nostril region in the thermal image, wherein the thermal image is the first frame thermal image in the target time window, and the target time window is the time window corresponding to the sliding window.

5. The method according to claim 1, characterized in that, The chest and abdomen region is determined based on the head region, including: The area of ​​a preset size directly below the head region in the thermal image is determined to be the chest and abdomen region.

6. The method according to claim 2, characterized in that, Determining the first respiratory airflow signal based on pixels in the nostril region of each frame of thermal image includes: For each frame of thermal image, the average value of all pixels in the nostril region of the thermal image is determined as the respiratory airflow signal value corresponding to the thermal image; The first respiratory airflow signal is obtained by arranging the respiratory airflow signal values ​​corresponding to all thermal images in chronological order.

7. The method according to claim 6, characterized in that, The respiratory airflow characteristics include at least one of the following: variance, waveform area, 90th percentile, and instantaneous respiratory interval. Multiple respiratory airflow characteristics of different dimensions are determined based on the second respiratory airflow signal, including: The second respiratory airflow signal is subjected to noise filtering to obtain the third respiratory airflow signal; Determine the variance of all respiratory airflow signal values ​​in the third respiratory airflow signal; The area enclosed by the waveform of the third respiratory airflow signal and a preset baseline is defined as the waveform area. All respiratory airflow signal values ​​in the third respiratory airflow signal are sorted from smallest to largest, and the 90th percentile of all sorted respiratory airflow signal values ​​is determined. Identify all first respiratory airflow signal peaks greater than a second preset threshold in the first respiratory airflow signal. Iterate through all first respiratory airflow signal peaks. If there are multiple consecutive first respiratory airflow signal peaks with time intervals less than a third preset threshold, retain only the maximum value among the multiple first respiratory airflow signal peaks. Determine the remaining first respiratory airflow signal peaks after iteration as second respiratory airflow signal peaks. Determine whether the third respiratory airflow signal includes the second respiratory airflow signal peaks. If it includes them, determine the instantaneous breathing interval as 1; if it does not include them, determine the instantaneous breathing interval as 0.

8. The method according to claim 1, characterized in that, The methods for determining the respiratory effort intensity characteristics include: The breathing effort intensity characteristic is determined based on the relative power index of each grid.

9. The method according to claim 8, characterized in that, The respiratory effort intensity characteristic is determined based on the relative power index of each grid, including: Determine a second number of grids whose relative power index is greater than a fourth preset threshold; The ratio of the second quantity to the first quantity is determined as the respiratory effort intensity characteristic.

10. The method according to claim 1, characterized in that, Clustering is performed on the feature vectors corresponding to multiple time windows to obtain the classification result for each time window, including: Three initial cluster centers are preset, corresponding to the three states of central sleep apnea, obstructive sleep apnea, and normal sleep breathing. Based on the three initial cluster centers, K-means clustering is performed on the multiple feature vectors to obtain three clusters. The state of the cluster to which the feature vector of each time window belongs is the classification result of the time window.

11. A sleep apnea detection device, characterized in that, include: An acquisition module is used to acquire thermal imaging video of a target object during sleep, and to determine a first respiratory airflow signal characterizing the respiratory airflow state of the target object and a first respiratory motion signal characterizing the chest and abdominal movement state of the target object in the thermal imaging video. The determination of the first respiratory motion signal includes: for each frame of the thermal image in the thermal imaging video, determining the head region in the thermal image, and determining the chest and abdominal region based on the head region; for each frame of the thermal image, determining the displacement of each pixel in the chest and abdominal region of the thermal image using dense optical flow, and acquiring the time interval between two consecutive frames in the thermal imaging video, determining the movement velocity of each pixel based on the displacement and the time interval; dividing the chest and abdominal region into a first number of grids; for each grid, determining the average velocity of the movement velocities of all pixels in the grid as the respiratory motion signal value corresponding to the grid, arranging the respiratory motion signal values ​​corresponding to the grid in all thermal images in chronological order to obtain a third respiratory motion signal corresponding to the grid; and using the third respiratory motion signal corresponding to each grid as one channel of the first respiratory motion signal. The segmentation module is used to segment the first respiratory airflow signal and the first respiratory motion signal using a preset sliding window to obtain the second respiratory airflow signal and the second respiratory motion signal under multiple time windows. The feature extraction module is used to determine multiple respiratory airflow features of different dimensions based on the second respiratory airflow signal within each time window, and to determine respiratory effort intensity features characterizing the amplitude of chest and abdominal movements and respiratory effort similarity / similarity features characterizing the phase of chest and abdominal movements based on the second respiratory motion signal. The multiple respiratory airflow features, respiratory effort intensity features, and respiratory effort similarity / similarity features are combined as the feature vector corresponding to the time window. The determination method for the respiratory effort similarity / similarity features includes: for each grid, determining the power spectrum corresponding to the third respiratory motion signal of the grid using autocorrelation and Fourier transform; if the peak frequency of the power spectrum is within a preset normal respiratory frequency band, determining the total power within the power spectrum corresponding to a preset width centered on the peak frequency as the normal respiratory power; if the peak frequency of the power spectrum is not within the normal respiratory frequency band, determining... Define the total power corresponding to the normal breathing frequency band as the normal breathing power; calculate the power difference between the total power of the power spectrum and the normal breathing power, and determine the ratio of the normal breathing power to the power difference as the relative power index corresponding to the grid; determine the maximum value among the relative power indices of all grids as the reference signal; for each grid, perform Butterworth low-pass filtering on the third respiratory motion signal corresponding to the grid to obtain the fourth respiratory motion signal, determine the cross-correlation function between the fourth respiratory motion signal and the reference signal, determine the ratio of the signal shift corresponding to the maximum value of the cross-correlation function to the period of the reference signal, binarize the ratio by 0 / 180° to obtain the signal phase difference corresponding to the grid; determine the third number of grids with a signal phase difference of 180°; determine the ratio of the third number to the first number as the respiratory effort difference feature; The classification module is used to cluster multiple feature vectors corresponding to multiple time windows to obtain a classification result for each time window. The classification result is used to indicate that the target object is in one of the following states within the time window: central sleep apnea, obstructive sleep apnea, or normal sleep breathing.

12. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the sleep apnea detection method according to any one of claims 1 to 10 by running the computer program.

13. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the sleep apnea detection method according to any one of claims 1 to 10 via the computer program.

Images