Method for classifying obstructive sleep apnea by using multimodal data and analysis device

Abstract

This method for classifying obstructive sleep apnea by using multimodal data comprises the steps of: receiving, by an analysis device, inputs of multimodal data of a subject; inputting, by the analysis device, the multimodal data into a trained learning model; and classifying, by the analysis device, whether the subject has obstructive sleep apnea, on the basis of a value output by the learning model. The multimodal data includes facial images, head X-ray images, and clinical data.

Description

Method and analysis device for classifying obstructive sleep apnea using multimodal data

[0001] The technique described below relates to a technique for diagnosing obstructive sleep apnea using head imaging.

[0002] Sleep apnea is a condition in which breathing temporarily stops during sleep. Obstructive sleep apnea (OSA) occurs when the airway in the mouth or throat narrows, temporarily blocking airflow during sleep. In Korea, the prevalence of OSA among adults aged 40 to 69 is high, at 27.1% in men and 16.8% in women.

[0003] Diagnosis of obstructive sleep apnea is usually made through symptoms such as daytime sleepiness, snoring, gasping or choking during sleep, fatigue, and difficulty concentrating, as well as through polysomnography.

[0004] The technology described below aims to provide a technique for diagnosing obstructive sleep apnea using multimodal data.

[0005] A method for classifying obstructive sleep apnea using multimodal data includes a step in which an analysis device receives multimodal data of a subject, a step in which the analysis device inputs the multimodal data into a learned learning model, and a step in which the analysis device classifies whether the subject has obstructive sleep apnea based on a value output by the learning model. The multimodal data includes a facial image, a head x-ray image, and clinical data.

[0006] An analysis device for analyzing obstructive sleep apnea includes an interface device for receiving multimodal data of a subject, a storage device for storing a learning model for classifying obstructive sleep apnea, and a calculation device for classifying whether or not the subject has obstructive sleep apnea based on a value output by inputting the multimodal data into the learning model.

[0007] The technology described below utilizes imaging and clinical information to diagnose obstructive sleep apnea with high accuracy by leveraging multifaceted features. The technology described below can diagnose obstructive sleep apnea in a short period of time using only medical imaging and clinical information.

[0008] Figure 1 is an example of a system for diagnosing obstructive sleep apnea.

[0009] Figure 2 is an example of a model for diagnosing obstructive sleep apnea.

[0010] Figure 3 is an example of the learning process of an obstructive sleep apnea diagnosis model.

[0011] Figure 4 is an example of the enhancement process of an x-ray image.

[0012] Figure 5 is an example of an analysis device for diagnosing obstructive sleep apnea.

[0013] The technology described below is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this does not limit the technology described below to specific embodiments, and it should be understood that all modifications, equivalents, and alternatives fall within the spirit and scope of the technology described below.

[0014] Terms such as first, second, A, and B may be used to describe various components, but these components are not limited by these terms and are used solely to distinguish one component from another. For example, without departing from the scope of the technology described below, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0015] As used herein, the singular expressions should be understood to include the plural expressions unless the context clearly dictates otherwise, and the term "comprises" and the like should be understood to mean the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0016] Before going into a detailed description of the drawings, it should be made clear that the division of components in this specification is merely a division based on the main function of each component. In other words, two or more components described below may be combined into a single component, or a single component may be further subdivided into two or more components with more detailed functions. In addition to its own main function, each component described below may additionally perform some or all of the functions of other components, and of course, some of the main functions of each component may be exclusively performed by other components.

[0017] Additionally, in performing a method or method of operation, each process constituting the method may occur in a different order than the stated order, unless the context clearly indicates a specific order. That is, each process may occur in the same order as the stated order, may be performed substantially simultaneously, or may be performed in the opposite order.

[0018] The technique described below is a technique for diagnosing obstructive sleep apnea using multiple domain information.

[0019] The technique described below can diagnose obstructive sleep apnea using at least one of the multimodal data.

[0020] Multimodal data may include facial images, head x-ray images, and clinical data about the subject (patient).

[0021] A facial image is a video captured of a patient's face. It can be a frontal image of the patient's face, or it can be a side view of the patient's face. It can be a color image captured by a camera. It can also be a two-dimensional (2D) image.

[0022] A head x-ray image is taken using an x-ray imaging device. The head x-ray image may be an x-ray image of the patient's side.

[0023] Clinical data may include general clinical information about the patient, obesity-related information, sleep-related indicators, etc. For example, clinical data may include the patient's age, sex, body mass index (BMI), neck circumference, abdominal circumference, Epworth Sleepiness Scale (ESS), and Stanford Sleepiness Scale (SSS).

[0024] The following describes how the analysis device uses a learning model to classify and diagnose obstructive sleep apnea. The analysis device can be implemented in various forms. For example, it can be implemented as a PC, a networked server, a smart device, or a chipset embedded with a dedicated program.

[0025] Learning models refer to machine learning models, and there are various types of machine learning models. For example, machine learning models include decision trees, random forests (RF), K-nearest neighbors (KNN), naive Bayes, support vector machines (SVM), and artificial neural networks (ANN). Meanwhile, ANNs are statistical learning algorithms that mimic biological neural networks. Various neural network models are being studied. Deep learning networks (DNNs), like general artificial neural networks, can model complex non-linear relationships. Various types of DNN models have been studied, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), generative adversarial networks (GANs), and relational networks (RLs). The following explanation focuses on CNN-based models.

[0026] A learning model is built through a specific learning process. Below, we explain how a learning device builds a learning model using learning data. A learning device is a computing device capable of image preprocessing and learning. A learning device can be implemented in a form factor such as a PC or server.

[0027] Fig. 1 is an example of a system (100) for diagnosing obstructive sleep apnea. In Fig. 1, the analysis device is illustrated as an example of a computer terminal (130) and a server (140).

[0028] The imaging device (110) captures an image of the patient's face. The imaging device (110) can store the facial image in an Electronic Medical Record (EMR, 120) or a separate database (DB). The imaging device (110) can be implemented in the form of an image sensor, a smart device, a user terminal, a wearable device, etc.

[0029] The x-ray equipment (112) takes an x-ray image of the patient's head. The x-ray equipment (110) takes an image of the patient's head area. The x-ray equipment (110) can store the head x-ray image in the EMR (120) or a separate DB.

[0030] The terminal device (113) can receive patient clinical data. Alternatively, the terminal device (113) can collect patient clinical data from patient terminals, IoT devices, etc. The terminal device (113) can store the clinical data in the EMR (120) or a separate database.

[0031] In FIG. 1, a user (A) can analyze at least one of multimodal data using a computer terminal (130). The computer terminal (130) can receive facial images, head x-ray images, and clinical data from an imaging device (110), an x-ray device (112), or a terminal device (113) via a wired or wireless network. Alternatively, the computer terminal (130) can receive at least one of the multimodal data from an EMR (120) via a wired or wireless network. The computer terminal (130) can preprocess at least one of the multimodal data. The computer terminal (130) can input at least one of the multimodal data into a pre-trained learning model. The computer terminal (130) can classify whether the patient has obstructive sleep apnea based on a value output by the learning model. User A can check the analysis results on the computer terminal (130).

[0032] The server (140) can receive facial images, head x-ray images, and clinical data from the imaging device (110), x-ray equipment (112), or terminal device (113). The server (140) can receive at least one of the multimodal data from the EMR (120) via a wired or wireless network. The server (140) can preprocess at least one of the multimodal data. The server (140) can input at least one of the multimodal data into a pre-trained learning model. The server (140) can classify whether the patient has obstructive sleep apnea based on a value output by the learning model. The server (140) can transmit the analysis result to the terminal of user A.

[0033] The computer terminal (130) and / or server (140) may also store the analysis results in the EMR (120).

[0034] Figure 2 is an example of a model for diagnosing obstructive sleep apnea.

[0035] Figure 2(A) is a model that classifies or diagnoses obstructive sleep apnea using a patient's single modal data.

[0036] Learning Model 1 can classify OAS using facial images. In Learning Model 1, the convolutional layer receives a facial image as input and outputs facial image features (Feature 1). The pre-processing layer then receives Feature 1 and classifies the subject as having OSA or being normal. For example, OSA patients have characteristics such as a thick neck, a wide face, or a wide chin. Feature 1 can reflect these structural characteristics.

[0037] Learning Model 2 can classify OAS using head X-ray images. In Learning Model 2, the convolutional layer receives a head X-ray image as input and outputs head X-ray image features (Feature 2), and the pre-coupling layer receives Feature 2 as input and can classify whether the subject has OSA or is normal. For example, an OSA patient may have a narrowed posterior airway space at the back of the tongue. Feature 2 can reflect this structural feature.

[0038] Learning Model 3 can classify OAS using clinical data. In Learning Model 3, the convolutional layer receives clinical data as input and outputs clinical data features (Feature 3). The pre-processing layer then receives Feature 3 as input and classifies whether the subject has OSA or is healthy.

[0039] Figure 2(B) is a model that classifies and diagnoses obstructive sleep apnea using multimodal data. The learning model receives facial images, head X-ray images, and clinical data as inputs, respectively. The learning model may have separate input layers according to modality. Alternatively, the learning model may embed data of different modalities in a certain order and input them into the input layer. In this case, the learning model can combine facial image features (feature 1), head X-ray image features (feature 2), and clinical data features (feature 3) and pass the integrated features to the pre-connected layer to ultimately classify whether the subject has OSA or is normal. Alternatively, the learning model can combine image features (feature 1), head X-ray image features (feature 2), and clinical data features (feature 3) and pass the fused features to the pre-connected layer to ultimately classify whether the subject has OSA or is normal.

[0040] Figure 3 illustrates an example of the learning process (200) for an obstructive sleep apnea diagnosis model. The learning model construction process is divided into a learning data construction process (210) and a model training process (220) using the learning data. The learning data construction process and the model training process can be performed on separate devices. For convenience of explanation, the learning device is described as performing the learning model construction process.

[0041] Training data can be constructed through certain preprocessing or augmentation. Image-type data, in particular, may require certain preprocessing or augmentation.

[0042] Among the training data, head X-ray images require particularly high preprocessing. The learning device can extract specific head X-ray images from the image database.

[0043] The learning device can consistently normalize head x-ray images taken under different conditions or devices.

[0044] The learning device can control the intensity of the head x-ray image. The learning device can control the intensity value of the image through a windowing clip.

[0045] The learning device can normalize a number of head x-ray images using min-max normalization. That is, the learning device can convert each pixel of the head x-ray image to a value between [0,1].

[0046] The learning device can normalize the head x-ray image to a certain size.

[0047] Depending on the shooting environment, head X-ray images may have certain areas cropped out. For example, some head X-ray images may have the occiput cropped out. In this case, the learning model may produce biased results depending on the input image. Therefore, the learning device can randomly crop the head X-ray images to augment the data.

[0048] The learning device can store preprocessed images and augmented learning data in a learning DB.

[0049] The learning database can store facial images, head X-ray images, and clinical data sets for specific subjects. Additionally, the learning database stores the correct answer to whether the subject has OSA.

[0050] Afterwards, the learning device performs the process of learning the learning model (220).

[0051] The learning device can extract multimodal data (facial images, head X-ray images, and clinical data) from the learning DB.

[0052] The learning device inputs the extracted multimodal data into a learning model. The learning model extracts features from each modality in the input multimodal data, combines or aggregates them, and then outputs a probability value for OSA based on this combined data. The learning device then compares the probability value output by the learning model with the correct answer and updates the learning model's parameters. The learning device repeatedly trains the learning model using multiple training data sets.

[0053] Figure 4 is an example of the enhancement process of an x-ray image.

[0054] A head x-ray image may be a form in which the entire head or a portion of the head is cropped, depending on the shooting environment. Figures 4(A) and 4(B) are images with different sizes (resolutions) of 2020×2012 and 1360×1840, respectively. Depending on the person shooting the image and / or the image resolution, the head x-ray image may be an image in which a specific portion is cropped. In order to prepare an image in which a specific portion is cropped for training data, as shown in Figure 4(A), a learning device can randomly crop an image to extract an image in which a certain portion is cropped. The learning device can generate a cropped image by randomly cropping an image with a large resolution. Thereafter, the learning device can randomly rotate the cropped image to produce multiple images.

[0055] As shown in Fig. 4(B), the learning device can also augment data by only performing random rotation for images with relatively low resolution.

[0056] The learning device can generate an image of a certain size (e.g., 336×224) by finally resizing while maintaining the width and height ratio during the cropping process.

[0057] The dataset used to build the learning model was collected from patients who underwent polysomnography at Samsung Medical Center from January 2016 to January 2024. The collected clinical data included physical information such as height, weight, abdominal circumference, and neck circumference, as well as Epworth Sleepiness Scale (ESS) and Stanford Sleepiness Scale (SSS) scores. OSA diagnosis was confirmed through polysomnography according to the American Academy of Sleep Medicine (AASM) guidelines. OSA severity was classified based on the Apnea-Hypopnea Index (AHI). The dataset consisted of 58 subjects with normal (0 ≤ AHI < 5), 145 with mild OSA (5 ≤ AHI < 15), 159 with moderate OSA (15 ≤ AHI < 30), and 264 with severe OSA (AHI ≥ 30). The population and clinical data used to build the model are shown in Table 1 below.

[0058] Item Dataset (OSA Severity Classification)Normal (0 ≤ AHI <5)Mild OSA (5 ≤ AHI <15)Moderate OSA (15 ≤ AHI <30)Severe OSA (30 ≤ AHI)Number of patients 58145159264Age (year) 43.36±13.1945.14±14.9851.84±13.2652.58±12.08BMI (kg / m) 2 )23.29± 2.3624.32 ± 2.7225.83 ± 3.1627.89 ± 4.00Gender (male / female)22 / 36108 / 37121 / 38237 / 27Neck circumference35.19± 3.4237.13 ± 2.8638.03 ± 3.0540.20 ± 3.26Abdominal circumference83.16± 7.7186.84 ± 7.9990.77 ± 8.6496.67 ± 9.20ESS9.78±4.929.77 ± 4.629.94 ± 4.8010.42 ± 4.84SSS2.95± 1.052.68 ± 0.942.62 ± 0.852.82 ± 1.16

[0059] Multimodal data, as described above, includes facial images, head X-ray images, and clinical data. Therefore, the dataset includes facial images and head X-ray images of the same patient in addition to clinical data.

[0060] To build a learning model, the training data underwent certain preprocessing. Facial images were converted to grayscale to minimize the influence of differences in clothing and morphological features. Facial images were augmented using data augmentation techniques (horizontal flipping and random rotation). As mentioned above, head X-ray images were randomly cropped and resized to a consistent size due to the various image sizes and imaging areas. Clinical data were normalized using a standard scaler that adjusted them to have a mean of 0 and a standard deviation of 1.

[0061] To compare model performance, the researchers built two models: one using a single modality and one using multimodal data. The models are illustrated in Figure 2. The learning models were trained to binary classify the target patients. The learning models were trained to classify patients into (i) normal and mild OSA groups (AHI < 15) or (ii) moderate and severe OSA groups (AHI ≥ 15).

[0062] The researchers performed K-fold 5-fold cross-validation and compared the means and standard deviations. Table 2 below shows the performance of each model. As shown in Table 2, models using multimodal data generally outperformed models based on a single modality. For example, models using multimodal data showed the highest performance based on the Area Under the ROC Curve (AUC).

[0063] modalityAUCSensitivitySpecificityAccuracyClinical data0.809±0.0370.796±0.0340.662±0.9210.758±0.041X-ray image0.801±0.0390.850±0.0350.544±0.1070.763±0.033Facial image0.757±0.0160.822±0.0520.570±0.1110.751±0.020Facial + X-ray + clinical data (multimodal data)0.820±0.0280.792±0.0630.704±0.0670.767±0.035

[0064] Figure 5 illustrates an example of an analysis device (300) for diagnosing obstructive sleep apnea. The analysis device (300) corresponds to the aforementioned analysis devices (130 and 140 of Figure 1). The analysis device (300) may be physically implemented in various forms. For example, the analysis device (300) may take the form of a computer device such as a PC, a network server, a data processing chipset, or the like.

[0065] The analysis device (300) may include a storage device (310), a memory (320), a computing device (330), an interface device (340), a communication device (350), and an output device (360).

[0066] The storage device (310) can store multimodal data of a specific subject. The multimodal data can include facial images, head X-ray images, and clinical data. The clinical data can include age, gender, BMI, neck circumference, abdominal circumference, Epworth Sleepiness Scale (ESS), and Stanford Sleepiness Scale (SSS).

[0067] The storage device (310) can store a code or program for classifying or diagnosing obstructive sleep apnea by analyzing at least one of the multimodal data.

[0068] The storage device (310) can store a trained learning model for diagnosing obstructive sleep apnea. The learning model may be a deep learning model. The learning model may be a CNN-based model.

[0069] The memory (320) can store data and information generated during the process of the analysis device classifying or analyzing obstructive sleep apnea.

[0070] The interface device (340) is a device that receives certain commands and data from the outside.

[0071] The interface device (340) can receive at least one of multimodal data from a physically connected input device or an external storage device.

[0072] The interface device (340) may analyze at least one of the multimodal data and transmit the results of classifying obstructive sleep apnea to an external object.

[0073] The interface device (440) may also refer to a device configuration that transmits certain data or information into the analysis device (400) through a communication device (450).

[0074] A communication device (350) refers to a configuration that receives and transmits certain information through a wired or wireless network.

[0075] The communication device (350) can receive at least one of the multimodal data from an external object.

[0076] Alternatively, the communication device (350) may analyze at least one of the multimodal data and transmit the result of classifying obstructive sleep apnea to an external object such as a user terminal.

[0077] An output device (360) is a device that outputs certain information. The output device (360) can output interfaces, multimodal data, analysis results, etc. required for the data processing process.

[0078] The computing device (330) can input at least one of the multimodal data into a learning model to classify or infer whether it is OSA.

[0079] The computing device (330) can input all facial images, head x-ray images, and clinical data into a learning model to classify or infer whether or not OSA is present.

[0080] The computational device (330) can input data from each modality (facial image, head X-ray image, and clinical data) and concatenate the features output by the convolutional layer of the learning model. The computational device (330) can input the connected features into a classification layer, such as a pre-processing layer, to classify or infer whether a specific subject has OSA.

[0081] The computational device (330) can input data of each modality (facial image, head X-ray image, and clinical data) and add the features output by the convolutional layer of the learning model. The computational device (330) can input the added features into a classification layer such as a pre-processing layer to classify or infer whether a specific subject has OSA.

[0082] The computing device (330) can classify the degree of obstructive sleep apnea for a specific subject based on the output value of the learning model. The learning model can be trained to perform binary classification or multi-classification according to the learning process. For example, the computing device (330) can classify whether a specific subject has OSA or normal. Furthermore, the computing device (330) can perform multi-classification for OSA. As in the learning process described above, the computing device (330) can determine a specific subject as having any of normal, mild OSA, moderate OSA, or severe OAS.

[0083] The computing device (330) may be a device such as a processor, AP, or chip embedded with a program that processes data and performs certain operations.

[0084] Additionally, the above-described obstructive sleep apnea diagnosis method can be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program can be stored and provided on a temporary or non-transitory computer-readable medium.

[0085] A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on a non-transitory readable medium, such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM), EEPROM (Electrically EPROM), or flash memory.

[0086] Temporarily readable media refers to various types of RAM, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM), and Direct Rambus RAM (DRRAM).

[0087] The present embodiment and the drawings attached to the present specification only clearly illustrate a part of the technical idea included in the above-described technology, and it is obvious that all modified examples and specific embodiments that can be easily inferred by a person skilled in the art within the scope of the technical idea included in the specification and drawings of the above-described technology are included in the scope of the rights of the above-described technology.

Claims

1. A step in which the analysis device receives multimodal data of the subject; A step in which the above analysis device inputs the multimodal data into a learned learning model; and The above analysis device includes a step of classifying whether the subject has obstructive sleep apnea based on the value output by the learning model. The above multimodal data is a method for classifying obstructive sleep apnea using multimodal data including facial images, head x-ray images, and clinical data.

2. In paragraph 1, A method for classifying obstructive sleep apnea using multimodal data including at least one of age, sex, BMI, neck circumference, abdominal circumference, Epworth Sleepiness Scale, and Stanford Sleepiness Scale.

3. In paragraph 1, The above analysis device inputs the multimodal data into the convolution layer of the learning model, and combines facial image features, head x-ray image features, and clinical data features to produce the multimodal data, and inputs the multimodal data into the classification layer of the learning model to classify obstructive sleep apnea.

4. In paragraph 1, The above analysis device inputs the multimodal data into the convolution layer of the learning model, adds the facial image features, head x-ray image features, and clinical data features, and inputs the multimodal data into the classification layer of the learning model to classify obstructive sleep apnea using the multimodal data.

5. In paragraph 1, The above learning model is a method for classifying obstructive sleep apnea using multimodal data, which is a model constructed using augmented learning data by performing random cropping of head x-ray images and random rotation of randomly cropped images during the learning process.

6. Interface device for receiving multimodal data of the subject; A storage device storing a learning model for classifying obstructive sleep apnea; and Including a calculation device that classifies whether the subject has obstructive sleep apnea based on the value output by inputting the above multimodal data into the above learning model, The above multimodal data is an analysis device for analyzing obstructive sleep apnea including facial images, head x-ray images, and clinical data.

7. In paragraph 6, The above clinical data is an analysis device for analyzing obstructive sleep apnea including at least one of age, sex, BMI, neck circumference, abdominal circumference, Epworth sleepiness scale, and Stanford sleep scale.

8. In paragraph 6, The above-mentioned computing device is an analysis device for analyzing obstructive sleep apnea by inputting the multimodal data into the convolution layer of the above-mentioned learning model, concatenating the facial image features, head x-ray image features, and clinical data features, and inputting them into the classification layer of the above-mentioned learning model.

9. In paragraph 6, The above-mentioned computing device is an analysis device for analyzing obstructive sleep apnea that inputs the multimodal data into the convolution layer of the above-mentioned learning model, adds the facial image features, head x-ray image features, and clinical data features, and inputs them into the classification layer of the above-mentioned learning model.

10. In paragraph 6, The above learning model is an analysis device for analyzing obstructive sleep apnea, which is a model constructed using augmented learning data by performing random cropping of head X-ray images and random rotation of randomly cropped images during the learning process.

Images