Method, electronic device and system for carrying out OSA early screening on multi-angle face photos based on deep learning
Through the MVCNN-based deep learning model, the OSA initial screening of multi-angle facial photos was performed, and combined with the reinforcement learning algorithm, the existing OSA diagnostic methods were solved, and efficient and accurate OSA screening and diagnosis.
Patent Information
- Application Number
- CN202410990712.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
The existing OSA diagnostic methods are costly, time-consuming and not suitable for large-scale low-cost screening. The traditional detection methods are complex and cannot accurately locate the airflow blocking areas. The existing screening methods such as blood oxygen bracelets and mobile phone software have low data validity. and environmental interference problems, the rationality of characteristic indicators of artificial design is difficult to verify and improve screening accuracy.
Using a deep learning model based on multi-view convolutional neural network (MVCNN) to initially screen patients' multi-angle facial photos. Combined with reinforcement learning algorithm, OSA severity is recognized and screened through multi-view convolutional neural network (MVCNN). Check and decide whether to perform PSG diagnosis based on the initial screening results, and continuously update the training data to improve the accuracy of the model.
Improve the accuracy and efficiency of OSA initial screening, reduce the characteristic design link of manual participation, identify the severity of OSA through multi-angle photos, provide PSG test recommendations, improve diagnostic efficiency and accuracy, and is suitable for large-scale screening and stratification. Screening.
Smart Images

Figure CN118942676A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a method, electronic equipment and system for performing early screening of OSA on multi-angle facial photos based on deep learning. Background Art
[0002] Obstructive sleep apnea syndrome (OSA) is a widespread and sleep-related breathing disorder. Its main feature is the repeated collapse of the pharyngeal upper airway, which is manifested as repeated apnea and hypopnea during sleep, accompanied by severe snoring at night and daytime sleepiness. OSA is not only a risk factor for chronic diseases (such as cardiovascular disease, hypertension, etc.), but also an important cause of acute diseases (such as stroke, myocardial infarction, cor pulmonale, congestive heart failure, and even sudden death). If it is not treated effectively in time, it will pose a serious threat to personal health, leading to a decline in the patient's quality of life, impaired work performance, and even an increased risk of traffic accidents.
[0003] Currently, the diagnosis of OSA is mainly based on the calculation of the apnea-hypopnea index (AHI), which is used to measure the frequency of reduced airflow associated with upper airway collapse or stenosis. When AHI ≥ 5 events / hour, OA can be diagnosed.
[0004] Although polysomnography (PSG) is the gold standard for diagnosing OSA and its severity worldwide, it is a costly and time-consuming method (currently there are less than 1,000 sleep technicians qualified to use PSG), and the collection and processing of neural electrical signals is extremely sophisticated and complex. Summary of the invention
[0005] One of the embodiments of the present disclosure is a method for early screening of OSA based on multi-angle face photos based on deep learning, comprising the steps of:
[0006] Obtain facial photographs of the patient, including at least one frontal and one side view of the patient;
[0007] The facial photo of the patient is input into the preliminary screening model to obtain the preliminary screening result of the patient's OSA.
[0008] The initial screening model is based on MVCNN.
[0009] Based on the OSA initial screening results, it is decided whether the patient needs a PSG diagnosis. When the patient has a PSG diagnosis, the patient's facial photo and PSG diagnosis report are used as new sample data and added to the training set of the initial screening model.
[0010] The initial screening model is trained by reinforcement learning using an updated training set. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:
[0012] Figure 1 Schematic diagram of an OSA preliminary screening system using MVCNN according to one embodiment of the present invention.
[0013] Figure 2 A schematic diagram of a CNN-based OSA classification method according to one embodiment of the present invention.
[0014] Figure 3 A schematic diagram of sample classification distribution data according to one embodiment of the present invention.
[0015] Figure 4 A schematic diagram of the test accuracy of the OSA primary screening model according to one embodiment of the present invention. DETAILED DESCRIPTION
[0016] PSG is considered the gold standard for the diagnosis of OSA, but it cannot detect the precise location of airflow obstruction. Imaging examinations such as X-ray lateral cranial film measurement, sleep video fluoroscopy, CT scan and MRI are important diagnostic methods for detecting OSA obstruction sites and other structural abnormalities. Among them, MRI requires children to maintain posture for a long time, and children sometimes cannot cooperate, resulting in the inability to conduct the examination smoothly. Lateral cranial films and CT are highly clinically practical and easy for children and their families to accept.
[0017] Since the existing PSG hardware equipment is expensive, although lower-cost portable sleep diagnostic equipment has appeared in recent years, it still requires patients to wear it for at least one night to measure the results, and is not suitable as a large-scale low-cost OSA screening method. The other detection methods mentioned above also have the problem of high cost and complicated detection process.
[0018] Existing solutions include OSA screening through snoring analysis using blood oxygen bracelets or mobile phone software. However, blood oxygen bracelets can only provide feedback on the degree of oxygen depletion at night and cannot determine whether the oxygen depletion is caused by OSA. At the same time, the blood oxygen collection process is greatly affected by skin color and it is difficult to cover all people. Mobile phone software analyzes the risk of OSA by absorbing snoring, which is to a certain extent greatly affected by environmental factors, and the effectiveness of the data obtained is low.
[0019] In addition, there are methods that use artificially designed feature indicators to analyze facial photos. However, because the design of these indicators relies too much on personal experience, their rationality is difficult to verify, and it is difficult to improve screening accuracy through big data.
[0020] According to one or more embodiments, Figure 1 As shown. An OSA (Obstructive Sleep Apnea) screening system based on a multi-view convolutional neural network (MVCNN). It includes building an OSA screening model based on MVCNN and performing a preliminary screening of OSA based on MVCNN. The preliminary screening process of the OSA screening system includes:
[0021] S101, take facial photos of the patient from the front and at 90° to the left or right. Note that the patient should look straight ahead and keep the head and neck as vertical as possible during the photo shoot.
[0022] S102, input the paired front and side photos into the pre-trained MVCNN network screening model for classification, and obtain the inference result of the OSA severity screening. In order to train the MVCNN model, the system requires a certain amount of training data, which includes facial photos of different patients and their corresponding OSA diagnosis results.
[0023] S103: Based on the initial screening results of OSA, for symptomatic severe patients, it is recommended or decided that the patients undergo further complex PSG diagnosis.
[0024] S104, simultaneously adding the facial photo data and the patient data of the PSG diagnosis report as new data into the training set.
[0025] S105, when the newly added data in the training set reaches a certain amount, a new round of reinforcement learning training is carried out to improve the prediction accuracy of the initial screening model.
[0026] S106, updating the results of reinforcement learning training into the screening model to improve the accuracy of OSA initial screening through facial photos.
[0027] In the OSA screening system of the disclosed embodiment, multi-angle facial photos of the patient are required as input data, which will be used to train the MVCNN model. MVCNN is a deep learning model that can process multi-angle input data to improve the ability to identify the severity of OSA.
[0028] The system uses the MVCNN model to infer the input multi-angle facial photos to determine the severity of OSA. In order to train the MVCNN model, the system requires a certain amount of training data, which may include facial photos of different patients and their corresponding OSA diagnosis results. During the training process, the system uses a reinforcement learning algorithm to optimize the parameters of the MVCNN model to improve the prediction accuracy of the model.
[0029] PSG is the abbreviation of Polysomnography, which is also used to diagnose sleep disorders. PSG follow-up means that after the initial screening, a PSG test is needed to further confirm the diagnosis of OSA.
[0030] The disclosed embodiment uses deep learning, especially multi-view convolutional neural network, to assist in the diagnosis of obstructive sleep apnea. The system analyzes multi-angle facial photos of patients and combines them with reinforcement learning algorithms to improve the accuracy of predicting the severity of OSA, and recommends patients to undergo PSG testing for further diagnosis after preliminary screening.
[0031] In the disclosed embodiment, only two photos of the front and 45° or 90° side view can be taken with a mobile phone or any camera in a normal indoor environment, and the severity level of OSA can be inferred through the online deep learning model. In addition, the deep learning model can automatically improve its accuracy through reinforcement learning as the data continues to increase. The manual feature design process is avoided, making the initial screening of OSA more efficient and reliable.
[0032] According to one or more embodiments, a method for constructing an OSA preliminary screening model. The construction of the preliminary screening model adopts the architecture of Multi-View CNN (MVCNN). The architecture compiles the information in multiple 2D views of the object into a compact object descriptor, and constructs a classifier for the 3D model from multiple 2D image sequences from different perspectives. The comparison shows that its performance is significantly better than the classifier built directly on the 3D representation method, and achieves an accuracy of 90.1% on the ModelNet dataset (a dataset of 3D CAD models with 40 classification categories maintained by Princeton University). Compared with the most advanced 3D model 3D ShapeNets at the time, MVCNN achieved an absolute gain of 12.8% in classification accuracy.
[0033] After testing the disclosed embodiments, using three facial photos from different angles does not significantly improve the OSA classification accuracy compared to two photos, one front and one side. In addition, deep learning frameworks based on a single photo, such as VGG and Resnet, also have a significant effect on OSA initial screening based on facial photos, but the accuracy is slightly lower than that of MVCNN, which considers photos from multiple angles at the same time.
[0034] The construction process of the OSA screening model of the disclosed embodiment includes cleaning and desensitizing the sample data. First, in order to train MVCNN for identifying OSA classification, it is necessary to prepare a data set grouped by the patient's PSG report, facial photos and other oral photos. Then use AI to automatically identify the faces in the photos, automatically classify the facial photos and other photos, and automatically crop and pre-process the original photos according to the position and size of the facial area, retaining only the face and neck to reduce the image size input to the neural network and reduce interference with effective features. At the same time, the eyes are desensitized by mosaic occlusion, retaining only the front, 45° side, and 90° side images of the face. Among them, the cropping preprocessing of the original photos of the patient can include the following steps:
[0035] Perform grayscale conversion, denoising, brightness adjustment and other steps on the photo images to ensure the accuracy of subsequent analysis;
[0036] Use face detection algorithms, including Haar cascade classifiers or convolutional neural networks (CNNs) in deep learning to accurately locate and identify faces in images;
[0037] Detect key points of face recognition, such as eyes and mouth, calculate the angle of the face, and use affine matrix to rotate it to achieve face alignment;
[0038] After face detection and alignment, the face part is cropped out to remove external interference information such as background, clothes or accessories;
[0039] Use the resize() function to scale the input image of the face to have the same shape and size;
[0040] The images are normalized by subtracting the mean and then dividing by the pixel-wise standard deviation so that the images input to MVCNN have a mean of 0 and a deviation of 1.
[0041] After the above preprocessing steps, the patient images are used by the MVCNN-based OSA screening model to extract facial feature size parameters related to OSA, which helps to improve the accuracy and efficiency of facial image recognition to predict OSA. In practical applications, these steps can be adjusted according to specific needs to achieve the best preprocessing effect.
[0042] The potential risk factors for OSA studied include increased age, male sex, obesity, and abnormalities in the craniofacial region and upper airway anatomy. Among them, the inharmonious craniofacial anatomical structure is an important predisposing factor for the occurrence of OSA. The pharyngeal airway of OSA patients is partially or completely blocked, leading to obstructive apnea / hypopnea episodes and awakenings. The anatomical relationship between the craniofacial skeleton and the upper airway soft tissue structure determines the patency of the upper airway. Normal nasal breathing function can ensure the normal development of the maxillofacial region. Mouth breathing due to various reasons can cause the maxillary arch to lose the normal dynamic balance of the internal and external muscles, so that the width of the maxillary arch cannot develop normally, gradually leading to narrow dental arches, high arches of the palate, and crowding or protrusion of the front teeth. During sleep, the most obvious manifestation of mouth breathing is open-mouth breathing, the tongue and mandible retreat, and the mandibular retrusion deformity is formed. Therefore, OSA patients often have their characteristic facial size ratios. Through the sample training of the disclosed embodiment, the OSA preliminary screening model based on MVCNN is judged and predicted after classification and identification based on these facial size features.
[0043] Craniofacial features of OSA patients can be acquired in a variety of ways, most commonly by cephalometric x-ray or by three-dimensional methods such as computed tomography (CT) and magnetic resonance imaging (MRI). Although these imaging modalities provide insight into anatomical abnormalities that may lead to impaired upper airway function, these examinations are labor-intensive and carry a certain amount of radiation, thus limiting their use to research settings.
[0044] Therefore, the idea adopted by the embodiments of the present disclosure is to use photographic images to perform craniofacial phenotype analysis on the subject to predict the risk of OSA in the subject. Craniofacial measurements are better at predicting the presence of OSA than other recognized risk factors, especially body mass index (BMI). Unlike more complex imaging methods, the craniofacial phenotype obtained by facial photography is a comprehensive assessment of craniofacial bones and soft tissue components (such as local obesity). Studies in the embodiments of the present disclosure have shown that facial surface dimensions convey information about pharyngeal airway structures, such as tongue volume. Therefore, facial photography can capture a range of phenotypic information related to OSA risk, which can be used for large-scale screening and stratification of OSA levels.
[0045] In the embodiment of the present disclosure, different patients have different angles and numbers of facial photos. In order to reduce the amount of sample processing calculations and the number of input photos, only a frontal photo and a 45° or 90° side photo are used as the input of MVCNN for training. Figure 2 shown.
[0046] Figure 2 A multi-view image processing pipeline for obstructive sleep apnea (OSA) classification using a convolutional neural network (CNN) is described. It includes:
[0047] S201, receiving front and side portrait photos of an OSA patient as input data.
[0048] S202, simultaneously processes the received image data from multiple perspectives (front and side) through the Multi-View CNN multi-view convolutional neural network.
[0049] S203, in Multi-View CNN, the first convolutional neural network layer CNN1 is used to extract features from the input image.
[0050] S204, in the Multi-View CNN, a pooling layer CNN Pooling in the convolutional neural network connected to the first convolutional neural network layer CNN1 is used to reduce feature dimensions while retaining important information.
[0051] S205, in the Multi-View CNN, the second convolutional neural network layer CNN2 of the pooling layer CNN Pooling in the convolutional neural network is connected to further extract and refine features.
[0052] S206, based on the features extracted by CNN2, the system classifies the input portrait photo for OSA, and the output results include:
[0053] OSA negative: means the person in the photo has no signs of OSA.
[0054] Mild OSA: Indicates that the person in the photo has mild OSA.
[0055] Moderate to severe OSA: Indicates that the person in the photo has moderate to severe OSA.
[0056] Therefore, one of the beneficial effects of the embodiments of the present disclosure is that, through image recognition and classification, deep learning is used to assist in screening and diagnosing obstructive sleep apnea to improve the efficiency and accuracy of diagnosis, especially in the preliminary screening stage.
[0057] In addition, according to the embodiment of the present disclosure, in order to better screen out the moderate to severe OSA population, the inference target of the primary screening model of the present disclosure is changed from a simple yin-yang binary classification to a three-target classification of yin (asymptomatic), mild, and moderate to severe, and the three OSA level classifications of asymptomatic, mild, and moderate to severe are guaranteed to cover all samples as much as possible, and the distribution of the photo groups by gender, BMI, and age statistics. Figure 3 shown.
[0058] The severity of OSA is classified based on the AHI value in the PSG report according to clinical criteria as follows:
[0059] Age Group Asymptomatic Mild Moderate to severe Youth (under 18 years old) AHI<1 1<=AHI<5 AHI>=5 Adults (18 years and above) AHI<5 5<=AHI<15 AHI>=15
[0060] In order to verify the training effect of the OSA screening model, the holdout method was used to process the valid data of 53 patients, of which 347 pairs of data from 40 people were used for training, and 102 pairs of photos from 13 people were used for testing. The best performance of the trained model on the test set can reach 86.3% accuracy, and the AUCs of the three categories of asymptomatic, mild, and moderate to severe are 0.894, 0.868, and 0.998, respectively. This preliminarily proves the diagnostic effect of the MVCNN method based on multi-angle photos on the three categories of OSA (asymptomatic, mild, moderate to severe). Figure 4 shown.
[0061] The beneficial effect of the disclosed embodiment is that multiple facial photos of patients from different angles are input into the deep learning network of Multi-ViewCNN, and the parameters that can be used for the OSA initial screening model are trained by comparing the differences with the control PSG results, so as to improve the accuracy of the OSA initial screening model.
[0062] In summary, the present disclosure relates to a system that uses deep learning, particularly multi-view convolutional neural networks, to assist in the diagnosis of obstructive sleep apnea. The system improves the accuracy of predicting the severity of OSA by analyzing multi-angle facial photos of patients combined with reinforcement learning algorithms, and may recommend patients to undergo PSG testing for further diagnosis after preliminary screening.
[0063] Compared with the prior art, the present invention does not use artificially designed feature indicators, but uses multi-angle facial photos of patients at the same time to preliminarily screen the severity of OSA in patients through a deep learning model. Moreover, when judging the degree of OSA in patients, it does not simply judge the yin and yang of OSA, but further distinguishes between mild and moderate to severe symptoms, providing guidance for further treatment method selection.
[0064] It should be understood that in the embodiments of the present invention, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0065] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.
[0066] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0067] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0068] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for early screening of OSA, characterized in that: The following steps are involved: Obtain facial photographs of the patient, including at least one frontal and one side view; The patient's facial photo is input into the primary screening model to obtain the patient's OSA primary screening result.
2. The method according to claim 1, wherein: The primary screening model is based on MVCNN.
3. The method according to claim 1, wherein: Based on the OSA initial screening results, decide whether the patient needs PSG diagnosis. When the patient undergoes a PSG diagnosis, the patient's facial photo and PSG diagnosis report are used as new sample data and added to the training set of the initial screening model.
4. The method according to claim 3, wherein: The initial screening model is trained by reinforcement learning using the updated training set.
5. The method according to claim 1, wherein: The side view of the patient includes a side view of the patient at 45° or 90°.
6. The method according to claim 1, wherein: The initial screening results of the patient's OSA include the classification of the patient's OSA, including OSA negative, mild OSA, or moderate to severe OSA.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 6.
8. An early screening system for OSA, characterized in that: The system includes a server having a memory; and A processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 6.
9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program, characterized in that The computer program is executed by a processor to implement the method according to any one of claims 1 to 6.
