Method and apparatus for artificial intelligence-based frontal and lateral craniofacial image processing and analysis
By performing local occlusion weighting processing on craniofacial images and combining face recognition and machine learning algorithms, the complexity and high cost of existing OSA diagnostic models are solved, achieving efficient and interpretable OSA diagnosis and improving the model's prediction accuracy and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING CHAOYANG HOSPITAL CAPITAL MEDICAL UNIVERSITY
- Filing Date
- 2022-06-30
- Publication Date
- 2026-06-16
Smart Images
Figure CN115205934B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence image processing, and in particular to a method and apparatus for processing and analyzing frontal and lateral craniofacial images based on artificial intelligence. Background Technology
[0002] Currently, the diagnosis of obstructive sleep apnea (OSA) relies on polysomnography (PSG) and manual data analysis at sleep centers. This involves analyzing overnight electroencephalogram (EEG), electrooculogram (EOG), mandibular EEG, nasal airflow, temperature signals, and chest movement signals to determine the presence of sleep problems. However, the PSG monitoring process, the use of sleep monitoring equipment, and the interpretation and analysis of results are costly in terms of manpower, resources, and time, making it difficult for many medical institutions to implement and hindering timely diagnosis and treatment for patients. Therefore, alternative methods are needed to conveniently and efficiently screen for OSA, enabling high-risk patients to receive further diagnosis and treatment. Furthermore, rapid and effective screening tools can assist in OSA epidemiological surveys.
[0003] Using clinical data to build diagnostic models for obstructive sleep apnea (OSA), or sleep questionnaires, is a common method for predicting OSA risk. OSA risk factors include demographic parameters, obesity-related parameters, clinical symptoms, and physical examination. Age, sex, snoring, sleep apnea, BMI, Mallampati rating, and pharyngeal wall stenosis play important roles in diagnosing OSA.
[0004] Using imaging images to establish an OSA diagnostic model, craniofacial anatomical abnormalities play an important role in the occurrence of OSA. Narrow space enclosed by bony structures, enlarged soft tissue, and excessive local fat accumulation can all lead to a reduction in upper airway space, increasing the likelihood of upper airway collapse.
[0005] Using craniofacial images to establish diagnostic models for obstructive sleep apnea (OSA), Lee et al. proposed a photometric analysis method for craniofacial images in 2009 to assess the craniofacial characteristics of OSA patients. By measuring calibrated images, they found that OSA patients had a smaller mandibular encirclement and a wider, flatter face. Furthermore, parameters measured from craniofacial images have predictive value for OSA. Since then, research on craniofacial images for OSA has continued to deepen.
[0006] Current research on image sampling processes is complex, involving a lengthy process of rigorous image calibration, manual image marking, and measurement of images of different structures. Furthermore, current models often input unprocessed craniofacial images or simply cropped craniofacial images. The required use cases are numerous, the model costs are high, and the models suffer from poor interpretability, weak generalization ability, and low robustness; they merely input craniofacial images into the model and output corresponding classification results. There is a lack of visualization analysis and interpretation of the image models.
[0007] Therefore, this invention addresses this drawback by weighting the image regions that provide positive feedback to the model, thereby reducing the negative feedback from other regions and enhancing the model's performance. Summary of the Invention
[0008] To address the aforementioned technical problems, this invention provides a method and apparatus for processing and analyzing frontal and lateral craniofacial images based on local occlusion weighting. This method preprocesses and weights the subject's craniofacial image using a face recognition algorithm, thereby highlighting positively contributing regions in the craniofacial image and improving the model's performance. Compared to existing technologies, this method offers advantages such as convenient and simple image acquisition, small data volume required, low model training cost, and excellent model visibility. After processing the initial frontal and lateral craniofacial image of the subject, the weighted frontal and lateral image of the subject can be obtained.
[0009] This invention, based on the occlusion of the following regions in a craniofacial image, investigates the contribution value of each region—eyes, nose, mouth, forehead, and cheeks—to the model in a frontal image. A threshold is established by calculating the average value of each region's contribution. Regions with contribution values exceeding the threshold are given increased weight, resulting in a weighted frontal image.
[0010] This invention, based on the occlusion of the following regions in craniofacial images, investigates the contribution value of each region—the preauricular region, ear, neck, temporal region, and occipital region—to the model in a lateral image. A threshold is established by calculating the average value of each region's contribution. Regions with contribution values exceeding the threshold are given increased weight, resulting in a weighted lateral image.
[0011] Compared to inputting raw craniofacial images, inputting craniofacial weighted images significantly improves the model's performance and greatly increases prediction accuracy.
[0012] The technical solution of this invention is: a method for processing and analyzing frontal and lateral craniofacial images based on artificial intelligence, comprising the following steps:
[0013] Step 1: Acquire frontal and lateral craniofacial images of the subject;
[0014] Step 2: Collect demographic information and obesity-related parameters of the subjects;
[0015] Step 3: Preprocess the original frontal and lateral craniofacial images collected from the subjects;
[0016] Step 4: Using the preprocessed images described above, a model is established through cross-validation;
[0017] Step 5: Occlude different regions of the preprocessed frontal and lateral craniofacial images with black rectangles;
[0018] Step 6: Determine the contribution of different regions to the model, i.e., the regional contribution.
[0019] Step 7: Weight the preprocessed frontal and lateral craniofacial images based on regional contribution.
[0020] Step 8: Perform facial feature analysis on the processed frontal and lateral craniofacial weighted images, demographic information, and obesity-related parameters.
[0021] Furthermore, this invention relies on image classification algorithms based on machine learning, including algorithms such as logistic regression, Naive Bayes, nearest neighbor, decision tree, support vector machine, and convolutional neural network. It should also be noted that in building this model, the input is a preprocessed dataset of all images. Cross-validation is used to train the model, and its performance is evaluated to obtain the optimal trained model.
[0022] This invention proposes a local contribution formula to quantify the contribution of local regions to the overall model. Multiple sets of occluded images at different locations are used as input to the model, and the output is an ROC curve. The area under the ROC curve (AUC) is then compared horizontally. The obtained AUC is substituted into the proposed formula to calculate the contribution value of each occluded region.
[0023] Furthermore, this invention employs a linear blending algorithm based on image processing to assign different weights to different regions in the frontal and lateral craniofacial images based on the calculated regional contribution values. This results in a weighted frontal and lateral craniofacial image. It should also be noted that the average value of the calculated contribution values is preset as a threshold. If the contribution value of a region exceeds the threshold, it indicates that the location has a positive impact on the model, and therefore the weight should be increased. For regions that do not exceed the threshold, it indicates that the location has a less positive impact on the model, and therefore the weight should be appropriately reduced. Through the above process, the weighting of regions is determined. After superimposing black rectangles to add weights to the image, a weighted frontal and lateral craniofacial film is obtained. A linear blending method is then applied to the film and the preprocessed image to obtain the weighted frontal and lateral craniofacial image of the subject.
[0024] Beneficial effects:
[0025] Compared to existing technologies, this invention requires fewer input test cases from subjects, and the image sampling is convenient and simple. Simultaneously, the model requires fewer input test cases, resulting in lower training costs, strong model interpretability, and excellent generalization and robustness. This model obtains a weighted frontal and lateral craniofacial image by analyzing the subject's original frontal and lateral craniofacial images, along with demographic information and obesity-related parameters, using the aforementioned model and methods. The model is then interpreted and visualized to produce this weighted image. Furthermore, combining the aforementioned craniofacial image processing techniques, the frontal and lateral craniofacial images, along with demographic information and obesity-related parameters, are input into the trained model to output the probability value of the subject having obstructive sleep apnea syndrome. This invention offers a simple method for acquiring craniofacial images, and the model test cases only require the acquisition of frontal and lateral craniofacial images, demographic information, and obesity-related parameters. Attached Figure Description
[0026] Figure 1 This paper presents a method for preprocessing subject craniofacial images based on a face recognition algorithm. This method reduces the interference of background clutter on the model in frontal and lateral craniofacial images. Simultaneously, the images are converted to grayscale to reduce color interference on the model and decrease the size of the data sample. The resulting preprocessed frontal and lateral craniofacial images are obtained.
[0027] Figure 2A Based on face recognition algorithms and image processing methods, multiple sets of frontal craniofacial occlusion images of different regions of the subjects were obtained;
[0028] Figure 2B Based on face recognition algorithms and image processing methods, multiple sets of lateral craniofacial occlusion images of different regions of the subjects were obtained;
[0029] Figure 3A As an image classification algorithm based on machine learning, frontal occlusion processing obtains contribution values of multiple occluded images. The frontal basis processing model here is the optimal frontal basis processing model obtained by training and evaluation based on all frontal preprocessed image instances through ten-fold cross operation.
[0030] Figure 3B Side occlusion processing yields contribution values for multiple sets of occluded images. The side occlusion base processing model is the optimal model obtained through training and evaluation based on all preprocessed side image instances, using a ten-fold cross-validation operation. It's important to note that multiple sets of frontal occluded images and multiple sets of side occluded images are input into their respective base processing models. The TPR and FPR values are calculated using formulas, multiple ROC curves are plotted, and the AUC value is calculated. Finally, the contribution value for each region is calculated using the formula provided in this invention.
[0031] Figure 4 A linear mixing algorithm based on image processing is used to obtain weighted images of the frontal and lateral craniofacial regions;
[0032] Figure 5 The algorithm is based on image classification in machine learning. It takes the subject's frontal craniofacial weighted image, lateral craniofacial weighted image, and the subject's demographic information and obesity-related parameters as input to the model to obtain the probability that the subject has obstructive sleep apnea syndrome.
[0033] Figure 6 This is a method for processing and analyzing craniofacial images from both the front and side views based on artificial intelligence.
[0034] Figure 7 ROC curves for multiple validation sets of frontal craniofacial occluded images. In a single test case, multiple ROC curves were generated from frontal craniofacial occluded images of multiple test subjects in the validation set, using the same frontal craniofacial training set and a frontal basis processing model trained through cross-validation, to explore the differences in different regions.
[0035] Figure 8 ROC curves for multiple validation sets of lateral craniofacial occlusion images. In a single test case, multiple ROC curves were generated from the same lateral base processing model trained on the same lateral craniofacial training set using multiple sets of lateral craniofacial occlusion images of test subjects through cross-validation, to explore the differences in different regions.
[0036] Figure 9 This image shows an example of frontal occlusion and its contribution to the occlusion area. In one test case, only one inventor, Li Yingjie, is used as a reference, and a partial frontal occlusion image of this test subject is shown. The AUC values in the image are based on... Figure 7 The contribution values in the figure are calculated using the ROC curve provided in the invention.
[0037] Figure 10 This image shows an example of lateral occlusion and its contribution to the occlusion area. In one test case, only one inventor, Li Yingjie, is used as a reference, and the image shows a partial lateral occlusion of this test subject. The AUC values in the image are based on... Figure 7 The contribution values in the figure are calculated using the ROC curve provided in the invention.
[0038] Figure 11 This is a comparison chart of weighted and unweighted craniofacial images. In one test case, the weighted craniofacial image test set, subject demographic information, and obesity parameters, and the unweighted craniofacial image test set, subject demographic information, and obesity parameters are input into the same base model, and the ROC curves are plotted.
[0039] Figure 12 This is a flowchart of a method for processing and analyzing frontal and lateral craniofacial images based on artificial intelligence according to the present invention. Detailed Implementation
[0040] To clearly illustrate specific implementation examples and methods, the models and methods involved in this invention will be described in detail below. Furthermore, examples will be provided in the following description to better illustrate the performance of the models and methods of this invention.
[0041] With the development of artificial intelligence, its corresponding subfields have also gradually developed. This invention combines the image processing subfield of artificial intelligence. The frontal and lateral craniofacial images are divided into regions. To explore the different effects of different regions on the overall model, this invention adopts a method of occluding craniofacial regions to investigate their contribution to the model. Then, different weights are assigned to the different contribution values of the obtained regions. In short, the weight of regions with contribution values exceeding a threshold is increased, while the weight of regions with contribution values below the threshold is relatively reduced. Based on this process, a superimposed image and the original image are linearly mixed to obtain a weighted craniofacial image, thereby increasing the weight of prominent contributing regions in the craniofacial image. Finally, the processed weighted frontal and lateral craniofacial images of the subjects, along with the subjects' demographic information and obesity-related parameters, are input into a machine learning model trained through cross-validation, thereby significantly improving the model's accuracy and outputting the probability value of whether the subject has obstructive sleep apnea syndrome.
[0042] Specific model operation examples and explanations will be described in detail below, and will be interpreted with actual operation procedures and model operation procedures.
[0043] See Figure 12 The technical solution of the present invention is: a method for processing and analyzing frontal and lateral craniofacial images based on artificial intelligence, comprising the following steps:
[0044] Step 1: Acquire frontal and lateral craniofacial images of the subject;
[0045] Step 2: Collect demographic information and obesity-related parameters of the subjects;
[0046] Step 3: Preprocess the original frontal and lateral craniofacial images collected from the subjects;
[0047] Step 4: Using the preprocessed images described above, a model is established through cross-validation;
[0048] Step 5: Occlude different regions of the preprocessed frontal and lateral craniofacial images with black rectangles;
[0049] Step 6: Determine the contribution of different regions to the model, i.e., the regional contribution.
[0050] Step 7: Weight the preprocessed frontal and lateral craniofacial images based on regional contribution.
[0051] Step 8: Perform facial feature analysis on the processed frontal and lateral craniofacial weighted images, demographic information, and obesity-related parameters.
[0052] Perform user facial feature recognition.
[0053] For step 1, see Figure 6 As shown, X601.
[0054] like Figure 1 As shown, frontal and lateral craniofacial images of the subject were collected. When collecting frontal craniofacial images of the subject, the background of the photographer should be white, the lighting should be sufficient, and the subject should have their neck and forehead exposed, and should not wear any accessories.
[0055] Step 2, see Figure 6 As shown, X602 collects demographic information and obesity-related parameters of the subjects;
[0056] Participants are required to input the following demographic information and obesity-related parameters. Demographic information includes: age, sex, height, and weight. Obesity-related parameters include: BMI, neck circumference, and waist circumference. Height is in meters, weight is in kilograms, neck circumference is in centimeters, and waist circumference is in centimeters.
[0057] Example of execution: age, gender, height, weight, neck circumference, waist circumference.
[0058] According to the formula, The calculated BMI should be used as the standard.
[0059] Step 3: Preprocess the original frontal and lateral craniofacial images acquired from the subjects, see [link to relevant documentation]. Figure 1 As shown.
[0060] To reduce sample noise in the frontal and lateral craniofacial images of subjects, X101 uses a face recognition algorithm to identify the head images from the front and sides. The original frontal and lateral craniofacial images are cropped, retaining only the complete head and neck regions. This reduces the interference of background and clutter on the model's feature analysis process.
[0061] To reduce the impact of lighting and brightness in the frontal and lateral craniofacial images of subjects on the model's feature analysis, X102 employs a grayscale conversion algorithm from image processing to convert the subjects' frontal and lateral craniofacial images into grayscale images. This improves the computational speed of the model algorithm and reduces the influence of color on image analysis.
[0062] After the above steps X101 and X102, the preprocessed images of the subject's frontal and lateral craniofacial features are obtained.
[0063] Step 4, Model building and training process;
[0064] This section describes the model construction and training process;
[0065] This invention is based on classification algorithms in machine learning, including algorithms such as logistic regression, Naive Bayes, nearest neighbor, decision tree, support vector machine, and convolutional neural network.
[0066] A model is built using all pre-processed frontal instance images through cross-validation, and its performance is evaluated. The corresponding parameters are then adjusted to obtain the optimal trained model, namely the frontal basis processing model.
[0067] The model is constructed using all preprocessed lateral base images through cross-validation, and its performance is evaluated. The corresponding parameters are then adjusted to obtain the optimal trained model, namely the lateral base processing model.
[0068] Step 5: To explore the contribution of different regions to the model, different regions of the preprocessed frontal and lateral craniofacial images were occluded using black (RGB(0,0,0)) rectangles. See Figure 2.
[0069] To investigate the contribution of different regions of the frontal craniofacial image to the model, the X201 model divides the frontal craniofacial region into the following areas: forehead, eyes, nose, cheeks, and jaw. In the craniofacial image representation, the acquired ear portion will be occluded in the lateral craniofacial image. This regional division ensures that the main contributing parts of the facial region are independently and non-interferingly separated. A face recognition algorithm is used to identify these regions in the frontal craniofacial image. The identified regions are then occluded with black rectangles. The RGB channel of black is (0,0,0), and the size of the rectangles is determined based on the different regions of different subjects. After face recognition, the following frontal craniofacial images are obtained: forehead occlusion image, eye occlusion image, nose occlusion image, cheek occlusion image, and jaw occlusion image.
[0070] To investigate the contribution of different regions of the lateral craniofacial image to the model, the lateral craniofacial region was divided into the following areas: preauricular region, ear region, neck region, temporal region, and occipital region. This division ensures that the main contributing parts of the facial region are independently and without interference. A face recognition algorithm was used to identify these regions in the lateral craniofacial image. The identified regions were then occluded with black rectangles. The RGB channel of black is (0,0,0), and the size of the rectangles was determined based on the different regions for different subjects. After face recognition, the following images were obtained: occluded images of the preauricular region, ear region, neck region, and temporal and occipital regions of the lateral craniofacial image.
[0071] Step 6: Explore the contribution of different regions to the model, see [link / reference]. Figure 3A , 3B As shown;
[0072] As shown in step X301 of the figure, the multiple sets of frontal occlusion images obtained in step 5 are used as the input dataset for cross-validation to establish a frontal basis processing model, thereby exploring the contribution of different regions to the frontal model. Based on the contribution formula proposed in this invention, the contribution value of each region is obtained.
[0073] As shown in step X302 of the figure, the complete frontal craniofacial image is input into the frontal model trained in step X301. After cross-validation, the average of ten cross-validations is taken. and Plot the RoC curve, where TP, called the true positive rate, indicates the number of samples that were actually positive but were predicted as positive. FP, called the false positive rate, indicates the number of samples that were actually negative but were predicted as positive. TN, called the true negative rate, indicates the number of samples that were actually negative but were predicted as negative. FN, called the false negative rate, indicates the number of samples that were actually positive but were predicted as negative.
[0074] According to the formula Where M is the number of positive samples and N is the number of negative samples. And rank... i This represents the sequence number of the i-th sample.
[0075] The AUC value of the complete frontal craniofacial image on the frontal model was calculated.
[0076] The frontal craniofacial image with forehead occlusion is input into a pre-trained machine learning model. and Plot the RoC curve, where TP, called the true positive rate, indicates the number of samples that were actually positive but were predicted as positive. FP, called the false positive rate, indicates the number of samples that were actually negative but were predicted as positive. TN, called the true negative rate, indicates the number of samples that were actually negative but were predicted as negative. FN, called the false negative rate, indicates the number of samples that were actually positive but were predicted as negative.
[0077] According to the formula Where M is the number of positive samples and N is the number of negative samples. And rank... i This represents the sequence number of the i-th sample.
[0078] The AUC value of the frontal cranial face occlusion image on the frontal model was calculated.
[0079] For the frontal craniofacial image with eye occlusion, the same procedure applies. The AUC value of the frontal craniofacial image with eye occlusion on the frontal basis processing model is obtained.
[0080] For the frontal craniofacial cheek occlusion image, the same procedure applies. The AUC value of the frontal craniofacial cheek occlusion image on the frontal basis processing model is obtained.
[0081] For the frontal craniofacial nose-occluded image, the same procedure applies. The AUC value of the frontal craniofacial nose-occluded image on the frontal basis processing model is obtained.
[0082] For the frontal craniofacial mandibular occlusion image, the same procedure applies. The AUC value of the frontal craniofacial mandibular occlusion image on the frontal basis processing model is obtained.
[0083] As shown in step X303 of the figure, the following formula is defined. The constant α is arbitrarily determined by the inventor to increase the contribution of each region. The AUC values of the frontal forehead occlusion image, frontal eye occlusion image, frontal nose occlusion image, frontal cheek occlusion image, and frontal jaw occlusion image obtained after step X302 are substituted into the above formula. The contribution value of the corresponding region to the overall model can then be obtained.
[0084] As shown in step X304 of the figure, the multiple sets of side occlusion images obtained in step 5 are used as the input dataset for cross-validation to establish a frontal basis processing model, thereby exploring the contribution of different regions to the model. Based on the contribution formula proposed in this invention, the contribution value of each region is obtained.
[0085] As shown in step X305 of the figure, the complete lateral craniofacial image is input into the base processing model trained in step 5. After cross-validation, the average of ten cross-validations is taken. and Plot the RoC curve, where TP, called the true positive rate, indicates the number of samples that were actually positive but were predicted as positive. FP, called the false positive rate, indicates the number of samples that were actually negative but were predicted as positive. TN, called the true negative rate, indicates the number of samples that were actually negative but were predicted as negative. FN, called the false negative rate, indicates the number of samples that were actually positive but were predicted as negative.
[0086] According to the formula Where M is the number of positive samples and N is the number of negative samples. And rank... i This represents the sequence number of the i-th sample.
[0087] The AUC value of the complete lateral craniofacial image on the lateral model was calculated.
[0088] The occluded image of the lateral craniofacial region in front of the ear is input into a pre-trained machine learning model. and Plot the RoC curve, where TP, called the true positive rate, indicates the number of samples that were actually positive but were predicted as positive. FP, called the false positive rate, indicates the number of samples that were actually negative but were predicted as positive. TN, called the true negative rate, indicates the number of samples that were actually negative but were predicted as negative. FN, called the false negative rate, indicates the number of samples that were actually positive but were predicted as negative.
[0089] According to the formula Where M is the number of positive samples and N is the number of negative samples. And rank... i This represents the sequence number of the i-th sample.
[0090] The AUC value of the lateral craniofacial image with preauricular occlusion was calculated on the lateral model.
[0091] For the lateral craniofacial ear-occluded image, the same procedure applies. The AUC value of the lateral craniofacial ear-occluded image on the lateral basis processing model is obtained.
[0092] For the lateral craniofacial-neck occluded image, the same procedure applies. The AUC value of the lateral craniofacial-neck occluded image on the lateral basis processing model is obtained.
[0093] For the occluded images of the temporal and occipital regions of the craniofacial region, the same procedure applies. The AUC values of the occluded images of the temporal and occipital regions of the craniofacial region on the lateral basis processing model are obtained.
[0094] As shown in step X306 of the figure, the following formula is defined. The constant α is arbitrarily determined by the inventor to increase the contribution of each region. The AUC values of the lateral preauricular occlusion image, lateral ear occlusion image, lateral neck region occlusion image, and lateral temporal and occipital region occlusion images obtained after step X305 are substituted into the above formula to obtain the contribution value of the corresponding region to the overall lateral model.
[0095] Step 7: Weight the preprocessed frontal and lateral craniofacial images based on their region contribution values. See [link / reference] Figure 4 As shown.
[0096] For X401, take the contribution values of each frontal region corresponding to X303 and calculate the average frontal contribution. For X306, take the contribution values of each side region and calculate the average side contribution.
[0097] X402 calculates the average value and sets it as a threshold. If a region exceeds the threshold, its weight is increased. If a region does not exceed the threshold, its weight is decreased. A black rectangle is added to obscure the region. Note that the black RGB channels are (0,0,0) at this point. This yields a frontal and lateral craniofacial weighted image.
[0098] X403, based on linear blending technology in image processing, linearly blends the pre-processed frontal craniofacial weighted image and the aforementioned frontal craniofacial preprocessed image to obtain a frontal craniofacial weighted image. The linear blending technology... g(x) is the digital matrix converted from the craniofacial weighted image, f0(x) is the digital matrix converted from the craniofacial weighted film, and f1(x) is the digital matrix converted from the craniofacial preprocessed image. The weights are those used in the linear blending process. Similarly, a linear blend is performed on the lateral craniofacial weighted image and the lateral craniofacial preprocessed image to obtain the lateral craniofacial weighted image.
[0099] Step 8: Based on the processed frontal and lateral craniofacial weighted images, and the subject's demographic information and obesity-related parameters, use this data as the dataset for model building. Cross-validation is used as the basis for building and adjusting the model to obtain the probability that the subject may have obstructive sleep apnea-hypopnea syndrome. See also... Figure 5 As shown.
[0100] X501, combined with the aforementioned frontal craniofacial weighted images, lateral craniofacial weighted images, and the subject's demographic information and obesity-related parameters.
[0101] X502, this invention is based on classification algorithms in machine learning, including algorithms such as logistic regression, Naive Bayes, nearest neighbor, decision tree, support vector machine, and convolutional neural network.
[0102] Using the data described in step X501 as input, a model is constructed through cross-validation, and the performance of the model is evaluated. The corresponding parameters are then adjusted to obtain the optimal model obtained through training, i.e., the base model.
[0103] X503 outputs the probability value that the test subject has obstructive sleep apnea syndrome.
[0104] Figure 6 The above eight process steps are integrated with the specific implementation plans of the five models described above.
[0105] When a new test subject is input, the sampled frontal and lateral craniofacial images undergo image preprocessing (Model 1) steps X101 and X102. Then, the images are weighted (Model 4) step X403. Here, the areas enhanced in the frontal image are the eyes, nose, and jaw. The areas weighted in the lateral image are the ears, preauricular region, and neck. The weighted image is then input into the image classification model trained by X502, which outputs the probability value of the subject having severe obstructive sleep apnea syndrome.
[0106] Training case examples
[0107] This section will illustrate a test example.
[0108] The model building, validation, and visualization processes will be illustrated with a series of examples. Furthermore, the processing flow for new tester models will be explained when this invention is applied.
[0109] First, we recruited data from 520 participants. This data included demographic information and obesity-related parameters, frontal and lateral craniofacial images, and a label indicating whether the participants had severe obstructive sleep apnea syndrome. This label was obtained from polysomnography throughout the night.
[0110] Of the 520 patients, 273 had non-severe OSA and 247 had severe OSA. The ratio of non-severe OSA to severe OSA was 1.10:1.00.
[0111] The demographic information and obesity-related parameters included the subject's age, sex, height, weight, BMI, neck circumference, and waist circumference. Both frontal and lateral craniofacial images fully captured the subject's face and neck against a pure white background, and the subject was not wearing any jewelry.
[0112] Step 1, as follows Figure 1 As shown, the 520 experimental cases were uniformly processed... Figure 1 The processing flow is shown below.
[0113] After performing face recognition processing on the image in step X101, only images containing the complete head and neck are cropped to reduce the interference of background and clutter on the model feature analysis.
[0114] Step X102 converts the image to grayscale to reduce the impact of color on the model and compress the image size. After processing, the result is 520 sets of images, each set containing preprocessed images of the subject's frontal and lateral craniofacial features.
[0115] Step 2: Using the 520 pre-processed frontal images, a frontal basis processing model is obtained through 10-fold cross-training. The training process is based on machine learning image classification algorithms, including logistic regression, Naive Bayes, nearest neighbor, decision tree, support vector machine, and convolutional neural network algorithms. The training set is input into any of the above algorithms, and the optimal model is obtained through training. The input of the frontal basis processing model is the frontal craniofacial image of the subject, and its output is the probability value of the subject having OSA (Orthopaedic Syndrome).
[0116] Using 520 preprocessed lateral images, a frontal basis processing model was obtained through ten-fold cross-training. The training process was based on machine learning image classification algorithms, including logistic regression, Naive Bayes, nearest neighbor, decision tree, support vector machine, and convolutional neural network algorithms. The training set was input into any of the above algorithms, and the optimal algorithm was obtained to produce the lateral basis processing model. The input of the lateral basis processing model is the lateral craniofacial image of the subject, and its output is the probability value of the subject having OSA.
[0117] Step 3, as follows Figure 2A , 2B As shown, 520 groups were processed Figure 2A , 2B The processing flow shown yielded 520 sets of frontal and lateral craniofacial occlusion images for the validation group.
[0118] Step 4, as follows Figure 3A , 3B As shown.
[0119] Step X301: Input 520 sets of multi-region frontal craniofacial occlusion images into the frontal base processing model.
[0120] Step X302, according to the formula and The TPR and FPR values for each occluded area are calculated.
[0121] According to the formula:
[0122]
[0123] Calculate the AUC value for each occluded region.
[0124] Based on the TPR, FPR, and AUC values calculated in the above steps, an ROC curve is plotted. In this example, the following image is plotted based on the frontal craniofacial occlusion image. The frontal occlusion group is as follows. Figure 7 As shown.
[0125] Step 5, as follows Figure 7 As shown, for Figure 7 Adding an auxiliary line (line 1) allows for a horizontal comparison of the contributions of different regions. The images clearly show a sharp decrease in model performance when the jaw, eyes, and nose are occluded, directly illustrating the significant contributions of these regions. However, the model's performance remains excellent when the cheeks and forehead are occluded, indicating that these areas have a relatively low impact on the model. It's important to note that this invention proposes a region contribution formula to quantify the contribution of each region.
[0126] Step 6, as shown in Figure 3, step X303, it is important to note that this invention proposes a formula for the regional contribution value in order to quantify the contribution of different regions to the model.
[0127] Define the following formula The constant 'a' is an arbitrary constant added to enhance the contrast of regional contributions. In this example, the constant value is set to 10. The contribution value of each region is obtained by applying the corresponding AUC value to the formula above, as follows: Figure 9 As shown. It should be noted that, in order to intuitively illustrate the occlusion method and the occlusion area, only multiple sets of frontal occlusion images by inventor Li Yingjie are listed here.
[0128] Step 7, as shown in Figure 3.
[0129] Step X304: Input the 520 sets of validation group lateral craniofacial occlusion images into the lateral base processing model.
[0130] Step X305, according to the formula and The TPR and FPR values for each occluded area are calculated.
[0131] According to the formula:
[0132]
[0133] Calculate the AUC value for each occluded region.
[0134] Based on the TPR, FPR, and AUC values calculated in the above steps, an ROC curve is plotted. In this example, the following image is plotted based on the lateral craniofacial occlusion image. The lateral occlusion group is as follows. Figure 8 As shown.
[0135] Step 8, as shown in Figure 3, step X306, it is important to note that this invention proposes a formula for the regional contribution value in order to quantify the contribution of different regions to the model.
[0136] Define the following formula The constant 'a' is an arbitrary constant added to enhance the contrast of regional contributions. In this example, the constant value is set to 10. The contribution value of each region is obtained by applying the corresponding AUC value to the formula above, as follows: Figure 10 As shown. It should be noted that, in order to intuitively illustrate the occlusion method and the occlusion area, only multiple sets of side occlusion images by inventor Li Yingjie are listed here.
[0137] Step 9, as follows Figure 4 As shown.
[0138] Step X401: Input the calculated contribution value into Model 4.
[0139] For the frontal craniofacial image, the contribution value of each occluded position was calculated to be an average of 33.2%.
[0140] For the contribution values of each occlusion location in the lateral craniofacial image, the average contribution value of the lateral side is calculated to be 40.8%. Step X402 sets the average contribution value of the frontal occlusion as a threshold. It can be seen that in the frontal craniofacial image, the areas exceeding the threshold are the eyes, nose, and jaw. Therefore, the weight value of these areas needs to be increased. It can be seen that in the frontal craniofacial image, the areas not exceeding the threshold are the cheeks and forehead. Therefore, the weight value of the cheeks and forehead areas needs to be decreased.
[0141] The average contribution value of lateral occlusion is set as a threshold. It can be seen that in lateral craniofacial images, the areas exceeding the threshold are the ear, neck, and preauricular region. Therefore, weight values for these areas need to be added. It can be seen that in lateral craniofacial images, the areas not exceeding the threshold are the temporal and occipital regions. Therefore, the weight values for the temporal and occipital regions need to be reduced.
[0142] Step 10, as follows Figure 4 As shown in X403. After step 4, the processing area requiring weight reduction is obtained. A black rectangle is added to mask this area. Note that the RGB channels of black are (0,0,0) at this point. This yields the frontal and lateral craniofacial weighted images.
[0143] Based on the linear blending technique of image processing, the above-processed frontal craniofacial weighted film and the frontal craniofacial preprocessed image are linearly blended to obtain the frontal craniofacial weighted image.
[0144] Based on linear hybridization techniques, where g(x) is the digital matrix converted from the craniofacial weighted image, f0(x) is the digital matrix converted from the craniofacial weighted film, and f1(x) is the digital matrix converted from the craniofacial preprocessed image. The weights are those used in the linear blending process. Similarly, a linear blend is performed on the lateral craniofacial weighted image and the lateral craniofacial preprocessed image to obtain the lateral craniofacial weighted image.
[0145] Step 11, as follows Figure 5 As shown in X501. The 520 sets of frontal and lateral craniofacial weighted images obtained in step 13, along with the demographic information and obesity-related parameters of the 520 groups of subjects, were used as input to construct a model through cross-validation. The performance of the model was evaluated, and the corresponding parameters were adjusted to obtain the optimal model obtained after training, i.e., the base model.
[0146] The output of the basic model is the probability that the subject has obstructive sleep apnea syndrome.
[0147] Step 12: To compare the performance of weighted and unweighted images within the same model, 520 preprocessed images (as a control group), demographic information, and obesity parameters were used as a blank control group, and 520 weighted images (as a weighted experimental group) obtained through the aforementioned weighting process were used as a weighted experimental group. These were input into the base model, cross-validated, and ROC plots were generated for both groups based on the TPR and FPR calculation formulas. Figure 11 As shown.
[0148] Step 13: Calculate the area under the ROC curve for both groups using the AUC calculation formula. The results show that the AUC value of the model is 0.81 when the blank control group is input, and 0.85 when the weighted experimental group is input.
[0149] In summary, when image processing is weighted as described above, the model's performance improves by 4 points, resulting in better performance. It also demonstrates stronger interpretability and superior robustness.
[0150] At the same time, it should be noted that when dealing with new subject images, the frontal over-threshold regions obtained in the usage examples of this invention, namely the eyes, nose, and jaw, are weighted in the frontal craniofacial region of the new subject to obtain a side-weighted image.
[0151] It should be noted that, for new subject images, the side over-threshold region obtained in the example of this invention, namely the preauricular region, neck, and ear, is weighted on this region of the new subject's side craniofacial region to obtain a side-weighted image.
[0152] The subject's demographic information and obesity-related indices, front-weighted images, and side-weighted images. A set of test cases for an actual test for a subject are input into the base model obtained in step 5 to output the probability that the subject has obstructive sleep apnea syndrome.
[0153] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.
Claims
1. A method for processing and analyzing frontal and lateral craniofacial images based on artificial intelligence, characterized in that, Includes the following steps: Step 1: Acquire frontal and lateral craniofacial images of the subject; Step 2: Collect demographic information and obesity-related parameters of the subjects; Step 3: Preprocess the original frontal and lateral craniofacial images collected from the subjects; Step 4: Using the preprocessed images described above, establish the frontal basis processing model and the lateral basis processing model through cross-validation; The model building and training process in step 4, wherein... The model is built using all pre-processed frontal instance images through cross-validation, and its performance is evaluated. The corresponding parameters are then adjusted to obtain the optimal model trained from these images, which is the frontal basis processing model. The model is built using all preprocessed lateral base images through cross-validation, and its performance is evaluated. The corresponding parameters are then adjusted to obtain the optimal model trained from these images, which is the lateral base processing model. Step 5: Occlude different regions of the preprocessed frontal and lateral craniofacial images with black rectangles; Step 6: Determine the contribution of different regions to the model, i.e., the regional contribution. Step 7: Weight the preprocessed frontal and lateral craniofacial images based on regional contribution. Step 8: Based on the processed frontal and lateral craniofacial weighted images, demographic information, and obesity-related parameters, perform facial feature analysis on the user and output the probability value of the test subject having obstructive sleep apnea syndrome. Step 5 includes the following steps: Step 5.1: Divide the frontal region into the following areas: forehead, eyes, nose, cheeks, and mouth. After dividing the region into these areas, ensure that the main contributing parts of the facial region are separated independently without interference. Based on the face recognition method, identify the above regions in the frontal craniofacial image. Occlude the identified regions with black rectangles to obtain multiple sets of forehead occlusion images, eye occlusion images, nose occlusion images, mouth occlusion images, and cheek occlusion images. Step 5.2: Divide the lateral region as follows: preauricular region, ear, neck, temporal region, and occipital region. Based on face recognition technology, identify the above regions in the lateral craniofacial image of the preauricular region, ear, neck, temporal region, and occipital region. Occlude the identified regions with black rectangles to obtain multiple sets of preauricular region occlusion images, ear occlusion images, neck occlusion images, and temporal and occipital region occlusion images. Step 6 includes the following steps: Step 6.1: Perform visualization analysis on the model. Use multi-region frontal craniofacial occlusion images and input them into the frontal basis processing model. Use multi-region lateral craniofacial occlusion images and input them into the lateral basis processing model. Calculate the contribution of different regions to the model. Step 6.2: Calculate the average contribution of the occluded areas on the front and side of the model respectively, set the average value as the threshold, and increase the weight value of the front and side image areas whose contribution to the model exceeds the threshold in the weighted image part. Step 7 includes the following steps: Step 7.1: Select the regions in each region whose contribution value does not exceed the threshold; Step 7.2: Cover the area with a black rectangle to reduce the weight level of the area, thereby obtaining the frontal image weighted negative and the side image weighted negative; Step 7.3: Based on the linear blending method of image processing, perform linear blending on the above-processed frontal and side-view weighted negative and the frontal and side-view preprocessed image to obtain the frontal and side-view weighted image. Based on the linear blending technique, wherein: , The digital matrix converted from craniofacial weighted images. A digital matrix converted from craniofacial weighted radiographs. The digital matrix converted from the craniofacial preprocessed image. The added weighting coefficients; Step 8 includes the following steps: Step 8.1, according to the formula The subject's BMI value was calculated; it also includes... Using the aforementioned frontal and lateral craniofacial weighted images, the subject's demographic information, obesity-related parameters, and BMI as input, a model is constructed through cross-validation. The model's performance is then evaluated, and corresponding parameters are adjusted to obtain the optimal model obtained through training, i.e., the base model. Step 8.2: Combine the weighted frontal and lateral images of the test subject with their demographic information, obesity-related parameters, and BMI value. The user's facial features are analyzed by inputting them into the base model trained through cross-validation, and the probability value of the test subject having obstructive sleep apnea syndrome is output.
2. The method for processing and analyzing frontal and lateral craniofacial images based on artificial intelligence according to claim 1, characterized in that, Step 1 includes the following steps: Step 1.1: Collect frontal and lateral craniofacial images of the subject according to the preset conditions.
3. The method for processing and analyzing frontal and lateral craniofacial images based on artificial intelligence according to claim 1, characterized in that, Step 2 includes the following steps: Step 2.1: Collect demographic information and obesity-related parameters of the subjects; the data format should be uniformly formatted as: age, gender, height, weight, neck circumference, and waist circumference.
4. The method for processing and analyzing frontal and lateral craniofacial images based on artificial intelligence according to claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: Extract the original frontal and side craniofacial images, retaining only the entire head and neck, and unify the image size; Step 3.2: Apply grayscale conversion from the field of image processing to convert the frontal and lateral craniofacial images of the subject into grayscale images.