Determining severity of gingivitis

Abstract

The subject matter of the present disclosure relates to a computer-implemented method of determining the severity of gingivitis from an image showing the gingiva. A computer-implemented method includes receiving an image showing a gingiva; determining, for each unit area of the gingiva, a color from among a plurality of unit areas of the gingiva; clustering the plurality of unit areas according to the correspondingly determined colors by using a clustering algorithm; calculating the distance between the color of each cluster and the reference color; calculating a total distance value of the gingiva based on the distance of each cluster from the reference color and the respective size of each cluster; and determining the severity of gingivitis based on the total distance.

Description

Technical Field

[0001] The subject matter of the present disclosure relates to computer-implemented methods of determining the severity of gingivitis from an image showing gums, computer-implemented methods of generating training examples of images of gums, computer-implemented methods of training a machine learning model to segment an image showing gums in order to extract the gums from the image, and transitory or non-transitory computer-readable media. Background Art

[0002] Gingivitis is a common early form of gum disease that affects the gums, which are the area between the gum line and the crowns of the teeth. Some common symptoms involve inflammation, swelling, and irritation of the gums. Generally, there are two types of gingivitis: plaque-induced gingivitis and non-plaque-induced gum damage. Plaque is a sticky bacterial film that accumulates on the teeth if proper oral hygiene is not maintained regularly. In plaque-induced gingivitis, plaque forms when starches and sugars in food interact with bacteria present in the mouth. Over time, plaque hardens to form tartar, which accumulates bacteria and acts as a protective barrier for bacteria, causing irritation along the gum line. Ultimately, untreated irritation can lead to gum inflammation, and this symptom is one of the main causes of gingivitis. Non-plaque-induced gum damage can be caused by fungal infections and allergic reactions to certain foreign objects (such as dentures).

[0003] The main cause of gingivitis on the gums is poor oral hygiene. A healthy lifestyle, including brushing twice a day and regular dental checkups, can prevent and, in most cases, reverse the disease. Healthy gums are typically light pink in color and fit tightly against the teeth. However, when a person's gums become infected with gingivitis, they can turn a dark or crimson red. In addition, the gums may become swollen, tender, and bleed when probed or brushed. Gingivitis is also characterized by swelling, ulcers, and thickening of the gum tissue. If left untreated, these symptoms can lead to a more serious gum disease called periodontitis, and ultimately, tooth loss. Research on gum disease has shown that the bacteria that cause periodontitis can enter the bloodstream through the gum tissue, where they can affect the lungs, heart, and other vital organs.

[0004] Gingivitis is diagnosed by examining the gums for signs of inflammation. The dentist may also review the patient's medical history, including whether the patient has diabetes or a smoking addiction. and J. Silness, Periodontitis in pregnancy I. Prevalence and severity. Acta odontologica scandinavica, 21(6), 533-551 (1963), describe a gingival index score that indicates the stage or severity of gingivitis. The score measures gingival health on a scale of 0-3, with 0 indicating healthy gums and 3 indicating severe gingivitis characterized by bleeding and edema.

[0005] DMAlalharith et al., A Deep Learning-Based Approach to Detect Early Signs of Gingivitis in Orthodontic Patients Using Faster Region-Based Convolutional Neural Networks, International Journal of Environmental Research and Public Health, 17(22), 8447(2020), proposed an AI-based approach for detecting inflammation of the gums. Their model consists of two Faster Region-Based Convolutional Neural Networks (R-CNN) and a ResNet-50 backbone. The first model detects teeth and the second model detects inflammation of the gums. A gums dataset containing 134 images was collected by the College of Dentistry at Imam Abdulrahman bin Faisal University. In each image, the distal gingiva, mesial gingiva, and mesial gingiva of the maxillary central incisor were annotated as follows. Figure 1 (a) to (c). More specifically, Figure 1 Predicted inflamed and non-inflamed bounding boxes in the distal (marked 1), middle (marked 2), and mesial (marked 3) regions. Figure 1 In (a), inflamed and non-inflamed areas are correctly detected. Figure 1 In (b), the non-inflamed distal area of ​​tooth p is mistakenly detected as inflamed. Figure 1 In (c), the distal region of tooth r is not detected, and the non-inflamed middle region of tooth s is mistakenly detected as inflamed.

[0006] This study focused on detecting gingivitis in specific areas of the gums surrounding the maxillary central incisor. A Faster R-CNN model was trained to predict whether the distal, mesial, and mesial regions were inflamed. Alalharith et al.'s work has two major shortcomings: a) The AI ​​model predicted gingivitis in very specific areas of the upper gums. The model was unable to detect inflammation in other areas of the upper and lower gums, and b) the AI ​​model could not predict the stage of gingivitis or the severity of the inflammation, which could help users take appropriate treatment options.

[0007] A. Rana et al., “Automatic Segmentation of Gum Disease from Oral Images,” IEEE Healthcare Innovations and Point of Care Technologies, pp. 144-147 (2017), developed an autoencoder with a convolutional neural network (CNN) architecture to predict inflamed areas in the gums. The oral dataset was collected from dental patients aged 18-90 years in Nashik, India. In each image, dental plaque is indicated by shades of yellow and orange, and gingivitis is indicated by shades of magenta and red. A dental expert annotated the ground truth masks of the inflamed areas on each image. The dataset consists of 400 images and the corresponding mask annotations of the inflamed areas of the gums. An autoencoder-based CNN model was trained to learn the mapping between the input images and the ground truth gingivitis segmentation. In Figure 2 In , the trained classifier predicts the pixel-wise segmentation of the gingival region with gingival inflammation. More specifically, in Figure 2 In (a), each row shows images showing gum samples from a specific individual. Figure 2 (b) shows the ground truth labels of gum inflammation given by dental experts. Figure 2 (c) shows the inflamed area predicted by the AI ​​model. Figure 2 (d) shows the prediction errors indicated by TP, FP and FN.

[0008] One of the main drawbacks of the above approach is that the oral dataset includes images with highly localized areas in the gums, which makes it difficult to train an AI model to predict inflamed segments across the entire area of ​​both the upper and lower gums. Another drawback is that the model cannot predict the degree of gum inflammation.

[0009] It is an object of the presently disclosed subject matter to improve upon the prior art. Summary of the Invention

[0010] According to a first aspect of an embodiment of the present invention, there is provided a computer-implemented method for determining the severity of gingivitis from an image showing gums, the computer-implemented method comprising: receiving an image showing gums; determining, for each unit area of ​​the gums, a color from a plurality of unit areas of the gums; clustering the plurality of unit areas according to the corresponding determined colors using a clustering algorithm; calculating the distance between the color of each cluster and a reference color; calculating a total distance value of the gums based on the distance between each cluster and the reference color and the corresponding size of each cluster; and determining the severity of gingivitis based on the total distance.

[0011] Advantageously, by using the total distance to determine the severity of gingivitis, it is possible to determine the severity of gingivitis in a continuous manner. In other words, the difference in severity between images can be assessed at a more granular level. Furthermore, metadata associated with identifying the user is not required, as embodiments of the present invention operate on a single image showing the gums.

[0012] In one embodiment, the computer-implemented method further comprises sending the determined severity of the gingivitis to a display for displaying the severity of the gingivitis to a user.

[0013] In one embodiment, the clustering algorithm is a k-means clustering algorithm.

[0014] In one embodiment, calculating the distance between the color of each cluster and the reference color includes: assigning a color space value to the color of the cluster; and determining the distance between the color space value of the color of the cluster and the color space value of the reference color, wherein the color space value is optionally a hexadecimal number. Using a color space is a particularly effective way to label colors for use by various machine learning algorithms.

[0015] In one embodiment, calculating the distance between the color of each cluster and the reference color includes calculating the Euclidean distance between the color of each cluster and the reference color. Compared with other tried methods, the Euclidean distance has been shown to be the most useful method for comparing different colors.

[0016] In one embodiment, the total distance value is a weighted average distance WAD, which is defined as follows: WAD = ∑ i∈S f i ·d i , where f i is the fraction of the area of ​​the ith cluster compared to the total area of ​​multiple unit areas of the gingiva, d i is the distance between the color of the ith cluster and the reference color, S is a set of indices of color clusters present in the gingival image, and ∑ i∈S f i = 1. Using WAD is computationally more efficient than SWAD (see below) because fewer parameters are required.

[0017] In one embodiment, the computer-implemented method further comprises generating a gingival score for each cluster based on distance, wherein the total distance value is a selected weighted average distance SWAD defined by the following formula: where f i is the fraction of the area of ​​the ith cluster to the total area of ​​the gingiva, d i is the distance between the color of the ith cluster and the reference color, is the weight, where c i is the gingival score of the th cluster, S is a set of indices of color groups present in the gingival image, and ∑i∈S f i =1.

[0018] Localized gingivitis can be diagnosed more accurately using SWAD because each cluster is additionally weighted according to its own gingival score.

[0019] In one embodiment, each of the plurality of unit areas is a pixel, and determining the color comprises determining the value of the pixel using a red, green, blue color model or a luminance, red, green, blue, and yellow color model. The term luminance may be replaced with lightness to conform to the terminology of the CIELAB color space. Pixel-level unit area makes color determination easier and more computationally efficient.

[0020] In one embodiment, the computer-implemented method further comprises, before determining the color of each unit area, segmenting the image to extract only the gums using a machine learning model.

[0021] Extracting only the gums from the image reduces the risk of the clusters being influenced by other features of the mouth, such as the tongue, which can also be different shades of red and pink.

[0022] In one embodiment, segmenting an image using a machine learning model to extract only gums from the image includes: segmenting the image using a first machine learning model of the machine learning model to extract an upper gum region of interest of the gums; and segmenting the image using a second machine learning model of the machine learning model to extract a lower gum region of interest of the gums.

[0023] In one embodiment, the computer-implemented method further includes filtering the clusters to remove one or more clusters having a color greater than a threshold distance from a reference color, wherein the threshold distance corresponds to a color associated with healthy gums.

[0024] It is computationally more efficient to remove clusters with colors associated with healthy gums before calculating the total distance rather than after.

[0025] In one embodiment, the computer-implemented method further includes determining a total gingival classification as severe gingivitis if the total distance is less than a first threshold distance; moderate gingivitis if the total distance is greater than the first threshold distance and less than a second threshold distance, wherein the second threshold distance is greater than the first threshold distance; and mild gingivitis if the total distance is greater than the second threshold distance and less than a third threshold distance, wherein the third threshold distance is greater than the second threshold distance.

[0026] According to one aspect of the present invention, a computer-implemented method for generating training examples of images of gums is provided, the computer-implemented method being used to train a machine learning model to segment images of gums so as to extract gums therefrom, the computer-implemented method comprising: receiving, through a cyclic generative adversarial model, a plurality of images labeled as healthy gums and a plurality of images of gums labeled as having gingivitis; training the cyclic generative adversarial model using one or more of the plurality of images of gums labeled as having gingivitis as true samples to transfer images from the plurality of images labeled as healthy gums to images showing gums having gingivitis; and storing the transferred images and the received plurality of images labeled as healthy gums and gums having gingivitis as training examples.

[0027] Examples of healthy gums tend to be easier to find than examples of gums with gingivitis. Therefore, generating examples of gums with gingivitis balances the training set.

[0028] According to one aspect of the present invention, a computer-implemented method for training a machine learning model is provided, the computer-implemented method segmenting an image showing gums so as to extract gums from the image, the computer-implemented method comprising: receiving, by the machine learning model, a training example of an image of gums; receiving, by the machine learning model, a plurality of gum masks, each of the plurality of gum masks being paired with a training example of an image, the training example of the image showing gums from the training example of the image of gums; running the machine learning model to generate a gum mask from each of the plurality of images of gums; and modifying parameterization settings of the machine learning model to reduce an error between the gum masks generated by the machine learning model and the corresponding paired gum masks.

[0029] In one embodiment, the computer-implemented method further comprises generating training examples of images of gums using the computer-implemented method of any preceding aspect or embodiment.

[0030] According to one aspect of the present invention, a transient or non-transitory computer-readable medium is provided, on which instructions are stored, which, when executed by a computer, cause the computer to perform a computer-implemented method of any of the aforementioned aspects or embodiments.

[0031] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Embodiments of the invention may best be understood with reference to the accompanying drawings, in which:

[0033] Figure 1(a) to (c) show images of gums using prior art methods to classify inflamed and non-inflamed areas;

[0034] Figure 2 (a) to (d) show images of gums using prior art methods to classify inflamed and non-inflamed areas;

[0035] Figure 3 shows a flowchart briefly outlining a training process for predicting gingival index in gingiva according to one or more embodiments;

[0036] Figure 4 Another flowchart briefly summarizes the reasoning method for gingivitis detection according to one or more embodiments is shown;

[0037] Figure 5 A flowchart is shown according to one or more embodiments, which briefly summarizes the process of Figure 4 The training model used in the flowchart;

[0038] Figure 6 shows several images of gums including healthy and pathological samples from a gums dataset according to one or more embodiments;

[0039] Figure 7 (a) shows two images of gums as real healthy samples arranged in a row according to one or more embodiments, Figure 7 (b) shows a graph of the CycleGAN according to one or more embodiments. Figure 7 Two images of the gums of the synthetic gingivitis gum sample generated from the corresponding healthy gum image in (a);

[0040] Figure 8 (a) shows an original image showing gums according to one or more embodiments, Figure 8 (b) shows a display according to one or more embodiments Figure 8 (a) The mask of the upper gum line extracted from the original image of the gums, Figure 8 (c) shows a display according to one or more embodiments Figure 8 (c) The mask of the lower gum line extracted from the original image of the gums;

[0041] Figure 9 shows a table illustrating color clusters extracted in RGB and LAB color spaces using k-means clustering according to one or more embodiments;

[0042] Figure 10 (a) shows the color associated with the color space hexadecimal number #800000, Figure 10(b) shows the color associated with the color space hexadecimal number #FFA093;

[0043] Figure 11 shows a table showing color clusters extracted in RGB and LAB spaces after color filtering according to one or more embodiments;

[0044] Figure 12 The color associated with the color space hexadecimal number #AD1D1D is shown;

[0045] Figure 13 shows a gingival index diagram known in the prior art;

[0046] Figure 14 A table is shown that includes gingival threshold values ​​based on a distance D (units) relative to a reference color #AD1D1D, where severe, moderate, mild, and normal categories correspond to Figure 13 The corresponding gingival status in the gingival index diagram;

[0047] Figure 15 shows a table including gingival index predictions for healthy and pathological samples in LAB space according to one or more embodiments;

[0048] Figure 16 shows a table including gingival index predictions for healthy and pathological samples in LAB space according to one or more embodiments;

[0049] Figure 17 A flowchart outlining a computer-implemented method for determining the severity of gingivitis based on an image showing gums is shown in accordance with one or more embodiments;

[0050] Figure 18 A flowchart outlining a training example computer-implemented method for generating an image of gingiva for training a machine learning model to segment an image of gingiva for extracting gingiva therefrom is shown in accordance with one or more embodiments;

[0051] Figure 19 A flowchart outlining a computer-implemented method of training a machine learning model to segment an image showing gums to extract the gums from the image is shown in accordance with one or more embodiments; and

[0052] Figure 20 A schematic diagram of the device is shown. DETAILED DESCRIPTION

[0053] At least some exemplary embodiments described herein can be constructed using dedicated hardware in part or in whole. Terms such as 'components', 'modules' or 'units' used herein can include but are not limited to hardware devices, such as circuits, field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs) in the form of discrete or integrated components, which perform specific tasks or provide associated functionality. In certain embodiments, the element described can be configured to reside on a tangible, persistent, addressable storage medium, and can be configured to execute on one or more processors. In certain embodiments, these functional elements can include, for example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcodes, circuits, data, databases, data structures, tables, arrays and variables. Although exemplary embodiments have been described with reference to components, modules and units discussed herein, this type of functional element can be combined into fewer components or separated into additional components. Various combinations of optional features have been described herein, and it should be understood that the features described can be combined in any suitable combination. In particular, features of any one exemplary embodiment may be appropriately combined with features of any other embodiment, unless such combinations are mutually exclusive. Throughout the specification, the term "comprising" or "comprises" means including the specified component(s) but does not exclude the presence of other components.

[0054] The training and inference methods described herein may be provided as instructions stored on a transitory or non-transitory computer-readable medium, which, when executed by a processor, causes the processor to perform the corresponding method. Therefore, these methods may be computer-implemented methods.

[0055] A hybrid AI approach based on state-of-the-art computer vision and unsupervised machine learning algorithms was designed to predict the gingival index of a patient's upper and lower gums with high accuracy in a single pass. To predict the gingival index, the criteria established by Loe and Silness were followed. A smartphone was used to capture an image of the gums, which was then fed into the hybrid AI approach for analysis and gingival index prediction.

[0056] refer to Figure 3 The training process includes the following steps. First, a training dataset is provided at 20. Image augmentation using CycleGAN occurs at 22 to balance the dataset. This is because there are many more images of healthy gums than images of gums with gingivitis. Next, at 24, regions of interest are identified and segmented using DeepLab. At 26, a machine learning module, which can be a clustering algorithm, detects gingival inflammation. At 28, a gingival index is predicted.

[0057] refer to Figure 4 , an inference method is implemented using the following modules.

[0058] One module is the Image Augmentation Module 32: Typically, in healthcare datasets, the number of pathological samples is far less than that of healthy samples. A pre-trained CycleGAN model on human gum images was fine-tuned to augment the dataset with samples from the pathology category. This model generates synthetic pathological samples from genuine healthy gum images, which can further improve the quality of the masks predicted by the AI ​​model, enabling more accurate analysis of gingivitis.

[0059] Another module is the ROI Segmentation Module34: It fine-tunes a pre-trained DeepLab AI model to detect ROIs in the gums. The ROI consists of image pixels representing the gums, gum line, and tooth crowns.

[0060] Another module is a color detection module 34: a k-means clustering algorithm is used to learn the colors present in the mask of the upper and lower gums. Healthy gums appear pink and firm, while gums with gingivitis tend to be red and swollen.

[0061] Another module is the color filtering module 36: colors with a red hue in the cluster predicted in the previous step are filtered with appropriate features.

[0062] Finally, we propose a new metric to predict the gingival index of the upper and lower gingiva at 28 based on the filtered colors in the upper and lower gingiva. Compared with previous works that focus on highly localized regions in the gingiva (e.g. Figure 1 and Figure 2 Compared to the present invention (shown in Figure 2), embodiments of the present invention predict the health of the upper and lower gums by considering the entire gum line area in the upper and lower gums. They are also the first to predict the severity of gingivitis.

[0063] exist Figure 3 In , the training process is shown involving deep learning and classical machine learning models for predicting gingival index in gingiva. Figure 4 In

[15] , an inference flowchart is presented that highlights how the proposed method processes an input gingival image and predicts the gingival index of both the upper and lower gingiva.

[0064] It should be pointed out that Figure 4 There are two branches in . These branches correspond to embodiments of the present invention performed in RGB and LAB color spaces, which will be described in more detail below.

[0065] refer to Figure 5In this study, we propose a simple yet effective method for accurately predicting the gingival index of both the upper and lower gums. The proposed method consists of three algorithms: a) CycleGAN image enhancement, b) DeepLab mask segmentation, and c) color detection with k-means clustering. The proposed method leverages state-of-the-art AI models for image enhancement and semantic segmentation tasks, as well as a novel metric for classifying the gingival index of both the upper and lower gums.

[0066] exist Figure 5 , which shows the workflow for training and fine-tuning AI models.

[0067] In more detail, Figure 5 The workflow includes receiving a pre-trained AI model at 40, fine-tuning the AI ​​model on a custom dataset at 42, saving the best AI model at 44, and then using the AI ​​models in production at 46. In production, a user gum image is received at 48, and gum prediction is performed at 50 using the AI ​​model in production at 46.

[0068] In the above workflow, the pre-trained CycleGAN and DeepLab AI models are fine-tuned on a custom dataset of human gum images. Further details of fine-tuning the AI ​​models and the custom human gum dataset are discussed below.

[0069] A. Gum Dataset

[0070] refer to Figure 6 , one embodiment of the present invention fine-tunes a state-of-the-art AI model on a customized human gingiva dataset for predicting gingival index. The gingiva dataset consists of 150 publicly available human gingiva images shared by various medical websites. All the images were obtained by web scraping. The dataset was manually prepared and includes images of patients from different age groups and socio-demographic backgrounds. Both healthy samples 52 and pathological samples 54 of varying severity were included in the dataset. However, the dataset is unbalanced with 110 healthy samples and 40 pathological samples.

[0071] Some of the challenges in training AI models on the GUM dataset are described below.

[0072] The gum line is not distinct, making accurate analysis and prediction impossible. Ambient lighting conditions, especially dark backgrounds. Light reflected from saliva on the gums. The number of samples in each category is highly unequal. Pathological samples have progressed to periodontitis.

[0073] B. Image enhancement with CycleGAN

[0074] refer to Figure 7(a) and (b), the main challenges of medical datasets are limited data availability and class imbalance between healthy and pathological samples. Limited datasets can cause overfitting of deep neural networks, which affects scoring performance on test data. Image or data augmentation provides a set of techniques that can increase the quality and size of training datasets to build powerful deep learning models. This embodiment leverages the potential of generative adversarial networks (GANs), which have been widely used for data augmentation in medical datasets.

[0075] Generative adversarial networks learn to generate new data that roughly follows the same distribution as the training dataset. In other words, a GAN trained on original images can generate new synthetic images that humans can detect as realistic. A common use case in healthcare and medicine is image-to-image translation, where the task is to identify salient features from one image domain and predict how these features will translate to another. In healthcare datasets, there are two primary image domains: healthy and pathological. Because healthcare datasets are highly skewed towards the healthy class, all algorithms used for image-to-image translation can be used to generate synthetic pathological samples from genuine healthy image data.

[0076] Cycle GAN is a special type of GAN that can perform image-to-image translation without paired images. These models can translate images from one domain to another without requiring a one-to-one mapping between the source and target domains. A dataset of source and target images (the source and target images do not need to be related) can be used to train a CycleGAN. This embodiment helps CycleGAN generate synthetic samples with gingivitis from images in the healthy domain, thereby enhancing the size and quality of the gingivitis dataset. The original gingivitis dataset includes 110 healthy samples and 40 pathological samples. A pre-trained CycleGAN (e.g., horse2zebra or PyTorch) model is fine-tuned on the gingivitis dataset, and 70 synthetic samples with gingivitis are generated. In Figure 7 In a, a truly healthy gum sample is shown. Figure 7 b shows the corresponding synthetic pathological sample with similar characteristics to the real gingivitis sample.

[0077] C. Mask detection of gums

[0078] refer to Figure 8 (a) to (c), Region of Interest (ROI) mask detection is a computer vision task in which an AI model maps each pixel of an image to a specific category. Mask detection models are widely used to train AI-enabled autonomous vehicles and medical image diagnosis. Figure 8 In a, a sample from the gingival dataset is shown, while Figure 8 b and Figure 8c represents the corresponding masks of the upper and lower gum lines predicted by the AI ​​model. The mask of an object consists of pixels belonging to the same class or category.

[0079] To train the AI ​​model, the original gum dataset was split into training and validation sets. Next, K. Wada's Labelme open-source package was used to annotate the upper and lower gum line ROIs in the training and validation sets. The Labelme package allows users to manually label image pixels that correspond to specific categories.

[0080] The original gums dataset was used to fine-tune a pre-trained DeepLabv3 PyTorch model with an FCN-ResNet101 backbone to learn a mask for the upper gum line 56 in each image. A second DeepLab model was trained to learn a mask for the lower gum line 58. The fine-tuned DeepLab model achieved IOU scores of 0.60 and 0.62 at a threshold of 0.95 on the validation gums dataset. The enhanced gums dataset (with synthetic pathology samples generated by CycleGAN) was then used to fine-tune the DeepLab model and achieved improved IOU scores of 0.66 and 0.67 at a threshold of 0.95 on the same validation dataset. The above results empirically demonstrate that image augmentation with CycleGAN can enhance the predictive performance of AI models on test data.

[0081] At inference time, according to the method of one or more embodiments, a machine learning model or a mask detection model receives an image showing gums and, before determining the color of each unit area, segments the image to extract only the gums. Figure 8 (a) and Figure 8 (b) Segmenting an image using a machine learning model so as to extract only the gums from the image may include: segmenting the image using a first machine learning model of the machine learning model to extract the upper gum 56 region of interest of the gums; and segmenting the image using a second machine learning model of the machine learning model to extract the lower gum 58 region of interest of the gums.

[0082] D. Color detection

[0083] In the previous section, masks of the upper and lower gums were obtained using a deep learning model. The next step involves determining the color and corresponding hue of the gum masks. Healthy gums typically appear pink and firm, while gums with gingivitis tend to be red and swollen. The main goal of this section is to detect hues ranging from light pink to dark red in each mask.

[0084] In this manner, the method includes, for each unit area of ​​gingiva, determining a color from a plurality of unit areas of gingiva. Each unit area of ​​the plurality of unit areas may be a pixel, wherein the color of each pixel may be determined by determining a value of the pixel using a red, green, blue (RGB) color model or a luminance / brightness, red, green, blue, and yellow color model. As will be understood below, the method may further include assigning a color space value to the color of the cluster, and optionally, the color space value is a hexadecimal number.

[0085] A color space diagram is a mathematical function that projects colors into a coordinate space. Since humans have trichromatic vision, with the highest sensitivity to red, green, and blue wavelengths, most color spaces are three-dimensional. The most common color space is RGB, or red, green, and blue, which modern computers and mobile applications use to display and store color images. CIELAB is another color space where the three channels are luminance (black to white), a (green to red), and b (blue to yellow). The CIELAB color space was designed to approximate human vision, and it facilitates the prediction of small differences in color. For simplicity, CIELAB will be referred to as LAB below.

[0086] K-means clustering is an unsupervised machine learning algorithm that groups n unlabeled observations into K predetermined clusters. A cluster refers to a group of observations that are grouped together due to the similarity of some features. Each observation is assigned to the cluster with the nearest mean or cluster center. One embodiment of the present invention utilizes k-means clustering to detect the different shades present in the upper and lower gum masks. The black background color in the masks is redundant for gingival index prediction and is removed for further analysis. The next step involves setting an appropriate threshold to extract the different shades of healthy and inflamed gums. In Figure 10 , hexadecimal color thresholds or limits #800000 and #FFA093 are shown. These thresholds ensure that only those chromaticities within the limits will be considered for analyzing gingival color using k-means clustering.

[0087] exist Figure 9 In the gingival mask, colors of similar hues and tones are clustered together based on their RGB and LAB characteristics. The colors in the pie chart represent the shades and tones of the different clusters in the healthy and inflamed gingival masks in RGB and LAB color space. These clusters are marked by the corresponding hexadecimal color next to the pie chart. The numerical value representing each cluster in the pie chart indicates the percentage of area or pixels in the upper or lower gingival mask represented by that particular cluster. The numerical value within a cluster helps determine the spatial spread of gingival inflammation.

[0088] exist Figure 9In the gingival mask, colors of similar hues and tones are clustered together based on their RGB and LAB features. The colors in the pie chart represent the shades and tones of the different clusters in the healthy and inflamed gingival masks in RGB and LAB color space. These clusters are marked by the corresponding hexadecimal color next to the pie chart. The numerical value representing each cluster in the pie chart indicates the percentage of area or pixels in the upper or lower gingival mask represented by that particular cluster. The numerical value within a cluster helps determine the spatial spread of gingival inflammation.

[0089] E. Color Filtering

[0090] The method includes clustering a plurality of unit areas according to corresponding determined colors using a clustering algorithm.

[0091] The primary goal of the k-means clustering algorithm is to obtain color clusters representing different shades of healthy and inflamed gums. Healthy gum color consists of different shades and tones of pink, while inflamed gums are represented by different shades of red. The next step involves color filtering the healthy gum color clusters from the RGB and LAB space pie charts obtained in the previous section. Clusters representing healthy gum color in the upper and lower masks are filtered out from the pie charts obtained in the previous section, and the updated pie charts are composed of color clusters representing various shades and tones of inflamed gums.

[0092] Several color space features were considered to implement color filtering, with the goal of clearly identifying healthy gum color clusters from their inflamed counterparts in the pie chart. The red (R) component of the RGB color space ranges from 0 to 255, where the RGB coordinates for pure red are (255, 0, 0). The R value in the hexadecimal color code was considered the decisive feature for distinguishing between various shades of pink and red. However, further research showed that the red feature was not optimal, as various shades of pink and red can have similar R values. For example, the hexadecimal color #FF1493 (255, 20, 147), which represents a dark pink shade, and the hexadecimal color #FF0000 (255, 0, 0), which represents pure red, have the same R value.

[0093] The next feature to consider is the yellowness of a color cluster. In the RGB color space, pure yellow (255, 255, 0) is a secondary color composed of two primary colors: red and green. Therefore, the yellowness of a color is related to its redness. CMYK (cyan, magenta, yellow, primary) is another color space based on the three primary colors of red, green, and blue. Each of the C, M, Y, and K components varies between 0% and 100%. For pure yellow, the Y value is 100% and is assumed to be a feature that can distinguish between shades of pink and red. However, the yellowness feature cannot accurately identify several pink-shaded color clusters from their red counterparts. For example, the pink-shaded color #F811AB (248, 17, 171) has the same 30% Y value in CMYK space as the hexadecimal color #810835 (129, 8, 53), which represents a variant of the red shade.

[0094] Another feature to consider is the lightness of the color group. HSL is a relative color space and an alternative representation to RGB space. HSL refers to the hue, saturation, and lightness of a color. The lightness of a color ranges from 0 to 1, where 1 is the lightest hue and vice versa. Healthy gums are typically light pink, while inflamed gums are darker. Therefore, it is natural to assume that the lightness of healthy gums should be greater than that of inflamed gums. However, the lightness of some shades of pink can be similar to the lightness of many shades of red, making it unsuitable for distinguishing between healthy and inflamed gum colors. For example, the hexadecimal color #E21EBD (226, 30, 189), which is a dark pink shade, has a lightness of 0.5, which is similar to the lightness of pure red #FF0000 (255, 0, 0).

[0095] As can be understood from the following description, the method may include determining the distance between the color space value of the cluster's color and the color space value of the reference color. In view of the limitations of the above-mentioned features, a distance-based color feature is considered for distinguishing healthy and inflamed gums. The Euclidean distance of the color representing the cluster of interest is calculated relative to the reference color in RGB and LAB space. In other words, the method includes calculating the Euclidean distance of the color of each cluster from the reference color. Figure 12As shown, the reference color considered for the illustrated embodiment is #A91D1D. Renowned dentist Willi Geller provided a colorimetric chart for detecting gingivitis. However, the chart does not mention the hexadecimal color codes corresponding to the stages of gingivitis. The hexadecimal color code that Willi Geller indicated is closest to the gum color for the advanced stage of gingivitis is #A91D1D. In this embodiment, the hexadecimal color #A91D1D is considered to represent the gum color in the advanced stage of gingivitis, and the predicted severity of all stages of gingivitis is based on the hexadecimal color #A91D1D in RGB and LAB space. The current hexadecimal color code benchmark is subjective, and alternative benchmarks may be considered that best suit a particular use case.

[0096] The feature analysis based on Euclidean distance resulted in the observation that the distance of the healthy gum color cluster is larger than the distance of the inflamed gum cluster relative to the reference color #A91D1D in both RGB and LAB space. Detailed studies in RGB space indicate that the number of healthy gum color shades that differ by more than 80-100 units from the reference color is likely to be much higher than the number of inflamed gums at similar distances. Similarly, healthy gums in LAB space are 40 to 50 units more than inflamed gums. In one embodiment, the above empirical observations are taken into account for setting a distance threshold of 80 units in RGB space and 40 units in LAB space to distinguish between healthy and inflamed gum color shades. The color filtering step removes those clusters in the pie chart that are beyond 80 units and 40 units relative to the reference color in RGB and LAB space. Based on the RGB and LAB thresholds, the remaining clusters in the filtered pie chart represent inflamed gum color shades that appear in the gum mask. In Figure 11 , filtered pie charts with inflamed gum clusters are shown for upper and lower gums in RGB and LAB space. In other words, the method can include filtering the clusters to remove one or more clusters that have a distance from a reference color greater than a threshold distance, where the threshold distance corresponds to a color associated with healthy gums.

[0097] F. Gingival Index Classification

[0098] Referring to the description below, the method includes calculating a total distance value of the gums based on the distance of each cluster from a reference color and the corresponding size of each cluster, and determining the severity of gingivitis based on the total distance.

[0099] The Gingival Index (GI) score introduced by Loe and Silness is a measure of the stage of gingivitis (normal or healthy, mild, moderate, and severe), as Figure 13As shown. The GI score ranges from 0 to 3, where a score of 0 indicates healthy gums and a score of 3 indicates severe gingivitis, which is characterized by swollen, inflamed, and bleeding gums. The color filtering method introduced in the previous section removes clusters representing color shades of healthy gums (GI score = 0). The goal of this section is to classify the gingival index categories (GI score = 1, 2, or 3) of the pathological upper and lower gums represented by the remaining clusters in the filtered pie chart. This section proposes a rule-based gingival index classification method in RGB and LAB space.

[0100] Based on empirical observations of the RGB and LAB space distances of clusters in the filtered pie chart relative to the reference color #A91D1D and the gingivitis stage in the upper or lower gingiva, Figure 14 We propose an optimal gingival index threshold for accurately classifying gingival samples. Note that the gingival threshold used to determine the severity of gingivitis can also be considered a hyperparameter. Machine learning methods for hyperparameter optimization can be used to effectively learn the gingival threshold, provided that the gingival dataset is labeled with the corresponding gingival index by domain experts.

[0101] In the previous section, the color filtering method produced a modified pie chart containing clusters of inflamed gum colors. The numerical value representing each cluster indicates the percentage of pixels or area assigned to that particular cluster. This numerical value can also be used to determine the fraction f of pixels or area in the gum mask assigned to the i-th cluster in the filtered pie chart. i . Assume d i represents the distance of the color of the i-th cluster relative to the reference color #A91D1D. Let S represent a set of indices for the inflamed gum clusters in the modified or filtered pie chart, which represents the gum mask after color filtering. Then, the weighted average distance WAD (which can be the total distance) of the set of clusters in S relative to the reference color is given by:

[0102] WAD=∑ i∈S f i ·d i (1)

[0103] where ∑ i∈S f i = 1. On a qualitative level, the plaque distance indicates the stage or severity of gingivitis in the gums. Figure 14 The gingival threshold bucket and WAD distance shown are analyzed to determine the severity of gingivitis. However, the weighted average distance has a disadvantage that it may affect the final gingival prediction score of the gingival sample. The WAD distance gives equal weight to each cluster in the set S regardless of the score (severe: 3, moderate: 2, mild: 1 and normal: 0) and does not distinguish between clusters with higher scores and clusters with lower scores. Note that it can be obtained from Figure 14Gingival scores were obtained for each individual group in the 2012 study. On an intuitive level, even highly localized gingival areas affected by severe inflammation could be diagnosed as moderate to severe gingivitis, which may not be indicated by the WAD.

[0104] To overcome the limitations of WAD, a new metric is proposed for accurately predicting the gingival score or index of gingival samples. The selected weighted average distance SWAD (which can be another type of total distance value) gives higher weights to clusters with higher gingival scores and vice versa. Let c i Indicates based on Figure 14 The gingival score of the ith cluster for the threshold value shown. In this way, the method can include generating a gingival score for each cluster based on the distance. The selected weighted average distance is given by:

[0105]

[0106] The normalization factor Please note that c i = 0 is not considered in the above equation because the clusters representing healthy gums are removed after color filtering and therefore they are not included in the set S. In the above equation, An additional weight of is assigned to the i-th cluster in S. Qualitatively, the SWAD distance can capture the severity of gingivitis in the gingiva at a finer level than WAD. Based on the SWAD distances calculated for the clusters in the filtered pie chart, the gingival index of the upper and lower gingiva is predicted.

[0107] Further references Figure 13 and Figure 14 , the total gingival classification can be determined as: if the total distance is less than a first threshold distance, it is severe gingivitis; if the total distance is greater than the first threshold distance and less than a second threshold distance, it is moderate gingivitis, where the second threshold distance is greater than the first threshold distance; if the total distance is greater than the second threshold distance and less than a third threshold distance, it is mild gingivitis, where the third threshold distance is greater than the second threshold distance.

[0108] The method may further include sending the determined severity of the gingivitis to a display for displaying the severity of the gingivitis to a user. Figure 20 Such a display is shown in FIG, which is shown purely for illustrative purposes. Display 72 may be part of a mobile device, such as a smartphone. Device 70 may also include a camera 74, which may be a camera that captures images used to perform the inference method for determining the level of gingivitis.

[0109] G. Results

[0110] exist Figure 15 and Figure 16In Figure 2, the predicted gingival index for several healthy and gingivitis-affected samples from the Gingiva dataset is displayed in both RGB and LAB space. The pie charts shown in the table were obtained after implementing the color filtering method. The first four images in the table represent gingivitis-affected gingiva, while the last image is a healthy gingiva sample. Note that lower SWAD distance values ​​indicate a more severe predicted stage of gingivitis, compared to higher SWAD distances. No SWAD distance is defined for the cluster representing healthy gingiva color. Therefore, by default, the SWAD distance for healthy gingiva is undefined.

[0111] Gingival index prediction based on both RGB and LAB space can be considered to make an informed decision about the gingival health status. The proposed method can help patients assess the severity of gingivitis and track the gingival health status regularly during and after treatment.

[0112] refer to Figure 17 According to one or more embodiments, a computer-implemented method for determining the severity of gingivitis from an image showing gingiva can be summarized as including the following steps: receiving S100 an image showing gingiva; for each unit area of ​​gingiva, determining S102 a color from a plurality of unit areas of gingiva; clustering S104 the plurality of unit areas according to the corresponding determined colors using a clustering algorithm; calculating S106 a distance between the color of each cluster and a reference color; calculating S108 a total distance value of the gingiva based on the distance of each cluster from the reference color and the corresponding size of each cluster; and determining S110 the severity of gingivitis based on the total distance.

[0113] refer to Figure 18 According to one or more embodiments, a computer-implemented method for generating training examples of images of gums is summarized, the computer-implemented method being used to train a machine learning model to segment the images of gums in order to extract gums therefrom. The computer-implemented method includes the following steps: receiving (S200) a plurality of images labeled as healthy gums and a plurality of images labeled as having gingivitis via a recurrent generative adversarial model; training (S202) the recurrent generative adversarial model using one or more of the plurality of images labeled as having gingivitis as true samples to transfer images from the plurality of images labeled as healthy gums to images showing gums having gingivitis; and storing (S204) the transferred images and the received plurality of images labeled as healthy gums and gums having gingivitis as training examples.

[0114] refer to Figure 19According to one or more embodiments, a computer-implemented method for training a machine learning model is summarized, the computer-implemented method segmenting an image showing gums so as to extract gums from the image, the computer-implemented method comprising: receiving S300 a training example of an image of gums by the machine learning model; receiving S302 a plurality of gum masks by the machine learning model, each gum mask in the plurality of gum masks being paired with a training example of an image, the training example of the image showing gums from the training example of the image of gums; running S304 the machine learning model to generate a gum mask from each of the plurality of images of gums; and modifying S306 parameterization settings of the machine learning model to reduce an error between the gum masks generated by the machine learning model and the corresponding paired gum masks.

[0115] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments.

[0116] Other variations of the disclosed embodiments may be understood and implemented by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may perform the functions of several items listed in a claim. The mere fact that certain measures are cited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method for determining the severity of gingivitis from an image showing gums, the computer-implemented method comprising: receiving the image showing the gums (S100); For each unit area of ​​the gum, determining (S102) a color from a plurality of unit areas of the gum; Using a clustering algorithm, clustering the plurality of unit areas according to the corresponding determined colors (S104); Calculating (S106) the distance between the color of each cluster and a reference color; calculating (S108) a total distance value of the gum based on the distance between each cluster and the reference color and the corresponding size of each cluster; as well as The severity of gingivitis is determined ( S110 ) based on the total distance.

2. The computer-implemented method of claim 1, further comprising sending the determined severity of gingivitis to a display (72) for displaying the severity of gingivitis to a user.

3. The computer-implemented method of claim 1 , wherein calculating the distance between the color of each cluster and a reference color comprises: assigning a color space value to the color of the cluster; as well as determining a distance between the color space value of the color of the cluster and the color space value of the reference color, Optionally, the color space value is a hexadecimal number.

4. The computer-implemented method of any one of the preceding claims, wherein calculating the distance of the color of each cluster from a reference color comprises: The Euclidean distance between the color of each cluster and the reference color is calculated.

5. The computer-implemented method of any one of the preceding claims, wherein the total distance value is a weighted average distance (WAD) defined by: WAD=∑ i∈S f i ·d i , where f i is the fraction of the area of ​​the ith cluster compared to the total area of ​​the plurality of unit areas of the gingiva, d i is the distance between the color of the i-th cluster and the reference color, S is a set of indices showing color clusters present in the image of the gum, and ∑ i∈S f i =1.

6. The computer-implemented method according to any one of claims 1 to 4, further comprising: Generate a gingival score for each cluster based on the distance, The total distance value is a selected weighted average distance SWAD, which is defined by the following formula: where f i is the fraction of the area of ​​the ith cluster compared to the total area of ​​the gingiva, d i is the distance between the color of the i-th cluster and the reference color, is the weight, where c i is the gum score of the i-th cluster, S is a set of indices showing the color clusters present in the image of the gum, and ∑ i∈S f i =1.

7. A computer-implemented method according to any one of the preceding claims, wherein each unit area of ​​the plurality of unit areas is a pixel, and wherein the determining the color comprises determining the value of the pixel using a red, green, blue color model or a luminance, red, green, blue and yellow color model.

8. The computer-implemented method of any preceding claim, further comprising: Before determining the color of each unit area, the image was segmented using a machine learning model to extract only the gums.

9. The computer-implemented method of claim 8, wherein segmenting the image using the machine learning model to extract only the gums from the image comprises: segmenting the image using a first machine learning model of the machine learning model to extract an upper gingival region of interest of the gingiva; as well as The image is segmented using a second machine learning model of the machine learning model to extract a lower gingival region of interest of the gingiva.

10. The computer-implemented method of any preceding claim, further comprising: The clusters are filtered to remove one or more clusters where the color is more than a threshold distance away from the reference color, wherein the threshold distance corresponds to a color associated with healthy gums.

11. The computer-implemented method of any preceding claim, further comprising: The total gingival classification was determined as: If the total distance is less than a first threshold distance, it is severe gingivitis; moderate gingivitis if the total distance is greater than the first threshold distance and less than a second threshold distance, wherein the second threshold distance is greater than the first threshold distance; and If the total distance is greater than the second threshold distance and less than a third threshold distance, it is mild gingivitis, wherein the third threshold distance is greater than the second threshold distance.

12. A computer-implemented method of generating training examples of images of gingiva, the computer-implemented method for training a machine learning model to segment images of gingiva for extracting gingiva therefrom, the computer-implemented method comprising: Receiving ( S200 ) a plurality of images ( 52 ) labeled as healthy gums and a plurality of images ( 54 ) labeled as gums suffering from gingivitis through a recurrent generative adversarial model; training ( S202 ) the recurrent generative adversarial model using one or more of the plurality of images labeled as gums with gingivitis as real samples to transfer images of the plurality of images labeled as healthy gums to images showing gums with gingivitis; as well as The transferred image and the received plurality of images labeled as healthy gums and gums with gingivitis are stored ( S204 ) as training examples.

13. A computer-implemented method of training a machine learning model, the computer-implemented method segmenting an image showing gums to extract the gums from the image, the computer-implemented method comprising: Receiving ( S300 ) a training example of an image of gums by the machine learning model; Receiving (S302) a plurality of gum masks by the machine learning model, each gum mask of the plurality of gum masks is paired with a training example of an image, the training example of the image showing gums from the training example of the image of gums; running ( S304 ) the machine learning model to generate a gingival mask from each of the plurality of images of the gingiva; as well as Modifying (S306) parameterization of the machine learning model to reduce an error between the gingival mask generated by the machine learning model and a corresponding paired gingival mask.

14. The computer-implemented method of claim 13, further comprising: Training examples of the images of gums are generated using the computer-implemented method of claim 12 .

15. A transitory or non-transitory computer-readable medium having stored thereon instructions which, when executed by a computer, cause the computer to perform the computer-implemented method according to any one of the preceding claims.

Images