Method for generating training data set for determining periodontal structure of patient

By generating training data sets for machine learning models, using out-oral image data to align with intraoral scanning data, the safety and efficiency problems of using radiation to evaluate periodontal structures in the prior art are solved, and non-invasive and fast teeth are achieved Weekly structure diagnosis.

CN118974841APending Publication Date: 2024-11-153SHAPE AS
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202380031658.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-02-18
Filing Date
2023-02-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing clinical methods are often used when evaluating bone levels and bone loss, radiation techniques are often used, resulting in patients being exposed to ionizing radiation, which is invasive and time-consuming, making it difficult to achieve non-invasive, safe and rapid assessment.

Method used

By generating training data sets for machine learning models, using extraoral image data to determine periodontal structures and aligning with intraoral scanning data, a training data set is generated to train machine learning models that recognize periodontal structures, and then diagnose periodontal disease , avoid radiation.

Benefits of technology

The rapid and reliable assessment of periodontal structure and bone loss without radiation is achieved, improving the safety and efficiency of diagnosis and reducing radiation exposure to patients.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118974841A_ABST
    Figure CN118974841A_ABST
Patent Text Reader

Abstract

According to one embodiment, a dental scanning system and a computer-implemented method of generating a training set for a machine learning model are disclosed. According to another embodiment, a dental scanning system and a computer-implemented method for providing a diagnostic data set of a patient are disclosed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to generating a training data set for a machine learning model and providing a diagnostic data set including periodontal attributes to a patient. In particular, the present disclosure relates to a method, a computer program, a data processing device, and a computer-readable storage medium for generating a training data set for diagnosing or at least assisting in diagnosing one or more periodontal structures of a patient's oral cavity. Background Art

[0002] Currently, clinical methods for assessing bone levels and associated bone loss are time-consuming, unpleasant for patients, and expensive (Shayeb et al., 2014; Guo et al., 2016, Fiorellini et al., 2021). Specifically, radiography (intraoral / panoramic) or even cone beam computed tomography (CBCT) scans are used to study bone levels or assess possible bone loss (Fiorellini et al., 2021). However, ionizing radiation doses should be avoided as it increases cancer risk, and while CBCT scans offer the highest quality for determining bone levels, they also inflict the highest radiation dose. In addition, more invasive methods such as transgingival probing or surgical exploration are also considered the clinical gold standard, for example, when assessing bone levels in relation to implant placement (Kloukos et al., 2021).

[0003] Based on the above, there is a clear need to develop a clinically applicable, noninvasive, and reproducible method that can estimate bone level and possible bone loss without the use of ionizing radiation. Summary of the invention

[0004] Typically, an assessment of the alveolar bone is performed by a dentist using either invasive methods or radiography to evaluate the pattern and extent of the alveolar bone (i.e., the appearance and position of the bone). Typically, bone levels are measured from the cemento-enamel junction (CEJ) to the crest of the alveolar bone, and bone loss defines such a process - meaning that if the bone level is abnormal, some bone has been resorbed, resulting in bone loss. Typically, bone loss is considered to be present when the bone level, measured as the distance between the CEJ to the crest of the alveolar bone, exceeds 3 mm. Conventional methods of using radiography to assess bone levels should be avoided as it exposes the patient to unnecessary radiation doses. The method described in this article is essentially intended to eliminate the use of radiation equipment when evaluating periodontal structures.

[0005] Therefore, one aspect of the present disclosure is to obtain a fast and reliable method for determining one or more periodontal structures of a patient and for assisting in diagnosing a disease or dental condition of the patient based on the one or more periodontal structures. The one or more periodontal structures may be bone loss and / or bone level.

[0006] Another aspect of the present disclosure is to achieve a method for determining one or more periodontal structures of a patient that is safer for the patient, i.e., minimizes exposure to ionizing radiation that is commonly used in current methods of assessing, inter alia, bone loss.

[0007] The present disclosure addresses the above challenges by providing a computer-implemented method for evaluating bone level and associated bone loss in intraoral surface scan images using a machine learning model. According to these aspects, a computer-implemented method for generating a training data set for a machine learning model is disclosed. The method includes receiving extraoral image data of a plurality of candidate patients provided by an extraoral imaging device, and determining one or more periodontal structures of each of the plurality of candidate patients based on the received extraoral image data.

[0008] The extraoral imaging device may be an X-ray scanner configured to scan a patient based on ionizing radiation. The extraoral imaging device may be a cone beam computed tomography (CBCT) scanner configured to scan a patient's head, an X-ray imaging device, or an intraoral radiography (X-ray) configured to scan a portion of a patient's teeth. The extraoral image data may be provided by an imaging device (e.g., an extraoral imaging device as previously defined). The extraoral image data is extraoral scan data as an output from the extraoral imaging device.

[0009] In addition, the method includes receiving intraoral scan data of a plurality of candidate patients provided by an intraoral scanner, and generating a training data set by combining extraoral image data with the intraoral scan data of each of the plurality of candidate patients to align one or more periodontal structures with the intraoral scan data.

[0010] In the context of the present application, combining is considered to include a process of image alignment in which elements in one image are mapped into a meaningful correspondence with elements in a second image. In an example according to the present disclosure, one or more periodontal structures determined from the extraoral image data may be considered as elements or more specifically image labels that are associated or mapped with elements in the intraoral scan data. The elements of the intraoral scan data may be, for example, teeth and / or gums.

[0011] Considering an example, the combination may include: identifying corresponding teeth in the extraoral image data and the intraoral scan data, determining one or more periodontal structures (i.e., labels, such as bone levels) in the extraoral image data, and mapping or associating the determined periodontal structures with corresponding teeth in the intraoral scan data using an image alignment process.

[0012] The image alignment process may result in the generation of, for example, a mapping matrix comprising information in the form of tooth numbers and associated label information - i.e., a matrix representing the periodontal structures determined from the extraoral image data together with the teeth at which the periodontal structures have been determined. The mapping matrix may be used together with the intraoral scan data to form a training data set. That is, for each intraoral scan data, at least one corresponding mapping matrix with extraoral periodontal structure (i.e., label) information is provided. In this way, it is ensured that the periodontal structures determined from the extraoral image data can be associated with the intraoral scan data in the form of image alignment.

[0013] In another example, the combination may include: identifying corresponding teeth in the extraoral image data and the intraoral scan data. Determining one or more periodontal structures (i.e., labels, such as bone levels) in the extraoral image data, and mapping or associating the determined periodontal structures with corresponding teeth in the intraoral image data using an image alignment process. In this example, the alignment process may include an image registration step, in which the periodontal structures determined from the extraoral image data are transferred to the intraoral scan data in an image warping process. In both of the above examples, a transformation (i.e., mapping) matrix is ​​generated that ensures that the tooth numbers and corresponding labels of the extraoral image data are associated with the teeth of the intraoral scan data, and that the determined periodontal structures of each tooth of the extraoral image data are associated with the intraoral data.

[0014] Through these combined methods, the intraoral scan data can be used as the input of the machine learning model to train the machine learning model, where the periodontal structures determined from the extraoral image data can be regarded as labels constituting the training data set (i.e., the labels can be regarded as the true values ​​(ground truth) of the data). A mapping matrix is ​​generated for each candidate patient's extraoral and intraoral image data to ensure that the output of the machine learning model (generated from the intraoral scan data input) can be compared with the true values ​​extracted from the extraoral image data during the training stage of the machine learning model.

[0015] The training data set is generated by combining extraoral and intraoral scan data, ensuring that extraoral information such as bone levels (which cannot be determined from the intraoral data) is associated with the intraoral scan data to generate the training data set. The machine learning model is trained based on a training data set having extraoral image information that can be used with the intraoral scan data information, ensuring that the machine learning model can learn how to extract periodontal feature information from the intraoral scan data after training without using the extraoral image data as an information source. Typically, periodontal structures cannot be identified from intraoral scans alone, but when a machine learning model is trained on two different data sets (i.e., extraoral image data and intraoral image data), a machine learning model can be created that is able to identify such periodontal structures when only provided with intraoral surface scan information.

[0016] Therefore, in order to train the machine learning model, the method includes: for each intraoral scan data acquired, a corresponding extraoral image of the same tooth taken at substantially the same time is acquired. In this way, it is ensured that the determined periodontal structure of each extraoral image data can be combined with the corresponding intraoral scan data. Therefore, the training data set includes a plurality of extraoral image data and intraoral scan data, wherein the preprocessing step includes determining one or more periodontal structures in the extraoral image data, and then generating a mapping matrix, which includes the determined periodontal structure and the associated determined periodontal structure of each of the teeth in the extraoral image data.

[0017] In one example, the mapping matrix may include further information, such as patient information, such as age, gender, diseases identified from previously acquired extraoral image data and / or intraoral scan data, etc.

[0018] Furthermore, the mapping matrix may include information regarding the location of the determined periodontal structures, for example, whether the periodontal structure is identified on the buccal or lingual side of the tooth, at which given reference site the periodontal structure is identified, and the like.

[0019] The intraoral scanner described herein may be a scanner configured to scan the anatomical structures of a patient's oral cavity without the use of ionizing radiation. The intraoral scanner may be configured to scan the patient's oral cavity by arranging at least a scanner tip of the intraoral scanner within the patient's oral cavity and moving the scanner tip around and within the oral cavity during scanning. The intraoral scanner may be a 2D intraoral camera or a 3D intraoral scanner. Preferably, the intraoral scanner is a handheld intraoral scanner that may be used in clinical practice and manually controlled by a dental practitioner when scanning a patient during a clinical visit.

[0020] As previously described, aspects described herein address the problem of eliminating the use of ionizing radiation when, for example, evaluating a patient for periodontal disease and periodontal structural changes. Rather than using extraoral image data that exposes the patient to radiation, a trained machine learning model is able to identify periodontal structures in intraoral scan data that could previously only be identified on, for example, CBCT data, X-ray data, or the like, thereby eliminating exposure of the patient to radiation.

[0021] Thus, the general idea described is to use extraoral image data in generating a training dataset to identify periodontal structures, such as bone levels and associated bone loss, and then form a mapping transformation (i.e., a mapping matrix) that transforms the extraoral image data (i.e., periodontal feature labels) into annotations of, for example, intraoral scan data of the training dataset. In this way, a training dataset is generated that includes intraoral scans with labels (also considered annotations) generated from an extraoral data source. This allows the use of intraoral scan data together with periodontal structure information (which is typically not available in intraoral scan data) to be input into a machine learning model to train such a model.

[0022] By using a training setup that combines descriptions from extraoral images and intraoral scan data, a diagnostic dataset can be output from the trained machine learning model that includes information about periodontal structures obtained directly from the intraoral surface scan without using previously acquired extraoral image data information for that scan. This eliminates the risks associated with radiography because radiation images taken by extraoral imaging equipment are no longer required to evaluate a patient's periodontal structure.

[0023] The term "candidate patient" in the context of the present disclosure refers to a patient who is being scanned with an extraoral imaging device and an intraoral scanner during a visit to a dental office. The two scans may not be performed during the same visit, but it may be preferred that the two scans are acquired within a short period of time to avoid large changes in periodontal structure, for example, large changes in bone levels between the acquisition of the extraoral image data and the intraoral image data. In order to ensure accurate training of the machine learning model, it is important that the periodontal structure information determined from the extraoral data is evaluated for training purposes at substantially the same time as the intraoral scan data is acquired. The data provided by these scans is used to train the machine learning model, and more specifically, to train the machine learning model using the training data set. Therefore, in order to obtain reliable combined extraoral image data and intraoral scan data, the time between the extraoral image scan and the intraoral scan should be as short as possible.

[0024] As previously described, combining the extraoral image data with the intraoral scan data may include aligning, merging, or overlapping the extraoral image data with the intraoral scan data to obtain a training data set. Thus, the training data set may be obtained in a preprocessing step, including aligning one or more periodontal structures with the intraoral scan data by combining information from extraoral image data that is not part of the intraoral scan data with the intraoral scan data as previously described. For example, the extraoral image data may include information about teeth and bones, and the intraoral scan data may include information about teeth and soft tissue, and the training data set may include information about teeth and soft tissue provided by the intraoral scan data combined with bone information provided by the extraoral image data.

[0025] In other words, in the context of the present application, the combination also includes a process of pre-processing the extraoral and intraoral data to create a training data set based on the information available in the two data types (i.e., the extraoral image data and the intraoral image data). Creating the training data set includes mapping the periodontal structures determined extraorally (in the form of labels or annotations) with the intraoral surface scan data to create the training data as described above, and then inputting the training data into the machine learning model (as described above).

[0026] When the training data set has been generated, the generated training data set is used to train the machine learning model to output at least the probability of identifying periodontal structure in the input data (i.e., intraoral surface scan data) for one or more teeth. Therefore, the input of the machine learning model is the patient's previously unseen intraoral scan data, which is input into the trained machine learning model. The term "unseen" intraoral scan data should be understood as data that has not been used to train the machine learning model. Subsequently, the trained machine learning model is configured to process the intraoral scan data and output a diagnostic data set associated with the input data. The diagnostic data set may include a prediction of the presence of periodontal structure on one or more of the teeth in the data set and where the prediction exists.

[0027] Thus, in view of one aspect, a computer-implemented method for providing a patient diagnostic dataset based on a training dataset is disclosed. The method includes receiving intraoral scan data of a patient, and outputting a diagnostic dataset of the patient, wherein the diagnostic dataset can be determined by processing the intraoral scan data of the patient based on the training dataset. In other words, a machine learning model can be trained on the training dataset and subsequently used in a clinical practice setting, wherein a patient intraoral scan that does not form part of the training process is input into a previously trained machine learning model. Based on the training of the model, the machine learning model is then able to output a diagnostic dataset that identifies periodontal structures present in the intraoral scan of the patient (particularly bone levels due to bone loss). More particularly, the diagnostic data includes a prediction of the presence of periodontal structures on one or more teeth in the intraoral scan data and where the prediction is present. Thus, when trained as described herein, the machine learning model does not require radiographic imaging to assess whether a patient is suffering from bone loss, etc.

[0028] The term "patient" in this disclosure refers to a person who is scanned by an intraoral scanner during a visit to a dental office. The intraoral scan data provided by the scan is used as input to a previously trained machine learning model, which then processes the data based on the training dataset and outputs a diagnostic dataset. The diagnostic dataset may include information about one or more periodontal structures, such as bone levels and / or bone loss. The information may be configured to provide a diagnostic status of one or more periodontal structures, which a dentist or physician may use to treat the patient to improve the diagnostic status of one or more periodontal structures. The machine learning model may be represented by a deep learning model or an artificial intelligence model (e.g., a neural network).

[0029] In addition, the machine learning model includes a training data set, and the model can be configured to provide a patient's diagnostic data set by processing the patient's intraoral scan data based on the training data set, as described above. The machine learning model can include one or more of the following known topologies, networks, or models, such as a convolutional neural network (CNN), a three-ring model, a continuous homology method, an algebraic topology, and the like.

[0030] In the examples given herein, extraoral and intraoral scan data are preferably image data input to the machine learning model. Periodontal structures determined from the extraoral image data may be viewed as labels and / or annotations as described, and may be viewed as annotations and / or labels aligned with the intraoral scan data by imaging warping techniques and / or by generating a mapping matrix including information about periodontal structures and locations on the teeth in the data.

[0031] Therefore, the data structure input to the machine learning model is preferably image data, which is best processed by different types of convolutional neural networks. Depending on the given task, the neural network can be generally configured to perform one of the tasks of image classification, object recognition and / or image segmentation.

[0032] In image classification, the training dataset is configured to allow the neural network to perform single-label classification on the input images. That is, each image in the dataset used for training has a corresponding label and / or annotation, and the model outputs a single prediction for each image encountered. In another setting, the neural network is configured to perform multi-label classification, where each image in the training dataset has multiple labels and / or annotations, allowing the neural network to output multi-label predictions for each image encountered.

[0033] In another example of object recognition, a neural network can be configured to perform an object detection task, where the neural network model is configured to detect objects and their locations. Using bounding boxes, the target object in the image is contained within a small rectangular box with a descriptive label.

[0034] In image segmentation, a neural network model uses an algorithm to separate an object in an image from its background and other objects. Using labels and / or annotations to map the pixels of an input image to specific features allows the model to divide the input image into subgroups called segments. Since the model uses the shape of the segments to predict different segments, it is important to annotate the shapes with shapes, not just with labels.

[0035] That is, the training data set described herein can be processed in a preprocessing step so as to be suitable for input into at least one convolutional neural network using one of the processing methods described above.

[0036] In an example using, for example, single-label classification, the input image to the neural network may be an intraoral surface scan along with its corresponding label extracted from the extraoral image data as the periodontal features present in the corresponding image. Multiple such intraoral scan data may be input to the neural network along with their corresponding periodontal structure labels. The output of such a neural network would be a single prediction of the presence of periodontal structure for each image it encounters (i.e., in each new intraoral scan data image that the neural network encounters during a patient's clinical visit).

[0037] In an alternative scheme using, for example, multi-label classification, each input image in the training data set can be intraoral scan data together with corresponding multiple labels of periodontal structures extracted separately from extraoral scan data and / or given by further annotations from, for example, a dentist. The multi-label configuration can also include patient-specific information, such as gender, age, dental diseases, etc. In this example, the neural network can output a multi-label prediction for each image it encounters, i.e., the output can include predictions of periodontal structures, age, gender, other diseases, etc. of the input intraoral scan image.

[0038] In examples using image segmentation, the input image can be intraoral scan data and corresponding labels in the form of periodontal features extracted from an extraoral image, for example, which are drawn to align with the corresponding intraoral scan data. Taking into account the warping process, the "drawn" periodontal features can be aligned directly with the intraoral scan data, but can also be provided to the neural network as a mapping matrix. In any case, the neural network model outputs a segmentation map that identifies segmented subgroups. The segmentation map can include label information, tooth information, and object information, which in the described example means that the segmentation map can include one or more periodontal reference structures, position information, tooth information, etc.

[0039] The training data set can be used not only for a machine learning model, but also for an artificial learning model or a deep learning model including a topology for training data and providing diagnostic data.

[0040] In more detail, to generate the training data set, preprocessing of the extraoral image data and the intraoral image data may be performed, as previously described. The preprocessing of the training data set may be based on receiving extraoral image data of a plurality of candidate patients. The extraoral image data may be collected from a plurality of scans of the plurality of candidate patients at different time points and may be stored in an image database for further processing.

[0041] Furthermore, in addition to receiving the extraoral image data, the method may also include receiving intraoral scan data of a plurality of candidate patients, wherein the intraoral scan data is provided by an intraoral scanner (e.g., a 3Shape A / STrios handheld intraoral scanner). As previously described, the intraoral scan data and the extraoral image data are collected substantially simultaneously to ensure that the data represent the same stage of the teeth and potential development of bone loss and / or other periodontal diseases. This ensures that the transfer of the extraoral image annotations and / or labels to the intraoral scan data accurately represents the state of the teeth in the intraoral scan data obtained substantially simultaneously.

[0042] Therefore, in order to create an intraoral surface scan dataset with annotated and / or labeled extraoral periodontal structures, the method may include associating extraoral image data with intraoral image data of the same candidate patient taken at substantially the same time point. In an example, each extraoral and intraoral dataset of a single candidate patient may be stored in an image database together with an identification tag associating the extraoral data, the intraoral data, and the identification tag. The identification tag may be an anonymized unique tag for a particular dataset (extraoral and intraoral data) of a particular candidate patient. Therefore, a large database of extraoral images and associated intraoral images of multiple candidate patients (e.g., including 500, 1000 or more samples) may be created.

[0043] In addition, the extraoral image data used to generate the training data set may include information about teeth and bones, such as information about bone levels and bone loss estimates for the extraoral image data. Accordingly, the extraoral image data may include extraoral annotations that identify bone levels and corresponding landmarks of bone loss on the extraoral image. Such bone landmark identifications (also referred to herein as labels) may be stored in a database together with the image to which they belong. The landmark identifications may be evaluated on the extraoral image for all teeth in the image, and the landmark identifications may include values ​​for any of the following parameters: position on the tooth, average bone loss, average bone level, buccal site, lingual site, and / or a spline representing the behavior of the landmark, etc.

[0044] In one example, these landmarks can be annotated by, for example, a human annotator who is presented with extraoral image data before creating a training dataset for a machine learning model. The annotator can mark the landmark areas in a number of different ways, while the processor ensures that information about the landmark locations, values ​​(e.g., levels or estimates of the landmarks) are correctly stored in the database along with the annotated extraoral image data. Thus, the image database also contains landmark information (also referred to as information about the teeth) for each extraoral image data collected for a plurality of candidate patients.

[0045] In an alternative approach, the landmark identification just described for extraoral images can be accomplished automatically by a machine learning method trained to recognize these landmarks in extraoral image data.

[0046] In order to create a training data set, the extraoral and intraoral images stored in the database can be processed to combine the extraoral image data with the intraoral scan data, for example, by aligning, merging and / or overlapping the extraoral image data with the intraoral scan data to obtain a training data set. In more detail, each extraoral image and the corresponding intraoral image of a plurality of candidate patients in the database are input into a processor together with the extraoral landmark identification (i.e., information about teeth and bones) obtained from the automatic method or annotation method. That is, the combining step is performed by a processor configured to process data from the extraoral images and the intraoral scan data to ensure that the annotations generated extraorally are merged (e.g., transferred) to the intraoral image data.

[0047] In one example, the method includes reading extraoral image data and corresponding landmark information into a processor, and reading intraoral image data into a processor. The landmark information data from the extraoral image data is then transformed into a data representation (e.g., a matrix representation) that includes, for example, tooth number information, identification of periodontal structure landmarks in the form of yes or no, the location of the tooth at which the periodontal structure is identified (i.e., buccal or lingual location), and / or identification of one or more of the three locations on each of the buccal and lingual locations where the periodontal structure landmark exists. Other information such as the patient's age, gender, etc. may also be represented.

[0048] The processor is then configured to transfer the data representation to the intraoral scan data to generate a training data set for training the machine learning model. In this way, a training data set is generated that combines extraoral image data with intraoral scan data for each of a plurality of candidate patients to align one or more periodontal structures with the intraoral scan data. The training data set can then be used as an input to the machine learning model to train the machine learning model. The machine learning is configured to learn from the training data set when a given structure in an input data set (e.g., intraoral scan data of a patient obtained in a clinical session) contains a periodontal structure of interest. Thus, the machine learning model can be configured to assign a probability value (e.g., a numerical value ranging from 0 to 1, where larger values ​​are associated with greater confidence) of the presence of a periodontal feature to each pixel in given input data in the form of intraoral scan data that does not form part of the training data set. The machine learning model can be configured to perform single-label or multi-label classification, image segmentation, and / or object recognition as previously described.

[0049] The intraoral scan data can then be post-processed into various forms for display in, for example, a graphical user interface. Thus, the machine learning model outputs a diagnostic data set that includes a set of probability scores, object recognition maps, and / or segmentation maps for the presence of detected periodontal structures and / or maps encoding the locations of any detected periodontal structures in the intraoral scan data. By using back propagation on the output of the machine learning model, the output of the machine learning model can be visualized on previously unseen image inputs. Thus, an input image of, for example, the maxilla or mandible can be visualized and the output of the neural network fused to the image by, for example, image warping techniques, to efficiently illustrate to the user that a periodontal structure was found at a particular input image.

[0050] Thus, by using a trained neural network (as described herein) on new image data, it is possible to assess whether a patient has periodontal disease without the use of radiation. Depending on the neural network used and the input data on which the neural network is trained, the assessment of periodontal disease can include identifying where bone loss exists on a tooth, the amount of bone level of a particular tooth, the age of the person, the sex of the person, and potential other diseases associated with a particular tooth of the input image data acquired from an intraoral scanner.

[0051] As previously described, the method described herein can also use more specific patient information as input to the neural network. This may improve the reliability of the method because the method may also include receiving candidate patient information for multiple candidate patients, and generating a training data set by combining the received patient information for each of the multiple candidate patients with the combined extraoral image data and intraoral scan data for each of the multiple candidate patients. The candidate patient information can be age, gender, health status, disease, race, etc. By applying the candidate patient information to the method, the possibility of grouping the training data set into one or more candidate patient information can be provided. The training of the training data set can be performed in parallel for each group of training data sets. Each group of the training data set can be labeled with one of the gender information. This improves the accuracy and reliability of the method and the machine learning model.

[0052] The method may include segmenting the extraoral image data of multiple candidate patients into teeth and bones based on an extraoral segmentation algorithm, and segmenting the intraoral scan data of multiple candidate patients into teeth and soft tissue based on an intraoral segmentation algorithm. The segmentation of data is known and will not be further described in the present disclosure. However, examples are given below. The extraoral segmentation algorithm can be configured to perform segmentation of data provided by an extraoral image device, and the intraoral segmentation algorithm can be configured to perform segmentation of data provided by an intraoral scanner. The segmentation process may include segmenting the extraoral image and / or intraoral image data into multiple segments representing object types, such as by applying a pixel-based segmentation technique, a block-based segmentation technique, or any other conventionally known segmentation technique, selecting a segment associated with multiple different features from the multiple segments; calculating an aggregation of the multiple different features to generate a single feature describing the object type, and identifying the type of the object based on the aggregation by comparing the aggregation with features corresponding to known object types from an object-type-specific trained data set. In this way, teeth and bones in view of the extraoral image data and gums and teeth in view of the intraoral image data can be separated from each other.

[0053] In another example, segmentation of extraoral image data and intraoral scan data can be provided by a segmentation algorithm configured for both scan types (i.e., intraoral scanning and extraoral imaging). Segmentation provides the ability to distinguish teeth from bone in extraoral image data and to distinguish teeth from soft tissue in intraoral scan data. This improves the reliability of the training data set and the accuracy of the diagnostic data. Without segmentation, the model can combine and align the extraoral image data and intraoral scan data based on random data used as reference points or surfaces.

[0054] Thus, the pre-processing step may further comprise the step of segmenting the extraoral image data into teeth and bones. This may allow the creation of a 2D surface representation of the extraoral image data, wherein bone levels and corresponding bone loss may be assessed. Thus, the extraoral image data to be used in the training dataset may be raw extraoral image data and / or segmented extraoral image data. In both cases, the bone level information of the candidate patient may be assessed by a dental practitioner in a manual annotation process and / or identified, for example, using a trained neural network configured to identify bone levels in extraoral image data.

[0055] In one example, the combination of extraoral image data and intraoral scan data can use the segmented teeth of the extraoral image data and the segmented teeth of the intraoral scan data as reference surfaces. The combination of extraoral image data and intraoral scan data can align the intraoral scan data with the extraoral image data based on using the segmented teeth of the extraoral image data and the intraoral scan data as reference surfaces. Both the extraoral image device and the intraoral scanner provide reliable tooth scan data, and using the segmented teeth as reference surfaces will produce even more reliable training data sets and even more accurate diagnostic data sets. Extraoral image devices cannot provide reliable soft tissue data, while intraoral scanners cannot provide reliable bone data, and if segmented soft tissue or segmented bone is used as a reference surface, less reliable training data and less accurate diagnostic data will be produced compared to the previous example. In other words, the extraoral image data is suitable for obtaining periodontal structure information, such as information about bones, such as bone levels and estimated bone loss, while the intraoral scan data is suitable for obtaining soft tissue diagnoses, such as gum recession, gum redness, etc. The use of these two types of data in data preprocessing to generate training data to be used in the machine learning model ensures that periodontal structures that are typically not identifiable in intraoral scan data can be identified in such data.

[0056] Furthermore, if soft tissue information from the intraoral scan data is additionally used, the development of more periodontal diseases can be assessed. That is, the intraoral scan data can provide information about, for example, gum recession and gingivitis, which are at least one clinical precursor to the development of, for example, bone loss. From a clinical perspective, inflammation, such as gingivitis and associated gum recession are often the main causes of tooth bone loss. Therefore, inflammation and changes occurring in the soft tissue that can be identified in the intraoral scan can be associated with bone loss identified from the extraoral image data. Therefore, the relationship between soft tissue findings (i.e., in the intraoral scan data) and hard tissue findings (i.e., in the extraoral scan data) can be defined to train a neural network, thereby being able to estimate, for example, the average value of bone loss per tooth from the extraoral image data and to estimate, for example, the associated average value of gum recession and / or gingivitis from the intraoral scan data. In this way, a mathematical relationship can be created that associates findings of bone loss with gum recession or gingivitis. Such a mathematical relationship can be input as a parameter in the form of, for example, a label into a machine learning model during a training process, thereby creating a machine learning model that can output average estimates of bone loss and associated gum recession and / or gingivitis in intraoral scan data.

[0057] Based on the segmented tooth data provided by segmentation of the intraoral scan data and / or the extraoral image data, the cemento-enamel junction can be determined for each segmented tooth. The cemento-enamel junction (CEJ) represents the anatomical boundary between the crown and root surfaces of the tooth, and is defined as the area of ​​union of cementum and enamel at the cervical region of the tooth. The bone margin can be determined based on the segmented bone provided by the segmentation of the extraoral image data. Determine one or more periodontal structures including a bone level, which is determined for each segmented tooth in the extraoral image data by the distance between the bone margin determined by the segmented bone and the cemento-enamel junction, and wherein the cemento-enamel junction is determined by the segmented tooth. In other words, the computer-implemented method may include determining one or more periodontal structures, including determining the bone level of each tooth in the extraoral image data, wherein the bone level is determined by the distance between the identified bone reference point of the extraoral image data and the identified cemento-enamel junction, and wherein the determined bone level for the extraoral image data is merged with the intraoral scan data. The cemento-enamel machine learning algorithm can be configured to determine the cemento-enamel junction based on extraoral image data. The combination of the extraoral image data with the intraoral scan data can include aligning the determined distance for the extraoral image data with the intraoral scan data. Therefore, the segmentation of the extraoral image data can help, for example, to perform data labeling from an automated process and / or data labeling performed manually by a dental practitioner. The identification of the CEJ can be input as a label into the machine learning model to ensure that the machine learning model is able to predict the location of the CEJ in new, unseen intraoral scan data. Therefore, the CEJ is considered to form part of the previously described periodontal structure. In addition, the mentioned bone margins can be considered to be labels and / or annotations identified as periodontal structures in the extraoral image data for input into the machine learning model.

[0058] Determining one or more periodontal structures for each of a plurality of candidate patients may include determining a first relationship between segmented teeth and bones of the extraoral image data using one or more reference sites for each identified tooth; and combining the extraoral image data with the intraoral scan data by aligning the first relationship with the intraoral scan data. The first relationship may include a distance between a tooth reference point disposed on a tooth (which may be a segmented tooth) of the extraoral image data and a bone reference point disposed on a bone that is closest to the tooth reference point or at least near the tooth reference point. The first relationship may be considered as a label of the periodontal structure that is input into a machine learning model during a training phase.

[0059] Determining one or more periodontal structures for each of a plurality of candidate patients includes identifying a plurality of teeth in the extraoral image data; and determining one or more reference locations for each identified tooth of the extraoral image data; and determining one or more periodontal structures at each of the one or more reference locations. In one embodiment, the determination of the periodontal structures may be found on a dataset comprising segmenting the extraoral image data into segmented teeth of the extraoral image data.

[0060] The one or more reference sites may include tooth reference points, bone reference points and / or soft tissue points as described above. Therefore, in one embodiment, the method includes determining tooth reference points and bone reference points for each reference site using extraoral image data. All of these mentioned reference sites can be considered as labels and / or annotations suitable for input into the machine learning model described herein. The reference site can indicate where one or more periodontal structures have been determined, that is, the location of the teeth where the periodontal structures are found. In order to improve the accuracy of determining one or more periodontal structures around and / or at the teeth, at least three reference sites can be used, and more ideally, about 6 reference sites. The reference sites can be arranged around at least one of the segmented teeth. In more detail, preferably at least 3 sites on the buccal side of the tooth and at least 3 sites on the lingual side of the tooth. This ensures that potential conditions (such as bone loss) can be identified on all sides of the tooth to assess the level of condition development on the entire tooth. Therefore, when preprocessing data for machine learning model training, annotations and / or labels for the extraoral image data can be given for all 6 sites of the tooth to form labels to be input into the machine learning model during the training phase. These tags can be associated with the intraoral scan data, as described previously, for example in a matrix transformation.

[0061] It should be noted that in the context of the present application, a tooth reference point can be a placement of a CEJ identified from extraoral image data, and a bone reference point can be, for example, a bone edge identified from extraoral image data, and each of the tooth reference point and / or bone reference point is considered to be a label used as input to a machine learning model.

[0062] In order to improve the accuracy and reliability of the method and machine learning model, the training data set can be customized for the patient. The method can also include receiving intraoral scan data of the patient at a first time and over a period of time, and customizing the training data set for the patient by including the intraoral scan data of the patient. In other words, the customization can be configured as an update to an already trained machine learning model to include information collected from the patient scan during a clinical visit. In this way, the clinical data identified by the dentist in the patient's intraoral scan can be directly transferred to the machine learning model to update the training data with the new information. In this way, the machine learning model will be continuously updated with new information and will automatically learn other clinical aspects of the dental condition. The customization described throughout can generally be viewed as automatically updating a previously trained neural network using additional information obtained, for example, during further scanning of the patient and / or when new extraoral image data is acquired. Customization can include utilizing unsupervised learning, wherein the neural network used is configured to automatically update the network without knowledge of labels and annotations.

[0063] To even further improve the accuracy and reliability of the method and machine learning model, the training data set can be customized for the patient by receiving intraoral scan data of the same patient at a second time and during that period of time. Improvements will continue as long as the method and machine learning model receive more intraoral scan data of the same patient during that period of time.

[0064] To even further improve the accuracy and reliability of the method and machine learning model, a training data set can be customized for a patient by receiving multiple intraoral scan data of the same patient, wherein the intraoral scans used to generate the multiple intraoral scan data are performed over a period of time. The multiple intraoral scan data received over time provides reliable detection of bone loss over time because differences in bone levels between two or more scans can be detected.

[0065] The machine learning model can be configured to generate a first training data set configured for a single intraoral scan for each of a plurality of candidate patients, and a second training data set configured for multiple scans for each of the plurality of candidate patients. For example, if multiple intraoral scans are being performed on the same patient, the machine learning model is configured to use the second training data set to provide a diagnostic data set.

[0066] The method may also include receiving intraoral scan data of the patient at a first time and over a period of time, receiving intraoral scan data of the patient at a second time and over the period of time, aligning the intraoral scan data received at the first time and the second time, and customizing a training data set for the patient based on differences in the alignment of the intraoral scan data received at the first time and the second time. Multiple intraoral scan data received over time (e.g., at a first time and a second time) provides reliable detection of bone loss over time because differences in bone levels between two or more scans can be detected. It is at least believed that the presence or absence of bone loss can be detected in this manner.

[0067] In order to even further improve the accuracy and reliability of the method and machine learning model, it would be beneficial to distinguish the scan data that the method should use to train the training data set. For example, the scan data may be of lower quality, which would result in incorrect training of the training data set, resulting in reduced accuracy and reliability of the method. Therefore, it would be advantageous if the method also included determining whether to customize the training data for the patient based on one or more criteria, such as the quality of the patient's intraoral scan data and the degree of mismatch between the patient's intraoral scan data and the training data set.

[0068] In one embodiment, a machine learning model including training data can be configured to provide a patient's diagnostic data set by processing the patient's intraoral scan data based on the training data set. The method can also include receiving patient information of the patient, and can determine the diagnostic data set by processing the patient's intraoral scan data and the patient information based on the training data set that also includes patient information of multiple candidate patients. Including patient information necessarily leads to a more accurate diagnostic data set. The diagnostic data set can include one or more periodontal structures determined by processing the patient's intraoral scan data based on the training data set.

[0069] In order to improve the accuracy of the diagnostic data set, the method may also include determining the periodontal structure in the following manner: receiving intraoral scan data of the patient at a first time and over a period of time; segmenting the intraoral scan data received at the first time into teeth and soft tissue based on an intraoral segmentation algorithm; aligning the segmented teeth of the intraoral scan data received at the first time with the segmented teeth of the training data set; retrieving at least segmented bones corresponding to the segmented teeth of the training data set from the training data set; combining the segmented bones with the segmented teeth of the intraoral scan data received at the first time; and determining one or more periodontal structures based on the retrieved segmented bones and the segmented teeth of the patient's intraoral scan data.

[0070] To even further improve the accuracy of the diagnostic data, the method may also include receiving intraoral scan data of the patient at a second time and within the period of time, segmenting the intraoral scan data received at the second time into teeth and soft tissue based on an intraoral segmentation algorithm, aligning the segmented teeth of the intraoral scan data received at the second time with the segmented teeth of the training data set, determining a difference in the alignment of the segmented teeth and soft tissue of the intraoral scan data received at the first time and the second time, retrieving at least a segmented bone corresponding to the determined difference in alignment from the training data, combining the segmented bone with the segmented teeth of the intraoral scan data received at the second time, and determining one or more periodontal structures based on the retrieved segmented bone and the segmented teeth of the intraoral scan data of the patient. By receiving multiple intraoral scan data at different times and within the period of time, a more customized training data set will be generated, which will ultimately generate a more accurate diagnostic data set. The period of time should exceed days, weeks, or years. The time between the first time and the second time should exceed days, weeks, or years.

[0071] Determining one or more periodontal structures of a patient may include determining one or more reference locations for each identified tooth of segmented teeth of intraoral scan data received at a first time or a second time, and determining one or more periodontal structures of the patient by associating the intraoral scan data received at the first time or the second time with a training data set at each of the one or more reference locations.

[0072] Determining one or more periodontal structures at each of one or more reference locations may include identifying a cemento-enamel junction for each segmented tooth of an intraoral scan data of a patient, determining a distance between the cemento-enamel junction and segmented tissue surrounding each of the segmented teeth, and associating the distance with a training data set to determine the one or more periodontal structures.

[0073] Determining one or more periodontal structures at each of one or more reference sites may include: determining a relationship between a patient's segmented teeth and soft tissue, associating the relationship with training data, which includes correlations between the relationship between segmented teeth and segmented bones provided by extraoral image data of multiple candidate patients and the relationship between segmented teeth and segmented soft tissue provided by intraoral scan data of multiple candidate patients.

[0074] One or more periodontal structures identified using the trained machine learning model can be presented on a graphical user interface, which can include presenting a 2D model or a 3D model of the oral anatomy along with the identified periodontal structures determined by the diagnostic data. Thus, the analyzed data of the intraoral scan input to the machine learning model can be used in a variety of ways: First, the computer system can use one or more presentation techniques to present the analyzed data; for example, various types of user interfaces, graphical representations, etc. can be used to efficiently present the data and quickly alert the professional to potential areas of interest, signaling on a monitor potential detected features that require the user's immediate attention. In another example, the graphical user interface can include a dental chart on which one or more periodontal structures identified from the machine learning model are visualized.

[0075] One or more periodontal structures can be visualized at one or more reference sites. One or more periodontal structures can be visualized with color or text. If visualization criteria are met, one or more periodontal structures can be visualized. The visualization criteria can define that the bone level is above a threshold level, and the threshold level can be between 2 mm and 3 mm, around 3 mm, or above 3 mm. The bone level can be determined between the CEJ and the bone (e.g., the bone edge identified by the segmented bone).

[0076] When trained on training data as described herein, the machine learning model described herein can be used to provide a diagnostic dataset. The trained machine learning model can be used in different clinical processes, as described above. First, the machine learning model can be used in a cross-sectional approach, in which the patient is scanned at the first clinical visit using an intraoral scanner. In this case, the computer-implemented method described herein provides a diagnostic dataset for the patient by: receiving intraoral scan data of the patient; inputting the intraoral scan data into the trained machine learning model described herein; and receiving the patient's diagnostic dataset from the trained machine learning model. The diagnostic dataset as an output of the machine learning model may include one of the aforementioned probability maps, segmentation maps, etc. that describe the input intraoral scan data including predictions of one or more periodontal structures. The diagnostic dataset can be visualized as described above.

[0077] In a second use of the machine learning model, the patient is scanned at a first visit, as described above, and scanned again at a second clinical visit, where the first visit at the first time is different from the second visit at the second time. At both visits, the intraoral scans obtained are input into the machine learning model to create a diagnostic data set reflecting the periodontal condition at the respective visits. Therefore, at the second visit, there are two diagnostic data sets obtained at two different times in the data storage. In order to assess whether the periodontal structure has changed from the first visit to the second visit, at least two diagnostic data sets are compared to each other. This can provide a comparative diagnostic data set that describes changes and potential progression in any of the identified periodontal structures of the second scan of the patient at the second time.

[0078] In one example, second scan data obtained at a second time may be used for customization of the machine learning model previously described.

[0079] In another example, the machine learning model may also be trained based on a comparison between the first and second visit scans and the diagnostic dataset output so as to be able to directly output periodontal changes through the diagnostic dataset.

[0080] The method and machine learning model can be configured to be implemented into a computing device, such as a laptop, PC, tablet, smartphone, etc.

[0081] The computer program may comprise instructions which, when a computer executes the program, cause the computer to perform the method.

[0082] The dental scanning system may include an apparatus for performing the method. Extraoral image data of multiple candidate patients and / or intraoral scan data of multiple candidate patients may be stored on a server, a memory of a computer, or a cloud server portion of the dental scanning system. The dental scanning system may include an extraoral image device interface configured to receive extraoral image data, and the system may also include an intraoral scanner interface configured to receive intraoral scan data. The two interfaces may share one interface. The interface may be wired and / or wireless. The training data set may be stored on a server, a memory of a computer, or a cloud server. The computer may be configured to perform operations on one or more artificial techniques, including one or more machine learning models, artificial networks, which can be used to analyze the combined extraoral and intraoral data and present the resulting output.

[0083] A data processing device may comprise means for performing the method.

[0084] The computer-readable storage medium may include instructions that, when executed by a computer, cause the computer to perform the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Aspects of the present disclosure may be best understood from the following detailed description in conjunction with the accompanying drawings. The drawings are schematic and simplified for clarity, and they only show details to improve understanding of the claims, while other details are omitted. Throughout, the same reference numerals are used for identical or corresponding parts. The various features of each aspect may each be combined with any or all of the features of the other aspects. These and other aspects, features and / or technical effects will be apparent and clarified from the illustrations described below, in which:

[0086] Figure 1A and 1B A computer-implemented method is shown;

[0087] Figure 2 Methods including machine learning models are shown;

[0088] Figures 3A-3D Different examples of how the method can generate training data sets or diagnostic data are shown;

[0089] Figure 4 An example of a method in which a training data set is customized is shown;

[0090] Figure 5 An example of a method in which a training data set is customized is shown;

[0091] Figure 6 Another example of this method is shown;

[0092] Figures 7A-7C Another example of this method is shown;

[0093] Figures 8A-8C A graphical visualization of the periodontal structure is shown;

[0094] Fig. 9 A dental scanning system is shown;

[0095] Fig.10 Combining extraoral and intraoral data to create a training data set is shown;

[0096] Fig.11 shows the machine learning model used in the training phase, and

[0097] Fig.12 The use of a trained machine learning model to determine changes in periodontal structure over time at two different clinical visits is shown. DETAILED DESCRIPTION

[0098] The detailed description set forth below in conjunction with the accompanying drawings is intended to describe various configurations. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be appreciated by those skilled in the art that these concepts can be practiced without these specific details. Several aspects of the devices, systems, media, programs, and methods are described by various boxes, functional units, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as "elements"). Depending on the specific application, design constraints, or other reasons, these elements may be implemented using electronic hardware, computer programs, or any combination thereof.

[0099] Electronic hardware may include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described in the present disclosure. Computer programs should be broadly interpreted to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0100] Scanning for providing extraoral image data may include scanners based on ionizing radiation technology, such as X-ray or CBCT scanners. Intraoral scan data may be obtained by a dental scanning system, which may include an intraoral scanning device, such as the TRIOS series scanners (i.e., handheld intraoral scanners) from 3Shape A / S or a laboratory-based scanner, such as the E series scanners from 3Shape A / S. The scanning device described herein may adopt scanning principles such as triangulation-based scanning, confocal scanning, focused scanning, ultrasonic scanning, X-ray scanning, stereoscopic vision, motion recovery structure, optical coherence tomography (OCT) or any other scanning principle. In an embodiment, the scanning device operates by projecting a pattern and translating the focal plane along the optical axis of the scanning device and capturing multiple 2D images at different focal plane positions so that each captured 2D image series corresponding to each focal plane forms a 2D image stack. The acquired 2D image is also referred to herein as an original 2D image, where original in this context means that the image has not been image processed. The focal plane position is preferably shifted along the optical axis of the scanning system so that the 2D images captured at multiple focal plane positions along the optical axis form the 2D image stack (also referred to herein as sub-scans) for a given view of the object (i.e., for a given arrangement of the scanning system relative to the object). After moving the scanning device relative to the object or imaging the object in a different view, a new 2D image stack for the view can be captured. The focal plane position can be changed by at least one focusing element (e.g., moving a focusing lens). The scanning device is typically moved and tilted during a scanning session so that at least some sets of sub-scans overlap at least partially so that stitching can be achieved in post-processing. The result of the stitching can be a digital 3D representation of a surface that is larger than a single sub-scan can capture, i.e., larger than the field of view of the 3D scanning device. Stitching, also known as registration, works by identifying overlapping areas of the 3D surface in the various sub-scans and transforming the sub-scans to a common coordinate system so that the overlapping areas match, ultimately producing a digital 3D model. An iterative closest point (ICP) algorithm can be used for this purpose. Another example of a scanning device is a triangulation scanner, where a time-varying pattern is projected onto the dental object and a sequence of images of different pattern configurations is acquired by one or more cameras positioned at an angle relative to the projector unit.

[0101] A scanning device, in particular an intraoral scanning device, may include one or more light projectors configured to generate an illumination pattern during a scanning session to be projected onto a three-dimensional dental object. (One or more) light projectors preferably include a light source, a mask having a spatial pattern, and one or more lenses, such as a collimating lens or a projection lens. The light source may be configured to generate light of a single wavelength or a combination of wavelengths (monochromatic or polychromatic). The wavelength combination may be generated by using a light source configured to generate light containing different wavelengths (e.g., white light). Alternatively, (one or more) light projectors may include multiple light sources, such as LEDs, which individually generate light of different wavelengths (e.g., red, green, and blue), which may be combined to form light containing different wavelengths. Therefore, the light generated by the light source may be defined by different wavelength ranges defining a wavelength of a specific color or defining a color combination (e.g., white light). In an embodiment, the scanning device includes a light source configured to excite a fluorescent material of a tooth to obtain fluorescent data from a dental object. Such a light source may be configured to generate a narrow range of wavelengths. In another embodiment, the light from the light source is infrared (IR) light, which is capable of penetrating dental tissue. The (one or more) light projectors can be DLP projectors that use a micromirror array to generate a time-varying pattern, or a diffractive optical element (DOF), or a back-illuminated mask projector, in which a light source is placed behind a mask having a spatial pattern so that the light projected onto the surface of the dental object is patterned. The back-illuminated mask projector can include a collimating lens for collimating the light from the light source, the collimating lens being placed between the light source and the mask. The mask can have a checkerboard pattern so that the generated illumination pattern is a checkerboard pattern. Alternatively, the mask can have other patterns, such as lines or dots, etc.

[0102] The scanning device preferably also includes an optical component for directing light from the light source to the surface of the dental object. The specific arrangement of the optical component depends on whether the scanning device is a focused scanning device, a scanning device using triangulation, or any other type of scanning device. The same applicant further describes a focused scanning device in EP2442720B1, which is incorporated herein in its entirety. Therefore, the intraoral scanning device described herein is preferably a handheld intraoral scanner that a dentist can operate during a clinical visit to scan a patient's oral cavity including teeth.

[0103] Using the optical components of the scanning device, light reflected from the dental object in response to illumination of the dental object is directed to (one or more) image sensors. The (one or more) image sensors are configured to generate multiple images based on the incident light received from the illuminated dental object. The image sensor can be a high-speed image sensor, such as an image sensor configured to acquire images with an exposure time of less than 1 / 1000 second or a frame rate of more than 250 frames per second (fps). For example, the image sensor can be a rolling shutter (CCD) or a global shutter sensor (CMOS). The (one or more) image sensor can be a monochrome sensor, including a color filter array, such as a Bayer filter and / or additional filters, which can be configured to substantially remove one or more color components from the reflected light before converting the reflected light into an electrical signal, and retain only other unremoved components. For example, such additional filters can be used to remove a portion of the white light spectrum, such as a blue component, and retain only red and green components in the signal generated in response to exciting the fluorescent material of the tooth.

[0104] The dental scanning system preferably also includes a processor configured to generate scan data (e.g., extraoral image data and / or intraoral scan data) by processing a two-dimensional (2D) image acquired by the scanning device. The processor may be part of the scanning device. As an example, the processor may include a field programmable gate array (FPGA) and / or an advanced RISC machine (ARM) processor located on the scanning device. The scan data includes information related to a three-dimensional dental object. The scan data may include any of the following: a 2D image, a 3D point cloud, depth data, texture data, intensity data, color data, and / or a combination thereof. As an example, the scan data may include one or more point clouds, each of which includes a set of 3D points describing a three-dimensional dental object. As another example, the scan data may include images, each of which includes image data, such as image data described by image coordinates and a timestamp (x, y, t), wherein depth information may be inferred from the timestamp. The (one or more) image sensors of the scanning device may acquire multiple original two-dimensional images of the dental object in response to illuminating the object using one or more light projectors. Multiple original two-dimensional images may also be referred to as two-dimensional image stacks herein. The two-dimensional image may then be provided as an input to a processor that processes the two-dimensional image to generate scan data. The processing of the two-dimensional image may include a step of determining which portion of each two-dimensional image is in focus in order to infer / generate depth information from the image. The depth information may be used to generate a three-dimensional point cloud, which includes a set of three-dimensional points in space, for example, described by Cartesian coordinates (x, y, z). The three-dimensional point cloud may be generated by a processor or other processing unit. Each two-dimensional / three-dimensional point may also include a timestamp indicating the recording time of the two-dimensional / three-dimensional point, i.e., which image in the two-dimensional image stack the point originated from. The timestamp is associated with the z coordinate of the 3D point, i.e., the z coordinate can be inferred from the timestamp. Therefore, the output of the processor is scan data, and the scan data may include image data and / or depth data, for example, described by image coordinates and timestamps (x, y, t) or alternatively described as (x, y, z). The scanning device may be configured to transmit other types of data in addition to the scan data. Examples of data include 3D information, texture information (e.g., infrared (IR) images), fluorescent images, reflected color images, x-ray images, and / or combinations thereof.

[0105] Obviously, the scanning devices described herein (e.g., extraoral scanning devices and intraoral scanning devices) can be used to generate extraoral image data and intraoral scan data, respectively. It is understood that the extraoral image data can be used together with the intraoral scan data obtained substantially simultaneously with the extraoral image data to generate a training data set. The training data set can be generated during the training phase and can include one or more preprocessing steps configured to adjust the data to a data format suitable for input into a machine learning model, which will be described below.

[0106] Figure 1A and 1B Computer-implemented methods (100, 200) for generating a training data set for a machine learning model and for providing a diagnostic data set for a patient based on the training data set are respectively shown. Figure 1A A computer-implemented method 100 for generating a training data set for a machine learning model 1 is shown, wherein the method comprises: receiving 100A extraoral image data of a plurality of candidate patients provided by an extraoral image device; determining 100B one or more periodontal structures (in Figure 1A 100C receives intraoral scan data of multiple candidate patients provided by an intraoral scanner, and generates a 100D training data set by combining extraoral image data with the intraoral scan data of each of the multiple candidate patients to align one or more periodontal structures with the intraoral scan data.

[0107] As previously discussed, combination is considered to include a process of image alignment in which elements in one image are mapped into meaningful correspondence with elements in a second image. In other words, combining extraoral image data with intraoral scan data includes merging data structures together to transfer information collected from the extraoral dataset to the intraoral dataset. In the context of machine learning, it should be understood that the extraoral data information can be used to annotate and / or label the intraoral dataset with one or more periodontal structure information determined. That is, because the intraoral scan data does not provide information about periodontal structures, the dataset cannot be trained using only the intraoral scan data. Instead, the extraoral scan data is collected to collect information about periodontal structures so that the intraoral scan data can be annotated and / or labeled using that information.

[0108] Alternatively, the method 100 also includes receiving 100E candidate patient information for a plurality of candidate patients, and generating a training data set by combining the received patient information for each of the plurality of candidate patients with combined extraoral image data and intraoral scan data for each of the plurality of candidate patients.

[0109] Figure 1B A computer-implemented method 200 is shown, which includes receiving 200A intra-oral scan data of a patient and processing 200B the intra-oral scan data of the patient based on a training data set. The method 200 also includes outputting 200C diagnostic data. Thus, Figure 1B It shows that the machine learning model is based on Figure 1A Here, a computer uses a machine learning model to provide a diagnostic data set from unseen intraoral scan data obtained by scanning patients during clinical visits. Figure 1A, it is shown how to obtain a diagnostic dataset by the method: receiving intraoral scan data of a patient; inputting the intraoral scan data into a trained machine learning model; and receiving (in the form of output) the patient's diagnostic dataset from the trained machine learning model.

[0110] The diagnostic data can be visualized on a graphical user interface or further analyzed by other processing units.

[0111] exist Figure 1B In another illustrated embodiment, the method further comprises receiving 200D patient information, and determining a diagnostic data set by processing 200B the intra-oral scan data and the patient information of the patient based on the training data set.

[0112] As described herein, outputting a patient's diagnostic data set includes one or more periodontal structures determined by processing the patient's intraoral scan data based on a training data set forming a machine learning model. Thus, the diagnostic data set includes a prediction of the presence of a periodontal structure on one or more teeth in the intraoral scan data and the location at which the prediction exists.

[0113] Periodontal structure may include determination of bone level or bone loss for a plurality of candidate patients.

[0114] Now go to Fig.10 , the preprocessing 300 of the extraoral image data and the intraoral image data for training the machine learning model is shown in more detail. It is shown here that the extraoral image data 10 includes bone, root and tooth information, while the intraoral image scan data 20 only includes surface information in the form of the surface of the teeth and gums. According to the preprocessing described herein, the two data components (i.e., information collected from two different data sets) can be combined to form a training data set 100D for the machine learning model. That is, the preprocessing step includes combining the two data types in the following manner: identifying 300F, 300G corresponding teeth in the extraoral image data and the intraoral scan data; and determining one or more periodontal structures 3 in the extraoral image data 302; and using an image alignment process to map or associate the determined periodontal structures with the corresponding teeth in the intraoral scan data. For illustrative purposes only, the alignment process is not performed in detail. Fig.10 The image warping process is shown in Figure 2. In image warping, two datasets are merged together, such as Fig.10 310 in FIG. However, as previously mentioned, such a distortion may not necessarily be necessary, as the information may be contained in the mapping matrix which is not necessarily shown.

[0115] In any case, it should be understood that the periodontal structures (in Fig.10 The dashed lines in the figure can be used as annotations and / or labels as input to machine learning models. Fig.11As shown, the machine learning model 300 is configured to use intraoral scan data 401 as input to, for example, a convolutional neural network during a training phase. The convolutional neural network then generates predictions 403, which are input to a loss function 404. The loss function simultaneously receives extraoral scan data 402 as annotations and / or labels, which provide these annotations and / or labels to the loss function, i.e., the true values ​​that the neural network should be able to estimate. Based on how the machine learning model manages to predict the true values, the machine learning model is updated until a satisfactory prediction is achieved. This process during the training phase is run on all candidate patient data that form part of the training data set. When the machine learning model obtains satisfactory predictions, the model is able to predict diagnostic data sets for new, unseen intraoral scan data, such as Fig.11 As shown in the dashed box.

[0116] Figure 2 An example of a method 100 including a machine learning model 1 is shown. In this example, the method includes segmenting extraoral image data 10 of a plurality of candidate patients into teeth 2 and bones 3 based on an extraoral segmentation algorithm, and segmenting intraoral scan data 20 of a plurality of candidate patients into teeth 5 and soft tissue 4 based on an intraoral segmentation algorithm. In this example, the segmented teeth (2, 5) are used as reference surfaces for a combination or alignment 100D of the extraoral image data 10 and the intraoral scan data 20. Furthermore, in this example, the extraoral image data 10 includes segmented bone information 3 that is not part of the intraoral scan data 20, and thus the training data set 30 includes data related to the segmented bones 3 aligned with the segmented soft tissue 4 and the segmented teeth 5 of the intraoral scan data 20 of the plurality of candidate patients.

[0117] In the example given, a training dataset can be created that includes extraoral image data, i.e., periodontal structures identified in the extraoral image data, as annotations and / or labels for the intraoral scan data. This ensures that the machine learning model can be trained on the intraoral scan data while being able to detect periodontal structures. This is not possible with the intraoral scan data alone, but by combining the extraoral information with the intraoral scan data, this is possible.

[0118] Figure 3A and 3B Different examples of how the method 100 generates a training data set by aligning one or more periodontal structures with intraoral scan data of multiple candidate patients are shown. Figure 3A and 3BIt can be seen that determining one or more periodontal structures for each of a plurality of candidate patients includes identifying a plurality of teeth in extraoral image data; determining one or more reference locations for each identified tooth in the extraoral image data; and determining one or more periodontal structures at each of the one or more reference locations, all of which will be described in more detail below. Figure 3C and 3D Various examples are shown regarding how method 200 may generate diagnostic data.

[0119] exist Figure 3A , an example of segmenting the extraoral image data 10 into segmented teeth (2A, 2B) and segmented bones 3 (which includes at least a bone edge 41) is shown. In this particular example, by performing surface analysis on the teeth (2A, 2B), it is identified that each segmented tooth (2A, 2B) has a cementoenamel junction 40. At least three reference sites (31A, 31B, 31C) have been determined and arranged on the cementoenamel junction 40, and another three reference sites (32A, 32B, 32C) have been arranged on the bone edge 41 (i.e., the segmented bone 3). In this example, one or more periodontal structures include a bone level, which is determined for each segmented tooth of the extraoral image data 10 by determining the distance (33A, 33B, 33C) between the bone edge 41 and the cementoenamel junction 40. In this example, three distances (33A, 33B, 33C) are determined between three reference sites (31A, 31B, 31C) and three other reference sites (32A, 32B, 32C). The combination 100D of the extraoral image data 10 with the intraoral scan data 20 includes combining the segmented bone 3 with the segmented teeth (5A, 5B) and the segmented soft tissue 4, which then results in alignment of the bone levels with the segmented teeth (5A, 5B) and the segmented soft tissue 4. If one or more of the distances (33A, 33B, 33C) is greater than 1 mm, greater than 2 mm, greater than 3 mm, greater than 4 mm, or greater than 5 mm, bone loss is diagnosed. In this example, one or more periodontal structures include the determined bone levels and bone loss. Therefore, if Figure 3A As shown, the method includes determining a tooth reference point (40) and a bone reference point (41) for each reference site (31A, 31B, 31C). In this example, for each reference site (31A, 31B, 31C), a first relationship between the tooth and the bone in the extraoral image data is determined, wherein the first relationship includes a distance (33A, 33B, 33C) between the tooth reference point (i.e., CEJ) (40) and the bone reference point (i.e., bone edge) (41). The determined distances, in this example, are three distances (33A, 33B, 33C), constituting the bone level of each reference site.

[0120] In another example, in Figure 3BIn the embodiment of the present invention, determining one or more periodontal structures of each of a plurality of candidate patients includes determining a first relationship between segmented teeth (2A, 2B) and segmented bones 3 of the extraoral image data 10. In this example, the extraoral image data has been segmented into segmented bones 3 and segmented teeth (2A, 2B), and the segmented bones 3 include at least one bone edge 41. At least three reference sites (31A, 31B, 31C) have been determined and arranged on tooth 2A of the segmented teeth (2A, 2B), and another three reference sites (32A, 32B, 32C) have been arranged on the bone edge 41 (i.e., the segmented bone 3). In this example, the one or more periodontal structures include a bone level, which is determined for each segmented tooth of the extraoral image data 10 by determining a distance (33A, 33B, 33C) between the bone edge 41 and the tooth 2A. In this example, three distances (33A, 33B, 33C) are determined between three reference sites (31A, 31B, 31C) and three other reference sites (32A, 32B, 32C). Combining 100D the extraoral image data 10 with the intraoral scan data 20 includes combining the segmented bone 3 with the segmented teeth (5A, 5B) and the segmented soft tissue 4, which then results in alignment of the bone levels with the segmented teeth (5A, 5B) and the segmented soft tissue 4.

[0121] For both examples, if one or more distances (33A, 33B, 33C) are above a threshold, bone loss is diagnosed. In this example, the one or more periodontal structures include the determined bone level and bone loss. The threshold may be between 3 and 5 mm, between 5 and 8 mm, between 3 and 10 mm, or above 10 mm.

[0122] The examples described above explain the use of segmentation to determine periodontal structures and generate combined data for training data sets. However, as described and discussed above, segmentation of the extraoral image data may not be required as long as a mapping matrix containing tooth information and associated determined periodontal structures is obtained. The examples given are for ease of illustration and should not be considered as the only possible way to implement the invention described herein.

[0123] exist Figure 3A and 3BIn the two examples shown, the combined 100D extraoral image data 10 and the intraoral scan data 20 use the segmented teeth (2A, 2B) of the extraoral image data 10 and the segmented teeth (5A, 5B) of the intraoral scan data 20 as reference surfaces. Alternatively, the combined 100D extraoral image data 10 and the intraoral scan data 20 align the intraoral scan data 20 with the extraoral image data 10 based on using the segmented teeth (2A, 2B) of the extraoral image data 10 and the segmented teeth (5A, 5B) of the intraoral scan data 20 as reference surfaces. Alternatively, the segmentation process can be omitted, and the training data set includes a mapping matrix that includes information about the teeth and the periodontal structures identified for each tooth. That is, the mapping matrix can include tooth numbers and corresponding periodontal structures determined for each tooth in the mapping matrix.

[0124] Figure 3C An example of a method 200 for outputting diagnostic data 60 of a patient based on a training data set 30 is shown. In this example, the method 200 receives 200A intra-oral scan data 50 of a patient, and it is segmented into segmented teeth (6A, 6B) and segmented soft tissue 7, and in this example, the segmented soft tissue 7 includes a soft tissue edge 43. The segmented teeth 6A are then aligned with the segmented teeth (5A, 5B) of the training data set 30. In this example, the training data set 30 includes the segmented teeth (5A, 5B) and the segmented bones 3. The segmented teeth (6A, 6B) of the intra-oral scan data 50 are then aligned 200B with the segmented teeth (5A, 5B) of the training data set 30, and then at least the segmented bones 3 are retrieved from the training data set 30, which correspond to the segmented teeth (5A, 5B) of the training data set 30.

[0125] Alignment of the training data set 30 with the intra-oral scan data 50 includes finding the best or optimal correlation of the segmented teeth (5A, 5B) of the training data set with the segmented teeth (6A, 6B) of the intra-oral scan data 50 of the patient's mouth.

[0126] The segmented bone 3 of the training data set 30 is then combined with the segmented teeth (6A, 6B) and the segmented soft tissue 7. In this example, the diagnostic data 60 includes the combined data and the periodontal structure, and the periodontal structure includes the bone level determined by the distance (33A, 33B, 33C) between the reference site (31A, 31B, 31C) arranged on the tooth 6A and the reference site arranged on the bone edge 41. In another example, the diagnostic data only includes the periodontal structure, which is the bone level in this example. Figure 3D Shown with Figure 3C A similar example in Figure 3D In the embodiment of the present invention, the distance between the cemento-enamel junction 40 and the bone edge 41 is determined (33A, 33B, 33C).

[0127] Figure 4 An example of method 100 is shown, where the training data set is being trained for a specific patient. In this example, the training data set is customized for a specific patient. In this example, the machine learning model is still trained 100D by extraoral image data and corresponding intraoral scan data of multiple candidate patients, and alternatively, is trained 100D by patient information. In addition, the training data set of the machine learning model 1 is customized for the patient by receiving 100F the intraoral scan data of the patient.

[0128] Figure 5 An example of a method 100 is shown that generates 100D a training data set and customizes 100G a training data set for a specific patient. In this example, the method includes receiving 100F intraoral scan data of a patient at a first time and within a period of time. Segmentation 100H is performed on the intraoral scan data to customize 100G a training data set. The method 100 also includes receiving 100J intraoral scan data of a patient at a second time and within the period of time. Segmenting 100K the intraoral scan data received at the second time, and aligning the segmented intraoral scan data received at the second time with the previously segmented intraoral scan data received at the first time. The 100G training data set is then customized based on the difference in alignment 100L. Alternatively, customization of the training data set is only allowed 100I based on one or more of the following criteria: the quality of the patient's intraoral scan data, and the degree of mismatch between the patient's intraoral scan data and the training data set.

[0129] As previously mentioned, customizations can be configured as updates to already trained machine learning models to include information collected from patient scans during clinical visits and reference these sections.

[0130] Figure 6An example of a method 200 is shown, which includes: receiving 200A intra-oral scan data of a patient at a first time and over a period of time; segmenting 200E the intra-oral scan data of the patient into teeth and soft tissue based on an intra-oral segmentation algorithm; aligning 200F the segmented teeth of the intra-oral scan data of the patient with the segmented teeth of a training data set; retrieving 200G at least segmented bones corresponding to the segmented teeth of the training data set from the training data set; combining 200H the segmented bones with the segmented teeth of the intra-oral scan data of the patient; and determining 200C one or more periodontal structures based on the retrieved segmented bones and the segmented teeth of the intra-oral scan data of the patient. Then, the patient is scanned again using the intra-oral scanner. Method 200 includes: receiving 200A intraoral scan data of a patient at a second time and within that period of time; segmenting 200E the intraoral scan data of the patient received at the second time into teeth and soft tissue based on an intraoral segmentation algorithm; aligning 200F the segmented teeth of the intraoral scan data received at the second time with the segmented teeth of a training data set; determining 200H the difference in alignment of the segmented teeth and soft tissue of the intraoral scan data received at the first time and the second time; retrieving 200I at least segmented bones corresponding to the determined alignment difference from the training data; combining 200J the segmented bones with the segmented teeth of the intraoral scan data received at the second time; and determining 200C one or more periodontal structures based on the retrieved segmented bones and the segmented teeth of the patient's intraoral scan data.

[0131] Fig. 7A A method 200 of outputting a diagnostic data set using a trained machine learning model is shown. The method shown in this example includes further determining one or more reference sites for each identified tooth of the segmented teeth of the intraoral scan data of the patient 200K, and determining one or more periodontal structures of the patient 200C by associating the intraoral scan data of the patient with the training data set at each of the one or more reference sites.

[0132] Figure 7B A method of outputting a diagnostic data set using a trained machine learning model is shown 200. The method shown in this example includes further determining 200L a relationship between segmented teeth and soft tissue of a patient, associating the relationship with training data 200C, which includes correlations between the relationship between segmented teeth and segmented bones provided by extraoral image data of multiple candidate patients and the relationship between segmented teeth and segmented soft tissue provided by intraoral scan data of multiple candidate patients.

[0133] Figure 7C 200 as described in any of the previous figures. Figure 7CThe machine learning model 1 in includes a first training data set 30A configured to receive a single intraoral scan of patient 200A, and a second training data set 30B configured to receive multiple scans of each of a plurality of patients 200M. For example, if multiple intraoral scans are performed on the same patient, the machine learning model 1 is configured to use the second training data set 30B to provide a diagnostic data set.

[0134] Figures 8A-8C An example of one or more periodontal structures 90 as output of a trained machine learning model being presented on a graphical user interface 500 is shown. The one or more periodontal structures may be visualized with color or text. Fig. 8A In , the periodontal structure 90 is visualized on a 3D model of the oral anatomy determined by the diagnostic data 60 , which includes the segmented teeth and soft tissue of the patient's intra-oral scan data 50 combined with the segmented bones retrieved from the training dataset 30 . Figure 8C A 2D model of the oral anatomy determined from the diagnostic data 60 is shown.

[0135] In one example, the diagnostic data set and intraoral scan data as output of the machine learning model can be post-processed into various forms for display in, for example, the graphical user interface 500. By using, for example, the output of the machine learning model through back propagation, the Fig. 8A As shown, the output of the machine learning model is visualized on a previously unseen image input. That is, previously unseen images in the form of intraoral scan data that the machine learning model has not seen before can be visualized in the graphical user interface 500. Fig. 8A In FIG. 6 , intraoral scan data 60 of the mandible is shown, and in FIG. Figure 8C Intraoral scan data of the maxilla is shown in FIG. Fig. 8A and 8C As shown, the periodontal structure 90 as the output of the machine learning model in the form of a diagnostic data set can be visualized at the location of the tooth at which the periodontal structure was determined by the machine learning model. Thus, an input image of, for example, the upper or lower jaw can be visualized and the output of the machine learning model fused to the image by, for example, image warping techniques to efficiently illustrate to the user that the periodontal structure was found at a particular input image.

[0136] Thus, by using a trained neural network (as described herein) on new image data, it is possible to assess whether a patient has periodontal disease without the use of radiation. Depending on the neural network used and the input data on which the neural network is trained, the assessment of periodontal disease can include identifying where bone loss exists on a tooth, the amount of bone level of a particular tooth, the age of the person, the sex of the person, and potential other diseases associated with a particular tooth of the input image data acquired from an intraoral scanner.

[0137] Figure 8B A dental chart is shown including periodontal structures labeled at reference sites as output of diagnostic data 60. This illustrates the possibility of transferring the output of a machine learning model to a periodontal billing system (e.g., in a practice management system) to ensure that patient logs are updated with the latest medical information.

[0138] As previously mentioned Fig.11 The trained machine learning model is configured to output a diagnostic data set comprising predictions of the presence and potential additional locations of periodontal structures identified in the input intraoral scan data. Fig.12 , the left side shows a first clinical visit 601, when a patient is scanned by an intraoral scanner, and the method described herein allows the use of a trained machine learning model 603 to identify the presence of periodontal structures in an intraoral scan 604 obtained from the patient. If the machine learning model 603 identifies periodontal structures, these periodontal structures are output as part of a diagnostic dataset 606. As can be seen from the illustrated diagnostic dataset 606, the machine learning model has discovered that the intraoral scan data 604, which is the input to the machine learning model 603, contains periodontal structures, such as Fig.12 606 is shown in dashed lines.

[0139] It should be noted that periodontal structures can be related to Figure 3A and 3B Any of the periodontal structures mentioned, with particular reference to the bone margin, CEJ and reference sites. Fig.12 The illustrations given in are only for illustrative purposes to facilitate understanding of the present disclosure. Therefore, it should be understood that the diagnostic data output as the machine learning model can be any one of a segmentation map, an object recognition map, a label classification, etc.

[0140] The diagnostic data set 606 provides the dental practitioner with a first assessment of the patient's periodontal disease at a first point in time without the need to use ionizing radiation to assess the patient's periodontal structure. The diagnostic data obtained at the first clinic visit 601 can be stored in the practice management system in association with the patient history file. When the patient visits the dental practitioner for a follow-up visit at a second clinic visit 602, the dental practitioner can obtain a new second visit scan 605 of the patient. Fig.12As shown, the new second visit scan 605 is configured to be input into the same machine learning model 603 that was previously run on the first visit scan data 604. The machine learning model 603 is configured to process the second visit scan data 605 to output a diagnostic data set that provides a prediction of the current state of the periodontal structure of the second visit scan data. By comparing two diagnostic data sets 606, 607 obtained at two different times (i.e., the first and second visits), the method described herein includes inputting the diagnostic data set alone or together with the intraoral scan data into a comparison processor, which is configured to compare the first and second diagnostic data sets 606, 607 to evaluate and output changes in periodontal structure over time. In this way, without the use of ionizing radiation, it is possible to evaluate whether the potential periodontal disease identified at the first clinical visit has progressed from the first clinical visit to the second clinical visit.

[0141] Now go to Fig. 9 , which shows a dental scanning system 200, which includes an apparatus for performing the method (100, 200). The system includes a computer 210, which has a wired or wireless interface with a server 215, a cloud server 220, an intraoral scanner 225, and / or an extraoral image device 230. Extraoral image data of multiple candidate patients and / or intraoral scan data of multiple candidate patients can be stored on the server 215, the memory of the computer 210, or the cloud server 220. The training data set can be stored on the server 215, the memory of the computer 210, or the cloud server 220.

[0142] Computer 210 may include a data processing device configured to perform the method ( 100 , 200 ).

[0143] The computer 210 may include a computer-readable storage medium configured to cause the computer 210 to perform the method ( 100 , 200 ).

[0144] Although some embodiments have been described and shown in detail, the present disclosure is not limited to these details, but can also be embodied in other ways within the scope of the subject matter defined in the appended claims. In particular, it should be understood that other embodiments can be utilized and structural and functional modifications can be made without departing from the scope of the present invention.

[0145] Benefits, other advantages, and solutions to problems have been described herein with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any (one or more) components / (one or more) units that may cause any benefit, advantage, or solution to occur or become more apparent should not be construed as key, essential, or basic features or components / elements of any or all claims or inventions. Therefore, the scope of the present invention is limited only by the appended claims, in which references to singular components / units / elements are not intended to mean "one and only one", unless expressly stated, but "one or more". A claim may refer to any of the preceding claims, and "any" should be understood to mean "any one or more" of the preceding claims.

[0146] It is intended that structural features of the apparatus described above (whether in the detailed description or in the claims) may be combined with steps of the method when appropriately replaced by corresponding procedures.

[0147] When used, the singular forms "a", "an" and "the / said" are also intended to include the plural forms (i.e., have the meaning of "at least one"), unless otherwise expressly stated. It should also be understood that when the terms "include", "comprises", "contains" and / or "covers" are used in this specification, the presence of the features, integers, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof is not excluded. It should also be understood that when an element is referred to as "connected" or "coupled" to another element, it can be directly connected or coupled to another element, but there can also be intermediate elements, unless otherwise expressly stated. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. As used herein, the term "and / or" includes any and all combinations of one or more related listed items. The steps of any disclosed method are not limited to the exact order described herein, unless otherwise expressly stated.

[0148] It should be understood that references throughout this specification to "one embodiment" or "embodiment" or "aspect" or features that "may" be included mean that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present disclosure. In addition, the particular features, structures, or characteristics may be combined in one or more embodiments of the present disclosure as desired. The previous description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects.

[0149] The claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the claim language, where references to singular elements are not intended to mean "one and only one" unless expressly stated, but rather "one or more." The term "some" refers to one or more unless expressly stated otherwise.

[0150] ______________

[0151] Project List

[0152] Shayeb KNAA, Turner W, Gillam DG. "Periodontal Probing: A Review" .Primary Dental Journal. 2014;3(3):25-29. doi:10.1308 / 205016814812736619;

[0153] DimitriosKloukos,DMD, Dr Med Dent / George Koukos,DMD / Nikolaos Gkantidis,Priv-Doz Dr Med Dent / AndreasStavropoulos,Prof,DDS,PhD,Dr odont. "Transgingival probing: a clinical gold standard for assessing gingival thickness".Quintessence international,Volume 52,number5,May2021.

[0154] Item 1. A computer-implemented method for generating a training data set for a machine learning model, wherein the method comprises:

[0155] · receiving extraoral image data of a plurality of candidate patients provided by an extraoral imaging device;

[0156] · determining one or more periodontal structures of each of a plurality of candidate patients based on the received extraoral image data;

[0157] receiving intraoral scan data of a plurality of candidate patients provided by an intraoral scanner, and

[0158] • Generating a training data set by combining the extraoral image data with the intraoral scan data for each of a plurality of candidate patients to align one or more periodontal structures with the intraoral scan data.

[0159] Item 2. The computer-implemented method according to Item 1 further includes receiving candidate patient information of multiple candidate patients, and generating a training data set by combining the received patient information of each of the multiple candidate patients with combined extraoral image data and intraoral scan data of each of the multiple candidate patients.

[0160] Item 3. A computer-implemented method according to any of the preceding items, wherein one or more periodontal structures include bone loss or bone levels.

[0161] Item 4. The computer-implemented method according to any of the preceding items further includes segmenting the extraoral image data of multiple candidate patients into teeth and bones based on an extraoral segmentation algorithm, and segmenting the intraoral scan data of multiple candidate patients into teeth and soft tissue based on an intraoral segmentation algorithm.

[0162] Item 5. A computer-implemented method according to Item 4, wherein the combined extraoral image data and intraoral scan data use segmented teeth of the extraoral image data and segmented teeth of the intraoral scan data as reference surfaces.

[0163] Item 6. A computer-implemented method according to Item 4, wherein the combined extraoral image data and intraoral scan data are based on aligning the intraoral scan data with the extraoral image data using segmented teeth of the extraoral image data and the intraoral scan data as reference surfaces.

[0164] Item 7. A computer-implemented method according to any one of Items 4 to 6, wherein determining one or more periodontal structures includes a bone level, which is determined for each segmented tooth in the extraoral image data by the distance between the bone edge determined by the segmented bone and the cemento-enamel junction, and wherein the cemento-enamel junction is determined by the segmented teeth in the extraoral image data.

[0165] Item 8. A computer-implemented method according to any one of Items 4 to 7, wherein determining one or more periodontal structures for each of a plurality of candidate patients includes determining a first relationship between segmented teeth and bone of extraoral image data, and wherein combining the extraoral image data with intraoral scan data includes aligning the first relationship with the intraoral scan data.

[0166] Item 9. A computer-implemented method according to Item 7, wherein combining the extraoral image data with the intraoral scan data includes aligning the determined distance for the extraoral image data with the intraoral scan data.

[0167] Item 10. The computer-implemented method of Item 4, wherein determining one or more periodontal structures of each of a plurality of candidate patients comprises:

[0168] determining one or more reference locations for each identified tooth of the segmented teeth of the extraoral image data, and

[0169] • Determining one or more periodontal structures at each of one or more reference sites.

[0170] Item 12. The computer-implemented method according to any of the preceding items:

[0171] Receive the patient's intraoral scan data immediately and within a certain period of time, and

[0172] Customize the training dataset for the patient by including the patient’s intraoral scan data.

[0173] Item 13. A computer-implemented method according to any one of Items 1 to 11:

[0174] Receive the patient's intraoral scan data immediately and within a certain period of time,

[0175] receiving the intraoral scan data of the patient at a second time and during the period,

[0176] · aligning the intraoral scan data received at the first time and the second time, and

[0177] Customize a training data set for a patient based on the difference in alignment between the intraoral scan data received at a first time and a second time.

[0178] Item 14. The computer-implemented method of any one of Items 12 and 13, further comprising determining whether to customize the training data set for the patient based on one or more of the following criteria:

[0179] the quality of the patient’s intraoral scan data, and

[0180] The degree of mismatch between the patient’s intraoral scan data and the training dataset.

[0181] _________________________

[0182] Computer-implemented method for providing a diagnostic data set

[0183] __________________________

[0184] Item 15. A computer-implemented method for providing a patient's diagnostic dataset based on a computer-implemented method for generating a training dataset according to any one of Items 1 to 14, comprising receiving intraoral scan data of the patient, and outputting the patient's diagnostic dataset, wherein the diagnostic dataset is determined by processing the patient's intraoral scan data based on the training dataset.

[0185] Item 16. A computer-implemented method according to Item 15, comprising receiving patient information of a patient, and wherein the diagnostic data set is determined by processing the patient's intraoral scan data and the patient's patient information based on a training data set.

[0186] Item 17. A computer-implemented method according to any one of Items 15 and 16, wherein outputting the patient's diagnostic dataset includes one or more periodontal structures determined by processing the patient's intraoral scan data based on the training dataset.

[0187] Item 18. The computer-implemented method of any one of Items 15 and 16, wherein outputting the patient's diagnostic dataset includes one or more periodontal structures determined by:

[0188] Receive the patient's intraoral scan data immediately and within a certain period of time,

[0189] Based on the intraoral segmentation algorithm, the intraoral scan data received at the first time is segmented into teeth and soft tissues.

[0190] aligning the segmented teeth of the intraoral scan data received at the first time with the segmented teeth of the training data set,

[0191] Retrieving from the training dataset at least the segmented bones corresponding to the segmented teeth of the training dataset,

[0192] combining the segmented bone with the segmented teeth of the intraoral scan data received at the first time, and

[0193] • Determining one or more periodontal structures based on the retrieved segmented bone and segmented teeth of the patient's intraoral scan data.

[0194] Item 19. The computer-implemented method of Item 18, wherein the outputting of the patient's diagnostic data set includes one or more periodontal structures determined by:

[0195] receiving the intraoral scan data of the patient at a second time and during the period,

[0196] Segment the intraoral scan data of the patient received at the second time into teeth and soft tissue based on the intraoral segmentation algorithm,

[0197] aligning the segmented teeth of the intra-oral scan data received at a second time with the segmented teeth of the training dataset,

[0198] · determining a difference in alignment of the segmented teeth and soft tissue of the intraoral scan data received at the first time and at the second time,

[0199] retrieving from the training data at least the segmented bones corresponding to the determined alignment differences,

[0200] combining the segmented bone with the segmented teeth of the intra-oral scan data received at a second time, and

[0201] • Determining one or more periodontal structures based on the retrieved segmented bone and segmented teeth of the patient's intraoral scan data.

[0202] Item 20. The computer-implemented method of any one of Items 18 and 19, wherein determining one or more periodontal structures of the patient comprises:

[0203] determining one or more reference locations for each identified tooth of the segmented teeth of the intra-oral scan data received at the first time or the second time, and

[0204] • Determining one or more periodontal structures of the patient by correlating intraoral scan data received at a first time or a second time with a training data set at each of one or more reference locations.

[0205] Item 21. The computer-implemented method of Item 20, wherein determining one or more periodontal structures at each of the one or more reference sites comprises:

[0206] Determine the relationship between the patient’s segmented teeth and soft tissue,

[0207] · Associating the relationship with training data, which includes correlations between the relationship between segmented teeth and segmented bones provided by extraoral image data of multiple candidate patients and the relationship between segmented teeth and segmented soft tissue provided by intraoral scan data of multiple candidate patients.

[0208] Item 22. A computer-implemented method according to any one of Items 15 to 21, wherein one or more periodontal structures include bone loss or bone levels.

[0209] Item 23. A computer-implemented method according to any one of Items 15 to 22, wherein one or more periodontal structures are presented on a graphical user interface comprising a 2D model or a 3D model of the oral anatomical structure determined by the diagnostic data on which the one or more periodontal structures are visualized.

[0210] Item 24. A computer-implemented method according to Items 20 and 23, wherein one or more periodontal structures are visualized at one or more reference sites.

[0211] Item 25. A computer-implemented method according to any of Items 23 and 24, wherein one or more periodontal structures are visualized using color or text.

[0212] Item 26. A computer-implemented method according to any one of Items 23 to 25, wherein one or more periodontal structures are visualized when a visualization criterion is met.

[0213] Item 27. A computer-implemented method according to Items 22 and 26, wherein the visualization criteria defines a bone level above a threshold level.

[0214] Item 28. A computer-implemented method according to Item 27, wherein the threshold level is between 2 mm and 3 mm, around 3 mm, or above 3 mm.

[0215] _________________________

[0216] Equipment and system projects

[0217] _________________________

[0218] Item 29. A computer program comprising instructions which, when executed by a computer, cause the computer to perform a method according to any one of Items 1 to 28.

[0219] Item 30. A dental scanning system comprising a device for performing the method according to any one of Items 1 to 28.

[0220] Item 31. A dental scanning system according to Item 30, wherein extraoral image data of multiple candidate patients and / or intraoral scanning data of multiple candidate patients are stored on a server, a computer's memory, or a cloud server portion of the dental scanning system.

[0221] Item 32. A dental scanning system according to any one of Items 30 and 31, wherein the training data set is stored on a server, a computer's memory, or a cloud server.

[0222] Item 33. A data processing device comprising means for executing a method according to any one of Items 1 to 28.

[0223] Item 34. A computer-readable storage medium comprising instructions, which, when executed by a computer, cause the computer to perform a method according to any one of Items 1 to 28.

Claims

1. A computer-implemented method for generating a training data set for a machine learning model, the method comprising: - receiving extraoral image data of a plurality of candidate patients provided by an extraoral imaging device; - determining one or more periodontal structures of each of the plurality of candidate patients based on the received extraoral image data; - receiving intraoral scan data of the plurality of candidate patients provided by a handheld intraoral scanner, and - generating a training data set by combining the determined extraoral one or more periodontal structures with the intraoral scan data for each of the plurality of candidate patients to align the one or more periodontal structures with the intraoral scan data.

2. The computer-implemented method of claim 1 , wherein: The combination includes: - identifying corresponding teeth in the extraoral image data and the intraoral scan data; and - determining said one or more periodontal structures in said extraoral image data; and - Mapping or associating the determined periodontal structures with corresponding teeth in the intraoral scan data using an image alignment process.

3. The computer-implemented method of claim 1 or 2, wherein: Determining the one or more periodontal structures of each of the plurality of candidate patients comprises: - identifying a plurality of teeth in the extraoral image data; - determining one or more reference locations for each identified tooth of said extraoral image data; and - determining the one or more periodontal structures at each of the one or more reference sites.

4. The computer-implemented method of claim 2, wherein: The method includes determining, for each reference location, a tooth reference point and a bone reference point using the extraoral image data.

5. A computer-implemented method according to any one of claims 3 or 4, comprising: - determining a first relationship between teeth and bone in the extraoral image data using one or more reference locations for each identified tooth; and - combining the determined first relationship from the extraoral image data with the intraoral scan data by aligning the first relationship with the intraoral scan data.

6. The computer-implemented method of claim 5, wherein: The first relationship includes a distance between a tooth reference point arranged on a tooth of the extraoral image data and a bone reference point arranged on a bone closest to the tooth reference point or at least near the tooth reference point.

7. A computer-implemented method according to any one of claims 4 to 6, wherein: Determining the one or more periodontal structures includes determining a bone level for each tooth in the extraoral image data, wherein the bone level is determined by a distance between an identified bone reference point and an identified cemento-enamel junction of the extraoral image data, and wherein the determined bone level of the extraoral image data is merged with the intraoral scan data.

8. The computer-implemented method of any one of the preceding claims, further comprising: - segmenting the extraoral image data of the plurality of candidate patients into teeth and bones based on an extraoral segmentation algorithm, and - Segmenting the intraoral scan data of the plurality of candidate patients into teeth and soft tissues based on an intraoral segmentation algorithm.

9. The computer-implemented method of claim 8, wherein: The extraoral image data and the intraoral scan data are combined using segmented teeth of the extraoral image data and segmented teeth of the intraoral scan data as reference surfaces.

10. The computer-implemented method of claim 9, wherein: Combining the extraoral image data and the intraoral scan data is based on aligning the intraoral scan data with the extraoral image data using segmented teeth of the extraoral image data and the intraoral scan data as reference surfaces.

11. A computer-implemented method for providing a patient diagnostic data set, comprising: - Receive intraoral scan data of the patient; - inputting the intraoral scan data into a trained machine learning model according to any one of claims 1 to 9; and -Receive a patient's diagnostic dataset from a trained machine learning model.

12. The computer-implemented method of claim 11, wherein: Outputting the diagnostic data set of the patient includes one or more periodontal structures determined by processing the intraoral scan data of the patient based on the training data set.

13. The computer-implemented method of claim 11 or 12, wherein: The diagnostic data includes a prediction of the presence of periodontal structures on one or more teeth in the intraoral scan data and the location of the prediction.

14. A computer-implemented method according to any one of the preceding claims, wherein: The one or more periodontal structures are presented on a graphical user interface that includes a 2D model or a 3D model of the oral anatomy determined from the diagnostic data on which the one or more periodontal structures are visualized.

15. The computer-implemented method of claim 14, wherein: When the values ​​of the one or more periodontal structures satisfy the visualization criteria, the one or more periodontal structures are visualized at one or more reference sites.

Citation Information

Cited By

  • Methods, systems, and computer program products for flagging dental claims for further scrutiny based on processing of dental clinical images and periodontal charts using multiple artificial intelligence (AI) models

    US12718300B2