Automatic 3D tooth root shape prediction using deep learning methods

Abstract

The present invention relates to a computer-implemented method for automatic 3D tooth root shape prediction, the method comprising: a preprocessor receiving 3D data defining at least one 3D representation of a tooth and processing the 3D data, the processing comprising: transforming at least a portion of the 3D data into a voxel representation of a crown, the voxel representation defining a voxel space that fits an input space of a first 3D deep neural network executed on a computer; the preprocessor providing the voxel representation of the crown to an input of the 3D deep neural network, the 3D deep neural network being trained based on clinical 3D data defining a 3D representation of a real tooth; the first 3D deep neural network generating a predicted tooth root or a voxel representation of a complete tooth including the predicted tooth root based on the voxel representation of the crown, wherein the generation of the predicted tooth root or the voxel representation of the complete tooth comprises: determining voxel activations of voxels in the voxel space of an output of the 3D deep learning network, each voxel activation representing a probability measure defining a probability that the voxel is part of the root or the complete tooth; and determining whether the voxel activation is part of the root or the complete tooth by comparing the voxel activation to a voxel activation threshold.

Description

Technical Field

[0001] The present invention relates to automatic 3D root shape prediction using deep learning methods, and in particular, but not exclusively, to systems and methods for automatic 3D root shape prediction using deep learning, methods for training such deep learning neural networks, and computer program products using such methods. Background Art

[0002] Reliable tooth modeling plays a very important role in many dental applications, including (but not limited to) computer-aided orthodontics and dental treatment simulation systems. For example, in periodontology, diagnosis and treatment are directly related to the anatomy of the tooth root, in orthodontic treatment, the planned root structure, root shape and bone-root ratio have a huge impact on the biomechanics of the treatment, in dental surgery, tooth extraction, understanding the root shape is crucial, and in endodontics, root canals and associated root canal treatments are directly related to the anatomy and root structure of the tooth root.

[0003] An accurate 3D model of a complete tooth, i.e., a model including crowns and roots, can be obtained using a suitable 3D X-ray based CT scanning technique such as cone beam computed tomography (CBCT). However, accurate 3D CT data of the patient and software to process such 3D CT data into a representation of a complete tooth 3D model are not always available. X-ray based 3D imaging systems are expensive systems that are not always available to dental professionals. Moreover, it is often desirable to keep the patient's radiation exposure to a minimum. Therefore, a system has been developed that is able to determine a 3D representation of a complete tooth based only on the visible portion of the tooth (the crown).

[0004] For example, US8639477 describes a system for 3D modeling of complete teeth, in which a mathematical model of a reference tooth (a parameterized 3D surface mesh) is matched according to parameters and deformed according to 3D image data of the patient's crowns. Nowadays, such crown 3D image data can be easily obtained using optical intraoral tooth scanners. In the scheme of US8639477, the shape information of the crowns is used to reshape a generic 3D tooth model, including the shape of the roots. Similarly, US8135569 describes a system for 3D modeling of complete teeth by using a landmark-based method, which deforms the reference tooth based on image data of the patient's crowns and, optionally, X-ray data of the patient's teeth. The X-ray data allows reducing the difference between the root shape of the deformed generic 3D tooth model and the actual root shape of the patient's teeth, so that an improved 3D approximation of the complete tooth can be determined.

[0005] These known 3D modeling techniques are based on crown morphology, a special area in dental technology that examines the relationship between a limited number of macroscopic features of a tooth (dimensions of the crown, shape of the crown, etc.) and the shape of its root. Such a relationship can be used to define a mathematical reference model, which is used to approximate the shape of the root based on the features of the crown. Although prior art systems can provide an approximation of a 3D model of a root based on crown information, the accuracy and reliability of such an approximation are limited and may still show relatively large deviations compared to the true anatomical shape of the root. In addition, if a specific 3D reference model is used for a specific category (for a specific tooth category (e.g., molars, canines, bicuspids), and to describe the category of the patient (e.g., age, gender)), the approximation can only be reliable to a certain extent. This approach would require the development of a specific mathematical reference model for each specific tooth / patient category.

[0006] Another problem with the prior art is that when constructing a complete 3D tooth based on the estimated 3D root model and the 3D crown model, the surface in the transition area between the predicted root and the original crown may require additional smoothing and / or other processing that may result in shape changes of the complete 3D tooth that differs from the actual anatomical shape of the tooth. In addition, it is worth noting that for accuracy, the above system may require additional inputs in addition to the crown 3D image data, such as applicable categories (for the tooth and patient domains) and x-ray images.

[0007] Currently, in the field of imaging, especially medical imaging, more advanced techniques such as deep learning techniques can be used for 3D modeling. These neural networks are trained to learn features that optimally represent relevant aspects within the data. Such deep learning algorithms include multi-layer deep neural networks that transform input data into outputs while learning increasingly high-level features. A successful neural network model for image analysis is the so-called convolutional neural network (CNN). CNN contains many layers that transform their inputs using kernels (also called convolutional filters), which consist of relatively small matrices. An overview of the use of CNNs in medical imaging can be found in the article "A Survey on Deep Learning in Medical Image Analysis" by Litjens et al. (published on February 21, 2017, arXiv (submitted to Computer Vision and Pattern Recognition)).

[0008] The accuracy and reliability of trained deep neural networks depend heavily on the quality and quantity of the data used to train the neural network. Cone beam computed tomography (CBCT) is the most commonly used 3D imaging technology in dental applications, and in principle, large CBCT datasets containing 3D dento-maxillofacial structures can be used to train neural networks. However, image analysis of CBCT image data and the generation of large amounts of high-quality training data based on these CBCT image data pose substantial problems. In CBCT scans, the radio density measured in Hounsfield units (HU) is inconsistent because different areas in the scan appear at different grayscale values, depending on their relative position in the scanned organ. The HU measured from the same anatomical region with CBCT and medical-grade CT scanners are not exactly the same, and are therefore unreliable for determining site-specific radiographically identified measured densities.

[0009] Moreover, again considering the issue of the availability of accurate and sufficient training data, CBCT systems used to scan dento-maxillofacial structures do not employ a standardized system for scaling the gray levels representing the reconstructed density values. These values ​​are arbitrary, and without such standardization, it is difficult to interpret the gray levels or even impossible to compare values ​​produced from different machines. Moreover, the roots of teeth and (one or more) jaw bone structures have similar densities, making it difficult for the computer to distinguish between voxels belonging to teeth and voxels belonging to jaw bones. In addition, CBCT systems are very sensitive to so-called beam hardening, which produces dark fringes between two high-attenuation objects (such as metal or bone), surrounded by bright fringes.

[0010] The above mentioned problems make the implementation of a (fully trained) deep learning system that can automatically perform 3D root shape prediction based on 3D crown information very challenging. Therefore, there is a need in the art for a computer system that can automatically generate anatomically accurate predictions of 3D root shape based on image data of 3D crown shape. Summary of the invention

[0011] As will be appreciated by those skilled in the art, aspects of the present invention may be implemented as a system, method or computer program product. Thus, aspects of the present invention may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which aspects may be generally referred to herein as a "circuit", "module" or "system". The functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer, and in addition, aspects of the present invention may take the form of a computer program product implemented in one or more computer-readable media, on which a computer-readable program code is implemented (e.g., stored).

[0012] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this article, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0013] A computer readable signal medium may include, for example, a propagated data signal in baseband or as part of a carrier wave, the propagated data signal having a computer readable program code implemented therein. Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium, and may transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0014] Any suitable medium can be used to transmit the program code implemented on the computer-readable medium, including but not limited to wireless, wired, optical fiber, cable, RF etc., or any suitable combination of the foregoing. The computer program code for performing the operation of various aspects of the present invention can be written with any combination of one or more programming languages, including function-oriented or object-oriented programming languages ​​such as Java (TM), Scala, C++, Python etc., and conventional process programming languages ​​(such as " C " programming language or similar programming languages). The program code can be completely on the user's computer, partly on the user's computer, as an independent software package, partly on the user's computer and partly on the remote computer, or completely on the remote computer, server or virtual server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected (for example, by using the Internet of Internet service providers) with an external computer.

[0015] Various aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It will be understood that each box of the flowchart illustration and / or block diagram and the combination of boxes in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, in particular a microprocessor or a central processing unit (CPU) or a graphics processing unit (GPU) to produce a machine so that instructions executed by a processor of a computer, other programmable data processing device or other device create means for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0016] These computer program instructions may also be stored in a computer-readable medium, which may instruct a computer, other programmable data processing apparatus, or other device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce an article of manufacture, which includes instructions for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0017] Computer program instructions may also be loaded onto a computer, other programmable data processing device or other device so that a series of operating steps are performed on the computer, other programmable device or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0018] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a fragment or a portion of a code, which includes one or more executable instructions for realizing (one or more) specified logical functions. It should also be noted that in some alternative embodiments, the functions indicated in the box may not occur in the order indicated in the figure. For example, depending on the functions involved, the two boxes shown in succession can actually be executed substantially at the same time, or sometimes these boxes can be executed in the opposite order. It should also be noted that each box illustrated in the block diagram and / or the flowchart and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a combination of a dedicated hardware-based system or dedicated hardware and computer instructions that performs a specified function or action.

[0019] In a first aspect, the present invention relates to a computer-implemented method for automatic 3D root shape prediction. In an embodiment, the method may include: a preprocessor receives 3D data defining at least one 3D representation of a tooth and processes the 3D data, the processing including: transforming at least a portion of the 3D data into a voxel representation of a crown, the voxel representation defining a voxel space that fits an input space of a first 3D deep neural network executed on a computer; the preprocessor provides the voxel representation of the crown to an input of the 3D deep neural network, the 3D deep neural network being trained based on clinical 3D data defining a 3D representation of a real tooth; the first 3D deep neural network generates a predicted root or a voxel representation of a complete tooth including the predicted root based on the voxel representation of the crown, wherein the generation of the predicted root or the voxel representation of the complete tooth includes: determining voxel activations of voxels in the voxel space of the output of the 3D deep learning network, each voxel activation representing a probability measure defining the probability that the voxel is part of the root or the complete tooth; and determining whether the voxel activation is part of the root or the complete tooth by comparing the voxel activation to a voxel activation threshold.

[0020] In another embodiment, the method may include a preprocessor (104) receiving 3D data (102) defined in voxel space, the voxel space defining a 3D volume, the voxel space including a voxel representation of a tooth crown, and processing the voxel representation so that it is at a scale, position, and orientation corresponding to the voxel representation used to train a first 3D deep neural network (106, 210); the preprocessor (104) providing the voxel representation of the tooth crown to an input of the first 3D deep neural network (106), the first 3D deep neural network being trained based on a training set of preprocessed clinical 3D data (202) defining 3D representations of real teeth, the trained deep neural network (106, 210) being configured to predict An anatomically accurate voxel representation (212) corresponding to a tooth crown or a voxel representation (214) of a complete tooth; a first 3D deep neural network (106, 210) generating a predicted tooth root (212) voxel representation (212) or a voxel representation (214) of a complete tooth including the predicted tooth root based on the voxel representation of the tooth crown, wherein the generation of the predicted tooth root or the voxel representation of the complete tooth includes: determining voxel activations of voxels in a voxel space of an output of the first 3D deep learning network, each voxel activation representing a probability measure defining a probability that the voxel is at least a portion of a tooth root or a complete tooth; and determining whether the voxel activation is a portion of a tooth root or a complete tooth by comparing the voxel activation to a voxel activation threshold. In one embodiment, the threshold may represent a probability greater than 0.5.

[0021] Thus, a 3D deep neural network is trained to automatically learn anatomical features obtainable from a received voxel representation of a crown, thereby generating (at least) an anatomically accurate prediction of a root representation, wherein the resulting root model will fit closely to the crown model so as to form a complete tooth model. Due to the fact that such features can be learned from any potential training sample (the sample being a voxel representation of a crown) and the 3D deep neural network will determine which features are relevant, this method is able to more accurately use any relevant information in the voxel representation. In other words, where prior art techniques may be limited to a specific set of input parameters, the proposed method has the potential to utilize more input information and will determine during training which features are relevant.

[0022] Furthermore, the 3D deep neural network will learn a generalization of the 3D image data to represent the root portion based on the crown portion. However, this generalization is more flexible than the use of template roots in the prior art (given the potential different shapes that can be generated for predicted roots or complete teeth). It is also worth noting that the method does not require separate inputs to indicate the tooth class (canine, molar, etc.) and / or patient class (age, gender, etc.). In fact, it is possible for the 3D deep neural network to identify relevant features directly from the received voxel representations that are the result of such classification. However, this depends (at least) on the number of available training samples that vary across such classes, and the memory bandwidth available within the feature representation in the 3D deep neural network.

[0023] In an embodiment, the method may include a post-processor receiving a voxel representation of a predicted tooth root generated by a first 3D deep neural network and processing the 3D crown and the voxel representation of the predicted tooth root, wherein the processing may include: merging the voxels of the 3D crown model and the 3D root into a voxel representation of the complete 3D tooth. In yet another embodiment, the post-processor may transform the voxel representation of the complete 3D tooth in a 3D mesh of the complete tooth.

[0024] Thus, the present invention allows for the automatic generation of at least anatomically accurate predictions of 3D root shapes by a trained deep neural network executed on a computer. The deep neural network can generate 3D root shapes based on a 3D crown dataset (e.g., a 3D surface mesh generated by an intraoral optical scanner or 3D data generated by an X-ray scanner (e.g., a CBCT scanner)). When the neural network is trained based on clinical data of a 3D model of a whole tooth, the network will generate anatomically accurate predictions of a 3D root model, where the 3D root model precisely fits the input 3D crown data fed to the neural network. In this way, a computer can construct an accurate 3D model of the whole tooth based on the generated 3D root representation and 3D crown model. Accurate 3D tooth models are of great value in numerous dental applications, including but not limited to periodontal diagnosis and treatment, orthodontic treatment planning, endodontics, and endodontics.

[0025] In another embodiment, the two voxel spaces described previously (one voxel representation for the crown and one voxel representation for the root) are treated as one space, which is actually a 3D space that can potentially contain the entire individual tooth. Voxels that are part of the received 3D crown are appropriately represented in the complete tooth space. This complete tooth space, including the voxel representation of the 3D crown, is provided to the input of the 3D deep neural network. The trained 3D deep neural network can then generate a voxel representation of the complete tooth in a 3D space with the same dimensionality as the input space.

[0026] In an embodiment, the method may further include: a post-processor receiving a voxel representation of a predicted tooth root generated by a first 3D deep neural network, and processing the voxel representation of the 3D crown and the predicted tooth root, the processing comprising: merging the voxels of the 3D crown model and the 3D root into a voxel representation of a complete 3D tooth; and optionally, transforming the voxel representation of the complete 3D tooth in a 3D mesh of the complete tooth.

[0027] In an embodiment, the 3D data may define a 3D representation of at least a portion of a dentition, and the processing performed by the preprocessor further comprises: segmenting the 3D data into at least one 3D dataset representing 3D crowns of teeth of the dentition; and transforming the 3D dataset into a voxel representation of the crowns that matches the voxel space of an input to a first 3D deep neural network.

[0028] In an embodiment, the 3D data received by the pre-processor may be 3D data generated by an optical scanner, preferably an intra-oral optical scanner, the 3D data defining a 3D surface mesh representing a dentition including a plurality of crowns.

[0029] In an embodiment, the processing performed by the preprocessor may also include: dividing the 3D mesh into a plurality of segmented 3D meshes, wherein each segmented 3D mesh represents a 3D crown of the dentition; transforming each segmented 3D surface mesh into a voxel representation of the crown, which voxel representation matches the voxel space of the input of the first 3D deep neural network.

[0030] In an embodiment, the 3D data received by the preprocessor may be generated by an X-ray scanner (preferably a CBCT scanner), the 3D data defining a voxel representation of at least a portion of a dentofacial structure, the voxels being associated with radiation intensity values ​​or density values, the dentofacial structure comprising a plurality of teeth of at least a portion of a dentition.

[0031] In an embodiment, the processing performed by the preprocessor may also include: classifying at least a portion of the voxels representing the maxillofacial structure into at least one of jaw, tooth and / or nerve voxels; segmenting the classified tooth voxels into one or more 3D data sets, each of the one or more 3D data sets defining a voxel representation of a tooth in the dentition of the maxillofacial structure.

[0032] In an embodiment, a second 3D deep neural network is used to classify voxels representing dento-maxillofacial structures.

[0033] In an embodiment, the processing performed by the preprocessor may also include: using a second 3D deep neural network to classify at least a portion of the voxels representing the maxillofacial structure into at least one of jaw, tooth and / or neural voxels; the second 3D deep neural network is trained based on 3D image data of the maxillofacial structure, one or more 3D position features optionally derived from the 3D image data of a training set, and one or more 3D models of the portion of the maxillofacial structure of the 3D image data of the training set, and the one or more 3D models are used as targets during the training of the first deep neural network; and segmenting the classified voxels into one or more 3D data sets, each of the one or more 3D data sets defining a voxel representation of a tooth in the dentition of the maxillofacial structure.

[0034] In an embodiment, the processing performed by the preprocessor may also include: providing a further voxel representation of the dental and maxillofacial structure to an input of a third 3D deep neural network, the third deep neural network being trained to determine at least one 3D position feature for each voxel of the voxel representation at the input, the 3D position feature comprising a measurement indicating the likelihood that the voxel represents a jaw, teeth and / or neural tissue, wherein the further voxel representation of the dental and maxillofacial structure is a low resolution version of the voxel representation of the dental and maxillofacial structure.

[0035] In an embodiment, the resolution of the further voxel representation may be at least three times lower than the resolution of the first voxel representation.

[0036] In an embodiment, the third 3D deep neural network may be trained based on the 3D image data of the dental and maxillofacial structure and one or more 3D models of parts of the dental and maxillofacial structure of the 3D image data of the training set used to train the second deep neural network.

[0037] In an embodiment, the processing performed by the preprocessor may also include: providing the voxel representation of the maxillofacial structure and one or more 3D position features to a second 3D deep neural network, the second 3D deep neural network using the one or more position features to classify at least a portion of the voxels in the voxel space as at least one of jaw, tooth and / or neural voxels.

[0038] In an embodiment, the second deep neural network may include a plurality of first 3D convolutional layers, outputs of the plurality of first 3D convolutional layers being connected to at least one fully connected layer, wherein the plurality of first 3D convolutional layers are configured to process a first voxel block from the first voxel representation, and wherein the at least one fully connected layer is configured to classify voxels of the first voxel block as at least one of jaw, tooth and / or neural voxels.

[0039] In an embodiment, the second deep neural network may further include multiple second 3D convolutional layers, outputs of the multiple second 3D convolutional layers being connected to at least one fully connected layer, wherein the multiple second 3D convolutional layers are configured to process a second voxel block from the first voxel representation, the first and second voxel blocks having the same or substantially the same center point in the image volume, and a volume in the real-world dimension of the second voxel block representation is larger than a volume in the real-world dimension of the first voxel block, and the multiple second 3D convolutional layers are configured to determine context information associated with voxels of the first voxel block provided to the input of the multiple first 3D convolutional layers.

[0040] In an embodiment, the processing performed by the preprocessor may also include: determining one or more 3D position features based on voxels representing the dental and maxillofacial structure, the 3D position features defining position information of voxels in a voxel space of at least a portion of the dental and maxillofacial structure; providing the one or more 3D position features to a second 3D deep neural network, the second 3D deep neural network using the one or more position features to classify at least a portion of the voxels in the voxel space as at least one of jaw, tooth and / or neural voxels.

[0041] In an embodiment, the position information may define a distance, preferably a vertical distance, between a voxel in voxel space and a first dental reference plane in voxel space; a distance between a voxel in voxel space and a first dental reference object in voxel space; and / or a position of a cumulative intensity value in a second dental reference plane in voxel space, wherein the cumulative intensity value at a point in the second dental reference plane includes the cumulative intensity values ​​of voxels on or near a normal through the point in the second dental reference plane.

[0042] In an embodiment, the second deep neural network may include a first data processing path and a second data processing path parallel to the first path, the first data processing path including at least a first plurality of 3D convolutional layers, preferably a first set of 3D CNN feature layers, the second data processing path including a second plurality of 3D convolutional layers, preferably a second set of 3D CNN feature layers, for receiving one or more 3D position features, the second plurality of 3D convolutional layers being configured to encode relevant information based on position information associated with pixel blocks fed to the input of the first plurality of 3D convolutional layers.

[0043] In an embodiment, the first 3D deep neural network may comprise a plurality of 3D convolutional layers connected to a plurality of 3D deconvolutional layers via one or more densely connected layers, and wherein the first deep neural network is trained based on a voxel representation of a tooth crown and associated tooth root or based on a voxel representation of a tooth crown and associated tooth, preferably at least a portion of the voxel representation being derived from segmented 3D X-ray data (preferably 3D CBCT data) representing one or more dentofacial structures.

[0044] In another aspect, the present invention may relate to a computer-implemented method for training a 3D deep learning neural network to generate a prediction of a 3D tooth root shape, the method comprising: a computer receiving training data comprising clinical 3D data comprising or based on a voxel representation of a crown and associated tooth root, wherein at least a portion of the voxel representation is derived from segmented 3D X-ray data (preferably 3D cone beam CT (CBCT) data); providing the voxel representation of the crown to an input of the 3D deep neural network, and the 3D deep neural network generates a predicted voxel representation of the root; optimizing the values ​​of one or more network parameters of the 3D deep neural network by minimizing a loss function representing a deviation between a voxel representation of the predicted root and a voxel representation of the root associated with the voxel representation of the crown provided to the input of the 3D deep neural network.

[0045] In an embodiment, the method may include storing optimized values ​​in a computer-readable storage medium, the optimized values ​​defining one or more network parameters of a trained neural network configured to predict an anatomically accurate voxel representation of a tooth root corresponding to a tooth crown or a voxel representation of a complete tooth when provided with a voxel representation of the tooth crown.

[0046] In an embodiment, the 3D deep neural network may include a plurality of 3D convolutional layers connected to a plurality of 3D deconvolutional layers via one or more densely connected layers.

[0047] In another aspect, the present invention may relate to a computer system, preferably a server system, suitable for automatically predicting 3D root shape, the computer system comprising: a computer-readable storage medium having computer-readable program code implemented thereon, the program code comprising a preprocessing algorithm and at least one trained first 3D deep neural network, the computer-readable program code; and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein, in response to executing the first computer-readable program code, the processor is configured to perform executable operations, including: the preprocessor receives 3D data defining at least one 3D representation of a tooth and processes the 3D data, the processing comprising: transforming at least a portion of the 3D data into a voxel representation of a crown, the voxel representation defining a fit performed on the computer The invention relates to a method for generating a voxel space of an input space of a 3D deep neural network of the present invention; the preprocessor provides a voxel representation of a crown to the input of the 3D deep neural network, the 3D deep neural network being trained based on clinical 3D data defining a 3D representation of a real tooth; the 3D deep neural network generates a voxel representation of a predicted tooth root or a complete tooth including the predicted tooth root based on the voxel representation of the crown, wherein the generation of the voxel representation of the predicted tooth root or the complete tooth including the predicted tooth root includes: determining voxel activations of voxels in the voxel space of the output of the 3D deep learning network, each voxel activation representing a probability measure defining the probability that the voxel is part of the tooth root or the complete tooth; and determining whether the voxel activation is part of the tooth root or the complete tooth by comparing the voxel activation to a voxel activation threshold.

[0048] On the other hand, the present invention may relate to a computer system, preferably a server system, suitable for automatically predicting 3D root shape, the computer system comprising: a computer-readable storage medium having computer-readable program code implemented thereon, the program code comprising a preprocessing algorithm and at least one trained 3D deep neural network; and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein, in response to executing the computer-readable program code, the processor is configured to perform executable operations defined in any method steps as described above.

[0049] The invention may also relate to a computer program product comprising software code portions configured to perform any of the methods described above when run in the memory of a computer.

[0050] Throughout this document, where reference is made to "3D image data" or "3D data", it is intended to refer to 3D image data in any format, such as (a) 3D surface mesh(es), (a) 3D point cloud(es), data representing density values ​​at specific 3D coordinates, a surface or volume in 3D (voxel) space, etc. Where reference is made to a "3D model", it is intended to refer to (a) 3D surface mesh(es), unless otherwise specified. In addition, a complete tooth is considered to be composed of a combination of "crown" and "root" portions, where "root" is defined as any portion of the 3D volume occupied by the complete tooth that is not a "crown".

[0051] The invention will be further explained with reference to the accompanying drawings, which will schematically show embodiments according to the invention. It will be understood that the invention is by no means limited to these specific embodiments.

[0052] The invention will be further explained with reference to the accompanying drawings, which will schematically show embodiments according to the invention. It will be understood that the invention is by no means limited to these specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 depicts a high-level schematic diagram of a computer system configured for automatic 3D tooth root shape prediction in accordance with an embodiment of the present invention;

[0054] Figure 2 Depicting a flow chart for training a deep neural network for predicting individual 3D tooth root shapes according to an embodiment of the present invention;

[0055] Figure 3 An example of a 3D deep neural network architecture used in a method and system for automatic root shape prediction from 3D crown image data according to an embodiment of the present invention is depicted;

[0056] Figure 4A and Figure 4B depicts a schematic diagram illustrating pre-processing and post-processing of 3D data according to various embodiments of the present invention;

[0057] Figure 5A and Figure 5B An illustration of a 3D clinical voxel representation compared to a predicted voxel representation is shown;

[0058] Fig. 6A and 6B A table of dice coefficients determined according to an exemplary embodiment of the system is shown;

[0059] Figure 7 A 3D model of the clinical tooth combined with the predicted results is shown;

[0060] Figure 8Schematically depicts a computer system for classification and segmentation of 3D dental and maxillofacial structures according to an embodiment of the present invention;

[0061] Fig. 9A and 9B Examples of 3D CT image data and 3D optical scanning data are depicted respectively;

[0062] Fig. 10A and 10B Depicts an example of a deep neural network architecture for classifying dentofacial 3D image data;

[0063] Fig.11A and 11B A flowchart and a neural network for determining dental and maxillofacial features according to an embodiment of the present invention are illustrated;

[0064] Fig.12 Provides a visualization containing the total voxel values ​​from the 3D image stack and the curves fitted to the voxels representing the dental and maxillofacial arch;

[0065] Figures 13A-13E Depicting examples of dentofacial features according to various embodiments of the present invention;

[0066] Figures 14A-14D depicts an example of the output of a trained deep learning neural network according to an embodiment of the present invention;

[0067] Fig.15 A flowchart of post-processing classified voxels of a 3D dental and maxillofacial structure according to an embodiment of the present invention is depicted;

[0068] Fig.16 Describes a deep neural network architecture for post-processing voxels of classification of 3D dental and maxillofacial structures according to an embodiment of the present invention;

[0069] Figures 17A-17B depicts a surface reconstruction process of classified voxels according to an embodiment of the present invention;

[0070] Fig.18 Depicts a schematic diagram of a distributed computer system for processing 3D data according to various embodiments of the present invention;

[0071] Fig.19 is a block diagram illustrating an exemplary data computing system that may be used to execute the methods and software products described in the present disclosure;

[0072] FIG. 20 illustrates a representation of a visualization of a 3D teeth model toward an end user, according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In the present disclosure, embodiments of computer systems and computer-implemented methods are described that use a 3D deep neural network to predict an anatomically accurate 3D representation of a tooth root based on 3D data defining a 3D representation of a tooth crown (referred to as 3D crown data). The 3D crown data can be derived from different data sources. For example, the 3D crown data can be derived from a 3D surface mesh representing the crown of a dentition generated by a 3D optical scanner (e.g., an intraoral optical scanner). Alternatively, the 3D crown data can be derived from a 3D voxel representation of a dentofacial structure (including a crown) generated by a 3D X-ray scanner (such as a CBCT scanner), for example. The 3D deep neural network can be trained based on clinical 3D data (e.g., 3D data defining a real tooth representation). A computer system according to the present invention may include at least one 3D deep neural network that is trained to generate an anatomically accurate predicted voxel representation of a voxel representation of a tooth root based on a voxel representation of a crown. The computer system can be configured to perform a training process that iteratively trains a 3D deep neural network based on training data based on clinical 3D data (i.e., 3D representations of real teeth). The training data can be based on 3D CBCT data of segmented dentofacial structures, in particular, voxel representations of individual segments of teeth. The voxel representation of teeth can be divided into two 3D data sets, i.e., a voxel representation of a 3D crown and a voxel representation of a 3D root.

[0074] Using the training set generated in this way will result in a trained deep neural network that can accurately predict an anatomically accurate voxel representation of a tooth root or a voxel representation of a complete tooth. Therefore, the deep learning network is trained to process voxels in a predetermined voxel space (i.e., a volume that defines voxels, each voxel being associated with a 3D position in the volume). Since the computer system is capable of receiving 3D data of different structures and formats, such as 3D data generated by a CBCT scanner of an intraoral optical scanner. The computer system includes a preprocessor that processes the 3D data before presenting it to the input of the 3D deep neural network. The preprocessor can transform the 3D data into a voxel representation of each 3D crown, wherein the voxel representation matches the voxel space of the input of the deep neural network. Preprocessing can also include scaling, positioning and / or orienting the voxel representation in a unified manner (between the 3D data in the training set, which is intra-set consistency, and within different training sets, which is inter-set consistency).

[0075] Once the deep neural network is trained, it can receive a voxel representation of a crown and predict a corresponding voxel representation of a root shape or a voxel representation of a complete tooth, including the predicted root shape. Before presenting the 3D input data to the trained deep neural network, the 3D input data is preprocessed in a manner similar to the preprocessing of the training data, so that the 3D crown is set to a scale, position and orientation corresponding to the scale, position and orientation used during the training of the deep neural network. If the deep neural network is trained to generate a predicted voxel representation of the root, then a post-processor can merge the voxel representations of the root and crown (as provided to the input of the deep neural network) into a voxel representation of the complete tooth. Depending on the application, the post-processor can also transform the voxel representation into a 3D model or mesh representation of the tooth. Because the network is trained using clinical 3D data (particularly clinical 3D CBCT data), the predicted root shape will be based on features derived by the deep neural network during training.

[0076] The computer system, deep neural network, pre-processor and post-processor, training of the neural network, and methods performed by the network and the pre-processor and post-processor are described in more detail below.

[0077] Figure 1 A high-level schematic diagram of a computer system according to an embodiment of the present invention is depicted. In particular, Figure 1 A computer system is depicted that is configured to automatically generate 3D data defining a representation of a complete 3D tooth based on 3D input data representing, for example, a crown, a complete tooth, a dentofacial structure including a tooth or a dentition. The computer system 100 may include a preprocessor, a trained deep neural network, and a postprocessor. The preprocessor may preprocess the 3D input data 102.

[0078] For example, in an embodiment, the preprocessor 104 can derive individual 3D representations of the crowns of the dentition from 3D data representing the real-world dental and maxillofacial structures. Such 3D data can be generated using known techniques such as X-ray or CBCT scanners. Typically, such 3D data includes a volume of voxels (voxel space), where each voxel value represents a radiation intensity value. Since the data represents the complete dental and maxillofacial structure, the 3D data also includes spatial information about the position of the teeth (and crowns) in the dental and maxillofacial structure. In this case, the preprocessor can perform a segmentation process to segment the 3D data of the dental and maxillofacial structure into individually segmented dental structures (e.g., bones, teeth, and nerves), including individual representations of complete teeth, typically voxel representations of complete teeth. Due to the nature of CBCT data, the automatic segmentation of 3D CBCT data representing the dental and maxillofacial structure is not a trivial process. For example, in a CBCT scan, the radio density measured in Hounsfield units (HU) is inconsistent because different areas in the scan appear with different grayscale values, depending on their relative positions in the scanned organ. Another problem is that CBCT systems do not employ a standardized system for scaling the gray levels representing the reconstructed density values.

[0079] In order to achieve automatic segmentation without any human intervention and / or assistance, the preprocessor can use a separately trained deep neural network to automatically classify the 3D data into different 3D datasets, where the 3D datasets define a voxel representation of the teeth. Figure 8 The segmentation process, the deep neural network used to perform such segmentation process, and the pre-processing and post-processing of 3D data to achieve such accurate segmentation are further described in more detail.

[0080] Results from the above-described deep neural network for automatically segmenting teeth from CBCT data can be used to generate clinical 3D tooth representations that are used to train a deep neural network for root shape prediction 106 during a training phase (described in more detail below). The segmentation can also be used during an inference phase, where the 3DCBCT data can be segmented in order to determine a voxel representation of the crown that can be provided as input to the trained neural network.

[0081] In another embodiment, the preprocessor 104 can derive individual 3D models of crowns of the dentition from 3D data generated by an intraoral optical scanner. Typically, such a 3D data set can define a 3D polygonal mesh within a predefined 3D space, wherein the data forms a collection of vertices, edges, and faces that define the shape of the scanned crowns and portions of the gingiva in the 3D space. Since the data represents a portion of the 3D dentition structure, the 3D data also includes spatial information about the position of the crowns in the dentition. In order to automatically divide the 3D mesh of the scanned dentition into 3D meshes of individual 3D crown models, the preprocessor can perform a segmentation process, wherein a 3D surface mesh representing multiple crowns of the dentition (or a portion thereof) is segmented in multiple 3D surface meshes, wherein each 3D surface mesh represents an individual 3D crown model. This segmentation process can be performed using known methods, such as described in "Automatic Tooth Segmentation of Dental Mesh Based on Harmonic Fields" by Liao in 2015.

[0082] After the segmentation process, each 3D dataset defining a 3D representation of a dental crown can be converted into a data format that matches the expected input format of the trained 3D deep neural network 106. To this end, the preprocessor can transform the 3D dataset defining the 3D representation of the dental crown into a voxel representation of the dental crown using a voxel space that fits the 3D input space of the trained 3D deep neural network. In addition, the preprocessor can process the 3D dataset so that the orientation of each dental crown model is scaled, positioned and oriented in the voxel space in a uniform manner. This preprocessing can include scaling, rotating, mirroring and / or translating voxels so that the voxel representation of the dental crown is centered in the voxel space with a predetermined orientation, for example, so that the tooth (and dental crown) "up and down" direction (or axis in the coordinate system) (as can be derived from the orientation it will be in when the patient is standing) can be used to ensure that this direction will be consistent between all voxel representations of the dental crown presented as input to the deep neural network 106. The preprocessor may mirror the voxel representation along a plane perpendicular to the aforementioned "up-down" axis to ensure that the "crown to root" direction of the clinical tooth always points in the same direction in the space of the voxel representation.

[0083] The preprocessed voxel representation of the crown is then provided to the input of a 3D deep neural network 108, which is trained to produce a predicted voxel representation of the root. The 3D deep neural network can be trained to produce a voxel representation of the root, which can be combined with the voxel representation of the crown provided to the input of the 3D deep neural network. In another embodiment, the 3D deep neural network can be trained to generate a voxel representation that includes the crown voxels provided to the input of the 3D deep neural network and the voxel prediction of the root. When the deep neural network predicts the root shape for each individual 3D crown model, the post-processor 108 can be adapted to merge the predicted root shape with the 3D crown data for the input (e.g., the segmented crown shape from the original source, or the crown shape with the voxel representation presented to the deep neural network). The post-processor 108 can convert the merged voxel representation of the complete tooth into a 3D model (surface mesh), and can use well-known surface mesh operations such as smoothing and decimation. Furthermore, in some embodiments, based on the source 3D input, the created 3D model of the complete tooth can be placed back into the original source to maintain their relative positions to each other. This operation can be done in the case where the source contains multiple 3D crown representations (as may be the case where the input data is, for example, a CBCT scan or an intraoral scan). In that case, the post-processor 108 can create a 3D data set 110 in which each rendered crown is provided with a predicted root.

[0084] Figure 2 A flow chart for training a deep neural network for predicting a single 3D tooth root shape according to an embodiment of the present invention is depicted. In order to implement a trained 3D deep neural network, a large amount of accurate clinical training data is required. In order to train a 3D deep neural network to accurately predict a single 3D tooth based on a 3D crown model, various training data sources can be used.

[0085] The training data 200 may be based on clinical 3D data 202, i.e., 3D data of real teeth of a patient, which may be derived from various sources, in particular 3D data generated by an X-ray scanner such as a CBCT scanner and / or 3D data generated by a 3D optical scanner. The 3D data generated by an X-ray scanner typically includes a voxel representation of at least a portion of the dentofacial structure of a plurality of 3D tooth models comprising at least a portion of the dentition, wherein each voxel is associated with a radiation intensity value or a density value. Similarly, the 3D data of an optical scanner typically includes a 3D mesh representing a complete tooth.

[0086] Since the deep neural network can be trained based on the voxel representation of each tooth model, the preprocessor processing of the clinical 3D data is required. In an embodiment, the processing of the preprocessor 204 may include determining a plurality of 3D datasets based on the clinical 3D CBCT data, wherein each 3D dataset may define a 3D representation of the teeth of the dentition in the clinical 3D CBCT data. In this way, each 3D dataset defines a clinical 3D representation of the teeth. The determination of the 3D datasets requires an accurate automated segmentation process. However, as described in reference Figure 1 As mentioned above, accurate automatic segmentation of 3D CBCT data is not an easy task and can be achieved using a separately trained deep neural network. Figure 8 The segmentation process, the deep neural network used to perform such segmentation process, and the pre-processing and post-processing of 3D data to achieve such accurate segmentation are further described in more detail.

[0087] Thus, a preprocessor can process clinical 3D data, typically CBCT data representing dentofacial structures, to generate training data for a deep learning network. Preprocessing can include extracting clinical 3D datasets, each clinical 3D dataset defining a voxel representation 206 of a tooth. In addition, preprocessing can include generating a voxel representation 208 of a crown and an associated voxel representation 207 of a root based on each 3D dataset. In addition, preprocessing can include normalizing the orientation, position, and / or dimensions of samples (e.g., crowns, roots, and complete teeth) represented by various voxel representations used to train a deep neural network. Reorientation, repositioning, and scaling of the samples results in normalized samples for training. Normalization enables the deep neural network to accurately train the relevant features required for root prediction.

[0088] In an embodiment, the 3D deep learning network 210 may be trained using a voxel representation of a crown as input and a voxel representation of a root as a target label. In another embodiment, the 3D deep learning network may be trained using a voxel representation of a crown as input and a voxel representation of a complete tooth (including the root) as a target label.

[0089] During the training phase, the intermediate output of the deep neural network representing the predicted root voxel representation 212 or the predicted tooth voxel representation 214 may be iteratively evaluated relative to the target label. In this way, after training, during the inference phase, when an arbitrarily selected 3D crown voxel representation is provided as an input to the 3D deep neural network, the trained 3D deep learning network is able to accurately predict either the 3D root shape or the complete 3D tooth.

[0090] As described above, root voxels 207 and crown voxels 208 can be extracted from a volume of voxels representing a complete tooth 206 (the volume of tooth voxels). Different methods can be used to separate the volume of tooth voxels into a volume of crown voxels and a volume of root voxels. For example, the volume of tooth voxels can be divided based on a transverse plane (i.e., a plane (sometimes called a frontal plane) that is perpendicular to the patient's "real world" "up-down" axis (i.e., a longitudinal axis that can be defined as the intersection of the sagittal plane and the coronal plane)). The height of the plane (i.e., the location where the plane intersects the longitudinal axis) defines the size of the volume containing the crown and root voxels. This method of dividing the crown and root voxels may be most beneficial when a separate space containing the voxel representation of the crown is strictly used as the input to the deep neural network, and a separate space containing the voxel representation of the root is used as the output of the deep neural network 210.

[0091] In another embodiment, both the input space and the output space of the deep neural network represent voxel spaces that can include voxels representing complete teeth. In that case, the partitioning may not necessarily be based on planes. A more arbitrary (curved) partitioning will accommodate the input more tightly, as would be expected from a voxel representation derived from an intra-oral scan. In such a scan, the gum line (which is the boundary where the segmentation will occur) is not expected to be straight.

[0092] When the crown is pointing upward, the "highest" position along the longitudinal axis can be used, at which position a plane perpendicular to the longitudinal axis will intersect the entire circumference of the segmented crown. This plane can be used to divide the voxels, resulting in a "straight" segmentation. However, this way of selecting crown voxels releases a large amount of surface data of the segmented tooth, and therefore information that can potentially be used by a deep neural network to accurately predict the root shape. Further details of the preprocessing are described in more detail below with reference to Figure 4.

[0093] The position of the normal plane along the longitudinal axis can be varied (within practical boundaries) using, for example, the total tooth height, an absolute distance and / or relative factor of this height (e.g., one quarter of the total height plus or minus one sixteenth or a predetermined height in millimeters, etc.), an expected height of a tooth that will not be obscured by the patient's gums, or any combination of these that can be automatically generated. During training, the same clinical 3D tooth data can be presented to the 3D deep neural network multiple times while determining the position changes of the plane of the crown-root division, thereby providing enhanced robustness to the inputs for the 3D deep neural network training.

[0094] In another embodiment, the preprocessor 204 can also generate non-straight partitions, for example, simulating an input that would be received as a 3D crown model generated by segmenting an intraoral scan. In this case, the preprocessor can partition the received 3D model data along a more curved surface, using at least one 3D reference point at a height that can be determined as described above, and the centroid of the closed circle of the surface at this height. In addition, practical limits on the amount of curvature allowed can be selected (e.g., the maximum height difference between the highest and lowest points of the surface that intersects the received closed volume; the maximum allowed curvature, etc.). Also in this case, the same tooth can be presented to the input of the 3D deep neural network multiple times, where changes in the partition height can be introduced each time, thereby making the trained neural network robust.

[0095] In another embodiment, the input of the preprocessor can be configured to receive a 3D surface mesh generated by an optical scan of a complete clinical tooth. Such a surface mesh can be preprocessed as described above, including transforming such a surface mesh into a voxel representation suitable for the input space of a deep neural network.

[0096] In another embodiment, the training data 200 can be enhanced data from other more accurate sources. To this end, the clinical 3D data 202 can also include a 3D dataset of 3D crown models, which can be derived from optically generated 3D data generated by an intraoral scanner. Typically, the optically generated 3D data includes a 3D mesh containing multiple 3D crown models of the dentition, which can be segmented and transformed into multiple optically generated 3D datasets by a preprocessor 204. Each generated 3D dataset can produce a highly accurate voxel representation 208 of a 3D crown, which is formatted (in terms of size, position, and orientation) for input to a deep learning network and can be used to train a deep neural network.

[0097] Thus, if both an X-ray generated 3D dataset and a more accurate optically generated 3D dataset are available for a 3D tooth model, a deep neural network may be trained using a voxel representation of the crown derived from the optically generated 3D dataset, combined with a voxel representation of the root derived from an X-ray generated 3D dataset of an associated root model that is (part of) the target. In the case where the 3D crown data is derived from a relatively low resolution 3D CBCT scan, it may be beneficial to use the optically generated 3D data of the crown model (3D mesh of the crown) for training. In such a case, pre-processing may require an alignment step that sets the position and orientation of the crown 3D data to match the crown portion within the 3D image data of the full tooth.

[0098] The performance of the trained 3D deep learning network can be verified by comparing the predicted voxel representation 212 of the tooth root or the predicted voxel representation 214 of the entire tooth with the original (real world) 3D image data 202, as described below with reference to Figure 6. In the case where the input and output (target labels) of the deep neural network 210 are strictly divided into root parts and crown parts, this verification can be done by comparing the 3D image data of the predicted tooth root with the matching parts of the original (real world) 3D image data of the tooth root.

[0099] The preprocessor 204 can facilitate data enhancement by introducing relatively small transformations of the received 3D image data before applying the segmentation. Such transformations can, for example, include mirroring the 3D image data along the patient's sagittal plane and / or rescaling and relatively small rotations along any of three orthogonal axes.

[0100] like Figure 2 The training process depicted in produces a 3D deep neural network that is trained to accurately predict 3D root shapes based on an arbitrarily selected 3D crown (such as a voxel representation of the crown derived from 3D data generated by an intraoral scanner). The deep neural network will automatically learn the anatomical features available from the received voxel representation of the crown to generate an anatomically accurate prediction of (at least) the root representation, wherein the resulting root model will closely fit the crown model to form a complete tooth model.

[0101] Due to the fact that such features can be learned from any potential training sample (the sample is a voxel representation of a tooth crown), and the 3D deep neural network will determine which features are relevant, this method is able to more accurately use any relevant information in the voxel representation. In other words, where the prior art may be limited to a specific set of input parameters, the proposed method has the potential to utilize more input information and will determine during training which derivable features are relevant.

[0102] Furthermore, the 3D deep neural network will learn a generalization of the 3D image data representing the root portion based on the crown portion. However, this generalization is more flexible than the use of template roots in the prior art (taking into account the potentially different shapes that can be generated for predicted roots or complete teeth). It is also important to note that the method does not require separate inputs indicating the tooth class (canine, molar, etc.) and / or the patient class (age, gender, etc.). In fact, it is possible for the 3D deep neural network to identify the relevant features as a result of such classification directly from the received voxel representation. However, this depends (at least) on the number of available training samples, their variation in these categories, and the available memory bandwidth within the feature representation in the 3D deep neural network.

[0103] Figure 3An example of a 3D deep neural network architecture 300 used in a method and system for automatic root shape prediction from 3D crown image data according to an embodiment of the present invention is depicted. Received (pre-processed) 3D image data (e.g., a voxel representation that matches the voxel space of the input of the deep neural network) can be passed and processed through various (different types of) network layers that form the network. The network's architecture may include three different functional parts 301, 303, 305.

[0104] The first part 301 can be configured to gradually encode features with the help of 3D convolutional layers 304, 306, 308. These features are derived by a learned (obtained from training) set of 3D convolutional filters. The generated feature map obtained by such a layer will represent information relevant to the root shape prediction problem. Stacking multiple 3D convolutional layers leads to an increase in the (potential) field of view of each additional layer, and the ability to derive combined information from the information derived from the previous layers. In this regard, the input 302 can be considered as a first set of information.

[0105] Due to the nature of 3D convolutional layers, memory and processing requirements are relatively low compared to, for example, fully connected layers. Due to the desired prediction accuracy, there is a minimum size of the input (voxel) space. However, it is also desirable to perform processing on information derived from the entire input space, for example with the aid of densely connected layers. For this reason, a reduction in the size of the (3D) feature space can be achieved, for example, by using striding or subsampling within such 3D convolutional layers. Alternatively, the convolutional layer may be followed by a pooling layer, such as a max pooling layer well known in the art.

[0106] Thus, when data has passed through the convolutional layers, the internal representation may be passed to a series of densely connected layers 310, 312, which are configured to act as an intermediary for inferring root shape selection from previous convolutional activations. The densely connected layers may form the second portion of the network 303. At this point in the neural network, such layers have the ability to access information derived from the entire input space.

[0107] After the densely connected layers, the resulting activations may be passed through a series of 3D deconvolutional layers 314, 316, 318 forming the third part 305 of the network. Such layers are also referred to as transposed convolutional layers. The parameters for such layers may be chosen in a way that the (3D) space represented at one point in the network architecture may be expanded. Again, utilizing such layers reduces the hardware (memory, processing speed) requirements required for both the architecture and (in time) predictions that are actually possible. Alternatively, conventional 3D convolutional layers may be employed, followed by (3D) upsampling layers as known in the art.

[0108] In an alternative embodiment, layers from part one are also directly connected to layers from part three. In this way, the network (also referred to in the art as a U-network) "forwards" feature maps from the compression stage (represented by part one) to the decompression stage (represented by part three). This can be done between the results from the convolutional layer and the deconvolution layer when each set of convolutions and deconvolutions operates at at least the same encoded spatial resolution. This can improve the quality of the predictions and / or reduce the time it takes for training to converge to a desired accuracy. Reference Fig. 11B Describe an example of a U network architecture.

[0109] Variations in the number of layers and the defining parameters of these layers (e.g., different activation functions (e.g., sigmoid, tanh, elu, softmax), the amount and size of kernels (of filters), the use of subsampling and dilation (s), and other functional layers (such as dropout layers)) may be used in embodiments without losing the essence of deep neural network design. The same applies to varying parameter initiation methods known in the art. (e.g., (glorot) normal or uniform, He normal, etc.) The final or output layer 324 may result in individual voxel activations, where each voxel activation represents a probability measure defining the probability that the voxel is part of a 3D tooth root or complete tooth. The voxel activations may be thresholded to obtain a binary prediction of the tooth root or complete tooth 3D data 320. In the case where the final layer employs a softmax activation (which is common for classification problems), for the "tooth root" or "complete tooth" class, voxel activations represented above 0.5 are thresholded to a binary "1" or "true" indicating that the voxel is predicted to be a "tooth root" or "tooth". Alternatively, such binary attribution is done based on the argmax over the class to be predicted, effectively attributing the voxel to the class with the highest activation. Thus, if the activation of this class is highest for such voxel, then it can be predicted that the voxel is part of the root or complete tooth class.

[0110] Thus, in an embodiment, the first 3D deep neural network may comprise three sequential parts, the first part comprising at least a plurality of 3D convolutional layers configured such that the spatial resolution of the derived information obtained from the part is reduced compared to the input resolution, the second part being configured to have the ability to process all information obtained from the first part in parallel, preferably by means of densely connected layers, the third part comprising at least a plurality of 3D convolutional layers configured such that the spatial resolution of the final output of the total network has at least the same resolution as the input resolution, preferably by means of transposed convolutions.

[0111] You can use reference Figure 2The network is trained using the training data described. For each sample, a matching 3D representation of a single root or complete tooth can be used to determine the loss between the desired output 322 and the actual output 320. This loss can be used as a measure to adjust the parameters within each layer of the deep neural network during training. Optimizer functions can be used during training to help with the efficiency of the training effort (e.g., stochastic gradient descent, rmsprop, adam, etc.). The network can be trained for any number of iterations until the internal parameters result in the desired accuracy of the results. When properly trained, a voxel representation containing a representation of at least the crown of a tooth can be presented as an input to the deep neural network and can be used to derive predictions of the corresponding 3D voxel representation of the root or complete tooth.

[0112] Figure 4A and 4B Schematic diagrams of pre-processing and post-processing 3D data according to various embodiments of the present invention are depicted. In particular, Figure 4A A flow chart depicting a process performed by a preprocessor. In a first step 402, the preprocessor may derive individual 3D crown data from 3D data originating from different sources (e.g., a 3D surface mesh of the dentition generated by an intraoral scanner or CBCT data of the dento-maxillofacial structure). In such a case, the preprocessing may include performing a segmentation process, wherein the 3D data received by the preprocessor is segmented into individual 3D crown data. In the case of 3D CBCT data, the segmentation process may be performed by a separate neural network that classifies voxels into specific voxel types, such as voxels representing a portion of a tooth, and forms a 3D dataset of teeth of the dento-maxillofacial structure. Reference is made below to Figure 8 An example of such a segmentation process is further described. Where the 3D data comprises a 3D mesh representing a dentition generated by an intraoral scanner, the pre-processor may comprise a segmentation process wherein the 3D mesh of the dentition is segmented into individual 3D meshes of tooth crowns.

[0113] Thereafter, the preprocessor may transform the individual 3D tooth data in a voxel representation of the tooth crown (step 404). Thus, in the case of a 3D surface mesh, the 3D mesh may be transformed into a volume of voxels. An interpolation algorithm may be used to fill the voxel space with voxels having a predetermined first voxel value (e.g., a "zero" or "background" value) where no tooth surface is present, and having a second voxel value of "one" or "tooth present" value for those voxels that coincide or nearly coincide with the 3D surface defined by the mesh. In the case of a voxel representation of a complete tooth, as described in reference to FIG. Figure 2 As described above, the voxel representation of the crown data can be determined by dividing the data of the complete tooth into crown data and root data. Therefore, the voxel representation includes a volume of voxels, such as a rectangular box, wherein the 3D surface of the crown is represented by the voxels within the volume.

[0114] In an embodiment, the preprocessor may further perform step 406 of setting voxels enclosed by the surface mesh to a second voxel value such that the 3D voxel representation represents a solid object in 3D space. Note that a closed surface of, for example, a voxel representation of a portion of a crown may be a surface that is not yet completely enclosed (e.g., this may be the case when the voxel representation of the tooth is partitioned into crown and root portions, or when the voxel representation of the crown is based on a segmentation of a 3D mesh generated from an intra-oral scan). If during training the voxel representation of the tooth is partitioned along a plane into a volume of crown voxels and a volume of root voxels (see also above Figure 2 ), then step 406 may include closing the volume and determining a solid representation by creating faces within the dividing planes, thereby enclosing the existing surface.

[0115] In the event that the edges represented by such voxels are not in the same plane, step 406 may include determining a 3D point which will be the centroid of the "highest" surface (in the real-world "up-down" direction, with the crown side of the tooth oriented "up") defined from the highest available fully closed circle as intersecting a plane perpendicular to said "up-down" direction. The original (partitioned) volume may then be closed by creating faces defined by the determined 3D points (within the volume that would be enclosed by the original received surface) and the vertices along the open edges of the original received surface. Due to the nature of the tooth shape, this method will close and fill the open surfaces in a manner that will ensure that the resulting volume is indeed (a portion of) the tooth volume in the real world. In some embodiments, the received 3D crown model may already be closed, as is the case for a model generated, for example, by segmentation from CBCT data, as described below with reference to FIG. Figure 8 Further described.

[0116] In an embodiment, the (rectangular) volume of voxels may be associated with a coordinate system, such as a 3D Cartesian coordinate system, so that the 3D voxel representation of the tooth may be associated with an orientation and dimension. However, the orientation, position and / or dimension of the tooth model may not be standardized. The 3D deep neural network may be sensitive to the orientation of the teeth and may have difficulty predicting the root shape based on the non-standardized dimensions and random crown orientation in the 3D image volume. Therefore, it may be beneficial to present the 3D deep neural network (including during training and inference) with a voxel representation in which the dimensions of the voxels (i.e., the real-world length, width and height of the space represented by the voxels) are the same for each presented sample or representation. In addition, it may be beneficial to present the 3D deep neural network (during training and inference) with a voxel representation in which the orientation, position and dimension of the tooth sample are the same for each sample.

[0117] To address this issue, during preprocessing of the 3D input data, the orientation, position, and dimension of each crown sample can be normalized according to the normalization used to normalize the training data. The preprocessor can ensure that the voxels representing the crown (i.e., the voxel representation as described above with reference to steps 404 and / or 406) can be transformed using, for example, a canonical transformation, so that the orientation, position, and dimension of the transformed sample are unified and normalized (step 410). The preprocessor can use spatial information of the 3D data source to complete this normalization of orientation, position, and / or dimension. In an embodiment, the spatial information can be determined by the preprocessor by examining the orientation, position, and dimension of the data source (step 408). For example, when the crown samples are derived from a 3D (CB) CT data stack of a volume of custom voxels or from an intraoral scan, the orientation, position, and dimension of each tooth or crown sample can be determined by the system.

[0118] The preprocessor may check the orientation, position and / or dimensions derived from the original 3D image data set, and if these values ​​do not match the expected (normalized) input format for the deep learning network, a transformation may be applied. In an embodiment, the transformation may include a 3D rotation for reorienting the orientation of the sample in 3D space. The reorientation may result in a (normalized) orientation expected by the neural network (step 410, described in more detail above with reference to feature 102). In an embodiment, the transformation may include a 3D translation for repositioning the sample in 3D space. The repositioning ensures the (normalized) positioning of the 3D crown model within any spatial dimensions expected by the 3D neural network. In another embodiment, the transformation may include a 3D scaling for rescaling the dimensions of the sample in 3D space (step 412). The rescaling ensures the (normalized) dimensions of the sample expected by the 3D neural network.

[0119] In an embodiment, (part of) the pre-processor may comprise a (third) deep neural network trained to perform the transformation as described above. In an embodiment, the trained neural network as described in European patent application no. 18181421.1 entitled "Automated determination of a canonicalpose of a 3D dental structure and superimposition of 3D dental structures using deep learning", which is incorporated herein by reference.

[0120] Figure 4BA flow chart depicting a process performed by a post-processor. In an embodiment, the post-processor may combine (merge) a voxel representation of a crown 414 and a voxel representation of a root shape 416 generated by a 3D deep neural network in a merged voxel representation 420 of a complete tooth. In an embodiment, the merged voxel representation may be transformed into a 3D mesh 422 of the complete tooth. Alternatively, the post-processor may transform the voxel representation of the root shape into a 3D mesh and combine (merge) the 3D mesh representation of the root shape with the 3D mesh representation of the crown. In yet another embodiment, the 3D deep neural network may generate a voxel representation 418 of a complete tooth, and the post-processor may transform the output of the 3D deep neural network directly into a 3D mesh of the complete tooth.

[0121] Figure 5A and 5B Shown is a diagram of the 3D clinical voxel representation compared to the predicted voxel representation. Figure 5A are examples of "molar" type teeth, and Figure 5B are examples of "canine" type teeth. (All are holdout samples, or samples not used during training). These representations are rendered in voxel space (in an x, y, z coordinate system) defining a volume of 160x100x100 voxels. The x and y axes of the visualization make up the coronal plane of the patient. The slice number above the figure indicates the distance along the z axis at which the slice is visualized. For example, Figure 5A Three slices of voxel space (in the xy plane) at three different positions on the z-axis are depicted. These illustrations clearly indicate the accuracy of the root shape predicted by the 3D deep neural network. The voxel size in the case of these illustrations is 0.2 mm in each orthogonal direction.

[0122] The predicted teeth are shown as the resulting merge of the voxel representation of the crown provided as input to the system and the predicted root in the voxel space containing the complete tooth. Each figure includes a slice of the representation of the clinical 3D model and a slice of the complete predicted 3D model of the tooth, which is placed on the clinical 3D model. The figures so formed illustrate the overlap between the clinical root shape and the predicted root shape in medium gray shades, where lighter gray shades (representation of the clinical tooth is present, but the predicted tooth representation is not) and darker gray shades (representation of the clinical tooth is not present, but the predicted tooth representation is present) indicate the part where the predicted representation deviates from the clinical representation.

[0123] The partitioning used for training and prediction in the case of this exemplary embodiment utilizes a "straight" partitioning, as shown by lines 5021 and 5022 (which represent planes in the xz direction). Thus, the upper (first) portion 504 of the volume defines voxels representing the crowns of the teeth, which are used as inputs to the deep neural network. The lower portion 506 defines the voxels predicted by the neural network based on the inputs. Thus, in this example, the voxel space of the input to the deep neural network represents a volume of 60x100x100 voxels, while the voxel space of the output of the deep neural network represents a volume of 100x100x100 voxels. Further details of an exemplary system for generating these results are described below with reference to FIG6 .

[0124] Due to the nature of the processing performed by the deep neural network, there are no hard transition edges between the original crown portion and the predicted root portion, as is the case with systems that utilize a template 3D model, such as those in the prior art. The transition between the original crown portion learned and therefore predicted by the system and the predicted root portion is considered to be a smooth transition (see, e.g., 508, 510), with little or no need for additional post-processing, such as smoothing, etc. The illustrations also show the accuracy of the 3D image data throughout the prediction. The predicted root portion overlaps the clinical tooth to a large extent. More details on considering this accuracy can be found with reference to FIG6. However, it is worth noting that in these illustrations, the predicted 3D root portion shown (for Figure 5A or 5B) below line 5021 or 5022 respectively) is automatically predicted by the system based on the information above those same lines.

[0125] Fig. 6A and Figure 6B Shown from reference Figure 5A and 5B Table of dice coefficients determined for the same exemplary embodiment of . In more detail, by comparing voxel representations of actual 3D clinical teeth with voxel representations of 3D complete teeth predicted by a single trained 3D deep neural network, a voxel size of 0.2 mm was adopted in each dimension, these were the results on a retained set of teeth (which means that these teeth were not part of the training set). A "straight" division between crown and root was adopted, and the input and output voxel spaces were separated. About 900 3D clinical tooth models derived from CBCT scans were used for training, and in addition augmentation of 3D image data was also utilized.

[0126] This training data contains various tooth classes, including incisors, canines, premolars, and molars. For example, Fig. 6A The Dice coefficients in are shown divided into these categories. Figure 6BThe tables in show the same total set of results, which are aggregated into different sets of categories, namely according to the individual tooth types as can be found in the dentition and according to the FDI World Dental Federation symbol. It is important to emphasize, however, that all results are generated by a single 3D deep neural network. The Dice coefficient or Dice score is a measure of object similarity. This parameter is commonly used in image segmentation to compare the predicted results with the ground truth (3D clinical teeth in the case of this system). The Dice score can be described as the size of the overlap between two objects (in this case, taken at a per-voxel level) divided by the total size of the two objects. This metric takes into account true positives, total positives and false positives, with higher scores indicating more accurate results.

[0127] like Fig. 6A As shown in the results in , when the predictions of the teeth are compared to the clinical 3D model (or ground truth), in general, the class of premolars is on average the "least accurate" (in this case, the lowest Dice coefficient). However, note that this table only shows the results of an exemplary embodiment. The set of holdout samples has been randomly selected, and a different selection would lead to different results. (This can also be seen in Figure 6B , where it is clear that tooth indices 18, 38, and 48 are not present in the randomly selected held-out dataset). Moreover, the exact 3D deep neural network architecture and parameters used in such networks, training considerations such as the choice of input and output spatial dimensions, augmentation options during training, etc., will affect the accuracy of the prediction results. When a larger training sample dataset is used, the accuracy and Dice score are expected to be higher.

[0128] Figure 7 The 3D model of a clinical tooth generated by a system according to an embodiment of the present invention is illustrated. In particular, Figure 7 3D models of clinical teeth are illustrated, combined with predictions of complete teeth by receiving only the crown portion. These are results from the same exemplary embodiment as described above with reference to Figures 5 and 6. Individual examples from four different classes of teeth (incisors 702, canines 704, premolars 706, and molars 708) are shown, as determined by a system comprising a trained 3D deep neural network. These illustrations are rendered based on individual 3D models (i.e., 3D meshes) of the entire teeth, which were generated based on voxel representations of the predicted roots as described above. The original 3D clinical tooth models are rendered in a lighter grey, while the 3D tooth models generated based on the predicted voxel representations of the teeth are rendered in a darker shade. Each generated 3D tooth model is overlaid with its original 3D clinical tooth model to provide a visual comparison of the results. Again, it is worth mentioning that these are holdout samples. In Figure 7In the visualization of , some darker grey patches (726) are visible at the crown portions. This is the result of overlaying the original 3D clinical tooth model and the complete predicted 3D model of the tooth, the latter having undergone a step of merging the voxel representation of the predicted roots with the received voxel representation of the crown. Where such patches are visible, the rendering engine may have determined that the surface of the 3D predicted complete tooth is the least important part that is "higher" on the surface, hence the darker shade of the predicted tooth is visible. Comparison with the original 3D clinical crown model (710 to 716) shows that the trained deep learning network is able to generate accurate predictions of root shapes (718 to 724), including (where applicable) the number of roots (724).

[0129] As described above, in some embodiments, the 3D data provided to the input of the system may represent a dentofacial structure, including voxels associated with the jaw, teeth, and (a portion of) nerves. In those embodiments, it is required to segment the dentofacial structure into separate parts, such as the jaw and individual teeth, in order to determine the 3D voxel representation of a single tooth crown that can be used by a 3D deep learning network. In addition, for the purpose of training the 3D deep learning network, a large number of complete tooth shapes may be required in order to learn a deep neural network to predict the root shape based on the crown shape. Considering the availability and relative ease of creating CBCT scans, the automatic segmentation of 3D teeth from such CBCT scans will prove to be very beneficial. Therefore, in those cases, the present invention also includes a computer system and a computer-implemented method for classifying, segmenting, and 3D modeling individual teeth of a dentition in a dentofacial structure using a 3D deep neural network, wherein the dentofacial structure is represented by 3D image data, and the 3D image data is defined by a series of images forming a CT image data stack (particularly a cone beam CT (CBCT) image data stack). The 3D image data may include voxels forming a 3D image space of the dentofacial structure.

[0130] Such a computer system according to the present invention may include at least one deep neural network that is trained to classify a 3D image data stack of a dental and maxillofacial structure into different categories of voxels, wherein each category may be associated with a different part of the structure (e.g., teeth, jaws, nerves). The computer system may be configured to perform a training process that iteratively trains (optimizes) one or more deep neural networks based on one or more training sets, wherein the training set may include an accurate 3D model of the dental and maxillofacial structure. These 3D models may include optically scanned dental and maxillofacial structures (teeth and / or jaws). Once trained, the deep neural network may receive a 3D image data stack of a dental and maxillofacial structure and classify the voxels of the 3D image data stack. Before presenting the data to the trained deep neural network, the data may be preprocessed so that the neural network can efficiently and accurately classify the voxels. The output of the neural network may include different sets of voxel data, wherein each set may represent a different part of the 3D image data, such as teeth or jaws. The classified voxels may be post-processed to reconstruct an accurate 3D model of the dental and maxillofacial structure.

[0131] Described below in more detail is a computer system comprising a trained neural network for automatically classifying voxels of dentofacial structures, training of the network, preprocessing of 3D image data before feeding it to the neural network, and post-processing of voxels classified by the neural network.

[0132] Figure 8 A flow chart for training a deep neural network for classifying dentofacial 3D image data according to an embodiment of the present invention is depicted. Training data is used to train a 3D deep learning neural network so that voxels of a 3D CT scan of a dentofacial structure can be automatically classified. As shown in this figure, a representation of a dentofacial complex 802 can be provided to a computer system. The training data can include a stack of CT image data 804 of a dentofacial structure and an associated 3D model, such as 3D data 806 generated by an optical scan of the same dentofacial structure. Such 3D data can be referred to as 3D optical scan data. Examples of such 3D CT image data and 3D optical scan data are shown in FIG. Fig. 9A and 9B Shown in. Fig. 9A DICOM slices associated with different planes (eg, an axial plane 902 , a frontoparietal or coronal plane 904 , and a sagittal plane 906 ) of a 3D CT scan of the dento-maxillofacial structure are depicted. Fig. 9B3D optical scan data of the dental and maxillofacial structure is depicted. The computer can form a 3D surface mesh 808 of the dental and maxillofacial structure based on the optical scan data. In addition, an alignment function 810 can be used, which is configured to align the 3D surface mesh with the 3D CT image data. After alignment, the representation of the 3D structure provided to the input of the computer uses the same spatial coordinate system. Based on the aligned CT image data and the 3D surface mesh, the position features 812 and the classified voxel data 814 of the optically scanned 3D model can be determined. The position features and the classified voxel data can then be provided to the input of the deep neural network 816 together with the image stack 804.

[0133] Therefore, during the training phase, the 3D deep learning neural network receives the 3D CT training data and the position features extracted from the 3D CT training data as input data, and uses the classified training voxels associated with the 3D CT training data as target data. By minimizing the loss function representing the deviation of the output of the deep neural network to the target data (i.e., the classified voxel data) representing the expected output for the predetermined input, an optimization method can be used to learn the optimal values ​​of the network parameters of the deep neural network. When the minimum value of the loss function converges to a certain value, the training process can be considered suitable for the application. Figure 8 The training process described in the Figure 1 uses 3D position features in combination with training voxels that can be (at least partially) derived from 3D optical scan data to provide a high-quality training set for a 3D deep learning neural network. After the training process, the trained network is able to accurately classify voxels based on a 3D CT image data stack.

[0134] Fig. 10A and 10B A high-level schematic diagram of a deep neural network architecture used in a method and system configured to classify and segment 3D voxel data of a dentofacial structure is depicted. A deep neural network can be implemented using one or more 3D convolutional neural networks (3D CNNs). A convolutional layer can employ activation functions associated with neurons in the layer, such as a sigmoid function, a tanh function, a relu function, a softmax function, and the like. A deep neural network can include multiple 3D convolutional layers, where slight variations in the number of layers and their defining parameters (e.g., different activation functions, kernel amounts and sizes, and additional functional layers such as dropout and batch normalization layers) can be used in an implementation without losing the essence of the deep neural network design.

[0135] like Fig. 10AAs shown in , the network may include multiple convolution paths, each of which is associated with a set of 3D convolution layers. In an embodiment, the network may include at least two convolution paths, a first convolution path associated with a first set of 3D convolution layers 1006 and a second convolution path associated with a second set of 3D convolution layers 1008. The first and second convolution paths may be trained to encode 3D features derived from received 3D image data associated with voxels of inputs provided to the first and second convolution paths, respectively. In addition, in some embodiments, the network may include at least another (third) convolution path associated with a third set of 3D convolution layers 1007. The third convolution path may be trained to encode 3D features derived from received 3D position feature data associated with voxels of inputs provided to the third path.

[0136] Alternatively, in another embodiment, instead of another convolution path trained based on 3D position feature data, the 3D position feature data is also associated with the intensity value of the voxel provided to the input of the first and second convolution paths. Therefore, in this embodiment, the first and second convolution paths can be trained based on the following training data, which includes a 3D data stack of voxel values ​​containing intensity values ​​and position feature information.

[0137] exist Fig. 10B The functionality of the different paths is illustrated in more detail in . As shown in this figure, voxels are fed to the input of the neural network. These voxels are associated with a predetermined volume, which can be referred to as an image volume 10013. The total volume of voxels can be divided into first voxel blocks, and the 3D convolution layer of the first path 10031 can perform a 3D convolution operation on each first voxel block 10011 of the 3D image data. During processing, the output of each 3D convolution layer can be the input of a subsequent 3D convolution layer. In this way, each 3D convolution layer can generate a 3D feature map representing features of the 3D image data fed to the input. Therefore, a 3D convolution layer configured to generate such a feature map can be referred to as a 3D CNN feature layer.

[0138] like Fig. 10B As shown in , the convolutional layer of the second convolution path 10032 can be configured to process a second voxel block 10012 of 3D image data. Each second voxel block is associated with a first voxel block, wherein the first and second voxel blocks have the same central origin in the image volume. The volume of the second block is greater than the volume of the first block. Moreover, the second voxel block represents a downsampled version of the associated first voxel block. In other words, the voxel resolution of the second block is lower than the voxel resolution of the associated first block. The downsampling factor can be any suitable value. In an embodiment, the downsampling factor can be selected between 20 and 2, preferably between 10 and 3.

[0139] Thus, the 3D deep neural network may include at least two convolution paths. The first convolution path 10031 may define a first set of 3D CNN feature layers (e.g., 5-20 layers) configured to process input data (e.g., a first voxel block at a predetermined position in the image volume) at a first voxel resolution (e.g., the voxel resolution of the target (i.e., the resolution of the voxels of the 3D image data to be classified)). Similarly, the second convolution path may define a second set of 3D CNN feature layers (e.g., 5-20 layers) configured to process input data at a second voxel resolution (e.g., a second voxel block, wherein each block in the second voxel block 10012 has the same center point as its associated block in the first voxel block 10011). Here, the second resolution is lower than the first resolution. Therefore, the volume represented by the second voxel block in actual dimensions is larger than that of the first block. In this way, the second 3D CNN feature layer processes voxels to generate a 3D feature map including information about the (direct) neighborhood of the associated voxels processed by the first 3D CNN feature layer.

[0140] Thus, the second path enables the neural network to determine contextual information, i.e. information about the context (e.g. its surroundings) of a voxel of the 3D image data presented to the input of the neural network. By using multiple (parallel) convolution paths, the 3D image data (input data) and the contextual information about the voxels of the 3D image data can be processed in parallel. The contextual information is useful for classifying dento-maxillofacial structures, especially in the case of CBCT image data, which often include densely packed tooth structures that are difficult to distinguish.

[0141] In an embodiment, the neural network of 10B may also include a third convolution path 10033 of a third 3D convolution layer set, which is trained to process a specific representation of a 3D position feature 1004 that can be extracted from the 3D image data. Extracting 3D position features from 3D image data can be implemented as a preprocessing step. In an alternative embodiment, instead of using the third convolution path to process the 3D position features, 3D position information including the 3D position features can be associated with the 3D image data provided to the input of the deep neural network. In particular, a 3D data stack in which each voxel is associated with an intensity value and position information can be formed. Therefore, the position information can be paired according to each applicable received voxel, for example, by adding the 3D position feature information as an additional channel to the received 3D image information. Therefore, in this embodiment, the voxels of the voxel representation of the 3D maxillofacial structure at the input of the deep neural network can be associated not only with voxel values ​​representing, for example, radio intensity values, but also with 3D position information. Therefore, in this embodiment, during the training of the convolutional layers of both the first and second convolutional paths, information derived from both 3D image features and 3D position features may be encoded in these convolutional layers.

[0142] The outputs of the set of 3D CNN feature layers are then merged and fed to the input of a set of fully connected 3D CNN layers 1010, which are trained to derive the expected classification of voxels 1012 provided at the input of the neural network and processed by the 3D CNN feature layers.

[0143] The set of 3D CNN feature layers are trained (through their learnable parameters) to derive and deliver the best useful information that can be determined from their specific inputs, and the fully connected layers encode the parameters that will determine how the information from the previous paths should be combined to provide the best classified voxels 1012. Thereafter, the classified voxels can be presented in image space 1014. Thus, the output of the neural network is the classified voxels in the image space corresponding to the image space of the voxels at the input.

[0144] Here, the output of the fully connected layer (the last layer) can provide multiple activations for each voxel. Such voxel activations can represent a probability measure (prediction) defining the probability that the voxel belongs to one of multiple categories (e.g., tooth structure categories, such as teeth, jaws, and / or neural structures). For each voxel, the voxel activations associated with different tooth structures can be thresholded to obtain a classified voxel.

[0145] 11-13 illustrate a method of determining 3D positional features in a 3D image data stack representing a 3D dental and maxillofacial structure and examples of such positional features. Specifically, in the case of manually designed features, both the 3D image data stack and the associated 3D positional features are provided as input to a 3D deep neural network so that the network can accurately classify voxels without the risk of overfitting. The conversion based on real-world dimensions ensures comparable inputs that are independent of the input image resolution.

[0146] Manually designed 3D position features can provide information to the 3D deep neural network about the position of voxels in the image volume relative to a reference plane or reference object in the image volume. For example, in an embodiment, the reference plane can be an axial plane in the image volume that separates voxels associated with the upper jaw from voxels associated with the lower jaw. In another embodiment, the reference object can include a curve, such as a 3D curve, approximating at least a portion of a dental arch of a tooth in 3D image data of a dento-maxillofacial structure. In this way, the position features provide a means for the first deep neural network to encode an abstraction indicating the likelihood of a jaw, tooth, and / or neural tissue associated with each voxel in different locations in the image volume. These position features can help the deep neural network efficiently and accurately classify voxels of a 3D image data stack, and can reduce the risk of overfitting.

[0147] In order to determine reference planes and / or reference objects in the image volume that are useful in the classification process, the feature analysis function can determine a predetermined intensity value or voxels above or below a predetermined intensity value. For example, voxels associated with bright intensity values ​​can be associated with teeth and / or jaw tissue. In this way, information about the position and orientation (e.g., rotation angle) of the teeth and / or jaws in the image volume can be determined by the computer. If the feature analysis function determines that the rotation angle is greater than a predetermined amount (e.g., greater than 15 degrees), then the function can correct the rotation angle to zero, as this is more conducive to accurate results.

[0148] Fig.11A An example of a flow chart 1102 of a method for determining manually designed 3D position features in 3D image data 1104, such as a 3D CT image data stack, is illustrated. This process may include determining one or more 3D position features of a dental and maxillofacial structure, wherein the one or more 3D position features are configured for input to a 3D deep neural network (such as the one or more 3D position features of the dental and maxillofacial structure described above). Fig. 10B Discussed). The manually designed 3D position feature defines the position information of a voxel in the image volume relative to a reference plane or reference object in the image volume, such as the distance, such as the vertical distance, between a voxel in the image volume and a reference plane in the image volume that separates the upper jaw from the lower jaw. It can also define the distance between a voxel in the image volume and a dental reference object (a dental arch in the image volume). It can also define the position of a cumulative intensity value in a second reference plane of the image volume, the cumulative intensity value at a point in the second reference plane including the cumulative intensity values ​​of voxels on or near a normal passing through the point in the reference plane. Examples of 3D position features are described below.

[0149] In order to determine a reference object providing position information of a dental arch in 3D image data of a dental and maxillofacial structure. A fitting algorithm can be used to determine a curve that fits predetermined points in a point cloud of different (cumulative) intensity values, such as a curve following a polynomial formula. In an embodiment, a point cloud of intensity values ​​in an axial plane (xy plane) of the image volume can be determined. The cumulative intensity value of a point in the axial plane can be determined by adding the voxel values ​​of the voxels located on the normal line passing through the point in the axial plane. The intensity values ​​in the axial plane thus obtained can be used to find a curve that approximates the dental arch of the teeth.

[0150] Fig. 11B An example of a machine learning method according to an embodiment of the present invention is described, which can be used to generate (non-artificially designed) relevant 3D position features. In particular, Fig. 11BAn exemplary 3D deep neural network architecture is depicted that can be trained to generate desired features to be processed by a segmentation 3D neural network. After training, such a trained model can be used similarly to method 502 as a preprocessor that derives relevant 3D position features based on the entire received 3D data set.

[0151] Similar to manually designed 3D position features, the goal is to incorporate information related to the entire received 3D dataset (or at least a large part thereof) into the 3D position features for use in a segmentation 3D deep learning network, which may be related to the task of automatic classification and segmentation and may not otherwise be available from one or more subsample sets provided to the segmentation 3D deep learning network. Again, like manually designed 3D position features, such information should be made available for each voxel in the received 3D dataset.

[0152] One of the possible ways to implement such a machine learning approach to automatically generate 3D position features is a trained deep neural network. Such a network can be trained to derive 3D position features based on an input 3D dataset (e.g., a voxel representation of the maxillofacial structure) provided as input to a 3D segmentation deep neural network. In an embodiment, the pre-processed deep neural network can be such as Fig. 11B A 3D U-net type deep neural network is shown. Due to the limitations of available processing (mainly memory requirements), such an architecture will not be able to operate at the resolution of the received voxel representation. Therefore, an interpolation algorithm can be used to downsample the first input 3D data set (a first voxel representation of a first resolution (e.g., 0.2x0.2x0.2mm per voxel)) to a second voxel representation of a second lower resolution (e.g., a resolution of 1x1x1mm per voxel). Thereafter, a 3D deep neural network trained based on the voxel representation of the second resolution can generate 3D position feature information for each input voxel. An interpolation algorithm can be used to scale this information up to the original first resolution. In this way, the resulting 3D position features are (spatially) consistent with the voxels of the first voxel representation, thereby generating relevant information for each voxel of the first input 3D data set, while taking into account the information of the entire received 3D data set (its aggregated version).

[0153] Such a pre-processing 3D deep neural network can be trained to approximate a desired target value (desired 3D position feature). In this specific example, the target can be, for example, a class indication for each voxel at the resolution at which the pre-processing 3D deep neural network operates. Such a class indication can, for example, be derived from the same classified training voxel pool 136, but is downsampled in the same way as the received 3D data set is downsampled. It is noted that such a pre-processing machine learning method can effectively be considered a coarse pre-segmentation, specifically a pre-segmentation that can potentially access information from the entire received 3D voxel representation (or a large part thereof). Pairing the coarse pre-segmentation information with applicable voxels of the received 3D image space (e.g., by means of magnification) causes these 3D position features to be processed in parallel with the received 3D image data to move towards a result at the received 3D image resolution.

[0154] The preprocessing network can be implemented using various 3D neural network layers such as convolutional layers (3D CNN), 3D max pooling layers, 3D deconvolutional layers (3D inverse CNN), and densely connected layers. These layers can use various activation functions such as linear, tanh, ReLU, PreLU, sigmoid, etc. The number of filters, filter sizes, and subsampling parameters of the 3D CNN and inverse CNN layers can be different. The parameter initialization methods of the 3D CNN and inverse CNN layers and the densely connected layers can be different. Dropout and / or batch normalization layers can be used throughout the architecture.

[0155] Following the 3D U-Net architecture, during training, various filters in the 3D CNN and 3D Inverse CNN layers learn to encode meaningful features that will contribute to prediction accuracy. During training, the matched 3D image dataset 1122 and the encoded matched 3D position features 1160 are used to optimize the prediction of the latter from the former. A loss function can be used as a measure to be minimized. This optimization work can be assisted by using an optimizer (such as SGD, Adam, etc.).

[0156] Such an architecture can employ various internal resolution scales, thereby effectively reducing 1126, 1130, 1134 as the result from the previous set of 3D CNN layers 1124, 1128, 1132 by max pooling. The term "meaningful features" here refers to the (successive) derivative of information related to determining the target output value and is also encoded by 3D inverse CNN layers, which effectively perform upscaling while employing filters. By combining the data 1140, 1146, 1152 obtained from such 3D inverse CNN layers 1138, 1144, 1154 with the data from the "last" 3D CNN layer (1132 to 1140, 1128 to 1146 and 1124 to 1152) operating at the same resolution, highly accurate predictions can be achieved. Additional 3D CNN layers 1142, 1148, 1154 can be used throughout the upscaling path.

[0157] When used for inference, after being trained with the encoded internal parameters in a manner such that validation produces sufficiently accurate results, an input sample may be presented and the 3D deep learning network may produce predicted 3D position features 542 .

[0158] Fig.12 An example of a reference object for determining manually designed 3D positional features, in this case a curve that approximates a dental arch, is provided in . In this example, a cloud of points in the axial (xy) plane indicates that areas of higher intensity values ​​(bright white areas) may indicate areas of tooth or jaw structure. To determine the dental arch curve, a computer may determine areas in the axial plane of the image volume that are associated with bright voxels (e.g., voxels having intensity values ​​above a predetermined threshold), which may be identified as tooth or jaw voxels. These areas of high intensity may be used to determine a crescent-shaped arrangement of bright areas that approximate a dental and maxillofacial arch. In this manner, dental arch curves may be determined that approximate the average of the dental and maxillofacial arches for the maxilla and mandible, respectively. In another embodiment, separate dental arch curves associated with the maxilla and mandible may be determined.

[0159] Figures 13A-13E Examples of 3D position features of 3D image data according to various embodiments of the present invention are depicted.

[0160] Fig.13A Depicted are (left) an image of a slice of a sagittal plane of a 3D image data stack, and (right) an associated visualization of a so-called height feature of the same slice. Such a height feature may encode the z position (height 1304) of each voxel in the image volume of the 3D CT image data stack relative to a reference plane 1302. The reference plane (e.g., an axial or xy plane, which is determined to be (the best approximation of) the xy plane) is approximately equidistant from the maxilla and mandible and their constituent teeth.

[0161] Other 3D position features may be defined to encode spatial information in the xy space of the 3D image data stack. In an embodiment, such position features may be based on a curve that approximates (a portion of) a dental arch. Such position features may be defined in Fig. 13B , which depicts (left) a slice from a 3D image data stack, and (right) a visualization of a so-called progression feature for the same slice. This progression feature is based on a curve 1306 that approximates the dental arch and defines relative distances 1308 measured along the curve. Here, zero distance can be defined as the point 1310 where the derivative of a quadratic polynomial on the curve is (approximately) zero. When moving in either direction on the x-axis from this point (e.g., the point where the derivative is zero), the distance traveled increases.

[0162] Another 3D position feature based on the dental arch curve can define the shortest (perpendicular) distance from each voxel in the image volume to the dental arch curve 1306. Therefore, this position feature can be referred to as a "distance feature". Fig. 13C An example of such a feature is provided in , which depicts (left) a slice from a 3D image data stack, and (right) a visualization of a distance feature for the same slice. For this feature, a distance of zero indicates that the voxel is located on the dental arch curve 1308.

[0163] Another 3D position feature can define the position information of each tooth. Fig.13D An example of such a feature (which may also be referred to as a tooth feature) is provided in , which depicts (left) a slice in a 3D image data stack, and (right) a visualization of the tooth feature for the same slice. The tooth feature can provide information that will be used to determine the likelihood of finding a voxel for a certain tooth at a certain location in voxel space. After a reference plane is determined (such as 1302), this feature can encode the separate sum of the voxels on the normals of any plane (e.g., the xy plane or any other plane). Therefore, this information provides the neural network with a "view" of all the information in the original space as summed over the plane normals. This view is larger than the view to be processed if this feature is not included, and can provide a method for distinguishing the presence or absence of hard structure based on all the information in the selected spatial directions (such as 1312 for the xy plane). 1,2 as shown in ).

[0164] Fig.13E It is shown that the machine learning preprocessor (especially as regards Fig. 10BVisualization of 3D position features generated by the described 3D deep neural network. These 3D position features are computer rendered in 3D and the 3D volume shown is the result of thresholding the predicted values. It can be seen from the relative "roughness" of the surfaces defining the volume that such a network and its input and target data operate at a 3D resolution lower than the 3D resolution of the final voxel representation to be segmented (in this example, a resolution of 1x1x1mm per voxel). As a target, the same training data as that used for the segmentation 3D deep learning network can be used, but it will be downsampled to a suitable resolution to meet the processing requirements for the use of such a pre-processed 3D deep neural network. In practice, this results in such 3D position features containing a "rough" pre-segmentation, in this example case, a pre-segmentation of the jaw 1320, teeth 1322 and nerve 1324 structures. For the purposes of this illustration, the jaw of this particular patient is not rendered to show the voxels that are classified as most likely to be part of the neural structure.

[0165] This coarse pre-segmentation may be appropriately upsampled, e.g. by means of interpolation, to ensure that each voxel is at the desired segmentation resolution (the original received voxel resolution) and that information from this pre-segmentation is spatially consistent at the desired resolution. For example, in the visualization shown the information from one voxel may be spatially consistent with a 5x5x5 voxel at the desired resolution, and this information should be paired with all applicable 125 voxels at the desired resolution. This upsampled information may then be presented as or included in a 3D position feature set and as described in reference Fig. 10A and 10B As described above, it is fed into a segmentation 3D deep neural network as input.

[0166] Figures 14A-14D Depicted are examples of the output of a trained deep learning neural network according to an embodiment of the present invention. In particular, Figures 14A-14D A 3D image depicting voxels classified using a deep learning neural network that uses a reference Figure 8 The training method described is used for training. Fig. 14B and 14C As shown in , voxels can be classified as belonging to tooth structures by a neural network ( Fig. 14B )、Jaw structure( Fig. 14C ) or neural structures ( Fig.14D ) voxels. Fig.14A Depicted are 3D images, including voxels that a deep learning neural network has classified as teeth, jaws, and neural tissue. Figures 14B-14D As shown in Figure 2, the classification process is accurate, but there are still many voxels that are missed or misclassified. Fig. 14B and14C As shown in , voxels that may be part of the jaw structure are classified as tooth voxels, while the surface of the tooth root belonging to the tooth voxel is omitted. Fig.14D As shown in , this problem is more pronounced when classifying neural voxels.

[0167] To address the problem of outliers in the classified voxels (forming the output of the first deep learning neural network), the voxels can be post-processed. Fig.15 A flow chart of post-processing classified voxels of 3D dental maxillofacial structures according to an embodiment of the present invention is depicted. In particular, Fig.15 Describes the use of the present application as referenced Figure 7-1 4 Flowchart describing the post-processing of voxel data of dentofacial structures classified by deep learning neural network.

[0168] like Fig.15 As shown in , the processing may include the steps of dividing the classified voxel data 1502 of the 3D dento-maxillofacial structure into voxels classified as jaw voxels 1504, tooth voxels 1506, and voxels classified as neural data 1508. As will be described in more detail below, the jaw and tooth voxels may be post-processed using an additional second deep learning neural network 1510. Unlike the initial first deep learning neural network (which uses the 3D CT image data stack of the dento-maxillofacial structure and the associated positional features as input) that generates the best possible voxel classification based on the image data, the second "post-processing" deep learning neural network translates portions of the output of the first deep learning neural network into voxels so that the output more closely matches the desired 3D structure.

[0169] The post-processing deep learning neural network encodes a representation of the teeth and jaws. During training the post-processing deep learning neural network, the parameters of the neural network are tuned so that the output of the first deep learning neural network is translated into the most feasible 3D representation of these dento-maxillofacial structures. In this way, defects in the classified voxels can be reconstructed 1512. In addition, the surfaces of the 3D structures can be smoothed 1514 so that the most feasible 3D jaw model and tooth model can be generated. Omitting the 3D CT image data stack as a source of information for the post-processing neural network makes this post-processing step robust to undesired variations within the image stack.

[0170] Due to the nature of (CB)CT images, the output of the first deep learning neural network will be subject to potential artifacts (mentioned above), such as averaging due to patient motion, beam hardening, etc. Another source of noise is the difference in image data captured by different CT scanners. This difference leads to the introduction of multiple factors, such as variations in the amount of noise within the image stack, variations in the intensity values ​​of voxels representing the same (real world) density, and other factors that may be present. By post-processing the deep learning neural network, the impact of the above-mentioned artifacts and noise sources on the output of the first deep learning neural network can be eliminated or at least substantially reduced, resulting in segmented jaw voxels and segmented tooth voxels.

[0171] The segmented tooth voxels may consist of all voxels that may be considered part of any tooth and may therefore contain representations of all teeth present in the dentition. Processing may be applied to the entire set of tooth voxels in order to separate the tooth voxels into sets of voxels belonging to individual teeth 1520. In such a process, regions of connected voxels may be considered individual objects and may be partitioned into separate representations of individual teeth.

[0172] Individual tooth voxels may be connected and additional processing may be applied in order to divide all volumes into what are considered to be individual teeth. This may be done using known methods from the fields of image processing and morphology, in particular by applying watersheding and erosion.

[0173] In more detail, the expected volume of an individual tooth can be used to iteratively perform multiple steps until all sets of divided voxels match the expected volume of the tooth. Performing successive 3D erosion steps on the tooth voxels can separate previously connected sets of voxels. The coordinates of the centers of these (newly) separated areas can be used as the starting point for a 3D watershed of the (original) tooth voxels, which can produce a separate set of voxels when connected. Iteratively performing erosion until the set of voxels is separated, performing a watershed as described above, and checking whether the minimum and maximum volume conditions are met can produce a set of voxels that meets the requirements of being consistent with an individual tooth.

[0174] The classified neural data 1508 may be post-processed separately from the jaw and tooth data. The nature of the neural data, which represents long thin filament structures in the CT image data stack, makes this data less suitable for post-processing by a deep learning neural network. Instead, the classified neural data is post-processed using an interpolation algorithm in order to process the segmented neural data 1516. To this end, a fitting algorithm uses voxels that are classified as neural voxels and are associated with a high probability (e.g., 95% or higher probability) in order to construct a 3D model of the neural structure. Thereafter, the 3D jaw, tooth, and neural models are combined into a 3D model of the dento-maxillofacial structure.

[0175] Fig.16 An example of an architecture of a deep learning neural network according to an embodiment of the present invention is depicted, which is configured to post-process the classified voxels of a 3D maxillofacial structure. The post-processing deep learning neural network may have an architecture similar to the first deep learning neural network, including a first path formed by a first set of 3D CNN feature layers 1604 configured to process input data (in this case, a portion of the classified voxel data) at the resolution of the target. The deep learning neural network also includes a second set of 3D CNN feature layers 1606, which is configured to process the context of the input data processed by the first 3D CNN feature layer, but at a lower resolution than the target. The outputs of the first and second 3D CNN feature layers are then fed to the input of a set 1608 of fully connected 3D CNN layers to reconstruct the classified voxel data so that they closely represent the 3D model of the 3D maxillofacial structure. The output of the fully connected 3D CNN layer provides reconstructed voxel data.

[0176] The post-processing neural network can be trained using the same objective as the first deep learning neural network (which represents the same desired output). During training, the network is made as broadly applicable as possible by providing noise to the input to represent the exceptions to the norm. Inherent in the nature of the post-processing deep learning neural network is that the processing it performs also results in the removal of infeasible aspects from the received voxel data. Factors here include smoothing and filling of the desired dentofacial structure, as well as the outright removal of infeasible voxel data.

[0177] Fig.17A and 17B Depicted is an iteration of a post-processing network that results in surface reconstruction of classified voxels according to an embodiment of the present invention. In particular, Fig.17A A picture of classified voxels of tooth structure is depicted, where the voxels are the output of the first deep learning neural network. As shown, noise and other artifacts in the input data lead to irregularities and artifacts in the voxel classification and thus 3D surface structure including gaps in the set of voxels representing tooth structure. These irregularities and artifacts are particularly evident at the inferior alveolar nerve structure 17021 of the tooth and the root structure 17041 of the tooth (i.e., the area in which the deep learning neural network must distinguish between tooth voxels and voxels that are part of the jawbone).

[0178] Fig. 17B Depicted according to reference Fig.15 and Fig.16Result of post-processing of the described process. As shown in this figure, the post-processing deep learning neural network successfully removed artifacts present in the input data (classified voxels). The post-processing step successfully reconstructed portions that were substantially affected by irregularities and artifacts, such as the root structure 17041 of the tooth, which now presents a smooth surface that provides an accurate 3D model of the individual tooth structure 17042. The fitting algorithm uses high probability neural voxels 17021 (e.g., 95% or higher probability) to build a 3D model of the neural structure 17022.

[0179] Fig.18 A schematic diagram of a distributed computer system according to an embodiment of the present invention is depicted. The distributed computer system may be configured to process 3D data based on a trained 3D deep learning processor as described in the present application and to render the processed 3D data. Fig.18 As shown in FIG. , the trained 3D deep learning processor may be part of a distributed system including one or more servers 1802 and multiple terminals 1810 in a network. 1-3 , preferably a mobile terminal, such as a desktop computer, a laptop computer, an electronic tablet, etc. The (trained) 3D deep learning processor can be implemented as a server application 1804, 1806. In addition, a client application (client device) 1812 executed on the terminal 1-3 The system may include: a user interface that enables a user to interact with the system; and a network interface that enables a client device to communicate with a server application via, for example, one or more networks 1808 (e.g., the Internet). The client device may be configured to receive input data. The client device may transmit the data to the server application, which may process the data based on the methods and systems described in this application. The processed data may be sent back to the client device, and a rendering engine 1814 associated with the client device may be used to display the data. 1-3 The resulting 3D image data can be used to render a 3D tooth model. In another embodiment, some or data processing can be performed on the client side. For example, any pre-processing and / or post-processing described in this disclosure can be performed by a client device. In another embodiment, instead of a distributed computer system, a central computer system can be used to perform any processing described in this application.

[0180] Therefore, if Fig.18As shown in , the present invention provides a fully automatic pipeline for predicting root shapes based on 3D data representing crowns. The user can provide 3D image data representing the maxillofacial structure or dentition to the input of the system, such as (CB) CT 3D data or intraoral scans, and in response, the system will generate (one or more) root shapes to match the crowns as represented in the source data, which can be presented to the user in different graphical formats, such as as a 3D rendering or as markers in the displayed image slices. The input data is automatically optimized for input to a 3D deep neural network, so that the 3D deep neural network can accurately process the 3D image data without any human intervention. Moreover, the present invention allows 3D rendering of the output generated by the 3D deep neural network processor, that is, a full tooth containing the predicted root. Such visual information is essential for the latest dental applications in the fields of dental care and dental reporting, orthodontics, orthognathic surgery, forensics, biometrics, etc.

[0181] Fig.19 1900 is a block diagram illustrating an exemplary data processing system described in the present disclosure. Data processing system 1900 may include at least one processor 1902 coupled to memory element 1904 via system bus 1906. As such, the data processing system may store program codes in memory element 1904. In addition, processor 1902 may execute program codes accessed from memory element 1904 via system bus 1906. On the one hand, the data processing system may be implemented as a computer suitable for storing and / or executing program codes. However, it should be understood that data processing system 1900 may be implemented in the form of any system including a processor and a memory capable of performing the functions described in this specification.

[0182] Memory element 1904 may include one or more physical memory devices, such as, for example, local memory 1908 and one or more mass storage devices 1910. Local memory may refer to a random access memory or (one or more) other non-persistent memory devices commonly used during the actual execution of program code. Mass storage devices may be implemented as hard drives or other persistent data storage devices. Processing system 1900 may also include one or more cache memories (not shown) that provide temporary storage of at least some program codes to reduce the number of times program codes must be retrieved from mass storage devices 1910 during execution.

[0183] Input / output (I / O) devices, depicted as input devices 1912 and output devices 1914, may optionally be coupled to the data processing system. Examples of input devices may include, but are not limited to, for example, a keyboard, a pointing device (such as a mouse), etc. Examples of output devices may include, but are not limited to, for example, a monitor or display, a speaker, etc. The input devices and / or output devices may be coupled to the data processing system either directly or through an intermediate I / O controller. A network adapter 1916 may also be coupled to the data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and / or remote storage devices through an intermediate private or public network. The network adapter may include: a data receiver for receiving data transmitted to the data by the system, device, and / or network; and a data transmitter for transmitting data to the system, device, and / or network. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that may be used with data processing system 1900.

[0184] like Fig.19 , memory element 1904 can store application 1918. It should be appreciated that data processing system 2100 can also execute an operating system (not shown) that can facilitate the execution of applications. Applications implemented in the form of executable program code can be executed by data processing system 1900 (e.g., by processor 1902). In response to executing the application, the data processing system can be configured to perform one or more operations that will be described in further detail herein. On the one hand, for example, data processing system 1900 can represent a client data processing system. In that case, application 1918 can represent a client application that configures data processing system 1900 to perform various functions described herein with reference to "client" when executed. Examples of clients can include, but are not limited to, personal computers, portable computers, mobile phones, etc. On the other hand, a data processing system can represent a server. For example, a data processing system can represent a (HTTP) server, in which case application 1918 can configure the data processing system to perform (HTTP) server operations when executed. On the other hand, a data processing system can represent a module, unit, or function referenced in this specification.

[0185] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "comprise" and / or "include" when used in this specification specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0186] All means or steps in the appended claims plus the corresponding structures, materials, actions and equivalents of the functional elements are intended to include any structure, material or action for performing a function in combination with other claimed elements specifically claimed. The description of the present invention has been given for the purpose of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be clear to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The embodiments are selected and described in order to best explain the principles and practical applications of the present invention and to enable other persons of ordinary skill in the art to understand the various embodiments of the present invention with various modifications suitable for the intended specific use.

[0187] FIG. 20A to FIG. 20D A representation of the visualization of a 3D teeth model towards an end user according to an embodiment of the present invention is shown. Fig. 20A and Fig. 20B 104 and 105 show slices of 3D image data received by the system. These 3D data may be generated by a CBCT scanner in the coronal and sagittal planes, respectively. Both figures show the entire extent of the data received, and as can be seen, the "top side" of the data does not contain a complete tooth present in the upper jaw. The data does contain portions of the crowns of the teeth, which may be obtained as described with respect to system step 104 and in Figure 8 to Figure 1 7, and the Fig. 20C 4, the voxel representation of the incomplete crown model can be passed individually to the trained 3D deep neural network 106 to obtain the following: Fig.20D Root shape prediction is shown by individual crowns 2004 (as indicated in the source data as 2002 ) and predicted individual roots 2006 .

[0188] The system can automatically perform this process for all individual tooth entities that are considered to be crowns and are incomplete, for example, which can be determined by using known information that takes into account the expected minimum real-world dimensions of the volume that is considered to be a complete tooth (such as the expected maximum change in surface area that is considered to be a part of the tooth, which surface area is within the plane that intersects the model, and the change in area is the result of iteratively moving the normal plane along the "up-down" (in the real-world coordinate system) axis with a set step size. In other words, scanning the volume of (a part of) the tooth along the up-down axis will produce any abrupt changes / terminations in volume, which indicates that the tooth is not complete.

[0189] After generating predictions for all applicable crown portions, the post-processor 108 may obtain Fig.20D The 3D image data 110 of the representation is visualized, thereby providing an end user with a complete tooth with a predicted root portion based on the crown portion of the received 3D image data.

Claims

1. A computer-implemented method for automatic 3D tooth root shape prediction, comprising: receiving, by a preprocessor, 3D data defined in a voxel space, the voxel space defining a 3D volume, the voxel space including a voxel representation of a tooth crown, and processing, by the preprocessor, the voxel representation so that it is at a scale, position, and orientation corresponding to the voxel representation used to train a first 3D deep neural network; The preprocessor provides the voxel representation of the crown to an input of a first 3D deep neural network, the first 3D deep neural network is trained based on a training set of preprocessed clinical 3D data defining a 3D representation of a real tooth, the trained deep neural network is configured to predict an anatomically accurate voxel representation of a tooth root corresponding to the crown or a voxel representation of a complete tooth; generating, by a first 3D deep neural network, a voxel representation of a predicted tooth root or a voxel representation of a complete tooth including the predicted tooth root based on the voxel representation of the tooth crown, wherein the generating of the voxel representation of the predicted tooth root or the complete tooth comprises determining voxel activations of voxels in a voxel space of an output of the first 3D deep learning network, each voxel activation representing a probability measure defining a probability that the voxel is at least a portion of the tooth root or the complete tooth; as well as Whether the voxel activation is part of a tooth root or a complete tooth is determined by comparing the voxel activation to a voxel activation threshold.

2. The method of claim 1, further comprising: Receiving, by a post-processor, a voxel representation of a predicted tooth root generated by the first 3D deep neural network, and processing, by the post-processor, the 3D crown and the voxel representation of the predicted tooth root, the processing comprising: Merging the voxels of the 3D root and the 3D crown into a voxel representation of the complete 3D tooth; and Transform the voxel representation of the complete 3D tooth in the 3D mesh of the complete tooth.

3. The method of claim 1 or 2, wherein the 3D data defines a 3D representation of at least a portion of the dentition, the processing by the preprocessor further comprising: segmenting the 3D data into at least one 3D data set, the 3D data set representing 3D crowns of teeth of the dentition; as well as The 3D dataset is transformed into a voxel representation of the tooth crown that matches the voxel space of the input to the first 3D deep neural network.

4. The method of claim 1 or 2, wherein the 3D data received by the preprocessor is 3D data generated by an optical scanner, the 3D data defining a 3D surface mesh representing at least a portion of a dentition including a plurality of crowns, wherein the processing by the preprocessor further comprises: segmenting the 3D mesh into a plurality of segmented 3D meshes, wherein each segmented 3D mesh represents a 3D crown of a dentition; Each segmented 3D surface mesh is transformed into a voxel representation of the crown that matches the voxel space of the input to the first 3D deep neural network.

5. The method of claim 1 or 2, wherein the 3D data received by the preprocessor is generated by an X-ray scanner, the 3D data defining a voxel representation of at least a portion of a dento-maxillofacial structure, the dento-maxillofacial structure comprising a plurality of teeth of at least a portion of a dentition, wherein the processing by the preprocessor further comprises: using a second 3D deep neural network to classify at least a portion of the voxels representing the dento-maxillofacial structure as at least one of jaw, tooth, and / or neural voxels; the second 3D deep neural network is trained based on the 3D image data of the dento-maxillofacial structure, one or more 3D positional features derived from the 3D image data of the training set, and one or more 3D models of the portion of the dento-maxillofacial structure of the 3D image data of the training set, the one or more 3D models being used as targets during training of the first deep neural network; as well as The classified voxels are segmented into one or more 3D data sets, each of the one or more 3D data sets defining a voxel representation of a tooth in a dentition of a dento-maxillofacial structure.

6. The method of claim 5, wherein the processing by the preprocessor further comprises: providing a further voxel representation of the dento-maxillofacial structure to an input of a third 3D deep neural network, the third 3D deep neural network being trained to determine, for each voxel of the voxel representation at the input, at least one 3D position feature, the 3D position feature comprising a measure indicating the likelihood that the voxel represents a jaw, a tooth and / or neural tissue, wherein the further voxel representation of the dento-maxillofacial structure is a low resolution version of the voxel representation of the dento-maxillofacial structure; The voxel representation of the dento-maxillofacial structure and the one or more 3D position features are provided to a second 3D deep neural network, which uses the one or more position features to classify at least a portion of the voxels in the voxel space as at least one of jaw, tooth and / or neural voxels.

7. A method as claimed in claim 6, wherein the resolution of the further voxel representation is at least three times lower than the resolution of the first voxel representation, or wherein the third 3D deep neural network is trained based on the 3D image data of the maxillofacial structure and the one or more 3D models of the part of the maxillofacial structure of the 3D image data of the training set used to train the second deep neural network.

8. The method of claim 5, wherein the second deep neural network comprises a plurality of first 3D convolutional layers, outputs of the plurality of first 3D convolutional layers being connected to at least one fully connected layer, wherein the plurality of first 3D convolutional layers are configured to process a first voxel block from a first voxel representation, and wherein the at least one fully connected layer is configured to classify voxels of the first voxel block as at least one of jaw, tooth, and / or neural voxels.

9. The method of claim 8, wherein the second deep neural network further comprises a plurality of second 3D convolutional layers, outputs of the plurality of second 3D convolutional layers being connected to the at least one fully connected layer, wherein the plurality of second 3D convolutional layers are configured to process a second voxel block from the first voxel representation, the first voxel block and the second voxel block having a same center point in the image volume, and a volume in a real-world dimension represented by the second voxel block being larger than a volume in the real-world dimension of the first voxel block, and the plurality of second 3D convolutional layers are configured to determine context information associated with voxels of the first voxel block provided to the input of the plurality of first 3D convolutional layers.

10. The method of claim 1 or 2, wherein the first 3D deep neural network comprises a plurality of 3D convolutional layers connected to a plurality of 3D deconvolutional layers via one or more densely connected layers, and wherein the first deep neural network is trained based on a voxel representation of a tooth crown and an associated tooth root or based on a voxel representation of a tooth crown and an associated tooth.

11. A computer-implemented method for training a 3D deep learning neural network to generate a prediction of a 3D tooth root shape, comprising: The computer receives training data, the training data comprising clinical 3D data, the clinical 3D data comprising a voxel representation of a crown and associated root or based on a voxel representation of a crown and associated tooth, wherein at least a portion of the voxel representation is derived from segmented 3D X-ray data; and / or derived from an optically scanned complete tooth; providing a voxel representation of the crown to an input of the 3D deep neural network, and the 3D deep neural network generates a predicted voxel representation of the root; optimizing the value of one or more network parameters of the 3D deep neural network by minimizing a loss function representing the deviation between a predicted voxel representation of a tooth root or a predicted complete tooth and a voxel representation of a tooth root or a voxel representation of a complete tooth associated with a voxel representation of a tooth crown provided as an input to the 3D deep neural network; as well as The optimized values ​​are stored in a computer-readable storage medium, the optimized values ​​defining one or more network parameters of a trained neural network configured to predict an anatomically accurate voxel representation of a tooth root corresponding to the tooth crown or a voxel representation of a complete tooth when provided with a voxel representation of the tooth crown.

12. The method of claim 11, wherein the 3D deep neural network comprises a plurality of 3D convolutional layers connected to a plurality of 3D deconvolutional layers via one or more densely connected layers.

13. A computer system suitable for automatically predicting 3D tooth root shape, comprising: a computer readable storage medium having computer readable program code embodied thereon; and a processor coupled to a computer-readable storage medium, wherein, in response to executing the computer-readable program code, the processor is configured to perform executable operations including: receiving, by a preprocessor, 3D data defined in a voxel space, the voxel space defining a 3D volume, the voxel space including a voxel representation of a tooth crown, and processing the voxel representation so that it is at a scale, position, and orientation corresponding to the voxel representation used to train a first 3D deep neural network; The preprocessor provides the voxel representation of the crown to an input of a first 3D deep neural network, the first 3D deep neural network is trained based on a training set of preprocessed clinical 3D data defining a 3D representation of a real tooth, the trained deep neural network is configured to predict an anatomically accurate voxel representation of a tooth root corresponding to the crown or a voxel representation of a complete tooth; generating, by a first 3D deep neural network, a voxel representation of a predicted tooth root or a voxel representation of a complete tooth including the predicted tooth root based on the voxel representation of the tooth crown, wherein the generating of the voxel representation of the predicted tooth root or the complete tooth comprises determining voxel activations of voxels in a voxel space of an output of the first 3D deep learning network, each voxel activation representing a probability measure defining a probability that the voxel is part of the tooth root or the complete tooth; and Whether the voxel activation is part of a tooth root or a complete tooth is determined by comparing the voxel activation to a voxel activation threshold.

14. The computer system of claim 13, wherein the processor is further configured to perform executable operations including the method of any one of claims 2 to 10.

15. A computer-readable storage medium comprising software code portions configured to, when executed in a memory of a computer, perform the steps of the method of any one of claims 1 to 12.

Images