Classification and 3D modeling of 3D dentofacial structures using deep learning methods

Abstract

To provide a method executed by a computer for processing 3D image data of a dento-maxillofacial structure.SOLUTION: A method may include: a step which receives 3D image data defining a volume of voxel and in which the voxel is associated with radiation density and a position in the volume, and the voxel provides a 3D expression of a dento-maxillofacial structure; a step which uses the voxel of the 3D image data for determining one or more 3D position features to be input to a first deep layer neural network and in which the 3D position feature defines information aggregated from a whole received 3D data set; and a step in which the first deep layer neural network receives the 3D image data and one or more position features in its input and which uses one or more 3D position features for classifying at least a part of the voxel of the 3D image data into a jaw voxel, a tooth voxel, and / or neutral voxel.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to classification and 3D modeling of 3D dental maxillofacial structures using deep learning neural networks, particularly, but not exclusively, to systems and methods for classification and 3D modeling of 3D dental maxillofacial structures using deep learning neural networks, methods for training such deep learning neural networks, methods for pre-processing dental maxillofacial 3D image data and methods for post-processing classified voxel data of dental maxillofacial structures, and computer program products for using such methods. [Background technology]

[0002] In image analysis of dental and maxillofacial structures, visualization of specific regions or tissues and 3D image reconstruction are essential to enable accurate diagnosis and treatment. Prior to 3D image reconstruction, classification and segmentation processes are applied to the 3D image data, e.g., voxels, to form 3D models of different parts of the dental and maxillofacial structures (e.g., teeth and jaws) represented in the 3D image data stack. The task of segmentation is defined as identifying a set of pixels or voxels that constitute either the contour or the interior of an object of interest. However, the process of segmenting dental and maxillofacial structures, such as teeth, jawbone, and inferior alveolar nerve, from 3D CT scans is challenging. Manual segmentation methods are very time-consuming and involve manual threshold selection and general approximations with manual correction. The results of manual segmentation have low reproducibility and depend on human interpretation of the CT scan.

[0003] Different imaging methodologies have been used to generate 3D tooth and jaw models based on CT scan image data. Initially, the sequential application of low-level pixel processing and mathematical modeling was used to segment dental and maxillofacial structures. One example is described in an article by Pavaloiu et al., "Automatic segmentation for 3D dental reconstruction," IEEE 6th ICCCNT, July 13–15, 2015. These techniques include active contour tracking, watershedding, region growing, and level set modeling using shape and intensity priors. Currently, more advanced techniques, such as deep learning techniques, are used in medical imaging to segment objects of interest in medical images.

[0004] These neural networks are trained to learn features that best represent the data. Such deep learning algorithms involve multilayer deep neural networks that transform input data (e.g., images) into outputs (e.g., the presence or absence of disease) while learning increasingly higher-level features. A successful neural network model for image analysis is the so-called convolutional neural network (CNN). CNNs contain many layers that transform their input using kernels, also known as convolution filters, composed of relatively small matrices. An overview of the use of CNNs for medical imaging can be found in the archived article "A Survey on Deep Learning in Medical Image Analysis" by Litjens et al., published February 21, 2017 (submitted to Computer Vision and Pattern Recognition). However, 3D modeling of dental and maxillofacial structures using 3D CNNs is challenging due to the complexity of dental and maxillofacial structures. In their article, "Neural network based edge detection for CBCT segmentation," 5th IEEE EHB, November 19-21, 2015, Pavaloiu et al. describe the use of a very simple neural network for edge detection in 2D CBCT images. However, to date, automatic and accurate 3D segmentation of 3D CBCT image data based on deep learning has not been reported.

[0005] A challenge in 3D classification and modeling of dental and maxillofacial structures is that dental and maxillofacial images are generated using cone-beam computed tomography (CBCT). CBCT is a medical imaging technique that uses X-ray computed tomography, in which X-ray radiation is shaped into a low-dose diverging cone. Because different regions in the scan appear at different grayscale values ​​depending on their relative location within the organ being scanned, radiation density, measured in Hounsfield units (HU), is unreliable with CBCT. HU measured from the same anatomical region using both a CBCT scanner and a medical-grade CT scanner are not identical and therefore unreliable for determining bone density identified with site-specific radiographs.

[0006] Furthermore, CBCT systems for scanning dental and maxillofacial structures do not use a standardized system for scaling the grayscales representing reconstructed density values. These values ​​are arbitrary and do not allow for the assessment of bone quality. In the absence of such standardization, it is difficult to interpret gray levels or impossible to compare values ​​resulting from different machines. Furthermore, because tooth roots and jaw bone structures have similar densities, it is difficult for a computer to distinguish between voxels belonging to teeth and voxels belonging to jaws. In addition, CBCT systems are highly sensitive to so-called beam hardening, which produces dark stripes between two highly attenuating objects (such as metal or bone) along with surrounding bright stripes. The above issues make automated segmentation of dental and maxillofacial structures particularly challenging.

[0007] Therefore, there is a need in the art for a computer system adapted to accurately segment 3D CT image data of dental and maxillofacial structures into 3D models, and in particular, a computer system that can accurately segment 3D CT image data of dental and maxillofacial structures from different CBCT systems. Summary of the Invention [Means for solving the problem]

[0008] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Functionality described in this disclosure may be implemented as an algorithm executed by a computer microprocessor. Furthermore, aspects of the present invention may take the form of a computer program product having computer-readable program code embodied therein, e.g., embodied in one or more computer-readable medium(s) having stored thereon computer-readable program code.

[0009] Any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0010] A computer-readable storage medium may include a propagated data signal having computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable storage medium is not a computer-readable storage medium but may be any computer-readable medium that can communicate, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device.

[0011] Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wired, fiber optic, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, C++, Python, and conventional procedural languages ​​such as the “C” programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer, server, or virtualization server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0012] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, particularly a microprocessor or central processing unit (CPU) of a general-purpose computer, or a graphics processing unit (GPU) of a special-purpose computer, or other programmable data processing device, to create a machine such that the instructions, executed via the processor of the computer, other programmable data processing device, or other device, create means for performing the functions / acts specified in the flowchart block diagram blocks.

[0013] These computer program instructions may be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to function in a particular manner to produce an article of manufacture containing instructions that implement the functions / acts specified in the flowchart and / or block diagram blocks, where the instructions stored in the computer-readable medium.

[0014] The computer program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to generate a computer-implemented process into a series of operational steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions, executing on the computer or other programmable apparatus, provide a process for performing the functions / operations specified in the flowchart and / or block diagram blocks.

[0015] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the flowchart diagrams and / or block diagrams, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0016] The present disclosure provides systems and methods that implement automatic classification and segmentation techniques that do not require user input or interaction other than the input of a 3D image stack. Embodiments may be used to reconstruct target anatomy, such as the jawbone, teeth, and dentofacial nerves, such as the inferior alveolar nerve. The system automatically separates structures and builds a 3D model of the target tissue.

[0017] In one aspect, the present invention relates to a computer-implemented method for processing 3D image data of dental maxillofacial structures. In one embodiment, the method comprises the steps of: a computer receiving 3D input data, preferably 3D cone beam CT (CBCT) data, the 3D input data comprising a first voxel representation of the dental maxillofacial structure, the voxels being associated with radiation intensity values, the voxels of the voxel representation defining an image volume; a pre-processing algorithm using the 3D input data to determine one or more 3D position features of the dental maxillofacial structure, the 3D position features defining information about the position of a dental reference plane, e.g., an axial plane positioned relative to a jaw, or the position of a voxel of the first voxel representation relative to a dental reference object, e.g., a jaw, a dental arch, and / or one or more teeth, within the image volume; and a computer-implemented method for processing the first voxel representation and the one or more 3D position features associated with the first voxel representation. to an input of a first 3D deep neural network, preferably a 3D convolutional deep neural network, configured to classify voxels of the first voxel representation into at least jaw voxels, tooth voxels, and / or nerve voxels; training the first neural network based on a training set, the training set including 3D image data of the dental maxillofacial structure and one or more 3D position features derived from the 3D image data of the training set; and receiving the classified voxels of the first voxel representation from an output of the first 3D deep neural network and determining a voxel representation of at least one of jaw, tooth, and / or nerve tissue of the dental maxillofacial structure based on the classified voxels.

[0018] Thus, the 3D location features define information about the location of voxels within the received image volume relative to the tooth reference plane and / or tooth reference object. This information is relevant for enabling the deep neural network to automatically classify and segment voxel representations of dental maxillofacial structures. The 3D location features of voxels in the first voxel representation may be formed by aggregating information (e.g., location, intensity value, distance, gradient, etc.) based on the entire or substantial portion of the data set of voxel representations provided to the input of the first deep neural network. The aggregated information is processed for each voxel location within the first voxel representation. In this manner, each voxel in the first voxel representation may be associated with a 3D location feature that the first deep neural network considers during voxel classification.

[0019] In one embodiment, the training set may further comprise one or more 3D models of portions of the dental maxillofacial structures of the 3D image data of the training set. In one embodiment, at least a portion of the one or more 3D models may be generated by optically scanning the portions of the dental maxillofacial structures of the 3D image data of the training set. In one embodiment, the one or more 3D models may be used as targets during training of the first deep neural network.

[0020] The 3D position features may be determined using (manually) designed features and / or using (trained) machine learning methods such as a 3D deep learning network configured to derive such information from the whole or a substantial portion of the received 3D dataset.

[0021] In one embodiment, the 3D location feature may define a distance, preferably a vertical distance, between one or more voxels in the image volume and a first tooth reference surface in the image volume. In one embodiment, the 3D location feature may define a distance between one or more voxels in the image volume and a first tooth reference object in the image volume. In a further embodiment, the location information may include accumulated intensity values ​​in a reference surface of the image volume, wherein the accumulated intensity values ​​at a point in the reference surface include accumulated intensity values ​​of voxels on or near a normal passing through the point in the reference surface.

[0022] 3D location features extracted from 3D image data encode information about the image volume of voxels provided to the neural network's input. Specifically, 3D location features provide information derived partially or completely about the location of each voxel within (a subsection of) the 3D image data and evaluated by the deep neural network. The 3D location features provide the neural network with a means to utilize information determined (in part) by the location of voxels within the image volume to determine the likelihood that a voxel associated with a particular dental maxillofacial structure can be found in a particular volume. Without this information, a larger spatial context may not be available for use by the deep neural network. 3D location features are designed to minimize the risk of overfitting while significantly improving the network's accuracy. 3D location features allow the network to gain knowledge about the location of voxels within the image volume compared to reference objects associated with the dental maxillofacial context, thus making this information available to determine the likelihood of finding voxels associated with dental maxillofacial structures. This allows the network to learn how to maximize this provided information where it is relevant.

[0023] In one embodiment, the first dental reference plane may comprise an axial plane within the image volume, as represented by the 3D image data, positioned at a predetermined distance from the upper and / or lower jaw. Thus, the reference plane is positioned relative to the relevant portion of the dental maxillofacial structure within the 3D image data. In one embodiment, the first dental reference plane may have approximately equal distances to the upper and lower jaws.

[0024] In one embodiment, the dental reference object may include a dental arch curve that approximates at least a portion of the dental arch as represented by the 3D image data. Thus, in this embodiment, the 3D position features may provide information regarding the position of voxels in the image volume relative to the position of the dental reference object dental arch in the image volume. In one embodiment, the dental arch curve may be determined in an axial plane of the image volume.

[0025] The manually designed 3D location features may be supplemented or replaced by other 3D location features, such as may be derived from machine learning methods that aggregate information from all or a substantial portion of the 3D input data. Such feature generation may be performed, for example, by a 3D deep neural network that performs pre-segmentation on a downsampled version of all or a substantial portion of the first voxel representation.

[0026] Thus, in one embodiment, the pre-processing algorithm may include a second 3D deep neural network that receives at its input the second voxel representation and is trained to determine 3D location features for each voxel of the second voxel representation. In one embodiment, the 3D location features may include a measure of the likelihood that the voxel represents jaw, teeth, and / or neural tissue, and the second voxel representation is a lower resolution version of the first voxel representation.

[0027] In one embodiment, the second 3D deep neural network may have a 3D U-Net architecture. In one embodiment, the 3D U-Net may comprise multiple 3D neural network layers, including a convolutional layer (3D CNN), a 3D max-pooling layer, a 3D deconvolutional layer (3D de-CNN), and a densely connected layer.

[0028] In one embodiment, the resolution of the second voxel representation may be at least one-third lower than the resolution of the first voxel representation.

[0029] In one embodiment, the second 3D deep neural network may be trained based on 3D images of dental maxillofacial structures in a training set used to train the first deep neural network. In one embodiment, the second 3D deep neural network is based on one or more 3D models of portions of dental maxillofacial structures in the 3D image data of the training set used to train the first deep neural network. These one or more 3D models may be used as targets during training.

[0030] In one embodiment, providing the first voxel representation and one or more 3D location features associated with the first voxel representation to an input of the first 3D deep neural network may further include associating each voxel of the first voxel representation with information defined by at least one 3D location feature, dividing the first voxel representation into first blocks of voxels, and providing the first blocks of voxels to an input of the first deep neural network, wherein each voxel of the first block of voxels is associated with a radiation intensity value and the information defined by at least one 3D location feature. Thus, the first 3D deep neural network may process the 3D input data based on the blocks of voxels. To this end, the computer may divide the first voxel representation into a plurality of first blocks of voxels and provide each of the first blocks to an input of the first 3D deep neural network.

[0031] In one embodiment, the first deep neural network may include a plurality of first 3D convolutional layers, and outputs of the plurality of first 3D convolutional layers may be connected to at least one fully connected layer. In one embodiment, the plurality of first 3D convolutional layers may be configured to process a first block of voxels from the first voxel representation, and the at least one fully connected layer is configured to classify voxels of the first block of voxels into at least one of jaw, tooth, and / or nerve voxels.

[0032] In one embodiment, the voxels provided to the input of the first deep neural network may comprise a radiant intensity value and at least one 3D position feature.

[0033] In one embodiment, the first deep neural network may further include a plurality of second 3D convolutional layers, and outputs of the plurality of second 3D convolutional layers may be connected to at least one fully connected layer.

[0034] In one embodiment, the plurality of second 3D convolutional layers may be configured to process a second block of voxels from the first voxel representation, where the first and second blocks of voxels may have the same or substantially the same center point within the image volume, and the second block of voxels may represent a volume with real-world dimensions that is larger than the volume with real-world dimensions of the first block of voxels.

[0035] In one embodiment, the plurality of second 3D convolutional layers may be configured to determine context information associated with voxels of a first block of voxels provided to inputs of the plurality of first 3D convolutional layers.

[0036] In one embodiment, the first deep neural network may further include a plurality of third 3D convolutional layers, the outputs of which are connected to at least one fully connected layer. The plurality of third 3D convolutional layers may be configured to process one or more 3D position features associated with voxels of at least a first block of voxels provided to inputs of the plurality of first 3D convolutional layers.

[0037] In one embodiment, the first deep neural network may be trained based on a training set, the training set including 3D image data of dental maxillofacial structures, one or more 3D positional features derived from the 3D image data, and one or more 3D models of portions of the dental maxillofacial structures of the 3D image data of the training set, and the one or more 3D models may be used as targets during training of the deep neural network.

[0038] In one embodiment, at least a portion of the one or more 3D models may be generated by optically scanning portions of the dentofacial structures of the training set 3D image data. Thus, instead of manually segmented 3D image data, the optically scanned 3D models are used to train the neural network, thus providing high-resolution, accurate modules that can be used as target data.

[0039] In one embodiment, determining the one or more 3D position features may include determining a cloud of points of cumulative intensity values ​​in a plane of the image volume, preferably an axial plane, where the cumulative intensity value at a point in the plane may be determined by summing voxel values ​​of voxels located on or near a normal line passing through the point in the axial plane, determining cumulative intensity values ​​in the plane that exceed the determined value, and fitting a curve through the determined cumulative intensity values, where the curve approximates at least a portion of a dental arch in the dental maxillofacial structure represented by the 3D data image. Thus, a dental structure such as a dental arch may be determined by summing intensity values ​​of voxels located in a plane, for example, a normal direction to the axial plane.

[0040] In one embodiment, the one or more 3D position features may include a first 3D position feature that defines a relative distance in a plane in the image volume, preferably an axial plane in the image volume, between a voxel in the plane and an origin on a dental arch curve defined in the plane. In one embodiment, the origin may be defined as a point on the dental arch curve where the derivative of the curve is zero.

[0041] In one embodiment, the one or more 3D position features include a second 3D position feature that defines a relative distance in a plane within the image volume, preferably an axial plane within the image volume, the distance being the shortest distance in the axial plane between a voxel in the axial plane and the dental arch curve.

[0042] In one embodiment, the 3D position features may be determined based on automatic feature generation using all or a substantial portion of the 3D input data. In one embodiment, the automatic feature generation may include a 3D deep neural network performing pre-segmentation on a downsampled version of all or a substantial portion of the 3D input data.

[0043] In one embodiment, the first deep neural network may comprise at least a first data processing path including a first set of 3D convolutional layers, preferably 3D CNN feature layers, configured to determine progressively higher abstractions of information useful for deriving a classification of a voxel, and a second data processing path parallel to the first path, the second path comprising a second set of 3D convolutional layers, preferably the second set of 3D CNN feature layers, configured to determine progressively higher abstractions of information useful for deriving a classification of a voxel by utilizing a spatially contextually larger representation of the block of voxels fed to the input of the first set of 3D convolutional layers.

[0044] Thus, a second set of 3D CNN feature layers may process voxels to generate a 3D feature map containing information about the immediate neighborhood of the associated voxels processed by the first 3D CNN feature layer. In this way, the second pathway allows the neural network to determine contextual information, i.e., information about the context (e.g., surroundings) of the voxels of the 3D image data presented to the input of the neural network. By using two or even more pathways, both the 3D image data (input data) and the contextual information about the voxels of the 3D image data can be processed in parallel. Contextual information is important for classifying dental and maxillofacial structures, including closely packed dental structures that are typically difficult to distinguish.

[0045] In one embodiment, the first deep neural network may further comprise a third data processing path parallel to the first and second paths, including a third set of 3D convolutional layers, preferably a third set of 3D CNN feature layers, for receiving one or more 3D position features associated with the 3D image data, the third set of 3D convolutional layers configured to encode relevant information from a collection of information from the entire received 3D dataset that is associated with a block of voxels that is fed to the input of the first set of 3D convolutional layers.

[0046] In one embodiment, instead of using a third data processing path, the 3D position features may be added to the first voxel representation to pair with voxels in the first voxel representation, for example, by adding the 3D position feature information as an additional channel to the received 3D image information.

[0047] In one embodiment, outputs of the first, second, and (optionally) third set of 3D convolutional layers may be provided to inputs of a set of fully connected convolutional layers configured to classify at least a portion of the voxels in the image volume into at least one of jaw voxels, tooth voxels, and / or nerve voxels.

[0048] In one embodiment, the method may further include a step in which a third deep neural network post-processes the voxels classified by the first deep neural network, where the post-processing includes correcting voxels incorrectly classified by the first deep neural network. In one embodiment, the second neural network may be trained using the voxels classified during training of the first deep neural network as input and one or more 3D models of portions of the dental maxillofacial structure in the training set 3D image data as targets. Thus, in this embodiment, the second convolutional neural network may be trained to correct the voxels classified by the first neural network. In this manner, highly accurate 3D models of individual portions of the dental maxillofacial structure, including 3D models of the teeth and jaws, may be determined.

[0049] In one aspect, the invention may relate to a computer-implemented method for training a deep learning neural network system to process 3D image data of dental maxillofacial structures. In one embodiment, the method includes the steps of: receiving training data by a computer, the training data including 3D input data, preferably 3D cone beam CT (CBCT) image data, the 3D input data defining one or more voxel representations of one or more dental maxillofacial structures, respectively, the voxels being associated with radiation intensity values, and the voxels of the voxel representations defining an image volume; and scanning the one or more voxel representations of the one or more dental maxillofacial structures, respectively, to determine one or more 3D positional features for voxels within the one or more voxel representations. The method may include using a preprocessing algorithm to preprocess the data, wherein the 3D position features define information regarding the position of at least one voxel of the voxel representation of the dental maxillofacial structure relative to the position of a tooth reference plane (e.g., an axial plane positioned relative to the jaw) or the position of a tooth reference object (e.g., the jaw, the dental arch, and / or one or more teeth) within the image volume; and using the training data and the one or more 3D position features to train a first deep neural network to classify voxels into jaw voxels, tooth voxels, and / or nerve voxels.

[0050] In one embodiment, the training data may further include one or more 3D models of the portions of the dental maxillofacial structure represented by the 3D input data of the training data. In one embodiment, at least a portion of the one or more 3D models may be generated by optically scanning the portions of the dental maxillofacial structure in the 3D images of the training data. In one embodiment, the one or more 3D models may be used as targets during training of the first deep neural network.

[0051] In one embodiment, the method may include using the voxels classified during training of the first deep neural network and one or more 3D models of portions of the dentofacial structure of the training set 3D image data to train a second neural network for post-processing voxels classified by the first deep neural network, wherein the post-processing includes correcting voxels incorrectly classified by the first deep neural network.

[0052] In a further aspect, the present invention may relate to a computer system adapted to process 3D image data of a dental maxillofacial structure, the computer system comprising: a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising a pre-processing algorithm and a first deep neural network; and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein in response to executing the computer readable program code, the processor is configured to perform executable operations, the executable operations being: receiving 3D input data, preferably 3D Cone Beam CT (CBCT) data, the 3D input data comprising a first voxel representation of the dental maxillofacial structure, the voxels being associated with radiation intensity values, the voxels of the voxel representation defining an image volume; and the pre-processing algorithm using the 3D input data to determine one or more 3D position features of the dental maxillofacial structure, the 3D position features being associated with dental reference planes within the image volume. , for example, defining information about the position of an axial plane arranged relative to the jaw or the position of voxels of the first voxel representation relative to a dental reference object, for example, a tooth, a dental arch, and / or the position of one or more teeth; and providing the first voxel representation and one or more 3D position features associated with the first voxel representation to an input of a first 3D deep neural network, preferably a 3D convolutional deep neural network, wherein the first deep neural network associates voxels of the first voxel representation with at least the jaw voxels. , tooth voxels, and / or nerve voxels; training a first neural network based on a training set, the training set including 3D image data of the dental maxillofacial structure and one or more 3D position features derived from the 3D image data of the training set; receiving classified voxels of the first voxel representation from an output of the first deep neural network, and determining voxel representations of jaw, tooth, and / or nerve tissue of the dental maxillofacial structure based on the classified voxels.

[0053] In one embodiment, the training set may further comprise one or more 3D models of portions of the dental maxillofacial structures of the 3D image data of the training set. In one embodiment, at least a portion of the one or more 3D models may be generated by optically scanning the portions of the dental maxillofacial structures of the 3D image data of the training set. In one embodiment, the one or more 3D models may be used as targets during training of the first deep neural network.

[0054] In one embodiment, the pre-processing algorithm may include a second 3D deep neural network, the second deep neural network being trained to receive at its input the second voxel representation and to determine 3D location features for each voxel of the second voxel representation, preferably the 3D location features comprising a measure of the likelihood that the voxel represents jaw, tooth, and / or neural tissue, the second voxel representation being a lower resolution version of the first voxel representation, preferably the resolution of the second voxel representation being at least one-third lower than the resolution of the first voxel representation, preferably the second 3D deep neural network being trained based on the 3D image data of the dental maxillofacial structures and one or more 3D models of portions of the dental maxillofacial structures of the 3D image data of a training set for training the first deep neural network.

[0055] In one embodiment, the first deep neural network may comprise a plurality of first 3D convolutional layers, outputs of which are connected to at least one fully connected layer, and the plurality of first 3D convolutional layers are configured to process a first block of voxels from the first voxel representation, and the at least one fully connected layer is configured to classify voxels of the first block of voxels into jaw voxels, tooth voxels, and / or nerve voxels, and preferably each voxel is provided to an input of the first deep neural network comprising a radial intensity value and at least one 3D position feature.

[0056] In one embodiment, the first deep neural network may further comprise a plurality of second 3D convolutional layers, the outputs of the plurality of second 3D convolutional layers being connected to at least one fully connected layer, the plurality of second 3D convolutional layers being configured to process a second block of voxels from the first voxel representation, the first and second blocks of voxels having the same or substantially the same center point within the image volume, the second block of voxels representing a volume with real-world dimensions that is larger than the volume with real-world dimensions of the first block of voxels, and the plurality of second 3D convolutional layers being configured to determine contextual information associated with voxels of the first block of voxels provided to the inputs of the plurality of first 3D convolutional layers.

[0057] The invention may also relate to a computer program product comprising software code portions configured to perform any of the methods described above when executed in a computer memory.

[0058] The invention will be further illustrated with reference to the accompanying drawings, which show, in a simplified manner, embodiments according to the invention, it being understood that the invention is in no way limited to these particular embodiments. [Brief explanation of the drawings]

[0059] [Figure 1] FIG. 1 is a diagram illustrating a schematic of a computer system for classification and segmentation of 3D dental maxillofacial structures, according to an embodiment of the present invention. [Figure 2] FIG. 1 is a flow diagram of training a deep neural network for classifying 3D dental maxillofacial structures according to an embodiment of the present invention. [Figure 3A] 1A and 1B show examples of 3D CT image data and 3D optical scan data, respectively. [Figure 3B] 1A and 1B show examples of 3D CT image data and 3D optical scan data, respectively. [Figure 4A]FIG. 1 illustrates an example of a deep neural network architecture for classifying 3D dental and maxillofacial image data. [Figure 4B] FIG. 1 illustrates an example of a deep neural network architecture for classifying 3D dental and maxillofacial image data. [Figure 5A] 1 illustrates a method for determining 3D position features according to various embodiments of the present invention. [Figure 5B] 1 illustrates a method for determining 3D position features according to various embodiments of the present invention. [Figure 6] FIG. 10 provides a visualization including total voxel values ​​from a 3D image stack and a curve fitted to the voxels representing the dental maxillofacial arch. [Figure 7A] 1A-1C illustrate examples of 3D position features according to various embodiments of the present invention. [Figure 7B] 1A-1C illustrate examples of 3D position features according to various embodiments of the present invention. [Figure 7C] 1A-1C illustrate examples of 3D position features according to various embodiments of the present invention. [Figure 7D] 1A-1C illustrate examples of 3D position features according to various embodiments of the present invention. [Figure 7E] 1A-1C illustrate examples of 3D position features according to various embodiments of the present invention. [Figure 8A] FIG. 2 illustrates an example output of a trained deep neural network, according to one embodiment of the present invention. [Figure 8B] FIG. 2 illustrates an example output of a trained deep neural network, according to one embodiment of the present invention. [Figure 8C] FIG. 2 illustrates an example output of a trained deep neural network, according to one embodiment of the present invention. [Figure 8D] FIG. 2 illustrates an example output of a trained deep neural network, according to one embodiment of the present invention. [Figure 9] FIG. 1 is a flow diagram of post-processing classified voxels of a 3D dentofacial structure according to an embodiment of the present invention. [Figure 10]FIG. 1 illustrates a deep neural network architecture for post-processing classified voxels of a 3D dentofacial structure according to an embodiment of the present invention. [Figure 11A] FIG. 1 illustrates surface reconstruction processing of classified voxels according to one embodiment of the present invention. [Figure 11B] FIG. 1 illustrates surface reconstruction processing of classified voxels according to one embodiment of the present invention. [Figure 12] FIG. 1 is a block diagram illustrating an exemplary data computing system that may be used to implement the methods and software products described in this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0060] This disclosure describes embodiments of a computer system and a computer-implemented method that use deep neural networks for classification, segmentation, and 3D modeling of dental and maxillofacial structures based on 3D image data, e.g., CT image data stacks, particularly 3D image data defined by a series of images forming a cone-beam computed tomography (CBCT) image data stack. The 3D image data may comprise voxels forming a 3D image space of dental and maxillofacial structures. The computer system according to the present invention may comprise at least one deep neural network trained to classify the 3D image data stack of dental and maxillofacial structures into different classes of voxels, each class being associated with a distinct portion of the structure (e.g., teeth, jaws, nerves). The computer system may be configured to perform a training process that iteratively trains (optimizes) one or more deep neural networks based on one or more training sets, which may include accurate 3D models of the dental and maxillofacial structures. These 3D models may include optically scanned dental and maxillofacial structures (teeth and / or jawbone).

[0061] Once trained, the deep neural network may receive 3D image data stacks of dental maxillofacial structures and classify voxels of the 3D image data stacks. Before the data is presented to the trained deep neural network, the data may be preprocessed to enable the neural network to efficiently and accurately classify the voxels. The output of the neural network may include different collections of voxel data, each collection representing a different portion, e.g., a tooth or jawbone, of the 3D image data. The classified voxels may be post-processed to reconstruct an accurate 3D model of the dental maxillofacial structures.

[0062] A computer system with a trained neural network for automatically classifying voxels of dentofacial structures, the training of the network, the pre-processing of 3D image data before feeding it to the neural network, and the post-processing of voxels classified by the neural network are described in more detail below.

[0063] FIG. 1 schematically illustrates a computer system for classification and segmentation of 3D dental maxillofacial structures according to an embodiment of the present invention. Specifically, a computer system 102 may be configured to receive a 3D image data stack 104 of dental maxillofacial structures. The structures may include jaw structures, tooth structures, and nerve structures. The 3D image data may comprise voxels, i.e., 3D spatial elements associated with voxel values ​​representing radiation intensity or density values, e.g., grayscale or color values. Preferably, the 3D image data stack may include CBCT image data in a predetermined format, e.g., DICOM format or a derivative thereof.

[0064] The computer system may include a preprocessor 106 for preprocessing the 3D image data before it is provided to an input of a first 3D deep learning neural network 112 that is trained to generate a 3D set of classified voxels as output 114. As described in more detail below, the 3D deep learning neural network may be trained according to a predetermined training regime such that the trained neural network can accurately classify voxels in the 3D image into different classes of voxels (e.g., voxels associated with teeth, jawbone, and / or neural tissue). The 3D deep learning neural network may include multiple connected 3D convolutional neural network (3D CNN) layers.

[0065] The computer system may further include a post-processor 116 for accurately reconstructing 3D models of different parts of the dentofacial structures (e.g., teeth, jaws, and nerves) using the voxels classified by the 3D deep learning neural network. As described in more detail below, some of the classified voxels, e.g., voxels classified as belonging to tooth or jaw structures, are input to a second 3D deep learning neural network 120 trained to reconstruct 3D volumes of the dentofacial structures, e.g., the shape of the jaw 124 and the shape of the teeth 126, based on the voxels classified as belonging to such structures. Other parts of the classified voxels, e.g., voxels classified by the 3D deep neural network as belonging to nerves, may be post-processed by using an interpolation function 118 and stored as 3D nerve data 122. The task of determining volumes representing nerves from the classified voxels is currently beyond the capacity of (the processing power available to) deep neural networks. Furthermore, the presented classified voxels may not contain information suitable for a neural network to solve this particular problem. Therefore, interpolation of the classified voxels is used to accurately and efficiently post-process the classified nerve voxels. After post-processing the 3D data of various portions of the dentofacial structures, the nerve data, jaw data, and tooth data 122-126 may be combined and formatted into a separate 3D model 128 that accurately represents the dentofacial structures within the 3D image data provided to the input of the computer system.

[0066] In CBCT scans, radiodensity (measured in Hounsfield units (HU)) is inaccurate because different regions in the scan appear at different grayscale values ​​depending on their relative location within the organ being scanned. HU measured from the same anatomical region using both a CBCT scanner and a medical-grade CT scanner are not identical and therefore cannot be relied upon to determine site-specific radiographically identified bone density.

[0067] Furthermore, dental CBCT systems do not use a standardized system for scaling the grayscales that represent reconstructed density values. These values ​​are therefore arbitrary and do not allow for the assessment of bone quality. In the absence of such standardization, it is difficult to interpret gray levels or impossible to compare values ​​resulting from different machines.

[0068] Tooth and jaw structures have similar densities, making it difficult for a computer to distinguish between voxels belonging to teeth and voxels belonging to jaw. In addition, CBCT systems are very sensitive to so-called beam hardening, which produces dark stripes between two highly attenuating objects (such as metal or bone) along with surrounding bright stripes.

[0069] To make the 3D deep learning neural network robust to the above problems, the 3D neural network may be trained using module 138 to utilize a 3D model of a portion of the dental maxillofacial structure as represented by 3D image data. The 3D training data 130 may be properly registered to the CBCT image (e.g., 3D CT image data of the dental maxillofacial structure and an associated 3D segmented representation of the dental maxillofacial structure) presented in 104, for which the associated target output is known. Conventional 3D training data may be obtained by manually segmenting the input data, which may represent a significant amount of work. Additionally, manual segmentation results in low reproducibility and consistency of the input data used.

[0070] To address this issue, in one embodiment, optically generated training data 130, i.e., accurate 3D models of (portions of) the dental and maxillofacial structures, may be used instead of, or at least in addition to, manually segmented training data. The dental and maxillofacial structures used to generate the training data may be scanned using a 3D optical scanner. Such optical 3D scanners are known in the art and can be used to generate high-quality 3D jaw and tooth surface data. The 3D surface data may include a 3D surface mesh 132 that can be filled in and used by a voxel classifier 134 (determining which particular voxels are part of the volume enclosed by the mesh). In this way, the voxel classifier can generate high-quality classified voxels 136 for training. Additionally, as described above, manually classified training voxels may be used by a training module to train the network as well. The training module may use the classified training voxels as targets and the associated CT training data as input.

[0071] Additionally, during the training process, the CT training data may be preprocessed by a feature extractor 108, which may be configured to determine 3D positional features. The dental maxillofacial features may encode at least spatial information associated with one or more portions of the imaged dental maxillofacial structures (the received 3D data set). For example, in one embodiment, the manually designed 3D positional features may include a 3D curve representing (a portion of) the jawbone, particularly the dental arch, within a 3D volume containing voxels. One or more weighting parameters may be assigned to points along the 3D curve. The weighting values ​​may be used to encode translation in 3D space from voxel to voxel. For example, rather than incorporating an encoded version of the original space in which the image stack was received, the encoded space is specific to the dental maxillofacial structures detected in the input. The feature extractor may examine voxel values ​​representing radiation intensity or density values ​​and determine one or more curves that approximate one or more curves of the jaw and / or teeth (e.g., the dental arch) by fitting one or more curves (e.g., polynomials) through particular voxels. The derivative of (a part of) the dental arch curve of the 3D CT image data stack may be stored as a mapping 110 of position features.

[0072] In another embodiment, such 3D position features may be determined by (trained) machine learning methods, such as, for example, a 3D deep neural network designed to derive relevant information from the entire received 3D dataset.

[0073] FIG. 2 illustrates a flow diagram for training a deep neural network to classify dental and maxillofacial 3D image data according to an embodiment of the present invention. The training data is used to train a 3D deep learning neural network to automatically classify voxels of a 3D CT scan of dental and maxillofacial structures. As shown in this figure, a representation of a dental and maxillofacial complex 202 may be provided to a computer system. The training data may include a CT image data stack 204 of the dental and maxillofacial structures and an associated 3D model, such as 3D data 206 from an optical scan of the same dental and maxillofacial structures. Examples of such 3D CT image data and 3D optical scan data are shown in FIGS. 3A and 3B. FIG. 3A illustrates DICOM slices associated with different planes of a 3D CT scan of the dental and maxillofacial structures, such as an axial plane 302, a frontal or coronal plane 304, and a sagittal plane 306. FIG. 3B illustrates the 3D optical scan data of the dental and maxillofacial structures. The computer may form a 3D surface mesh 208 of the dental maxillofacial structure based on the optical scan data. Further, a registration function 210 configured to register the 3D surface mesh to the 3D CT image data may be used. After registration, the representation of the 3D structure provided to the input of the computer uses the same spatial coordinate system. Based on the registered CT image data and the 3D surface mesh, positional features 212 and classified voxel data of the optically scanned 3D model 214 may be determined. The positional features and classified voxel data may then be provided to the input of a deep neural network 216, along with the image data stack 204.

[0074] Thus, during the training phase, the 3D deep learning neural network receives 3D CT training data and location features extracted from the 3D CT training data as input data, and the classified training voxels associated with the 3D CT training data are used as target data. Optimization methods may be used to learn optimal values ​​for the network parameters of the deep neural network by minimizing a loss function that represents the deviation from the output of the deep neural network to target data (i.e., classified voxel data) that represents the desired output for a given input. If the minimization of the loss function converges to a particular value, the training process can be deemed suitable for the application.

[0075] The training process shown in Figure 2, which uses 3D location features in combination with training data that may be derived (at least in part) from 3D optical scan data, provides a high-quality training set for a 3D deep learning neural network. After the training process, the trained network is able to accurately classify voxels from a 3D CT image data stack.

[0076] 4A and 4B show high-level schematic diagrams of deep neural network architectures for use in the methods and systems described in this disclosure. The deep neural network may be implemented using one or more 3D convolutional neural networks (3D CNNs). The convolutional layers may use activation functions associated with the neurons in the layer, such as a sigmoid function, a tanh function, a relu function, or a softmax function. The deep neural network may include multiple 3D convolutional layers, and slight variations in the number of layers and their defining parameters, such as different activation functions, kernel amounts and sizes, and additional functional layers, such as dropout and batch normalization layers, may be used in the implementation without losing the essence of the deep neural network design.

[0077] As shown in FIG. 4A , the network may include multiple convolutional paths, each associated with a set of 3D convolutional layers. In one embodiment, the network may include at least two convolutional paths: a first convolutional path associated with a first set of 3D convolutional layers 406 and a second convolutional path associated with a second set of 3D convolutional layers 408. The first and second convolutional paths may be trained to encode 3D features derived from received 3D image data associated with voxels provided to inputs of the first and second convolutional paths, respectively. Additionally, in some embodiments, the network may include at least a further (third) convolutional path associated with a third set of 3D convolutional layers 407. The third convolutional path may be trained to encode 3D features derived from received 3D positional features associated with voxels provided to inputs of the third path.

[0078] Alternatively, in another embodiment, instead of further convolution paths being trained based on 3D location feature data, the 3D location feature data may be associated with intensity values ​​of voxels provided to the inputs of the first and second convolution paths. Thus, in this embodiment, the first and second convolution paths may be trained based on training data including a 3D data stack of voxel values ​​that includes intensity values ​​and location feature information.

[0079] The functionality of the different paths is shown in more detail in FIG. 4B. As shown in this figure, voxels are provided to the input of the neural network. These voxels are associated with a predetermined volume, sometimes referred to as image volume 4013. The total volume of voxels may be divided into first blocks of voxels, and a 3D convolutional layer of the first path 4031 may perform a 3D convolution operation on each of the first blocks of voxels 4011 of the 3D image data. During processing, the output of each 3D convolutional layer may be the input of a subsequent 3D convolutional layer. In this manner, each 3D convolutional layer may generate a 3D feature map that represents the features of the 3D image data provided to its input. Accordingly, a 3D convolutional layer configured to generate such a feature map may be referred to as a 3D CNN feature layer.

[0080] As shown in FIG. 4B, the convolution layer of the second convolution path 4032 may be configured to process second blocks 4012 of voxels of the 3D image data. Each second block of voxels is associated with a first block of voxels, and the first and second blocks of voxels have the same central origin within the image volume. The volume of the second block is larger than the volume of the first block. Furthermore, the second block of voxels represents a downsampled version of the associated first block of voxels. The downsampling may be based on using well-known interpolation algorithms. The downsampling factor may be any suitable value. In one embodiment, the downsampling factor may be selected between 20 and 2, preferably between 10 and 3.

[0081] Thus, the 3D deep neural network may comprise at least two convolutional paths. The first convolutional path 4031 may define a first set (e.g., 5-20 layers) of 3D CNN feature layers configured to process input data (e.g., a first block of voxels at a predetermined location within an image volume) at a first voxel resolution, e.g., a target voxel resolution (i.e., the resolution of the voxels of the 3D image data to be classified). Similarly, the second convolutional path may define a second set (e.g., 5-20 layers) of 3D CNN feature layers configured to process input data at a second voxel resolution (e.g., a second block of voxels, each of which has the same center point as its associated block from the first block of voxels). Here, the second resolution is lower than the first resolution. Thus, the second block of voxels represents a volume larger in real-world dimensions than the first block. In this way, the second 3D CNN feature layer processes voxels to generate a 3D feature map containing information about the (direct) neighborhood of the associated voxels processed by the first 3D CNN feature layer.

[0082] Thus, the second pathway allows the neural network to determine contextual information, i.e., information about the context (e.g., surroundings) of voxels of 3D image data presented to the neural network's input. By using multiple (parallel) convolutional pathways, both the 3D image data (input data) and the contextual information about the voxels of the 3D image data can be processed in parallel. Contextual information is useful for classifying dental and maxillofacial structures, including closely packed dental structures that are typically difficult to distinguish, especially in the case of CBCT image data.

[0083] In one embodiment, the neural network of 4B may further include a third convolutional path 4033 of a third set of 3D convolutional layers trained to process specific representations of 3D position features 404 that can be extracted from the 3D image data. Extraction of 3D position features from the 3D image data may be implemented as a preprocessing step. In an alternative embodiment, instead of using a third convolutional path for processing the 3D position features, 3D position information including the 3D position features may be associated with the 3D image data provided to the input of the deep neural network. Specifically, a 3D data stack may be formed in which each voxel is associated with an intensity value and position information. Thus, position information may be paired for each applicable received voxel, for example, by adding the 3D position feature information as an additional channel to the received 3D image information. Thus, in this embodiment, voxels of the voxel representation of the 3D dental maxillofacial structure at the input of the deep neural network may be associated not only with voxel values ​​representing radio wave intensity values, but also with 3D position information. Therefore, in this embodiment, during training of the convolutional layers of both the first and second convolutional paths, information derived from both 3D image features and 3D position features may be encoded in these convolutional layers.

[0084] The outputs of the set of 3D CNN feature layers are then merged and fed to the input of a set of fully connected 3D CNN layers 410, which are provided to the input of a neural network and trained to derive an intended classification 412 of the voxels processed by the 3D CNN feature layers.

[0085] A set of 3D CNN feature layers are trained (via their learnable parameters) to derive and pass on the optimally useful information that can be determined from their particular inputs, and the fully connected layers encode parameters that determine how information from previous paths should be combined to provide optimally classified voxels 412. The classified voxels may then be presented in image space 414. Thus, the output of the neural network is a classified voxel in image space that corresponds to the image space of the voxels in the input.

[0086] Here, the output of the fully connected layer (last layer) may provide multiple activations for each voxel. Such voxel activations may represent probability measures (predictions) defining the probability that a voxel belongs to one of multiple classes, e.g., tooth structure classes, e.g., teeth, jaws, and / or neural structures. For each voxel, the voxel activations associated with different tooth structures may be thresholded to obtain classified voxels.

[0087] 5-7 illustrate a method for determining 3D positional features within a 3D image data stack representing a 3D dento-maxillofacial structure, along with examples of such positional features. Specifically, in the case of manually designed features, as described with reference to FIG. 1, both the 3D image data stack and the associated 3D positional features are provided as input to a 3D deep neural network so that the network can accurately classify voxels without the risk of overfitting. A transformation based on real-world dimensions ensures comparable input regardless of input image resolution.

[0088] The manually designed 3D location features may provide the 3D deep neural network with information regarding the location of voxels within the image volume relative to a reference plane or reference object within the image volume. For example, in one embodiment, the reference plane may be an axial plane within the image volume that separates voxels associated with the upper jaw from voxels associated with the lower jaw. In another embodiment, the reference object may include a curve, e.g., a 3D curve, that approximates at least a portion of the dental arch of the teeth within the 3D image data of the dental maxillofacial structures. In this manner, the location features provide the first deep neural network with a means of encoding abstractions indicative of the likelihood of each voxel-associated jaw, tooth, and / or neural tissue at different locations within the image volume. These location features may help the deep neural network efficiently and accurately classify voxels in the 3D image data stack and are designed to reduce the risk of overfitting.

[0089] To determine reference surfaces and / or objects within the image volume that are useful in the classification process, the feature analysis function may determine voxels that are at or above or below a predetermined intensity value. For example, voxels associated with bright intensity values ​​may be associated with tooth and / or jaw tissue. In this manner, information regarding the position and orientation (e.g., rotation angle) of the teeth and / or jaw within the image volume may be determined by the computer. If the feature analysis function determines that the rotation angle is greater than a predetermined amount (e.g., greater than 15 degrees), the function may correct the rotation angle to zero, as this is more beneficial for accurate results.

[0090] FIG. 5A shows an example of a flow diagram 502 for determining manually designed 3D positional features in 3D image data 504, e.g., a 3D CT image data stack. This process may include determining one or more 3D positional features of dental maxillofacial structures, where the one or more 3D positional features are configured for input to a 3D deep neural network (as discussed above with reference to FIG. 4B). The manually designed 3D positional features define positional information of voxels in the image volume relative to a reference plane or object in the image volume, e.g., a distance, e.g., a vertical distance, between a voxel in the image volume and a reference plane in the image volume that separates the upper jaw from the lower jaw. They may also define a distance between a voxel in the image volume and a dental reference object, e.g., a dental arch in the image volume. The location of accumulated intensity values ​​in a second reference plane of the image volume may further be defined, where the accumulated intensity value at a point in the second reference plane includes accumulated intensity values ​​of voxels on or near a normal passing through the point in the reference plane. Examples of 3D position features are described below.

[0091] To determine a reference object that provides position information of the dental arch within the 3D image data of the dental maxillofacial structure, a fitting algorithm may be used to determine a curve that fits a given point within a cloud of points of different (cumulative) intensity values, for example a curve that follows a polynomial.

[0092] In one embodiment, a cloud of intensity values ​​in an axial plane (xy-plane) of the image volume may be determined. The cumulative intensity value of such an axial plane point may be determined by summing the voxel values ​​of voxels located on a normal line passing through the point in the axial plane. The intensity values ​​in the axial plane thus obtained may be used to find a curve that approximates the dental arch.

[0093] Figure 5B illustrates an example of a machine learning method that can be utilized to generate relevant 3D location features (not manually designed) according to an embodiment of the present invention. Specifically, Figure 5B illustrates an exemplary 3D deep neural network that can be trained to generate desired features to be processed by a segmented 3D neural network. After training, such a trained model can be used similarly to method 502 as a preprocessor to derive relevant 3D location features based on the entire received 3D dataset.

[0094] As with manually designed 3D location features, the objective is to consider the entire incoming 3D dataset (or at least a substantial portion thereof) and incorporate into it 3D location feature information for use in the segmented 3D deep learning network that is potentially relevant to the task of automatic classification and segmentation and that may not otherwise be available from the set of subsamples provided to the segmented 3D deep learning network. Again, as with manually designed 3D location features, such information should be made available for each voxel in the incoming 3D dataset.

[0095] One possible way to implement such a machine learning method for automatically generating 3D position features is through a trained deep neural network. Such a network may be trained to derive 3D position features based on an input 3D dataset (e.g., a voxel representation of dental and maxillofacial structures) provided to the input of a 3D segmentation deep neural network. In one embodiment, the preprocessing deep neural network may be a 3D U-net type deep neural network, as shown in FIG. 5B. Due to limitations in available processing (mainly memory requirements), such an architecture does not operate at the resolution of the received voxel representation. Therefore, a first input 3D dataset, a first voxel representation at a first resolution (e.g., 0.2 × 0.2 × 0.2 mm per voxel), may be downsampled using an interpolation algorithm to a second voxel representation at a second, lower resolution, e.g., 1 × 1 × 1 mm per voxel. The 3D deep neural network trained based on the second-resolution voxel representation may then generate 3D position feature information for each input voxel. An interpolation algorithm may be used to upscale this information to the original first resolution, such that the resulting 3D position features (spatially) correspond to the voxels of the first voxel representation, yielding relevant information for each voxel of the first input 3D dataset, while taking into account information that takes into account (an aggregated version of) the entire received 3D data.

[0096] Such a pre-preprocessing 3D deep neural network may be trained to approach a desired target value (which is a desired 3D location feature). In this particular example, the target may be, for example, a per-voxel class indicator at the resolution at which the pre-processing 3D deep neural network operates. Such class indicators may be sourced from the same pool of classified training voxels 136, but downsampled in the same way that the incoming 3D dataset was downsampled.

[0097] It should be noted that such exemplary implementations of pre-processing machine learning methods are effectively considered to be able to potentially access information from the coarse pre-segmentation, specifically the entire received 3D voxel representation (or a substantial portion thereof). Pairing the coarse pre-segmentation information with the corresponding voxels in the received 3D image space, e.g., by upscaling, results in these 3D location features being processed in parallel with the received 3D image data towards a result at the received 3D image resolution.

[0098] The preprocessing network may be implemented using various 3D neural network layers, such as convolutional layers (3D CNNs), 3D max pooling layers, 3D deconvolutional layers (3D de-CNNs), and densely connected layers. The layers may use various activation functions, such as linear, tanh, ReLU, PreLU, and sigmoid. The 3D CNN and de-CNN layers may differ in their amount of filters, filter size, and subsampling parameters. The 3D CNN and de-CNN layers, as well as the densely connected layers, may differ in their parameter initialization methods. Dropout and / or batch normalization layers may be used throughout the architecture.

[0099] According to the 3D U-Net architecture, during training, various filters within the 3D CNN layer and the 3D de-CNN layer learn to encode meaningful features that will aid in prediction accuracy efforts. During training, a matching set of 3D image data 522 and encoded matching 3D position features 560 is used to optimize the prediction of the latter from the former. A loss function may be used as the metric to be minimized. This optimization effort may be aided by using optimizers such as SGD, Adam, etc.

[0100] Such architectures may use various internal resolution scales that effectively downscale 526, 530, 534 the resulting data from a previous set of 3D CNN layers 524, 528, 532, for example, via max pooling or subsampled 3D convolutions. The term "meaningful features" here refers to the (successive) derivation of information relevant to determining target output values, which is also encoded via 3D de-CNN layers that effectively perform upsampling while using filters. By combining the resulting 540, 546, 552 data (532 to 540, 528 to 546, 524 to 552) from such 3D de-CNN layers 538, 544, 550 with data from a "final" 3D CNN layer operating at the same resolution, highly accurate predictions may be achieved. Additional 3D CNN layers 542, 548, 554 may be used throughout the upscaling path.

[0101] When used for inference, provided that the validation has been trained to encode the internal parameters to yield sufficiently accurate results, an input sample may be presented and the 3D deep learning network may yield predicted 3D position features 542.

[0102] An example of a reference object for use in determining manually designed 3D positional features is provided in FIG. 6, in this case, a curve approximating a dental arch. In this example, a cloud of points in the axial (xy) plane indicates that regions of high intensity (bright white regions) may represent regions of tooth structure or jaw structure. To determine the dental arch curve, the computer may determine regions in the axial plane of the image volume associated with bright voxels (e.g., voxels with intensity values ​​above a predetermined threshold) that may be identified as tooth voxels or jaw voxels. These regions of high intensity may be used to determine a crescent-shaped arrangement of bright regions approximating the dental maxillofacial arch. In this way, dental arch curves approximating the average of the dental maxillofacial arches of the upper and lower jaws, respectively, may be determined. In another embodiment, separate dental arch curves associated with the upper and lower jaws may be determined.

[0103] 7A-7E show examples of 3D position features of 3D image data according to various embodiments of the present invention.

[0104] 7A shows an image of a sagittal slice of a 3D image data stack (left) and an associated visualization of so-called height features of the same slice (right). Such height features may encode the z-position (height 704) of each voxel within the image volume of the 3D CT image data stack relative to a reference plane 702. The reference plane (e.g., an axial or xy-plane) is determined to be (the best approximation of) the xy-plane that has approximately equal distances to both the upper and lower jaws and their constituent teeth.

[0105] Other 3D position features may be defined to encode spatial information in the xy space of the 3D image data stack. In one embodiment, such a position feature may be based on a curve approximating (a portion of) the dental arch. Such a position feature is illustrated in FIG. 7B, which shows a slice from the 3D image data stack (left) and a visualization of a so-called translation feature for the same slice (right). This translation feature is based on a curve 706 approximating the dental arch and defines a relative distance 708 measured along the curve. Here, zero distance may be defined as the point 710 on the curve where the derivative of a quadratic polynomial is (approximately) zero. Moving from this point (e.g., the point where the derivative is zero) in either direction on the x-axis increases the translation distance.

[0106] An additional 3D position feature based on the dental arch curve may define the shortest (perpendicular) distance of each voxel in the image volume to the dental arch curve 706. This position feature may therefore be referred to as a "distance feature." An example of such a feature is shown in FIG. 7C , which shows a slice from the 3D image data stack (left) and a visualization of the distance feature for the same slice (right). For this feature, zero distance means that the voxel is located on the dental arch curve 706.

[0107] Additionally, additional 3D positional features may define the positional information of individual teeth. An example of such a feature (sometimes referred to as a dental feature) is shown in FIG. 7D , which shows a slice (left) from a 3D image data stack and a visualization of the dental feature for the same slice (right). The dental feature may provide information to be used to determine the likelihood of finding a particular tooth voxel at a particular location in voxel space. This feature may encode a separate summation of voxels across normals to any plane (e.g., the x-y plane or any other plane) following a determined reference plane such as 702. This information thus provides the neural network with a “view” of all information from the original space summed across plane normals. This view is larger than what would be processed if this feature were excluded, and may provide a means to distinguish whether hard structures are present based on all information in a selected direction in space (shown at 7121, 7122 for the x-y plane).

[0108] Figure 7E shows a visualization of 3D location features that may be generated by a machine learning preprocessor, specifically the 3D deep neural network described with reference to Figure 5B. These 3D location features have been computer-rendered in 3D, and the 3D volume shown is the result of thresholding the predicted values. The relative "coarseness" of the surfaces defining the volume indicates that such a network, as well as its input and target data, operated at a lower 3D resolution than that of the final voxel representation to be segmented (in this example, a resolution of 1 x 1 x 1 mm per voxel was used). The same training data used for the segmentation 3D deep learning network may be used as the target, but downsampled to an appropriate resolution that adheres to the processing requirements for use by such a preprocessing 3D deep neural network. This effectively results in such 3D location features including, in this example, a "coarse" pre-segmentation of the jaw 720, teeth 722, and nerve 724 structures. For purposes of this illustration, the mandible of this particular patient has not been rendered to show the voxels classified as most likely to be part of neural structures.

[0109] Such coarse pre-segmentation may be appropriately upsampled, for example, by interpolation, to ensure that for each voxel at the desired resolution (which is the originally received voxel resolution), information from such pre-segmentation is spatially matched at the desired resolution. For example, information from one voxel in the illustrated visualization may spatially match a 5x5x5 voxel at the desired resolution, and this information should be paired with all appropriate 125 voxels at the desired resolution. This upsampled information may then be presented as, or included within, a set of 3D position features and fed as input to a segmented 3D deep neural network, as described with reference to FIG. 4.

[0110] 5-7 show that the 3D position features define information about the voxels of the voxel representation that is provided to the input of a deep neural network trained to classify the voxels. The information may be aggregated from all (or a substantial portion of) the information available from the voxel representation, and during aggregation, the position of the voxel relative to the tooth reference object may be taken into account. Furthermore, the information is aggregated so that it can be processed for each voxel's position within the first voxel representation.

[0111] 8A-8D show examples of the output of a trained deep learning neural network according to one embodiment of the present invention. Specifically, FIGS. 8A-8D show 3D images of voxels classified using a deep learning neural network trained using the training method described with reference to FIG. 2. As shown in FIGS. 8B and 8C, voxels may be classified by the neural network as belonging to tooth structure (FIG. 8B), jaw structure (FIG. 8C), or nerve structure (FIG. 8D). FIG. 8A shows a 3D image including voxels classified by the deep learning neural network as tooth, jaw, and nerve tissue. As shown by FIGS. 8B-8D, the classification process is accurate, but there are still a significant number of voxels that are missed or misclassified. For example, as shown in FIGS. 8B and 8C, voxels that could be part of the jaw structure are classified as tooth voxels, and voxels at the surface that belong to the root of the tooth are missed. This problem is even more pronounced in the classified nerve voxels, as shown in FIG. 8D.

[0112] To address the issue of outliers in the classified voxels (which form the output of the first deep learning neural network), the voxels may be post-processed. Figure 9 shows a flow diagram for post-processing classified voxels of 3D dentofacial structures according to one embodiment of the present invention. Specifically, Figure 9 shows a flow diagram for post-processing voxel data of classified dentofacial structures using a deep learning neural network as described with reference to Figures 1-8 of the present application.

[0113] 9, the process may include dividing classified voxel data 902 of dental maxillofacial structures into voxels classified as jaw voxels 904, tooth voxels 906, and voxels classified as neural data 908. As described in more detail below, the jaw voxels and tooth voxels are post-processed using a further second deep learning neural network 910. In contrast to the initial first deep learning neural network, which generates the best possible voxel classification based on the image data (using as input the 3D CT image data stack of the dental maxillofacial structures and associated positional features), the second "post-processing" deep learning neural network transforms a portion of the output of the first deep learning neural network into voxels such that the output more closely matches the desired 3D structure.

[0114] The post-processing deep learning neural network encodes representations of both the teeth and jaws. During training of the post-processing deep learning neural network, the neural network's parameters are adjusted to transform the output of the first deep learning neural network into the most likely representation of these dental maxillofacial structures. In this way, defects in the classified voxels can be reconstructed (912). Additionally, the surfaces of the 3D structures can be smoothed (914) to generate the best possible 3D jaw and tooth models. Excluding the 3D CT image data stack from the information sources for the post-processing neural network makes this post-processing step robust to undesired variations in the image stack.

[0115] Due to the nature of (CB)CT images, the output of the first deep learning neural network is subject to potential artifacts (as discussed above), such as averaging due to patient motion, beam effects, etc. Another source of noise is inconsistency in the image data captured by different CT images. This inconsistency results in the introduction of various factors, such as varying amounts of noise within the image stack, variations in voxel intensity values ​​representing the same (real-world) density, and potentially others. The impact of the above artifacts and noise sources on the output of the first deep learning neural network is removed, or at least substantially reduced, by the post-processing deep learning neural network, resulting in segmented jaw voxels 918 and segmented tooth voxels 920.

[0116] The classified nerve data 908 may be post-processed separately from the jaw and tooth data. The nature of the nerve data, which represents long, thin filament structures within the CT image data stack, makes this data less suitable for post-processing with deep learning neural networks. Instead, the classified nerve data is post-processed (916) using an interpolation algorithm to generate segmented nerve data. To that end, voxels classified as nerve voxels and associated with a high probability (e.g., 95% or greater) are used by a fitting algorithm to construct a 3D model of the nerve structure. The 3D jaw, tooth, and nerve models are then combined into a 3D model of the dental maxillofacial structure.

[0117] FIG. 10 illustrates an example architecture of a deep learning neural network configured for post-processing classified voxels of 3D dentofacial structures according to an embodiment of the present invention. The post-processing deep learning neural network may have an architecture similar to the first deep learning neural network, including a first path formed by a first set of 3D CNN feature layers 1004 configured to process input data (in this case, a portion of the classified voxel data) at a target resolution. The deep learning neural network further includes a second set of 3D CNN feature layers 1006 configured to process the context of the input data processed by the first 3D CNN feature layer, but at a lower resolution than the target. The outputs of the first and second 3D CNN feature layers are then fed to the input of a set of fully connected 3D CNN layers 1008 to reconstruct the classified voxel data to closely represent a 3D model of the 3D dentofacial structures. The outputs of the fully connected 3D CNN layers provide reconstructed voxel data.

[0118] The post-processing neural network may be trained using the same target as the first deep learning neural network, representing the same desired output. During training, the network is made as broadly applicable as possible by providing noise to the input to represent exceptional cases that should be regularized. Inherent in the nature of the post-processing deep learning neural network, the processing it performs also results in the removal of improbable features from the received voxel data. Factors here include smoothing and filling of the desired dental and maxillofacial structures and the complete removal of improbable voxel data.

[0119] 11A and 11B illustrate iterations of a post-processing network resulting in a surface reconstruction of classified voxels, according to one embodiment of the present invention. Specifically, FIG. 11A shows a picture of classified voxels of a tooth structure, where the voxels are the output of a first deep learning neural network. As shown, noise and other artifacts in the input data result in irregularities and artifacts in the voxel classification, and thus a 3D surface structure that includes gaps in the set of voxels representing the tooth structure. These irregularities and artifacts are particularly visible in the inferior alveolar nerve structure 11021 and the tooth root structure 11041, i.e., areas where the deep learning neural network must distinguish between tooth voxels and voxels that are part of the jawbone.

[0120] FIG. 11B shows the results of post-processing according to the process described with reference to FIGS. 9 and 10. As shown in this figure, the post-processing deep learning neural network successfully removes artifacts that were present in the input data (classified voxels). The post-processing step accurately reconstructs areas substantially affected by irregularities and artifacts, such as the tooth root structure 11041, which here shows a smooth surface, providing an accurate 3D model of the individual tooth structure 11042. High-probability nerve voxels 11021 (e.g., greater than 95% probability) are used by the fitting algorithm to construct a 3D model of the nerve structure 11022.

[0121] Although the figures show the 3D deep neural networks as separate neural networks, each having a specific function, e.g., pre-processing, classification and segmentation, and post-processing, these neural networks may also be connected to each other to form one or more deep neural networks that include desired functions. In that case, different neural networks may be trained separately (e.g., as described with reference to figures in this disclosure). The trained networks may then be connected to each other to form one deep neural network.

[0122] 12 is a block diagram illustrating an exemplary data processing system described in this disclosure. Data processing system 1200 may include at least one processor 1202 coupled to memory elements 1204 via a system bus 1206. As such, the data processing system may store program code in memory elements 1204. Furthermore, processor 1202 may execute program code accessed from memory elements 1204 via system bus 1206. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be understood that data processing system 1200 may be implemented in the form of any system including a processor and memory capable of performing the functions described herein.

[0123] Memory elements 1204 may include one or more physical memory devices, such as, for example, a local memory 1208 and one or more mass storage devices 1210. Local memory may refer to random access memory or other non-persistent memory devices typically used during the actual execution of program code. The mass storage device may be implemented as a hard drive or other persistent data storage device. Processing system 1200 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code to reduce the number of times the program code must be retrieved from mass storage device 1210 during execution.

[0124] Input / output (I / O) devices, depicted as input devices 1212 and output devices 1214, may optionally be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, and the like. Examples of output devices may include, but are not limited to, a monitor or display, speakers, and the like. The input and / or output devices may be coupled to the data processing system directly or through intervening I / O controllers. A network adapter 1216 may also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote control devices, and / or remote storage devices through intervening private or public networks. A network adapter may comprise a data receiver for receiving data transmitted to the system, device, and / or network by the system, device, and / or network, and a data transmitter for transmitting data to the system, device, and / or network. Modems, cable modems, and Ethernet cards are examples of various types of network adapters that may be used in data processing system 1200.

[0125] 12, memory element 1204 may store application 1218. It should be understood that data processing system 1200 may further execute an operating system (not shown), which may facilitate the execution of applications. Applications implemented in the form of executable program code may be executed by data processing system 1200, for example, by processor 1202. In response to executing an application, the data processing system may be configured to perform one or more operations described in further detail herein.

[0126] In one aspect, for example, data processing system 1200 may represent a client data processing system. In that case, application 1218 may represent a client application that, when executed, configures data processing system 1200 to perform various functions described herein with respect to a “client.” Examples of clients may include, but are not limited to, a personal computer, a portable computer, a mobile phone, etc.

[0127] In another aspect, the data processing system may represent a server. For example, the data processing system may represent an (HTTP) server, in which case the application 1218, when executed, may configure the data processing system to perform (HTTP) server operations. In another aspect, the data processing system may represent a module, unit, or function referred to herein.

[0128] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0129] The corresponding structure, material, acts, and equivalents of all means or step-plus-function elements in the following claims are intended to include any structure, material, or acts for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments were chosen and described to best explain the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in various embodiments with various modifications as suited to the particular specifications intended. [Explanation of symbols]

[0130] 102 Computer Systems 104 3D image data stacks 106 Preprocessor 108 Feature Extractor 110 Mapping 112 The First 3D Deep Learning Neural Network 114 Output 116 Post Processor 118 Interpolation Functions 120 Second 3D Deep Learning Neural Network 122 3D neural data 124 Jaw 126 teeth 128 3D models 130 3D training data 132 3D surface meshes 134 Voxel Classifier 136 voxels, training voxels 202 Dentomaxillofacial complex 204 CT image data stacks 206 3D data 208 3D surface meshes 210 Alignment function 212 Location Features 214 3D models 216 Deep Neural Networks 302 axial plane 304 Frontal or Coronal Plane 306 sagittal plane 4011 1st Block 4012 Second Block 4013 image volume 4031 First Path, First Convolution Path 4032 Second Convolution Path 4033 Third Convolution Path 404 3D location features First set of 406 3D convolutional layers Third set of 407 3D convolutional layers A second set of 408 3D convolutional layers A set of 410 fully connected 3D CNN layers Intended classification of 412 voxels, optimally classified voxels 504 3D image data 522 3D image data Previous set of 524 3D CNN layers 526 Downscaling Previous set of 528 3D CNN layers 530 Downscaling Previous set of 532 3D CNN layers 534 Downscaling 538 3D de-CNN layers 540 Data 542 3D CNN layers, predicting 3D position features 544 3D de-CNN layers 546 Data 548 3D CNN layers 550 3D de-CNN layers 552 data 554 3D CNN layers 560 Encoding Matching 3D Position Features 702 Reference plane 704 height 706 Curve approximating dental arch, dental arch curve 708 Relative Distance 710 points 7121 space 7122 space 720 Jaw 722 teeth 724 Nerves 902 voxel data 904 Jaw Voxels 906 tooth voxels 908 Neural Data 910 Deep Learning Neural Networks 912 Reconstruction 914 Smoothing 916 Post-processing 918 segmented jaw voxels 920 segmented tooth voxels First set of 1004 3D CNN feature layers Second set of 1006 3D CNN feature layers A set of 1008 fully connected 3D CNN layers 11021 Inferior alveolar nerve structure, nerve voxel 11022Neural Structure 11041 Root structure 11042 Tooth structure 1200 Data processing system, processing system 1202 processor 1204 memory elements 1206 System Bus 1210 Mass Storage Device 1212 Input Devices 1214 output device 1216 Network Adapter 1218 Applications

Claims

1. 1. A computer-implemented method for processing 3D data representing dental and maxillofacial structures, comprising: receiving 3D input data, the 3D input data including a first voxel representation of the dental-maxillofacial structure, voxels associated with radiation intensity values, and voxels of the first voxel representation defining an image volume; - determining, by a pre-processing algorithm, one or more 3D position features of the dental maxillofacial structure using the 3D input data, the 3D position features defining information about the positions of voxels of the first voxel representation relative to positions of tooth reference surfaces or tooth reference objects within the image volume; providing the first voxel representation and the one or more 3D location features associated with the first voxel representation to an input of a first 3D deep neural network, wherein the first 3D deep neural network is trained to generate classified voxels of the first voxel representation at an output of the first 3D deep neural network, the classified voxels including at least jaw voxels, tooth voxels, and / or nerve voxels; receiving the classified voxels of the first voxel representation from the output of the first 3D deep neural network and determining a voxel representation of at least one of a jaw, a tooth, and / or a neural tissue of the dentomaxillofacial structure based on the classified voxels; A method comprising:

2. determining one or more 3D location features by a preprocessing algorithm, determining distances between voxels of said first voxel representation and tooth reference surfaces and / or tooth reference objects within said image volume; 2. The method of claim 1, comprising:

3. The method of claim 2 , wherein the dental reference plane comprises an axial plane within the image volume, positioned at a predetermined distance from the upper and / or lower jaw of the dental maxillofacial structure.

4. The method of claim 2 , wherein the dental reference object comprises a dental arch curve that approximates at least a portion of a dental arch represented by the dental maxillofacial structure.

5. 2. The method of claim 1, wherein the pre-processing algorithm comprises a second 3D deep neural network trained to receive at its input a second voxel representation and to determine, for each voxel of the second voxel representation, a 3D location feature of the second voxel representation, the 3D location feature of the second voxel representation comprising a measure of the likelihood that the voxel represents jaw, teeth, and / or neural tissue, and the second voxel representation is a lower resolution version of the first voxel representation.

6. providing the first voxel representation and the one or more 3D location features associated with the first voxel representation as an input to a first 3D deep neural network, associating each voxel of the first voxel representation with information defined by at least one 3D position feature; dividing the first voxel representation into first blocks of voxels; providing the first block of voxels to the input of the first 3D deep neural network; 10. The method of claim 1, further comprising: each voxel of the first block of voxels being associated with a radiation intensity value and information defined by at least one 3D position feature.

7. 7. The method of claim 1, wherein the first 3D deep neural network comprises a plurality of first 3D convolutional layers, outputs of which are connected to at least one fully connected layer, the plurality of first 3D convolutional layers being configured to process a first block of voxels from the first voxel representation, the at least one fully connected layer being configured to classify voxels of the first block of voxels into at least one of jaw voxels, tooth voxels, and / or nerve voxels.

8. 8. The method of claim 7, wherein the first 3D deep neural network further comprises a plurality of second 3D convolutional layers, outputs of the plurality of second 3D convolutional layers connected to the at least one fully connected layer, the plurality of second 3D convolutional layers configured to process a second block of voxels from the first voxel representation, the first block of voxels and the second block of voxels having the same or substantially the same center point within the image volume, the second block of voxels representing a volume with larger real-world dimensions than a volume with real-world dimensions of the first block of voxels, the plurality of second 3D convolutional layers configured to determine contextual information associated with voxels of the first block of voxels, and the voxels of the first block of voxels provided to inputs of the plurality of first 3D convolutional layers.

9. 9. The method of claim 7 or 8, wherein the first 3D deep neural network further comprises a plurality of third 3D convolutional layers, outputs of the plurality of third 3D convolutional layers connected to the at least one fully connected layer, the plurality of third 3D convolutional layers configured to process one or more 3D location features associated with voxels of at least the first block of voxels, and the voxels of at least the first block of voxels provided to inputs of the plurality of first 3D convolutional layers.

10. 10. The method of claim 1, further comprising a third 3D deep neural network for post-processing voxels classified by the first 3D deep neural network, the third 3D deep neural network receiving at its input the voxels classified by the first 3D deep neural network and being trained to correct voxels incorrectly classified by the first 3D deep neural network.

11. 1. A computer-implemented method for training a 3D deep neural network system to process 3D data representing dentofacial structures, comprising: receiving training data, the training data including 3D data, the 3D data including one or more voxel representations of one or more dentofacial structures, each voxel associated with a radiation intensity value, the voxels of the one or more voxel representations defining an image volume; receiving classified training voxels associated with the training data, the classified training voxels defining target data; - preprocessing the one or more voxel representations of the one or more dental maxillofacial structures, respectively, using a preprocessing algorithm to determine one or more 3D position features for voxels within the one or more voxel representations, the 3D position features defining information about a position of at least one voxel of the voxel representation of the dental maxillofacial structure relative to a position of a tooth reference surface or a tooth reference object; training a first 3D deep neural network to classify voxels of a voxel representation of a dentofacial structure into jaw voxels, tooth voxels, and / or nerve voxels, said training comprising providing the training data and the one or more 3D position features as inputs to said first 3D deep neural network, and learning optimal values ​​of network parameters of said first 3D deep neural network by minimizing a loss function representing deviation between an output of said first 3D deep neural network and said target data; A method comprising:

12. 1. A computer system for processing 3D data representing dental and maxillofacial structures, comprising: a computer-readable storage medium having computer-readable program code stored thereon, the computer-readable program code including a pre-processing algorithm and a first 3D deep neural network; and a processor coupled to the computer-readable storage medium, the processor configured to perform executable operations in response to executing the computer-readable program code, the executable operations including: an operation of receiving 3D input data, the 3D input data including a first voxel representation of the dental-maxillofacial structure, voxels associated with radiation intensity values, and voxels of the first voxel representation defining an image volume; - determining, by the pre-processing algorithm, one or more 3D position features of the dental maxillofacial structure using the 3D input data, the 3D position features defining information about positions of voxels of the first voxel representation relative to positions of dental reference surfaces or dental reference objects within the image volume; providing the first voxel representation and the one or more 3D location features associated with the first voxel representation to an input of the first 3D deep neural network, wherein the first 3D deep neural network is trained to generate classified voxels of the first voxel representation at an output of the first 3D deep neural network, the classified voxels including at least jaw voxels, tooth voxels, and / or nerve voxels; receiving the classified voxels of the first voxel representation from the output of the first 3D deep neural network, and determining a voxel representation of at least one of a jaw, a tooth, and / or a neural tissue of the dentomaxillofacial structure based on the classified voxels; A computer system comprising:

13. 13. The computer system of claim 12, wherein the pre-processing algorithm comprises a second 3D deep neural network trained to receive at its input a second voxel representation and to determine, for each voxel of the second voxel representation, a 3D location feature of the second voxel representation, the 3D location feature of the second voxel representation comprising a measure of the likelihood that the voxel represents jaw, teeth, and / or neural tissue, and the second voxel representation is a lower resolution version of the first voxel representation.

14. 14. The computer system of claim 13, wherein the first 3D deep neural network comprises a plurality of first 3D convolutional layers, outputs of the plurality of first 3D convolutional layers connected to at least one fully connected layer, the plurality of first 3D convolutional layers configured to process a first block of voxels from the first voxel representation, and the at least one fully connected layer configured to classify voxels of the first block of voxels into at least one of jaw voxels, tooth voxels, and / or nerve voxels.

15. 15. The computer system of claim 14, wherein the first 3D deep neural network further comprises a plurality of second 3D convolutional layers, outputs of the plurality of second 3D convolutional layers connected to the at least one fully connected layer, the plurality of second 3D convolutional layers configured to process a second block of voxels from the first voxel representation, the first block of voxels and the second block of voxels having the same or substantially the same center point within the image volume, the second block of voxels representing a volume with larger real-world dimensions than a volume with real-world dimensions of the first block of voxels, the plurality of second 3D convolutional layers configured to determine contextual information associated with voxels of the first block of voxels, and the voxels of the first block of voxels provided to inputs of the plurality of first 3D convolutional layers.

16. A computer program comprising software code portions which, when executed in a memory of a computer, cause said computer to carry out the steps of the method according to any one of claims 1 to 10.

Images