Tooth Segmentation Using Neural Networks

Abstract

A computer-implemented method and system for digitally segmenting teeth in a digital model includes generating a panoramic image from a 3D digital model of a patient's dentition, labeling the panoramic image using a first trained neural network to provide a labeled panoramic image, mapping the labeled panoramic image to corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model, and segmenting the labeled 3D digital model to provide a segmented 3D digital model. The computer-implemented method and system for generating a panoramic image includes determining, using a trained neural network, digital tooth bounding regions corresponding to the digital teeth from a 2D depth map of the patient's dentition, connecting the digital tooth bounding regions with splines, determining sampled digital surface points from the sampled spline points, and determining associated digital surface points corresponding to each sampled digital surface point.

Description

[Technical field]

[0001] This application claims priority to and the benefit of U.S. Utility Application No. 16 / 778,406, filed January 31, 2020, and U.S. Utility Application No. 17 / 140,739, filed January 4, 2021, the disclosures of which are incorporated by reference herein in their entireties for all purposes. [Background technology]

[0002] Professional dental laboratories typically use computer-aided design (CAD) to design dental prostheses based on patient-specific instructions provided by the dentist.

[0003] In a typical workflow, a dental laboratory receives information from a dentist regarding the patient's oral condition. To use a CAD / CAM system, a digital model of the patient's dentition is used as an input to the process. It may be desirable to segment and label the individual teeth and gingival areas in the digital model. However, labeling the teeth and finding precise dental and gingival borders can be difficult. Traditional approaches of manual segmentation can be slow, cumbersome, and error-prone. Newer approaches can be less cumbersome, but can produce inaccurate and / or missing boundaries of teeth and other areas of interest in the digital model. Summary of the Invention

[0004] A computer-implemented method for digitally segmenting teeth in a digital model includes receiving a 3D digital model of patient scan data of at least a portion of the patient's dentition; generating a panoramic image from the 3D digital model; labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image; mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model; and segmenting the labeled 3D digital model to provide a segmented 3D digital model.

[0005] A system for digitally segmenting teeth in a digital model comprises a processor and a computer-readable storage medium comprising instructions executable by the processor to perform steps including receiving a 3D digital model of patient scan data of at least a portion of the patient's dentition; generating a panoramic image from the 3D digital model; labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image; mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model; and segmenting the labeled 3D digital model to provide a segmented 3D digital model.

[0006] A non-transitory computer readable medium storing executable computer program instructions for segmenting a digital model, the computer program instructions can include receiving a 3D digital model of patient scan data of at least a portion of the patient's dentition; generating a panoramic image from the 3D digital model; labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image; mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model; and segmenting the labeled 3D digital model to provide a segmented 3D digital model.

[0007] A computer-implemented method for generating a panoramic image may include receiving a 2D depth map of patient scan data of at least a portion of the patient's dentition, determining using a trained neural network one or more digital tooth bounding regions corresponding to one or more digital teeth, connecting the one or more digital tooth bounding regions by a spline, sampling one or more spline points on the spline, determining one or more sampled digital surface points from the one or more sampled spline points, and determining one or more associated digital surface points corresponding to each sampled digital surface point. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 shows a perspective view of a digital model of teeth and gums. [Diagram 2] FIG. 1 illustrates a diagram of an example of a convolutional neural network in accordance with some embodiments. [Diagram 3] FIG. 1 illustrates an example diagram of at least a portion of a YOLO network according to some embodiments. [Figure 4(a)]FIG. 1 illustrates an example diagram of at least a portion of a U-Net network in some embodiments. [Figure 4(b)] FIG. 1 illustrates an example diagram of at least a portion of a U-Net network in some embodiments. [Figure 5(a)] FIG. 1 is a diagram of an example of a pyramidal attention network. [Figure 5(b)] FIG. 2 is a diagram of an example of a pyramid attention feature. [Figure 5(c)] FIG. 13 is a diagram of an example of a global attention upsample feature. [Figure 6(a)] FIG. 1 is a diagram of an example of a ResNet neural network. [Figure 6(b)] FIG. 1 is a diagram of an example of a portion of a ResNet neural network. [Figure 7] FIG. 2 illustrates an example of a panoramic image with overlaid diagrams showing columns and rows in some embodiments. [Figure 8] 1 shows an example image of a 2D depth map. [Figure 9] 1 shows an image of an example of a labeled 2D depth map. [Figure 10(a)] FIG. 1 shows an image of a digital dental arch having one or more digital teeth with illustrated bounding regions. [Figure 10(b)] FIG. 1 shows an image of a digital dental arch with one or more digital teeth with illustrated bounding regions and digital bounding box centers. [Figure 10(c)] FIG. 2 illustrates digital bounding box centers. [Figure 10(d)] 1 shows one or more digital tooth bounding area centers connected to each other by splines. [Figure 11] FIG. 1 illustrates an example of a digital model with sampling points on a spline. [Figure 12(a)] FIG. 1 illustrates an example of a digital model with sample digital surface points that correspond to sampling points on a continuous smoothing spline. [Figure 12(b)] FIG. 1 illustrates an example of a digital model with additional sample points. [Figure 12(c)] FIG. 2 illustrates an example of a digital model having multiple rays extending from a sample depth point. [Figure 13] FIG. 2 illustrates an example of a panoramic image with illustrated rows and columns. [Figure 14] FIG. 2 is a diagram showing an example of a panoramic image having depth information. [Figure 15] FIG. 1 shows an example of a panoramic image with labeled digital tooth bounding regions. [Figure 16(a)] FIG. 1 illustrates an example of a panoramic image that may include one or more digital bounding regions. [Figure 16(b)] FIG. 1 illustrates an example of a panoramic image that may include one or more digital bounding regions. [Figure 17] FIG. 1 illustrates an example of some embodiments of mapping to a 3D digital model. [Figure 18(a)] FIG. 1 illustrates an example of identifying an uncertainty region. [Figure 18(b)] FIG. 1 illustrates an example of identifying an uncertainty region. [Figure 19] FIG. 13 shows an example of determining the curvature on an edge connecting two triangles. [Figure 20(a)] FIG. 13 illustrates an example of determining the average curvature of an edge in some embodiments. [Figure 20(b)] 13A-13C show examples of segments after merging triangles in some embodiments. [Figure 20(c)] FIG. 1 illustrates an example of a segmented digital surface. [Figure 21] 1 illustrates an example of some embodiments of a digital surface mesh that includes multiple segments according to some embodiments. [Figure 22(a)] FIG. 1 illustrates an example of two-point segmentation in some embodiments. [Figure 22(b)]FIG. 1 illustrates an example of two-point segmentation in some embodiments. [Figure 22(c)] FIG. 1 illustrates an example of two-point segmentation in some embodiments. [Figure 22(d)] FIG. 1 illustrates an example of two-point segmentation in some embodiments. [Figure 22(e)] FIG. 1 illustrates an example of two-point segmentation in some embodiments. [Figure 22(f)] FIG. 1 illustrates an example of two-point segmentation in some embodiments. [Figure 22(g)] FIG. 1 illustrates an example of two-point segmentation in some embodiments. [Figure 23(a)] FIG. 2 illustrates an example of a portion of a boundary loop in some embodiments. [Figure 23(b)] FIG. 2 illustrates an example of a vertex diagram of a dual graph. [Figure 24] FIG. 1 shows an example of a fully segmented and labeled set of digital teeth and gingival regions. [Figure 25(a)] FIG. 1 illustrates a flowchart of an example of a computer-implemented method for digitally segmenting teeth in a digital model in some embodiments. [Figure 25(b)] FIG. 2 illustrates a flowchart of an example of a computer-implemented method for generating a panoramic image in some embodiments. [Figure 26] 1 illustrates a diagram of an exemplary processing system in accordance with some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] For the purpose of this description, certain aspects, advantages, and novel features of the embodiments of the present disclosure are described herein. The methods, apparatus, and systems of the present disclosure should not be construed as limiting in any way. Instead, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, both alone and in various combinations and subcombinations with each other. The methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, and the embodiments of the present disclosure do not require that any one or more particular advantages exist or problems be solved.

[0010] Although some operations of the embodiments of the present disclosure are described in a particular sequential order for convenience of presentation, it should be understood that this manner of description encompasses rearrangements unless a particular ordering is required by the particular language set forth below. For example, operations described sequentially may in some cases be rearranged or performed simultaneously. Furthermore, for simplicity, the accompanying drawings may not show the various ways in which the methods of the present disclosure may be used in conjunction with other methods. In addition, the description sometimes uses terms such as "provide" or "achieve" to describe the methods of the present disclosure. The actual operations corresponding to these terms may vary depending on the particular implementation and are readily discernible by those of ordinary skill in the art.

[0011] In some embodiments, the computer-implemented method can include receiving a 3D digital model of patient scan data of at least a portion of the patient's dentition, generating a panoramic image from the 3D digital model, labeling one or more regions of the panoramic image with a first trained neural network to provide a labeled panoramic image, mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model, and segmenting the labeled 3D digital model to provide a segmented 3D digital model. Alternatively, in some embodiments, the computer-implemented method can include receiving a panoramic image, labeling one or more regions of the panoramic image with a first trained neural network to provide a labeled panoramic image, mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model, and segmenting the labeled 3D digital model to provide a segmented 3D digital model.

[0012] In some embodiments, the computer-implemented method can include receiving a 3D digital model ("digital model") of patient scan data of at least a portion of the patient's dentition. FIG. 1 illustrates an example of a digital model 100 that can be generated by scanning a physical impression using any scanning technique known in the art, including, but not limited to, optical scanning, CT scanning, etc. The digital model 100 can also be generated, for example, by intraoral scanning of the patient's dentition. An example is described in U.S. Patent Application Publication No. 20180132982(A1) to Nikolskiy et al., which is incorporated by reference herein in its entirety. Conventional scanners typically capture the physical impression / shape of the patient's dentition in three dimensions during scanning and digitize the shape into a three-dimensional digital model. The digital model 100 can include, for example, multiple interconnected polygons with a topology corresponding to the shape of the physical impression / patient's dentition. In some embodiments, the polygons can include two or more digital triangles. In some embodiments, the scanning process can generate STL, PLY, or CTM files that may be suitable for use with dental restoration design software, such as, for example, FastDesign™ dental design software offered by Glidewell Laboratories of Newport Beach, Calif.

[0013] Neural Networks In some embodiments, the computer-implemented method may use one or more trained neural networks. A neural network is a computational model that is part of machine learning. A neural network typically includes nodes organized in layers. The layers of nodes may include, for example, an input layer, one or more hidden layers, and an output layer. A neural network with more than one hidden layer - typically many more hidden layers - is a deep neural network ("DNN"). Information from one layer may be processed and provided to the next layer.

[0014] In some embodiments, the DNN can be a convolutional neural network ("CNN"), which is a network that uses convolution instead of common matrix multiplication in at least one of the hidden layers of a deep neural network. A convolutional layer can calculate its output values ​​by applying a kernel function to a subset of values ​​of a previous layer. The computer-implemented method can train the CNN by adjusting the weights of the kernel function based on the training data. The same kernel function can be used to calculate each value in a particular convolutional layer. One advantage of using a CNN can include learning fewer weights during training. Another advantage of using a CNN can be, for example, detecting edge features.

[0015] FIG. 2 illustrates an example of a CNN in some embodiments. The CNN may include one or more convolutional layers, such as a first convolutional layer 202. The first convolutional layer 202 may apply a kernel (also referred to as a filter), such as kernel 204, across an input image, such as input image 203, and optionally apply an activation function to generate one or more convolutional outputs, such as a first kernel output 208. The first convolutional layer 202 may include one or more feature channels. The application of a kernel, such as kernel 204, and optionally an activation function, may result in a first convolutional output, such as convolutional output 206. The kernel may then proceed to the next set of pixels in the input image 203 based on a stride length, and apply the kernel 204, and optionally an activation function, to result in a second kernel output. The kernel may proceed in this manner until it has been applied to all pixels in the input image 203. In this manner, the CNN may generate a first convolutional image 206, which may include one or more feature channels. The first convolved image 206 may, in some embodiments, include one or more feature channels, such as 207. In some cases, the activation function may be, for example, a RELU activation function. Other types of activation functions may also be used.

[0016] The CNN may also include one or more pooling layers, such as a first pooling layer 212. The first pooling layer may apply a filter, such as a pooling filter 214, to the first convolved image 206. Any type of filter may be used. For example, the filter may be a maximum filter (outputting the maximum value of the pixels to which the filter is applied) or a mean filter (outputting the mean value of the pixels to which the filter is applied). The one or more pooling layers may downsample and reduce the size of the input matrix. For example, the first pooling layer 212 may reduce / downsample the first convolved image 206 by applying a first pooling filter 214 and providing a first pooled image 216. The first pooled image 216 may include one or more feature channels 217. The CNN may optionally apply one or more additional convolutional layers (and activation functions) as well as pooling layers. For example, the CNN can apply a second convolutional layer 218 and, optionally, an activation function to output a second convolved image 220, which can include one or more feature channels 219. A second pooling layer 222 can apply a pooling filter to the second convolved image 220 to generate a second pooled image 224, which can include one or more feature channels. The CNN can include one or more convolutional layers (and activation functions) and one or more corresponding pooling layers. The output of the CNN can optionally be sent to a fully connected layer, which can be part of one or more fully connected layers 230. The one or more fully connected layers can provide an output prediction, such as output prediction 224. In some embodiments, the output prediction 224 can include, for example, labels of the teeth and surrounding tissue.

[0017] CNNs can be structured and used in different ways. For example, details of an example of a You Only Look Once ("YOLO") network are described in You Only Look Once: Unified, Real-Time Object Detection, by Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, submitted 8 Jun 2015, last revised 9 May 2016, v5, the entirety of which is incorporated herein by reference. Further details of the YOLO network can be found in YOLO9000: Better, Faster, Stronger, by Joseph Redmon, Ali Farhadi, University of Washington, Allen Institute for AI, published 25 Dec 2016, arXiv, the entirety of which is incorporated herein by reference. Details of an example of a YOLO network are also described in YOLOv3: An Incremental Improvement, by Joseph Redmon and Ali Farhadi, University of Washington, published 2018, ArXiv, the entirety of which is incorporated herein by reference. A trained YOLO network can, for example, receive a 2D digital model of a patient's dentition and output a digital model having digital bounding boxes as digital tooth bounding regions around each digital tooth.

[0018] The YOLOv3 network (hereinafter "YOLO network") as described above may include one or more convolutional networks that predict multiple bounding boxes and class probabilities for each bounding box. In some embodiments, the YOLO network may divide the input image into an S x S grid. Each of the grid cells may predict B bounding boxes and may determine a confidence score for the bounding boxes. The confidence score may indicate the model's confidence that the bounding boxes contain the object and the accuracy of the predicted boxes. The confidence may be expressed as follows:

[0019]

number

[0020] In some embodiments, YOLO can use dimension clusters as anchor boxes to predict bounding boxes. For example, YOLO uses four coordinates for a bounding box: x , t y , t w , t h It is possible to predict that the cell is located at the top left corner of the image (C x ,C y ), and the prior bounding box is offset by width p w and height p h If we have, the prediction can correspond to: b x =σ(t x )+c x b y =σ(t y )+c y

[0021]

number

[0022]

number

[0023] In some embodiments, YOLO can perform feature extraction using one or more convolutional layers. One or more of the convolutional layers can optionally include a residual operation. Figure 3 shows an example of YOLO feature extraction. The following indicates, by way of example, the operations performed in each layer:

[0024] [Table 1]

[0025] Layer 310 may be performed twice, layer 314 may be performed eight times, layer 318 may be performed eight times, and layer 322 may be performed four times, resulting in a total number of convolutions for the entire network of 53 convolutions. The avgpool may be global. Other configurations and variations are also contemplated in the YOLO network. In some embodiments, a trained YOLO network may receive an image and provide a bounding region around each feature in the image. The features may be defined during training. Training YOLO may include minimizing a loss function. The loss function only penalizes classification errors when an object is within a particular grid cell. The loss function penalizes bounding box coordinate errors if a particular predictor contributes to the ground truth box. For example, if a particular predictor has the highest IOU of all predictors in a particular grid cell.

[0026] In some embodiments, the computer-implemented method can use a semantic segmentation network. The semantic segmentation network can be any neural network that provides pixel-wise segmentation. Some examples of semantic segmentation networks can include DeconvNet, FPN, DeepLabV3, U-Net, and Pyramid Attention Network.

[0027] In some embodiments, the computer-implemented method may use, for example, U-Net as the semantic segmentation network. U-Net is a convolutional neural network that may be used for biomedical image segmentation and is described in U-Net: Convolutional Networks for Biomedical Image Segmentation, by Olaf Ronneberger, Philipp Fischer, and Thomas Brox, Computer Science Department and BIOSS Centre for Biological Signalling Studies, University of Freiburg, Germany, arXiv, 18 May 2015, the entirety of which is incorporated herein by reference. A standard CNN typically includes one or more convolutional / pooling layers, known as contraction layers. The U-Net architecture may combine one or more convolutional / pooling contraction layers with one or more convolutional / upsampling layers. Therefore, the U-Net architecture may increase the resolution output. Localization may be achieved by combining high-resolution features from the contraction path with the upsampled output. The U-Net architecture may also include multiple feature channels to provide contextual information to the higher resolution layers. In some embodiments, the expansion path can be symmetrical to the contraction path, resulting in a U-shaped architecture.

[0028] FIG. 4(a) illustrates an example of a U-Net CNN including a contraction path 402 and an expansion path 404 in some embodiments. The contraction path 402 can include one or more convolution layers, such as a first convolution layer 406. Each convolution layer can perform one or more convolutions, such as a first convolution 410. The convolutions can utilize kernels of any size. In some embodiments, for example, a 3×3 kernel can be used to perform one or more convolutions. In some embodiments, each convolution layer can perform two convolutions.

[0029] In some embodiments, each convolution can generate a feature map, and each feature map can include one or more feature channels. In some embodiments, each convolution in a convolution layer in a contraction path can maintain the same number of feature channels. For example, the network can receive an input image 401 that can have an image resolution and a number of feature channels n. In the convolution layer 406, the first convolution 410 can provide an n-channel feature map, such as the first feature map 411, and the second convolution 413 can provide an n-channel feature map, such as the last feature map 412. Both the first feature map 411 and the last feature map 412 can include n channels. In some embodiments, each convolution may not be padded. In some embodiments, one or more convolutions may be padded. Each of the one or more convolutions may also be followed by an activation function, such as ReLu, or other activation functions known in the art.

[0030] The U-Net CNN may also include performing a pooling operation, such as the first pooling operation 408. The pooling operation in some embodiments may downsample each feature map. In some embodiments, the pooling operation may be performed on the last feature map generated after the last convolution in a particular convolution layer. For example, in FIG. 4(a), the first pooling operation 408 may be performed on the last feature map 412, and the result may become the convolution layer input 414 to the next convolution layer 416. The first pooling operation 408 may be any pooling operation known in the art. For example, in some embodiments, the first pooling operation 408 may be a max pooling operation. In some embodiments, the first pooling operation 408 may be an average pooling operation. The pooling operation may be any value. For example, in some embodiments, the pooling operation may be a 2×2 pooling operation with a stride of 2.

[0031] In some embodiments, each pooling operation can be optionally followed by additional convolutional layers and pooling operations. For example, the first pooling operation 408 can be followed by a second convolutional layer 416, followed by a third convolutional layer 418, and a fourth convolutional layer 427, with a pooling operation between each convolutional layer. In some embodiments, the number of feature channels can be doubled for each convolutional layer in the contraction path 402. For example, the second convolutional layer 416 can double the number of feature channels to twice that of the first convolutional layer 406, the third convolutional layer 418 can double the number of feature channels to twice that of the second convolutional layer 416, and the fourth convolutional layer 427 can double the number of feature channels to twice that of the third convolutional layer 418. This can provide 2*n feature channels for each feature map provided by each convolution layer in the contraction path 402. For example, the first convolution layer 406 can receive an input image with one channel and create a feature map with 64 channels after performing one or more convolutions, the second convolution layer 416 can provide a feature map with 128 channels after performing one or more convolutions, the third convolution layer 418 can provide a feature map with 256 channels after performing one or more convolutions, and the fourth convolution layer 418 can provide a feature map with 512 channels. The dimensions of the image can be reduced at each convolution. For example, the input image can have a resolution of 572×572. The first convolution 410 can provide a feature map 411 with a resolution of 570×570. The second convolution can provide a final feature map 412 having a resolution of 568 x 568. In some embodiments, each pooling operation between convolutional layers can reduce the resolution by a factor of 2. For example, the first pooling operation 408 can reduce the final feature map 412 having a resolution of 568 x 568 to a convolutional layer input 414 having a resolution of 284 x 284.

[0032] The U-Net CNN can perform a final pooling operation 422 in the contraction path 402 to a final contraction path convolutional layer 427. The U-Net CNN can perform one or more convolutions in the convolutional layer 426. In some embodiments, the input to the convolutional layer 426 can have a resolution of 32x32 with 1024 feature channels. The output from the convolutional layer 426 can be a feature map with a resolution of 28x28 with 1024 feature channels.

[0033] The U-Net CNN can perform upsampling of the feature maps in the dilation path 404. The dilation path 404 can include one or more upsampling layers, such as upsampling layer 425. Each upsampling layer can halve the number of feature channels. For example, upsampling 424 can halve the number of feature channels for the final feature map from the convolutional layer 426 to provide the feature map 428. The upsampling can be, for example, a 2×2 deconvolution. Each upsampling layer can concatenate the cropped feature maps from the contraction path. For example, upsampling layer 425 can perform a concatenation operation 429 to concatenate the cropped feature map 430 from the contraction path 424 to the upsampled feature map 428. Each upsampling can double the resolution of the feature map from the previous upsampling layer. For example, upsampling 424 can double the feature map resolution from the convolutional layer 426 to provide the feature map 428. As an example, if the final feature map from the convolutional layer 426 is 28×28, the feature map 428 can be 56×56 after upsampling 424. Each upsampling layer can also perform one or more convolutions, each of which can be followed by an activation function, such as ReLU, or any other activation function known in the art. The convolutions in each upsampling layer can be, for example, 3×3 convolutions. The U-Net CNN can perform a 1×1 convolution as the last convolution in the final upsampling layer. For example, the final upsampling layer 432 can include a final convolution 434, which can provide, for example, an output segmentation map 435, in some embodiments. In some embodiments, the output segmentation map 435 can have, for example, a resolution of 388×388 and can include two channels.

[0034] FIG. 4(b) illustrates another embodiment of a U-net neural network 4000. The U-Net neural network 4000 can receive an input image, such as input image 4002, and perform one or more convolutions in one or more convolution layers. In some embodiments, the U-net neural network can include a contraction path, such as contraction path 4006, and an expansion path, such as expansion path 4021. The input image can be of any resolution and can include any number of channels. For example, in the figure, input image 4002 can have an input image resolution of 512×512 with n channels ("N×512×512"). In some embodiments, each downward arrow in the figure, such as convolution layer 4004 in contraction path 4006, can include 4×4 convolutions (stride 2×2) with a LeakyReLU activation function and normalization, such as InstanceNorm, as known in the art. In some embodiments, the convolution layers can also include pooling. In some embodiments, each upward arrow, such as deconvolution layer 4032 in the dilation path 4021, may include a 4x4 deconvolution (stride 2x2, padding 3x3) with a ReLU activation function and normalization such as InstanceNorm as known in the art. In some embodiments, the convolution layers may also include upsampling.

[0035] In some embodiments, each convolutional layer in the contraction path 4006 can halve the resolution of its input image. For example, the first convolutional layer 4004 can receive the input image 4002 and output a first feature map 4008 that can have a resolution of 128×256×256 (128 channels with an image resolution of 256×256). In some embodiments, the output feature map after each convolutional layer can be input to another convolutional layer. For example, the first feature map 4008 output from the first convolutional layer 4002 can be input to a second convolutional layer 4005, which can provide a second feature map 4010 with a resolution of 245×128×128. The second feature map 4010 can be input to a third convolutional layer 4007, which can output a third feature map 4012 with a resolution of 512×64×64. The third feature map 4012 may be input to a fourth convolutional layer 4009, which may output a fourth feature map 4014 having a resolution of 1024×32×32. The fourth feature map 4014 may be input to a fifth convolutional layer 4011, which may output a fifth feature map 4016 having a resolution of 1024×16×16. The fifth feature map 4016 may be input to a sixth convolutional layer 4013, which may output a sixth feature map 4018 having a resolution of 1024×8×8. The sixth feature map 4018 may be input to a seventh convolutional layer 4015, which may output a seventh feature map 4020 having a resolution of 1024×4×4.

[0036] In some embodiments, each deconvolution layer in the dilation path 4021 can double the resolution of its input image. In some embodiments, one or more deconvolution layers can concatenate cropped feature maps from the erosion path after deconvolution. In some embodiments, the output feature maps after each deconvolution layer can be input to another deconvolution layer.

[0037] For example, the first deconvolution layer 4032 may receive the feature map 4020, perform deconvolution, concatenate 4030 the sixth feature map 4018, and output a first deconvolution feature map 4038 that may have a resolution of 2048×8×8 (2048 channels with an 8×8 image resolution). The second deconvolution layer 4033 may receive the first deconvolution feature map 4038, perform deconvolution, concatenate 4030 the fifth feature map 4016, and output a second deconvolution feature map 4036 that has a resolution of 2048×16×16. The third deconvolution layer 4031 may receive the second deconvolution feature map 4036, perform deconvolution, concatenate the fourth feature map 4014, and output a third deconvolution feature map 4034 having a resolution of 2048×32×32. The fourth deconvolution layer 4039 may receive the third deconvolution feature map 4034, perform deconvolution, concatenate the fifth feature map 4012, and output a fourth deconvolution feature map 4042 having a resolution of 1024×64×64. The fifth deconvolution layer 4037 may receive the fourth deconvolution feature map 4042, perform deconvolution, concatenate the fifth feature map 4010, and output a fifth deconvolution feature map 4040 having a resolution of 512×128×128. A sixth deconvolution layer 4035 may receive the fifth deconvolution feature map 4040, perform deconvolution, and concatenate the fifth feature map 4008, and may output a sixth deconvolution feature map 4048 having a resolution of 256×256×256. A seventh deconvolution layer 4044 may receive the sixth deconvolution feature map 4048, perform convolution, and output a seventh deconvolution feature map 4050 having a resolution of 1×512×512.

[0038] In some embodiments, each deconvolution layer may be preceded by upsampling.

[0039] In some embodiments, training a U-Net CNN can determine a pixel-wise softmax energy function over the feature maps and combine it with a cross-entropy loss function, where softmax is:

[0040]

number

[0041]

number

[0042] In some embodiments, the semantic segmentation network can be a Pyramid Attention Network ("PAN"). An example of a PAN is described in Pyramid Attention Network for Semantic Segmentation by Hanchao Li, Rengfei Xiong, Jie An and Lingxue Wang, arXiv:1805.10180v3, submitted Nov 25, 2018, which is incorporated by reference in its entirety. Other PAN architectures can also be used. The PAN can include an encoder and decoder with Feature Pyramid Attention ("FPA") features and Global Attention Upsample ("GAU") features to increase the receptive field and effectively recover pixel localization details. In some embodiments, the GAU can be part of the decoder branch of the PAN, for example, to select a discriminative multi-resolution feature representation.

[0043] FIG. 5(a) illustrates an example of a PAN in some embodiments. The PAN may include an encoder, such as encoder 5002. In some embodiments, the encoder may receive an input image, such as input image 5001. The encoder may include one or more stages, where each stage may be a combination of layers that may perform, for example, a downsampling operation on the input data. During encoding, each stage may result in an array of features (tensors) that constitute feature maps at different levels. Each stage may provide one or more output feature maps corresponding to a feature layer. In some embodiments, the FPA features may be between the encoder and the decoder. In some embodiments, the decoder may include, for example, 32 channels.

[0044] In some embodiments, the encoder can be any kind of ResNet architecture. An example of a ResNet architecture is described in ResNets, by Pablo Ruiz-Harvard University, August 2018, the entirety of which is incorporated herein by reference. In some embodiments, the encoder can be a ResNet-101 encoder. In some embodiments, different stages of the ResNet-101 encoder can output feature maps of a particular number of channels. For example, in ResNet-101, Conv-1 5004 can output a 64-channel feature map, Res-2 5008 can output a 256-channel feature map, Res-3 5010 can output a 512-channel feature map, Res-4 5012 can output a 1024-channel feature map, and Res-5 5014 can output a 2048-channel feature map.

[0045] In a preferred embodiment, the encoder can be a ResNet-34 encoder. In some embodiments, the ResNet-34 encoder can include one or more stages, such as a convolution stage Conv-1 5004 followed by one or more stages such as Res-2 5008, Res-3 5010, Res-4 5012, and Res-5 5014. In a ResNet-34 encoder, for example, Conv-1 5004 can output a 64-channel feature map, Res-2 5008 can output a 64-channel feature map, Res-3 5010 can output a 128-channel feature map, Res-4 5012 can output a 256-channel feature map, and Res-5 5014 can output a 512-channel feature map.

[0046] FIG. 6(a) shows an example of a ResNet-34 neural network. An input image 6002 can be provided to a first ResNet convolutional layer Conv-1 6004. In some embodiments, the first ResNet convolutional layer can be a 7×7 convolution with 64 channels and a stride of 2. In some embodiments, this can be followed by a pooling step 6006 with a stride of 2. Some embodiments of a ResNet neural network can include, for example, four additional ResNet stages, or layers, each of which can perform one or more 3×3 convolutions and bypass the input every two convolutions. The height and width dimensions (i.e., image resolution) and the number of channels (feature map dimensions) can remain the same within each stage.

[0047] FIG. 6(a) illustrates an example of a ResNet-34 neural network that can include five stages: Conv-1 6004, Res-2 6010, Res-3 6012, Res-4 6014, and Res-5 6016. Each block in the Res-2 6010, Res-3 6012, Res-4 6014, and Res-5 6016 stages represents a pair of 3×3 convolutions performed in series (i.e., the output of the first convolution is the input of the second convolution in each box). For example, a first block 6018 can include a first Res-2 convolution 6020 and a second Res-2 convolution 6022 performed in series. In some embodiments, the first convolution in the first block of one or more ResNet stages can increase the stride from 1 to 2. For example, the first Res-3 convolution 6024 in stage Res-3 6012, the first Res-4 convolution 6026 in stage Res-4 6014, and the first Res-5 convolution 6027 in stage Res-5 6016 may each use a stride of 2, while the remaining convolutions use a stride of 1. In some embodiments, a ResNet stage may perform a bypass. In a bypass, the input of a first block is added to the output of the first block, and the sum is provided as an input to the next block. For example, as illustrated in FIG. 6(a), the input to the first block 6018 bypasses the first block via bypass 6028 and is added to the output of the first block 6018. This sum is then provided as an input to the next block 6040.

[0048] FIG. 6(b) shows an example of a bypass in more detail. A first block 6050 can receive a first block input feature map 6052. The input image 6052 can undergo a first convolution 6054, e.g., a 3×3 convolution, to provide a first convolution output 6056, which can undergo a second convolution 6058 to provide a first block output 6060. The ResNet neural network can sum the first block input feature map 6052 with the first block output 6060 via a bypass 6062 and provide the sum as a second block input feature map 6064 to a second block 6066. This pattern can be repeated throughout the ResNet stages, as illustrated in FIG. 6(a).

[0049] Referring to FIG. 6(a), the Conv-1 6004 stage can output an image size of 112×112 after performing a 7×7, 64 channel, stride 2 convolution. The Res-2 6010 stage includes three blocks, each performing two 3×3, 64 channel convolutions, and can output an image size of 56×56. The Res-3 6012 stage includes four blocks, each performing two 3×3, 128 channel convolutions, and can output an image size of 28×28. The Res-4 6014 stage includes six blocks, each performing two 3×3, 256 channel convolutions, and can output an image size of 14×14. The Res-5 6016 stage includes three blocks, each performing two 3×3, 512 channel convolutions, and can output an image size of 7×7. Finally, the pooling stage 6029 may, in some embodiments, perform average pooling, 1000-d fc, softmax, and output an image size of 1×1.

[0050] As illustrated in FIG. 5(a), the output of a ResNet stage may be input to the next stage of the encoder in some embodiments. In addition, the output of a particular stage may also be input to a GAU. For example, the output of Res-2 5008 may be input to Res-3 5010 and also to GAU-3 5016, the output of Res-3 5010 may be input to Res-4 5012 and also to GAU-2 5018, and the output of Res-4 5012 may be input to Res-5 5014 and also to GAU-1 5020. In some embodiments, the output feature map of the last stage of the encoder may be input to an FPA. For example, the output feature map of Res-5 5014 may be input to FPA 5022.

[0051] In some embodiments, the FPA can combine features from three pyramid scales by using a U-shaped structure. In some embodiments, the FPA can embed context features of different scales into a fully convolutional network ("FCN")-based pixel prediction framework. FIG. 5(b) illustrates an example of an FPA in some embodiments. The FPA 5030 can receive a feature map 5034 from an encoder network, such as, for example, Resnet 5032. The resolution of the feature map can be, for example, 32×32 in some embodiments. In some embodiments, the FPA can include downsampling followed by upsampling. For example, the FPA 5030 can include a downsampling branch 5038 and an upsampling branch 5040. In some embodiments, the downsampling branch 5038 may include one or more convolution layers, such as a first downsample convolution 5048 that may perform a 7x7 convolution, a second downsample convolution 5050 that may perform a 5x5 convolution, and a third downsample convolution 5052 that may perform a 3x3 convolution. Since high-level feature maps have lower resolution, a large kernel size does not increase the computational complexity. In some embodiments, the pyramid structure may integrate differently scaled information at each step, for example, allowing it to incorporate adjacent scales of context features more precisely. The output of each downsample convolution may be fed into another convolution layer with the same filter size. For example, the output of the third downsample convolution 5052 may be input to a 3x3 convolution 5054, the output of the second downsample convolution 5050 may be input to a 5x5 convolution 5056, and the output of the first downsample convolution 5048 may be input to a 7x7 convolution 5058. In some embodiments, the upsampling branches may include increasing feature map resolutions.For example, in some embodiments, the resolution can be 4×4 in the first upsample branch stage 5042, 8×8 in the second upsample branch stage 5044, and 32×32 in the third upsample branch stage 5046. Each upsampling branch can include concatenation in some embodiments, for example, as illustrated in the figure. The original features from the convolutional network can be multiplied pixel-by-pixel, for example, in per-pixel multiplier 5070, after passing through a convolutional layer, such as convolutional layer 5072, which can perform 1×1 convolutions. One or more advantages of the pyramidal structure can include, for example, at least fusing different scales of context information and creating pixel-level attention for high-level feature maps.

[0052] In some embodiments, the FPA may include a global pooling branch, such as global pooling branch 5036, which may include global pooling 5060, 1×1 convolution 5062, and upsampling 5064.

[0053] The provided filters and resolution sizes are merely examples. The filters and resolution sizes can be any suitable values. The downsampling and upsampling branches can include more or fewer stages in some embodiments. In some embodiments, batch normalization is performed after every convolution layer.

[0054] In some embodiments, the FPA may provide input to a decoder, such as decoder 5074 shown in FIG. 5(a). As shown in the figure, in some embodiments, the decoder may also receive input from one or more GAUs, such as, for example, GAU-1 5020, GAU-2 5018, and GAU-3 5016. In some embodiments, the FPA 5022 may provide its output as input to the decoder 5074 and to the GAU-1 5020. The GAU-1 5020 may also receive input from a ResNet layer, such as Res-4 5012. In some embodiments, the GAU-2 5018 may receive input from a corresponding ResNet layer, such as, for example, Res-3 5010, and from a first decoder stage, such as the first decoder stage 5076. In some embodiments, GAU-3 may receive inputs from Res-2 5008 as described above and from the second decoder stage 5078. In some embodiments, the third decoder stage 5080 may sum the output of GAU-3 5016 with the output from the second decoder stage 5078. In some embodiments, each decoder stage may perform the summation of its inputs.

[0055] In some embodiments, each GAU may include performing global average pooling, which may provide a global context of low-level features that may help select category localization details, for example. FIG. 5(c) illustrates an example of a GAU, such as GAU 50100. In some embodiments, the GAU may perform a convolution, such as convolution 50102, on one or more low-level features, such as low-level features 50101. In some embodiments, convolution layer 50102 may be a 3×3 convolution that may reduce channels of a feature map from a CNN, for example. In some embodiments, global context from high-level features, such as high-level features 50104, may be pooled in global pooling 50106, which may provide one or more feature maps, such as feature map 50108, for example. In some embodiments, feature map 50108 may have a resolution of, for example, C×1×1, where C may represent the number of channels. In some embodiments, the GAU can generate a global context from the high-level features through a convolution with batch normalization and ReLU nonlinearity that can be multiplied with the low-level features. For example, in some embodiments, the convolution layer 50110 can provide a 1×1 convolution with batch normalization and ReLU nonlinearity. The convolution can be multiplied with the low-level features from the convolution layer 50102 in multiplier 50112 to provide one or more weighted low-level features, such as weighted low-level features 50114. In some embodiments, the high-level features can be added to the weighted low-level features. For example, in some embodiments, the high-level features 50104 can be added to the weighted low-level features 50114 in sum 50116. One or more advantages of such a GAU can include, for example, unfolding feature maps at different scales and using the high-level features to provide guidance information to the low-level feature maps. In some embodiments, the PAN can provide a segmented image 5075.

[0056] In some embodiments, ResNet can be pre-trained on ImageNet using, for example, a dilated convolution strategy to extract feature maps. For example, in some embodiments, a dilated convolution rate of 2 is applied to the res5b block. This can result in, for example, feature maps whose output size from ResNet is 1 / 16 of the input image. In the case of ResNet-101 and ResNet-34 implementations, the 7x7 convolution layer can be replaced with, for example, three 3x3 convolution layers. In some embodiments, training can include a poly learning rate as known in the art. For example, the initial rate can be (1-iter / max_iter) power where the power is 0.9 and the initial rate is 4e-3. In some embodiments, the network can be trained using mini-batch stochastic gradient descent ("SGD") with a batch size of 16, momentum of 0.9, and weight decay of 0.0001. In some embodiments, for example, cross-entropy error at each pixel across categories can be applied as a loss function. During training, for example, random left-right flipping and random scaling of 0.5 to 2 for all datasets can be used. In some embodiments, mean pooling can be preferred over max pooling, especially for convolutions with a kernel size of 3x3. In some embodiments, the stride can be, for example, 16. One or more advantages of using PAN can include, for example, improved performance.

[0057] In some embodiments, the computer-implemented method may implement, include, and / or perform one or more features of one or more neural networks. In some embodiments, the computer-implemented method may train one or more neural networks. Although specific values ​​are discussed for one or more features in one or more neural networks, the values ​​are provided merely as examples. Other suitable values ​​and variations may be used.

[0058] Panoramic Images In some embodiments, the computer-implemented method can generate a panoramic image from the 3D digital model using a trained neural network. The panoramic image can be, for example, a 2D raster image including one or more columns and one or more rows. In some embodiments, the computer-implemented method can generate each column and row in the 2D panoramic image. The panoramic image can advantageously show, for example, more digital teeth in the digital model from more sides. The panoramic image can advantageously be, for example, a single image showing both the lingual and buccal gum-crown margins of all teeth. FIG. 7 shows an example of a panoramic image with superimposed views of columns and rows, such as row 5102 and column 5104. The number of columns and rows shown in FIG. 7 is for illustrative purposes only. In some embodiments, the desired number of columns and rows can be set in a configuration file or through a graphical user interface element, such as an input field for both values. In some embodiments, the computer-implemented method can load, for example, the desired number of columns and rows from a configuration file. A larger number of columns and rows can increase the resolution of the panoramic image. As an example, if the number of columns is set to, for example, 2048 and the number of rows is set to, for example, 320, then the panoramic image can be 2048 x 320 (width x height). This can mean that the computer-implemented method takes 2048 samples on the spline. Smaller dimensions can be faster to process, while larger dimensions can provide better spatial resolution. One advantage includes being able to make a choice between speed and spatial resolution.

[0059] In some embodiments, the neural network can be trained by providing a 2D depth map training data set that can include one or more 2D depth maps of at least a portion of a digital dental arch having one or more digital teeth, where each digital tooth is labeled with a labeled digital tooth bounding region, such as, for example, a rectangular boundary. Other shapes for the labeled digital tooth bounding region can also be used. Some embodiments of the computer-implemented method can optionally include generating a 2D image from the 3D digital model. In some embodiments, the 2D image can be a 2D depth map. The 2D depth map can include a 2D image that includes, in each pixel, a distance from an orthogonal camera to an object along a line that passes through the pixel. For example, the object can be, for example, a digital jaw model surface, in some embodiments. In some embodiments, the input can include an object, such as, for example, a 3D digital model of the patient's dentition ("digital model"), such as, for example, a jaw, and a camera orientation. In some embodiments, the camera orientation can be determined based on an occlusal direction. The occlusal direction is perpendicular to the occlusal plane, and the occlusal plane can be determined for the digital model using any technique known in the art. For example, one technique is described in AN AUTOMATIC AND ROBUST ALGORITHM OF REESTABLISHMENT OF DIGITAL DENTAL OCCLUSION, by Yu-Bing Chang, James J. Xia, Jaime Gateno, Zixiang Xiong, Fellow, IEEE, Xiaobo Zhou, and Stephen TC Wong in IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 29, NO. 9, SEPTEMBER 2010, the entirety of which is incorporated herein by reference.Alternatively, in some embodiments, the occlusal direction may be specified by a user manipulating a digital model on a display with an input device such as a mouse or touch screen, for example, as described herein. In some embodiments, the occlusal direction may be determined using, for example, the occlusal axis method described in Nikolskiy et al., PROCESSING DIGITAL DENTAL IMPRESSION, U.S. Patent Application Publication No. 16 / 451,968, the entirety of which is incorporated herein by reference.

[0060] The 2D depth map can be generated using any technique known in the art, including, for example, z-buffer or ray tracing. For example, in some embodiments, the computer-implemented method can initialize, for example, the depth of each pixel (j,k) to a maximum length and the pixel color to a background color. For each pixel in a projection of a polygon onto a digital surface, such as a 3D digital model, the computer-implemented method can determine the depth z of the polygon at (x,y) corresponding to pixel (j,k). If z<depth of pixel (j,k), then set the depth of the pixel to depth z. "z" can refer to the convention that the central viewing axis of the camera is in the direction of the camera's z-axis, but not necessarily the absolute z-axis of the scene. In some embodiments, the computer-implemented method can also, for example, set the pixel color to something other than the background color. In some embodiments, the polygon can be, for example, a digital triangle. In some embodiments, the depth in the map can be per pixel. FIG. 8 illustrates an example of a 2D depth map of a digital model in some embodiments.

[0061] To generate the training data set, each digital tooth in each 2D depth map in the training data set can be labeled with a digital tooth bounding region. FIG. 9 shows an example of a 2D depth map 700 with labeled digital tooth bounding regions 702. The labeled digital tooth bounding regions for each digital tooth in the 2D depth map can be labeled manually, for example, by a user or technician, or by an automated process, in some embodiments. In some embodiments, the computer-implemented method can provide the 2D depth map training set to a neural network to provide a 2D depth map trained neural network. In some embodiments, the 2D depth map trained neural network can receive one or more 2D depth maps and automatically generate digital tooth bounding regions for each digital tooth in the 2D depth map. In some embodiments, the computer-implemented method can train the YOLO network with one or more 2D depth maps, each having a labeled digital tooth bounding region shaped as a rectangle or box. In some embodiments, the computer-implemented method can train a semantic segmentation network using one or more 2D depth maps, each having a labeled digital tooth bounding region shaped to follow the overall shape of each digital tooth as it appears in the 2D depth map. For example, in some embodiments, the computer-implemented method can train a CNN, a U-Net CNN, or a PAN neural network. In some embodiments, the training data set can include, for example, 10,000 2D depth map images. For example, in some embodiments, other suitable numbers of 2D depth map images can be used as the training data set.

[0062] After training, in some embodiments, the 2D depth map trained neural network can receive one or more unlabeled 2D depth maps, each having a digital dental arch, and provide a digital tooth bounding region for each digital tooth within at least a portion of each digital dental arch. In some embodiments, the computer-implemented method can, for example, use the trained neural network to roughly define a digital tooth bounding region around each digital tooth. Each digital tooth bounding region can provide a rough approximation of the position of each tooth when viewed in an occlusal direction. FIG. 10(a) shows an example of a digital model 800 having a digital dental arch with one or more digital teeth including a digital tooth bounding region around each digital tooth. In this example, the digital tooth bounding region is a digital bounding box 802 that bounds a digital tooth 804 when viewed in an occlusal direction 806. Although a bounding box 802 is shown, each digital tooth bounding region may be of any suitable shape and / or size for bounding a particular digital tooth, depending on the embodiment.

[0063] In some embodiments, the 2D depth map trained neural network is a 2D depth map trained convolutional neural network as described above. In some embodiments, the 2D depth map trained CNN can be a 2D depth map trained YOLO network as described above. The trained 2D depth map YOLO network can receive a 2D depth map and provide a digital tooth bounding region for each digital tooth within at least a portion of the 2D depth map. The computer-implemented method can, for example, in some embodiments, label all pixels bounded by the digital tooth bounding region with a unique label. The digital tooth bounding region provided by the trained 2D depth map YOLO network can, for example, be a digital tooth bounding box.

[0064] In some embodiments, the 2D depth map trained neural network is a 2D depth map trained semantic segmentation network. In some embodiments, the 2D depth map trained semantic segmentation network can be a 2D depth map trained U-Net as described above. In some embodiments, the 2D depth map trained semantic segmentation network is a PAN network as described above. The trained 2D depth map semantic segmentation network can receive the 2D depth map and can provide a digital tooth bounding region for each digital tooth within at least a portion of the 2D depth map. The computer-implemented method can, for example, in some embodiments, label all pixels bounded by the digital tooth bounding region with a single label. The digital tooth bounding region provided by the trained 2D depth map semantic segmentation network can be shaped to generally follow, for example, the shape of the corresponding digital tooth in the 2D depth map. Therefore, a trained 2D depth map semantic segmentation network can provide pixel-wise segmentation / labeling.

[0065] The labels can, for example, distinguish each tooth from another and / or each tooth from the surrounding gums and gingiva. For example, a YOLO network can be faster to train and apply, but it cannot provide the pixel-by-pixel labeling that a semantic segmentation network can provide. Therefore, in some embodiments, a 2D depth map trained YOLO network may be suitable for generating panoramic images. However, for example, in some embodiments, a 2D depth map trained semantic segmentation network can also be used to generate panoramic images. In some embodiments, each digital tooth can be identified based on its size and location in the digital dental arch.

[0066] In some embodiments, the computer-implemented method can connect one or more of the digital tooth bounding regions to one another. In some embodiments, the computer-implemented method can connect one or more of the bounding region centers to one another. In some embodiments, the computer-implemented method can determine a center of each digital tooth bounding region. The digital tooth bounding region center can be, for example, in some embodiments, the geometric center of the digital tooth bounding region. FIG. 10(b) shows an example where the digital tooth bounding region is a digital bounding box 802 for a digital tooth 804. The computer-implemented method can determine a digital bounding box center 808. The digital bounding box center 808 can be, for example, in some embodiments, the geometric center of the digital bounding box 802. FIG. 10(c) shows an example of a digital bounding box center of the digital model 800 from FIG. 10(a), such as the digital bounding box center 808 of the digital bounding box 802.

[0067] In some embodiments, the computer-implemented method can connect neighboring digital tooth bounding area centers by passing a spline through one or more digital tooth bounding area centers. In some embodiments, the computer-implemented method can select one or more digital tooth bounding area centers through which to pass the spline such that the spline forms a smooth convex hull. In some embodiments, the computer-implemented method can pass the spline through all of the digital tooth bounding area centers. In some embodiments, the spline can be a smooth spline. In some embodiments, neighboring digital tooth bounding areas are adjacent to each other. This may be because the digital tooth bounding areas are of neighboring or adjacent teeth. The spline between two digital tooth bounding area centers can be determined using any technique known in the art. For example, a suitable spline between two digital tooth bounding area centers can include a cubic Hermite spline. The cubic Hermite spline may, for example, in some embodiments, include the locations of adjacent digital tooth bounding area centers and the first derivative at each of the two digital tooth bounding area centers. Thus, the computer-implemented method may provide one or more splines connected to each other to form a continuous spline that passes through two or more of the digital tooth bounding area centers. The computer-implemented method may determine the spline using any technique known in the art. For example, in some embodiments, the spline may be determined by determining the tangent at every center point and then constructing the spline between every two digital tooth bounding area centers as a Bezier curve. In some embodiments, if a digital tooth bounding area center has only one adjacent neighboring digital tooth bounding area center, the computer-implemented method may smoothly extend a straight line to make the spline continuous on the side that does not have an adjacent neighboring digital tooth bounding area center.

[0068] FIG. 10(d) shows an example of one or more digital tooth bounding area centers, such as a first bounding box center 808 and a second bounding box center 810, connected to each other by a first spline 812. The figure also shows a second spline 814 connecting the second digital bounding box center 810 with a third digital bounding box center 816. In the case of the first bounding box center 808, which is missing one neighbor, the computer-implemented method can extend the first spline 812 in a straight line 824. Similarly, in the case of the last bounding box center 822, which is missing one neighbor, the computer-implemented method can extend the last spline 826 in a straight line 828. The computer-implemented method can thus connect the neighboring digital bounding box centers into a continuous spline 820. As described above, the spline can be a continuous smooth spline.

[0069] In some embodiments, the computer-implemented method may sample one or more points on the spline. In some embodiments, the computer-implemented method may determine the number of points to sample based on the number of desired columns in the panoramic image. For example, in some embodiments, the number of sample points may be the same as the number of columns. In some embodiments, the computer-implemented method may sample a user-configurable number of sample points. In some embodiments, the number of samples may be set and loaded from a configuration file or set by a user using an input device to enter a value into an input field of the GUI. The number of samples may be any suitable value, with a higher number of samples corresponding to a higher resolution panoramic image and a lower number of samples corresponding to a lower resolution but faster generated panoramic image. As an example, the number of sample points may be 2048. FIG. 11 shows an example of sample points 902 on a continuous smooth spline 904 projected onto a digital model 906.

[0070] In some embodiments, the computer-implemented method can determine sampled digital surface points corresponding to the sampled points on the spline. In some embodiments, the computer-implemented method can determine the sampled digital surface points by projecting the sampled points onto the 3D digital model. FIG. 12(a) shows an example of sampled digital surface points 1002 corresponding to sampled points on a continuous smooth spline 1004 for a 3D digital model 1006. In some embodiments, the computer-implemented method can determine sampled digital surface points corresponding to the sampled points on the spline by determining a sample plane that is orthogonal to the spline at the sampled point on the spline and parallel to the occlusion direction in the digital model, and determining the intersection of the plane with the digital surface at the sampled point on the spline. For example, as illustrated in FIG. 12(a), the computer-implemented method can determine a sample plane 1008 that is orthogonal to the continuous smooth spline 1004 at the sampled digital surface point 1002 and parallel to the occlusion direction 1010.

[0071] In some embodiments, the computer-implemented method can determine one or more digital surface points associated with the sampled digital surface point. In some embodiments, the computer-implemented method can determine, or use, for each sampled digital surface point, a sample plane that is orthogonal to the continuous smooth spline at the sampled digital surface point and parallel to the occlusion direction in the digital model.

[0072] In some embodiments, the computer-implemented method can move the sampled digital surface point down along the occlusal direction by the sampled digital surface point depth along the sampled digital surface point depth ray to the sample depth point. For example, the computer-implemented method can move the sampled digital surface point 1002 down along the sampled point depth ray 1012 to the sampled depth point 1014, and in some embodiments, along the sampled point depth ray 1012 along the occlusal direction 1010. The sampled point depth ray can be parallel to the occlusal direction in some embodiments. In some embodiments, the computer-implemented method can move the sampled digital surface point in a sample plane. For example, the sampled digital surface point 1002 can be moved in the sample plane 1008. In some embodiments, the sampled point depth can be, for example, 15 mm along the occlusal direction from the sampled digital surface point. Other suitable sample point depths can be used.

[0073] In some embodiments, the computer-implemented method can determine one or more rays from the sample depth point to the digital surface and determine one or more digital surface points associated with the sampled digital surface point. In some embodiments, the computer-implemented method can determine the one or more rays by extending rays in the sample plane from the sample depth point to the digital surface of the 3D digital model at different angles relative to the sample point depth ray. In some embodiments, the computer-implemented method can form a tilt angle within a tilt angle range. In some embodiments, the tilt angle range can be -45 degrees to +45 degrees relative to the sample point depth ray. The tilt angle range is a user-configurable value that can be set, for example, in a configuration file that the computer-implemented method can read, and / or can be input by a user using an input device, for example, through the use of a GUI. In some embodiments, the computer-implemented method can determine the number of rays from the sample depth point to generate based on the number of desired rows in the panoramic image. In some embodiments, the computer-implemented method can set the number of rays from the sample depth point to the number of desired rows in the panoramic image. In some embodiments, the computer-implemented method can determine a tilt angle increment for one or more rays from a sample depth point. In some embodiments, the tilt angle increment can be determined as (total tilt angle range) / (desired number of rows). For example, if the desired number of rows is set to 320 and the total tilt angle range is set to -45 degrees to +45 degrees, then the tilt angle increment would be (90 degrees) / (320), which would be 0.28 degrees.

[0074] In some embodiments, the computer-implemented method may form a sample point depth ray and determine one or more tilt angles on either side of the sample point depth ray as multiples of the tilt angle increment to determine one or more rays extending from the sample depth point to at least the digital surface of the 3D digital model. In this example, the computer-implemented method would determine the one or more rays as multiples of 0.28 degrees on either side of the sample point depth ray. The one or more rays may be in the sample plane, for example, in some embodiments. The computer-implemented method may determine a digital surface point associated with the sampled digital surface point as a digital surface point that intersects with the ray. The computer-implemented method may determine one or more rays and their corresponding intersecting digital surface points until an end of range is reached on either side of the sample point depth ray.

[0075] For example, as illustrated in FIG. 12(a), the computer-implemented method may determine a first positive tilt angle ray 1016 that makes a first positive tilt angle 1018 from the sample point depth ray 1012, and a first negative tilt angle ray 1020 that makes a first negative tilt angle from the sample point depth ray 1012. The first positive tilt angle 1018 and the first negative tilt angle may be tilt angle increment values, since they are both the first tilt angles at which the ray is extended. The second positive and negative tilt angle ray will make an angle of plus / minus 2*(tilt angle increment) from the sample point depth ray 1012, the third positive and negative tilt angle ray will make an angle of plus / minus 3*(tilt angle increment) from the sample point depth ray 1012, and so on. The computer-implemented method can determine a first digital surface point 1022 at the intersection of the first positive tilt angle ray 1016 with the digital surface of the digital model 1006, and can determine a second digital surface point 1024 at the intersection of the first negative tilt angle ray 1020 with the digital surface of the digital model 1006. The first digital surface point 1022 and the second digital surface point 1024 can be associated with the sampled digital surface point 1002. Although only a single sample point and two rays are shown for illustration, many more rays at different angles are possible. In some embodiments, the number of sample points on the spline can be, for example, 2100, and the number of rays can be, for example, 320. For example, FIG. 12(b) shows a digital model 1050 with additional sample points, such as sample point 1052, sampled on a spline 1054. A sample point 1052 can have one or more rays, such as ray 1056, extending from a depth point corresponding to sample point 1052. Figure 12(c) shows multiple rays, such as ray 1056, extending from a sample depth point for sample point 1052.

[0076] In some embodiments, a central row of the panoramic image may contain the sampled digital surface points. In some embodiments, each sampled digital surface point and its associated digital surface point may be a column in the panoramic image. In some embodiments, rows on one side of the central row may contain digital surface points corresponding to rays with positive tilt angles, and rows on the other side of the central row may contain digital surface points corresponding to rays with negative tilt angles. In some embodiments, the same tilt angle increment value may be used to construct rays for all sampled digital surface points. In some embodiments, the depth of the sample depth points for each sampled digital surface point is the same. In some embodiments, the central row corresponds to rays that make a tilt angle of 0 with respect to the sample point depth ray.

[0077] FIG. 13 illustrates an example of a panoramic image, shown for illustrative purposes only. A central row 1102 may include sampled digital surface points, such as, for example, sampled digital surface point 1103. The digital surface points in column 1105 are digital surface points associated with sampled digital surface point 1103. The computer-implemented method may determine the digital surface points in column 1105 from rays that extend from the sample depth point of sampled digital surface point 1103 at different multiples of tilt angle increments. In some embodiments, for example, the digital surface points in row 1104 may be those determined from rays with positive tilt angles, and the digital surface points in row 1106 may be those determined from rays with negative tilt angles. For example, a first positive row 1110 may include a sample depth ray that is 0.01 mm to 0.01 mm. Slope angle = 1 * (slope increment) For example, the first negative row 1112 may include digital surface points that intersect with a first positive ray having a depth ray of Slope=-1*(Slope increment) In some embodiments, the tilt angle increment may be specified in a configuration file or through an editable input field by a user, for example, by using a GUI. The computer-implemented method may, for example, in some embodiments, determine the number of rows from the set tilt angle increment.

[0078] The panoramic image can include depth information for each sampled digital surface point and its associated digital surface point. In some embodiments, the computer-implemented method can determine the depth of a digital surface point as the length of its corresponding ray from the digital surface point to the sample depth point. In some embodiments, the computer-implemented method can determine the depth of a sampled / associated digital surface point as the length of the sample point depth ray extending from the sampled / associated digital surface point to the sample depth point. In some embodiments, the computer-implemented method can store the depth of each pixel of the panoramic image. In some embodiments, the computer-implemented method can store the depth information as luminance. For example, in some embodiments, lighter colored pixels represent digital surface areas closer to the camera field of view, and darker colored pixels represent digital surface areas further from the camera field of view. FIG. 14 shows an example of a panoramic image with depth information. For example, darker area 1202 is farther from the camera than lighter areas such as lighter area 1204. Alternatively, in some embodiments, lighter colored pixels may represent areas of the digital surface that are further from the camera field of view, and darker colored pixels may represent areas of the digital surface that are closer to the camera.

[0079] It is noted that the designations positive and negative tilt angles are used to distinguish rays on one side of the sample point depth ray from rays on the other side, and the designations can be interchanged. For example, in some embodiments, the digital surface points in row 1104 can be those determined from rays with negative tilt angles, and the digital surface points in row 1106 can be those determined from rays with positive tilt angles.

[0080] At least one advantage of using a panoramic image can include, for example, improved visibility of the tooth-gingival boundary because the light rays converge toward the root of each tooth, instead of being parallel, as in an orthogonal view, and diverging, as in an oblique view. This can improve the accuracy of the boundaries of the teeth and other areas, such as the gingiva.

[0081] Labeling Some embodiments of the computer-implemented method may include labeling one or more regions of the input image using a trained neural network to provide a labeled image.

[0082] In some embodiments, the trained neural network can be any kind of neural network. In some embodiments, the neural network can be, for example, a CNN, as described above. In some embodiments, the neural network can be, for example, a YOLO neural network, as described above. In some embodiments, the neural network can be, for example, a semantic segmentation network. In some embodiments, the semantic segmentation network can be, for example, a U-Net neural network, as described above. In some embodiments, the semantic segmentation network can be, for example, a PAN neural network, as described above.

[0083] In some embodiments, the neural network can be trained by providing a training data set that can include one or more images of at least a portion of a digital dentition, which can include, for example, one or more digital teeth, where each digital tooth is labeled with a labeled digital tooth bounding region. In some embodiments, the training data set can include one or more panoramic images, where each digital tooth is labeled with a labeled digital tooth bounding region. A neural network trained with the panoramic training data set can become a panoramic image-trained neural network.

[0084] In some embodiments, the labeled digital tooth bounding region can be, for example, a rectangular boundary. However, other shapes can be used as the labeled digital tooth bounding region. Each digital tooth in each image in the training data set is labeled with a digital tooth bounding region. FIG. 15 shows an example of a panoramic image 1300 with labeled digital tooth bounding regions 1302. The labeled digital tooth bounding region for each digital tooth in the panoramic image can be labeled manually or by an automated process in some embodiments. In some embodiments, the training data set can include 10,000 images, such as, for example, panoramic images. For example, other suitable numbers of panoramic images can be used as the training data set in some embodiments. The training of any neural network can be supervised or unsupervised.

[0085] In some embodiments, the YOLO network can be trained by providing it with labeled digital tooth bounding regions shaped as rectangles or boxes. In some embodiments, semantic segmentation networks, such as U-Net or PAN neural networks, can be trained by providing them with labeled digital tooth bounding regions labeled to follow the shape of each digital tooth in the panoramic image.

[0086] In some embodiments, the trained neural network can be, for example, a panoramic image trained CNN, as described above. In some embodiments, the trained neural network can be, for example, a panoramic image trained YOLO neural network, as described above. In some embodiments, the trained neural network can be, for example, a panoramic image trained semantic segmentation network. In some embodiments, the trained semantic segmentation network can be, for example, a panoramic image trained U-Net neural network, as described above. In some embodiments, the trained semantic segmentation can be, for example, a panoramic image trained PAN neural network, as described above. Alternatively, in some embodiments, the trained neural network can be a 2D depth map trained neural network.

[0087] Although a panoramic image is described as an example, in some embodiments, the training data set may alternatively include one or more 2D depth maps in which each digital tooth is labeled with a labeled digital tooth bounding region. A neural network trained with a 2D depth map training data set may be a 2D depth map trained neural network, e.g., trained in the same manner as described with respect to the panoramic image, except that the training data set includes one or more 2D depth maps.

[0088] In some embodiments, the computer-implemented method can use a trained neural network to label one or more regions of an input image to provide a labeled image. For example, in some embodiments, the computer-implemented method can receive an input image and use one or more trained neural networks to label one or more regions of the input image to provide a labeled image. In some embodiments, the trained neural network can be a panoramic image trained neural network. In some embodiments, the computer-implemented method can receive a panoramic image and use one or more panoramic trained neural networks to label one or more regions of the panoramic image to provide a labeled panoramic image.

[0089] Alternatively, in some embodiments, the computer-implemented method can receive a 2D depth map and use one or more 2D depth map trained neural networks to label one or more regions of the 2D depth map and provide a labeled 2D depth map.

[0090] In some embodiments, the trained neural network can receive one or more images and generate a digital tooth bounding region for each digital tooth in the image. The computer-implemented method can, for example, in some embodiments, label all pixels bounded by the digital tooth bounding region with a unique label. For example, in some embodiments, the panoramic image trained neural network can receive one or more panoramic images and automatically generate a digital tooth bounding region for each digital tooth in the panoramic image.

[0091] For example, in some embodiments, the panoramic image trained YOLO network can receive a panoramic image and provide a digital tooth bounding region for each digital tooth in at least a portion of the panoramic image. The computer-implemented method can, for example, in some embodiments, label all pixels bounded by the digital tooth bounding region with a unique label. The digital tooth bounding region provided by the panoramic image trained YOLO network can, for example, be a digital tooth bounding box. For example, the panoramic image trained YOLO network can receive the panoramic image illustrated in FIG. 14 and output a labeled panoramic image as illustrated in FIG. 16(a), which can include one or more digital bounding regions, such as digital bounding region 1402. The computer-implemented method can, for example, in some embodiments, label all pixels bounded by the digital tooth bounding region with a unique label to provide a pixel-by-pixel labeling. This can also be referred to as neural network segmentation, or rough segmentation.

[0092] In some embodiments, a panoramic image trained semantic segmentation network, such as a U-Net neural network or a PAN neural network, can receive the panoramic image and provide a digital tooth bounding region for each digital tooth in at least a portion of the panoramic image. For example, a panoramic image trained semantic segmentation network, such as a U-Net or a PAN, can receive the panoramic image illustrated in FIG. 14 and output a labeled panoramic image as illustrated in FIG. 16(b), such as digital tooth bounding region 1404. The computer-implemented method can, for example, in some embodiments, label all pixels bounded by the digital tooth bounding region with a unique label to provide a pixel-by-pixel labeling. This can also be referred to as neural network segmentation or rough segmentation.

[0093] In some embodiments, a 2D depth map trained neural network (e.g., CNN, YOLO, semantic segmentation, or any other type of neural network) can receive the 2D depth map image and provide digital tooth bounding regions for each digital tooth within at least a portion of the panoramic image as described with respect to the panoramic image trained neural network. The computer implemented method can, for example, in some embodiments, label all pixels bounded by the digital tooth bounding regions with a unique label.

[0094] In some embodiments, the 2D label map with the labels can be of the same dimensions as the 2D depth map.

[0095] In some embodiments, the computer-implemented method can include mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface polygon labels in the 3D digital model to provide a labeled 3D digital model. In some embodiments, the computer-implemented method can pass a virtual ray through each pixel of the 2D label map and label the digital surface polygons that the ray passed through with the same label as the pixel label. In some embodiments, the computer-implemented method can pass a virtual ray through every digital surface polygon and label each digital surface polygon based on the label of the pixel it passes through. In some embodiments, the computer-implemented method can project all labeled regions from the 2D image onto the 3D digital model mesh, including any uncertain regions. In some embodiments, the computer-implemented method can label one or more unlabeled digital surface polygons with the same label as the nearest digital surface polygon. In some embodiments, the digital surface polygons can be, for example, digital surface triangles.

[0096] In some embodiments, this mapping can include, for example, projecting the labeled panoramic image digital tooth boundary regions onto the 3D digital model. In some embodiments, the computer-implemented method can project the digital tooth bounding boxes as a prism. Since the panoramic image was generated from the 3D digital model using rays, the computer-implemented method can map the digital tooth bounding boxes back onto the 3D digital model using the same rays. In some embodiments, the computer-implemented method can determine four corners of each digital tooth bounding box. For each corner, the computer-implemented method can, in some embodiments, determine rays that pass through the corner. In some embodiments, the computer-implemented method can, in some embodiments, connect each set of rays in a plane to their corresponding depth points. In some embodiments, the computer-implemented method can label every digital surface that is within the constructed prism, for example, coarsely labeled as belonging to a single tooth. In some embodiments, the computer-implemented method can label every digital surface polygon (which in some embodiments can be digital surface triangles) that is within the constructed prism with the same label. In some embodiments, the computer-implemented method can label each constructed prism with a different label. In some embodiments, the computer-implemented method can individually label the digital surfaces that intersect with every ray within the box.

[0097] FIG. 17 shows an example of some embodiments of mapping. As illustrated in the figure, mapping can include, for example, projecting a labeled panoramic image digital tooth bounding region, such as, for example, digital tooth bounding box 1502, onto a 3D digital model, such as, for example, 3D digital model 1500. The computer-implemented method can determine a first digital tooth bounding box corner 1504 of the digital tooth bounding box 1502. The computer-implemented method can determine a first ray 1506 as a ray extending to and / or through the first digital tooth bounding box corner 1504. Similarly, the computer-implemented method can determine a second digital tooth bounding box corner 1508. The computer-implemented method can determine a second ray 1510 as a ray extending to and / or through the second digital tooth bounding box corner 1508. By construction, the first ray 1506 and the second ray 1510 are in the same plane, as described in the section of this disclosure regarding the generation of panoramic images. In the panoramic image, for example, the first ray 1506 and the second ray 1510 are in the same column. Thus, the first ray 1506 and the second ray 1510 meet at a depth point 1512. The computer-implemented method can similarly determine a third digital tooth bounding box corner 1513 with a corresponding third ray 1516, and a fourth digital tooth bounding box corner 1518 with a corresponding fourth ray 1520. The third ray 1516 and the fourth ray 1520 are in the same plane by construction, and thus meet at their own corresponding depth points. In some embodiments, the computer-implemented method can label every digital surface point that is in a prism, for example, with the same label. In some embodiments, the computer-implemented method can individually label every ray within the digital tooth bounding box, hi some embodiments, the computer-implemented method can label the rays that pass through the interior portion of the box.

[0098] In some embodiments, the computer-implemented method can determine one or more uncertainty regions. An uncertainty region can occur when the neural network does not provide enough information to label one or more regions, or when the neural network generates unreliable information. This can occur, for example, at the edges of the digital tooth bounding region. In some embodiments, the computer-implemented method can determine, for example, all digital surface points that are within a user-configurable uncertainty region buffer distance from any boundary of surface points in the digital tooth bounding region as part of one or more uncertainty regions for a panoramic image or a 2D depth map. In the case of a YOLO network, for example, the digital tooth bounding region can be a digital tooth bounding box. The distance is along the digital surface in some embodiments. In some embodiments, the uncertainty region buffer distance can be, for example, 6 mm. Other suitable distances may be used, and the uncertainty region buffer distance may be a user-configurable value that may, for example, be stored and loaded from a configuration file, or that may, for example, in some embodiments, be entered into a Graphical User Interface ("GUI") by a user using an input device.

[0099] Uncertain regions may arise, for example, for all digital surface polygons (e.g., digital surface triangles) that are not visible on the image provided to the neural network. As described above, the computer-implemented method may account for the uncertain regions in some embodiments. For example, in some embodiments, the computer-implemented method may leave the uncertain regions unlabeled or label them as uncertain regions. FIG. 18(a) illustrates an example of how the computer-implemented method identifies uncertain regions. In the figure, the trained neural network may determine digital tooth boundary regions, such as digital tooth boundary region 1602. The computer-implemented method may label certain regions within digital tooth boundary region 1602, such as certain region 1604, where the polygons are all visible. The computer-implemented method may determine uncertain regions, such as uncertain region 1606 based on distance 1605 and uncertain region 1607 based on distance 1603, and designate the regions as uncertain, for example, by leaving the uncertain region unlabeled or labeling it as uncertain. 16(b) illustrates mapping the 2D image to a labeled 3D digital model. In the labeled 3D digital model, the computer-implemented method can determine a first labeled digital surface region 1614, an unlabeled (or labeled as uncertain) region 1616, and a second labeled digital surface region 1618. There can also be additional labeled regions, such as a third labeled digital surface region 1620. The uncertain regions can be converted to labeled regions during segmentation in some embodiments. For example, the computer-implemented method can label the uncertain regions during geometric segmentation in some embodiments.

[0100] In some embodiments, the computer-implemented method can segment the labeled 3D digital model to provide a segmented 3D digital model. In some embodiments, the computer-implemented method can segment the 3D digital model after the neural network generates the labeled panoramic image / 2D label map from the panoramic image / 2D depth map. In some embodiments, the computer-implemented method can generate the segmented 3D digital model from the labeled 3D digital model using any type of segmentation technique known in the art. In some embodiments, the computer-implemented method can perform, for example, curvature-based segmentation.

[0101] In some embodiments, the computer-implemented method can apply a morphological shrinkage operation to one or more labeled regions. The one or more labeled regions can include, in some embodiments, a labeled gingival region. In some embodiments, the computer-implemented method can perform the morphological shrinkage operation on either the 2D label map or the labeled 3D digital model. In some embodiments, the computer-implemented method can perform the morphological shrinkage operation on the 2D label map as follows: for a pixel of the 2D label map, evaluate one or more nearby pixels within a user-configurable radius (e.g., 5 pixels), and if the one or more nearby pixels do not have the same label, then the computer-implemented method can designate the particular pixel and its surrounding pixels as unlabeled. In some embodiments, the computer-implemented method can repeat this process for one or more pixels of the 2D label map. In some embodiments, the computer-implemented method may perform a morphological shrinkage operation on the labeled 3D digital model as follows: for a particular polygon (e.g., digital triangle, etc.) of the labeled 3D digital model, evaluate one or more nearby polygons (e.g., digital triangle, etc.) within a user-configurable radius (e.g., 5 mm) of the particular polygon (e.g., digital triangle, etc.), and if the one or more nearby polygons do not have the same label, then the computer-implemented method may designate the particular polygon and its surrounding polygons as unlabeled. In some embodiments, the computer-implemented method may repeat this process for one or more polygons of the labeled 3D digital model. In some embodiments, the computer-implemented method may apply the morphological shrinkage operation to any labeled regions, including gingival regions.

[0102] In some embodiments, one or more unlabeled digital surface polygons may remain. One or more unlabeled digital surface polygons may result when the total number of digital surface polygons is greater than the number of pixels. One or more unlabeled digital surface polygons may result from one or more occluded digital surface polygons for which a label could not be determined. One or more unlabeled digital surface polygons may result from morphological erosion.

[0103] Curvature-Based Segmentation In some embodiments, the curvature-based segmentation may include determining a curvature of a digital surface region in the digital model. In some embodiments, the computer-implemented method may receive the digital model and determine a curvature of the digital surface region. The computer-implemented method may use any technique to determine the curvature of the digital surface region. In some embodiments, the curvature determination may be performed automatically by the computer-implemented method.

[0104] In some embodiments, the digital surface region comprises a triangle, the curvature of which may be determined by taking the average of the curvatures of the triangle's sides or the average of the curvatures of the triangle's vertices.

[0105] In some embodiments, the computer-implemented method may determine the curvature of a triangle by taking the average of the curvatures of its edges. FIG. 19 shows an example of determining the curvature at an edge 1786 connecting two triangles 1788 and 1790. In some embodiments, the computer-implemented method may determine the curvature at an edge 1786 based on a dihedral angle 1792 formed at the edge 1786 between a particular triangle 1790 and its adjacent neighboring triangle 1788 in the digital surface mesh, as shown. The dihedral angle 1792 may be determined by the computer-implemented method as the angle formed between the two adjacent triangles 1788 and 1790 in a third plane 1794 that is perpendicular to the edge 1786 formed by the two adjacent triangles 1790 and 1788. For example, in some embodiments, the computer-implemented method may take sin(φ), where φ is the dihedral angle 1792 between the two adjacent triangles 1790 and 1788. The computer-implemented method may repeat this curvature function at all of the triangle edges.

[0106] Alternatively, in some embodiments, the computer-implemented method can determine the curvature of a triangle by taking the average of the curvatures of the triangle's vertices. For example, in some embodiments, the computer-implemented method can determine the curvature at each vertex P by selecting a neighborhood (size N) of vertices around P, optionally using connectivity information to reduce the search space. The computer-implemented method can fit a quadric surface patch F(x,y,z)=0 onto the neighborhood of the point. The computer-implemented method can determine a projection P0 of P onto the patch, such that F(P0)=0. The computer-implemented method can determine the curvature properties of F at P0 and assign the curvature properties to P.

[0107] In some embodiments, since each datum (x,y,z) will not lie perfectly on the surface of F, the computer-implemented method may, for example, use a quadratic form a 2 +by 2 +cz 2+2exy+2fyz+2gzx+2lx+2my+2nz+d=0. The computer-implemented method can use the form A=D T From a 10x10 real symmetric eigenvalue problem for D, we can determine the coefficients of the patch surface (a,b,c,e,f,g,l,m,n,d), where D i is an N x 10 design matrix, each row of which is

[0108]

number

[0109]

number

[0110]

number

[0111]

number

[0112]

number

[0113]

number

[0114] The computer-implemented method then derives matrices A and B from E, F, G, L, M, and N as:

[0115]

number

[0116]

number

[0117] The computer-implemented method can apply a selected scalar function to the principal curvatures k1 and k2 to determine a selected curvature function ("SCF"). For example, for the principal curvatures k1 and k2, the computer-implemented method can determine the Gaussian curvature (K) as K=k1k2, or the mean curvature (H) as H=1 / 2(k1+k2).

[0118] The radius for either method of determining curvature can be up to, including, on average, 60 digital vertices in the vicinity of the vertex being evaluated, and can be a user-selectable value. Selecting a smaller number of points and a smaller radius can result in faster calculations, while selecting a larger number of points and a larger radius can give a more precise curvature estimate. The computer-implemented method can be repeated, for example, for all vertices of the digital surface mesh.

[0119] In some embodiments, the computer-implemented method can segment the entire digital dental impression surface into one or more digital segments. In some embodiments, the computer-implemented method can segment the digital dental impression surface in three dimensions (3D) using curvature-based segmentation. This can include, for example, watershed segmentation. Segmentation can be performed automatically by the computer-implemented method in some embodiments.

[0120] In some embodiments, the digital dental impression surface may include one or more triangles that connect at their edges and vertices to form a digital surface mesh. In some embodiments, the computer-implemented method determines the curvature of every triangle in the digital surface mesh. The computer-implemented method may determine the curvature of each particular triangle by determining either the average curvature of the vertices of the particular triangle or the average curvature of the edges of the particular triangle, as described above.

[0121] In one embodiment, the computer-implemented method can determine the curvature of a particular triangle by determining the curvature at each of the particular triangle's sides and calculating the average of the side curvatures, as described above in this disclosure. FIG. 20(a) illustrates an example of some embodiments of determining the average of the side curvatures, where a particular triangle 2402 includes a first side 2405, a second side 2407, and a third side at 2409. The computer-implemented method can determine the curvature at the first side 2405 based on a dihedral angle between the particular triangle 2402 and an adjacent triangle 2408. The computer-implemented method can determine the curvature at the second side 2407 based on a dihedral angle between the particular triangle 2402 and an adjacent triangle 2406 as described above in this disclosure. The computer-implemented method can determine the curvature at the third side 2409 based on a dihedral angle between the particular triangle 2402 and an adjacent triangle 2404. The computer-implemented method may then determine the average of the curvatures of the first edge 2405, the second edge 2407, and the third edge at 2409 to determine the curvature of the particular triangle 2402. In some embodiments, the computer-implemented method may, for example, store the curvature of the particular triangle 2402 in a look-up table. The computer-implemented method may repeat this process with every triangle in the digital surface mesh to determine the curvature at each triangle in the digital surface mesh.

[0122] In some embodiments, the computer-implemented method can assign a user-selectable positive or negative sign to the curvature of each triangle. For example, if the curvature is set to the most convex side, then any concave regions are assigned a negative sign and any convex regions are assigned a positive sign. If the curvature is set to the most concave side, then any convex regions are assigned a negative sign and any concave regions are assigned a positive sign. Concave / convex can be defined relative to the normals of the digital surface. For surface normals pointing outwardly of the digital surface, the computer-implemented method can, for example, assign positive values ​​to convex sides and negative values ​​to concave sides. For normals pointing inwardly of the digital surface, the computer-implemented method can, for example, assign positive values ​​to convex sides and negative values ​​to concave sides. In some embodiments, the segment boundary corresponds to the maximum curvature along the digital surface.

[0123] After determining the curvature of each particular triangle, the computer-implemented method may segment the triangles based on a 3D curvature-based segmentation. In some embodiments, watershed segmentation is used. For example, in some embodiments, the computer-implemented method determines the curvature for each triangle. The curvature of each triangle may be stored in a lookup table in some embodiments. The computer-implemented method may start with the triangle having the smallest curvature as the particular triangle to be evaluated. The computer-implemented method may search for the curvature of triangles in the vicinity of the particular triangle to be evaluated, for example, from a lookup table. In some embodiments, the computer-implemented method may determine the curvature of the neighboring triangles from the lookup table. Any neighboring triangles having a curvature greater than the particular triangle to be evaluated may be added to the segment to which the particular triangle to be evaluated belongs. Any neighboring triangles having a curvature less than the curvature of the particular triangle are not added to the segment of the particular triangle. The computer-implemented method then selects the neighboring triangle as the next particular triangle to be evaluated, and repeats the process for every triangle. In some embodiments, if a segment contains a sufficiently large number (e.g., a majority or any other user-configurable value) of triangles that can be attributed to a label, then the computer-implemented method can assign the segment to that label.

[0124] FIG. 20(a) shows an example of some embodiments of triangle watershed segmentation. As described herein, the computer-implemented method determines the curvature of all of the triangles in the digital surface mesh. In one embodiment, the computer-implemented method stores the triangle curvatures in a lookup table. The computer-implemented method identifies the triangle with the smallest curvature, e.g., the particular triangle 2402. In some embodiments, the computer-implemented method can use the lookup table to determine the triangle with the smallest curvature. The computer-implemented method determines the curvatures of neighboring triangles 2404, 2408, and 2406. In some embodiments, the computer-implemented method can determine the curvatures of the neighboring triangles from the lookup table. In this example, if the neighboring triangle 2406 has a greater curvature than the curvature of triangle 2402, then the neighboring triangle 2406 can be considered part of the same watershed as the particular triangle 2402. The computer-implemented method combines digital surface triangle 2402 with triangle 2406 into a single segment, such as segment 2411, as illustrated in FIG. 20(a).

[0125] The computer-implemented method may then, for example, compare the curvature of nearby triangle 2404 with the curvature of particular triangle 2402. For example, if the curvature of nearby triangle 2408 is greater than the minimum curvature (i.e., the curvature of 2402), then triangle 2408 is merged with segment 2411 that contains triangle 2402. As illustrated in FIG. 20(b), segment 2412 is formed after merging triangle 2408.

[0126] If a nearby triangle has a lower curvature than the particular triangle 2402 in question, then the nearby triangle is not merged by the computer-implemented method with the segment that contains the particular triangle 2402. For example, if nearby triangle 2404 has a lower curvature than triangle 2402, then 2404 is not merged with segment 2412 to which the particular triangle 2402 belongs.

[0127] After processing the first particular triangle, the computer-implemented method moves on to a new particular triangle, which may be a neighboring triangle of the first particular triangle. The computer-implemented method may repeat determining the segmentation with the new particular triangle being evaluated to segment the entire digital surface. Figure 20(c) illustrates an example of a segmented digital surface mesh 2414, including, for example, segment 2416.

[0128] After performing the triangular segmentation, the digital surface mesh may contain a large number of segments, as illustrated in FIG. 21. In some embodiments, the number of segments may be optionally reduced by the computer-implemented method by merging two or more segments together. In some embodiments, the computer-implemented method may merge smaller segments with larger ones based on geometric attributes such as their average curvature, average size, area, perimeter, perimeter-to-area ratio, and / or other geometric factors. In some embodiments, the computer-implemented method does not merge two segments that have been assigned to separate labels, except for the uncertain region. In some embodiments, the computer-implemented method may attribute the merged segment to the same label as one of its parts after each merger. In some embodiments, the computer-implemented method may label the merged segment as an uncertain region only if both segments before the merger were uncertain regions. Merging may be automatically performed by the computer-implemented method in some embodiments.

[0129] In some embodiments, the computer-implemented method determines a merge priority for any two neighboring segments. The computer-implemented method may determine the merge priority of two neighboring segments based on their attributes. If two segments can be merged based on their attributes, then in some embodiments, the computer-implemented method determines the priority based on geometric factors. For example, the computer-implemented method may determine the priority based on 1) the average curvature of each segment's interior and their common boundary (segments with smaller curvature difference between their boundary and the segment's interior will merge sooner), and 2) the ratio of the length of the common boundary to the minimum perimeter of the two segments (segments with a larger ratio will merge sooner).

[0130] In some embodiments, the computer-implemented method can store the priorities in a priority queue. The computer-implemented method can extract the highest priority from the queue, merge the two corresponding segments, and update the priorities between the newly formed segment and its neighbors in the queue. The computer-implemented method can repeat this process until no more two segments can be merged.

[0131] In some embodiments, for example, smaller segments may be merged until no segments are attributed to the uncertainty region.

[0132] In some embodiments, the computer-implemented method may provide a segmented 3D digital model from the labelled 3D digital model after performing curvature-based segmentation on the labelled 3D digital model.

[0133] Two-point segmentation In some embodiments, the computer-implemented method can perform two-point segmentation. For example, in some embodiments, the computer-implemented method can receive a labeled 3D digital model, perform two-point segmentation, and provide a segmented 3D digital model. In some embodiments, the labeled 3D digital model is a result of labeling performed by one or more neural networks as described above. The labels on the labeled 3D digital model can define local tooth regions. These can help provide a rough boundary of each digital tooth in the labeled 3D digital model. One or more examples of two-point segmentation in some embodiments can be found in SEMI-AUTOMATIC TOOTH SEGMENTATION, Nikolskiy et al., U.S. Patent Application Publication No. 16 / 778,406, the entirety of which is incorporated herein by reference.

[0134] In some embodiments, the computer-implemented method can receive a labeled 3D digital model of the patient's dentition, determine a center point of one or more labeled regions, construct a best-fit parabola to connect the center points of the one or more labeled regions, construct a plane at each center point such that the plane encompasses the center point and is orthogonal to the best-fit parabola at the center point, and for each plane, determine a lingual point and a buccal point where the plane intersects the segment boundary, and construct a first and second path on either side of the plane to connect the lingual point and the buccal point. In some embodiments, instead of constructing a plane at each center point, the computer-implemented method can utilize or reuse a sample plane corresponding to each center point (e.g., such as sample plane 1008 from FIG. 12(a)).

[0135] 20(a)-20(g) show an example of two-point segmentation. As illustrated in FIG. 22(a), the computer-implemented method can receive a labeled 3D digital model 2000, for example, encompassing one or more labeled regions 2002. The computer-implemented method can determine a center point, such as center point 2004, for each labeled region, such as labeled region 2002, as viewed from an occlusal direction 2012. This can provide the center points as illustrated in FIG. 22(b). The computer-implemented method can connect the one or more center points by a best-fit smooth curve, such as a best-fit analytical curve, such as a parabola. As illustrated in FIG. 22(c), the computer-implemented method can construct a best-fit parabola, such as parabola 2010, that connects the center points of the one or more labeled regions as viewed from an occlusal direction 2012. To determine the best-fit parabola, the computer-implemented method determines a least-squares plane. For example, in the case of a center point, the computer-implemented method projects the center point onto the plane. For example, as illustrated in FIG. 22(d), a center point 8502 (shown as a black dot in the figure) is arranged in a least-squares plane. The computer-implemented method generates a first x-axis 8504 in a first direction in the plane and determines a first y-axis 8506 that is 90 degrees relative to the x-axis in the plane. The computer-implemented method uses quadratic least-squares regression as known in the art to fit the equation y=ax 28. Determine the coefficients a, b, and c in +bx+c. For example, a parabola 8508 may be determined by the computer-implemented method after determining the coefficients a, b, and c. The computer-implemented method then determines, for example, the offset of the parabola 8508 from the center point 8502. The computer-implemented method repeats the steps for a user-selectable number of x-axis directions. For example, the computer-implemented method may rotate the x-axis 8504 by rotating the x-axis to a new x-axis 8510 with a corresponding y-axis 8512 to determine the parabola 8514. In some embodiments, the number of x-axis directions may be user-selectable and / or a pre-defined value. In some embodiments, the number of x-axis directions may be, for example, 100. The computer-implemented method may select a value such that a is 150 meters to avoid very sharp parabolas. -1 A parabola with a minimum deviation that is less than or equal to 20 mm may be selected. In some embodiments, the computer-implemented method optionally excludes center points that are located farther than a user-selectable and / or predefined maximum center point distance, which may be any value. In some embodiments, the maximum center point distance may be, for example, 5 mm. As illustrated in FIG. 22(c), the computer-implemented method may connect the center points with a best fit parabola 2010.

[0136] 22(e), the computer-implemented method may construct a plane, such as separation region 2020, at each center point, such as center point 2004, such that separation region 2020 encompasses center point 2004 and is orthogonal to best fit parabola 2010 at center point 2004. In some embodiments, instead of constructing a plane, the computer-implemented method may utilize or reuse a sample plane that corresponds to each center point, such as, for example, sample plane 1008 from FIG.

[0137] As illustrated in Fig. 22(f), for example, the computer-implemented method can determine, for each plane, such as plane 2030, a lingual point 2032 and a buccal point 2034 where the plane 2030 intersects with a first segment path 2036 and a second segment path 2072. Fig. 22(g) shows a digital tooth 2050 having a center point 2051 through which a parabola 2059 passes. The computer-implemented method constructs a first path 2053 on a first side of the separation region 2057 and a second path 2056 on a second side of the separation region 2057 to connect the lingual point 2052 and the buccal point 2062.

[0138] In some embodiments, the first segment path 2053 and the second segment path 2056 can be determined using, for example, Dijkstra's shortest path algorithm as known in the art, using either the facial point 2062 or the lingual point 2052 as the initial digital surface point. In some embodiments, the computer-implemented method can determine the first segment path 2053 using a shortest path algorithm, for example, by evaluating only edges between digital surface points of the first side 2054 of the separation region 2057. In some embodiments, the computer-implemented method can determine the second segment path 2056 using a shortest path algorithm, for example, by evaluating only edges between digital surface points of the second side 2058 as the digital surface points. In some embodiments, the computer-implemented method can perform, for example, the following:

[0139] 1. Set an initial digital surface point. In some embodiments, the initial digital surface point can be a facial point 2062. Alternatively, in some embodiments, the initial digital surface point can be a lingual point 2052.

[0140] 2. Mark all digital surface points on one side of the separation region 2057 as unvisited. Generate an unvisited set of all unvisited digital surface points. In some embodiments, the one side can be the first side 2054. Alternatively, in some embodiments, the one side can be the second side 2058.

[0141] 3. Assign temporary distance values ​​to every digital surface point on one side. The temporary distance value for the initial digital surface point is assigned to 0, and the temporary distance values ​​for all other digital surface points on one side are assigned to, for example, infinity, or the highest possible value or a value greater than the sum of the lengths of all the sides. Set the initial digital surface point as the current digital surface point.

[0142] 4. For the current digital surface point, consider all unvisited neighboring digital surface points on one side and determine their calculated tentative distance (e.g., edge length between the current digital surface point and the particular unvisited neighboring digital surface point) through the current digital surface point. In some embodiments, the calculated tentative distance may determine an edge length between the current digital surface point and the particular unvisited neighboring digital surface point based on the curvature of the digital surface. In some embodiments, the computer-implemented method may include, for example, an angle-dependent factor in determining the edge length. For example, in some embodiments, the computer-implemented method may determine the tentative distance as: f(e i )=|e i |exp(k sin α i ), or f(e i )=|e i |exp(k α i ) Here, f(e i ) is the edge e for the algorithm between the current digital surface point and a particular unvisited neighboring digital surface point. i is the length of |e i| is the 3D Euclidean distance between the two ends of the edge, and k is a constant, e.g., 1.8 in some embodiments. A larger magnitude of the constant k may result in choosing paths that pass through regions of higher curvature. The sign of the constant may dictate whether a concave or convex edge will be chosen by the algorithm, and α i is side e i is the dihedral angle of the triangular surface at sin α i For example, in some embodiments, α i The sine of that angle, which may be computationally faster to find than α itself. i and / or sin α i can be, for example, an angle-dependent factor. The boundary between the tooth and the gum can have a much higher curvature than the rest of the surface. i )=|e i |exp(k sin α i ), the length of the edges in the digital tooth-gumline boundary region is significantly smaller than in other regions. Thus, the computer-implemented method can determine the shortest path along the digital tooth-gumline junction.

[0143] 5. Compare the newly calculated tentative distance with the current assigned value and assign the smaller one. For example, if the current digital surface point A is labeled with a distance of 6, and the edge connecting it to a neighboring digital surface point B has length 2, then the distance through A to B is 6+2=8. If B was previously labeled with a distance greater than 8, then change it to 8. Otherwise, keep the current value.

[0144] 6. After considering all of the current digital surface point's unvisited neighbors, mark the current digital surface point as visited and remove it from the unvisited set. Visited digital surface points are never checked again.

[0145] 7. If the destination digital surface point is marked as visited (when planning a route between two specific digital surface points), or if the smallest tentative distance between digital surface points in the unvisited set is, for example, infinity, or the highest possible value or a value greater than the sum of all edge lengths (when planning a complete traverse; occurs when there is no connection between the initial digital surface point and the remaining unvisited digital surface points), then stop. The algorithm is finished.

[0146] Otherwise, select the unvisited digital surface point labeled with the smallest tentative distance, set it as the new “current digital surface point”, and go back to step 4.

[0147] The algorithm may stop once the destination digital surface point has the smallest tentative distance of all "unvisited" digital surface points (and therefore could be selected as the next "current" point).

[0148] The computer-implemented method can repeat the steps on the other side of the separation region 2057 (which can be performed simultaneously or sequentially). In this manner, the computer-implemented method can determine a first segment path 2053 along the first side 2054 and a second segment path 2056 along the second side 2058. The first segment path 2053 and the second segment path 2056 can be, for example, sides of a digital surface triangle in some embodiments.

[0149] In some embodiments, the first segment path 2053 connects to one side of the facial triangle side 2066 and one side of the lingual triangle side 2064. In some embodiments, the second segment path 406 connects to the other side of the facial triangle side 2066 and the other side of the lingual triangle side 2064. The connected first segment path 2053, second segment path 2056, facial triangle side 2066, and lingual triangle side 2064 may together form a boundary loop in some embodiments. The boundary loop may be closed, e.g., have an enclosed internal loop region 2068.

[0150] In some embodiments, the computer-implemented method may further include providing a segmented digital tooth from the first segment path and the second segment path. As illustrated in Fig. 22(f), the computer-implemented method may include providing a segmented digital tooth 2070 from the first segment path 2036 and the second segment path 2072. This may include, for example, providing a digital surface of the digital tooth 2070 by, for example, determining digital surface triangles within a boundary loop 2074 formed by the first segment path 2036 and the second segment path 2072.

[0151] 23(A) shows an example showing a portion of a boundary loop 502 with a digital dental region 504 within the portion of the boundary loop 502 and a digital non-dental region 506 outside the portion of the boundary loop 502. The digital dental region 504 and the digital non-dental region 506 can include digital surface triangles, such as, for example, digital surface triangle 508 and digital surface triangle 514, respectively.

[0152] In some embodiments, the computer-implemented method for determining the digital surface of the digital tooth can include generating a dual graph of the original digital surface triangles. The computer-implemented method can generate the dual graph such that the vertices of the dual graph correspond to the digital surface triangles of the original digital surface and the edges of the dual graph correspond to the edges of the shared digital surface triangles except where the edges of the shared digital surface triangles are part of a bounding loop. For example, as illustrated in FIG. 23(B), the computer-implemented method can generate a dual graph vertex 512 for a digital surface triangle 514 in the portion of the digital model 500. The computer-implemented method can, for example, generate a dual graph edge 516 for the shared digital surface triangle edge 518. The computer-implemented method does not generate a dual graph edge for the shared digital surface triangle edge 520 because edge 520 is part of the bounding loop 502. The computer-implemented method can repeat this process for every digital surface triangle in the digital model. Thus, in some embodiments, the computer-implemented method can generate a dual graph for the digital surface of the digital model. In some embodiments, each dual graph edge may cross any original surface edge, but no dual graph edge crosses a bounded loop. In some embodiments, the dual graph may include, for example, an inner dual graph region and an outer dual graph region separated by a bounded loop. The inner dual graph region may be an interconnected dual graph region enclosed by a bounded loop, and the outer dual graph region may be an interconnected dual graph region outside of the bounded loop. The bounded loop may, in some embodiments, separate the inner dual graph region from the outer dual graph region. For example, as illustrated in FIG. 23(B), the dual graph may include an inner dual graph region 522 and an outer dual graph region 524 separated by a bounded loop 502.

[0153] In some embodiments, determining the digital tooth surface may include performing a search on the dual graph. The computer-implemented method may, for example, determine the digital tooth surface based on the interconnections between the vertices of the dual graph. In some embodiments, the search may, for example, be a conventionally known breadth-first search. In a breadth-first search, the computer-implemented method may, for example, load the dual graph and the root of the starting vertex. In some embodiments, the root vertex of the starting dual graph may be any dual graph vertex. The computer-implemented method may explore all neighboring dual graph vertices at the current depth and label them as discovered before moving to the next depth level. The computer-implemented method may then proceed to the next depth level and repeat the process, for example, until there are no more undiscovered dual graph vertices. The computer-implemented method may thus determine the interconnected vertices of the dual graph and determine the discovered dual graph. The breadth-first search technique may, for example, be applied to the digital tooth without having one or more tunnels. In some embodiments, the computer-implemented method may determine a conventionally known breadth-first search as follows: The procedure Breadth_First_Search(Dual_Graph, start_vert) is as follows. Let Q be a queue Label start_vert as discovered Q.enqueue(start_vert) While Q is non-empty, do the following v:=Q.dequeue() For every edge in Dual_Graph.adjacentEdges(v) from v to y, do If y has not been labeled as discovered, then Label y as discovered y.parent:=v Q.enqueue(y)

[0154] In some embodiments, the search can be, for example, a conventionally known depth-first search. In a depth-first search, the computer-implemented method can, for example, load the dual graph and the root of the starting vertex. In some embodiments, the root of the starting vertex can be any dual graph vertex. In some embodiments, the computer-implemented method can perform a conventionally known depth-first search on the dual graph as follows: the input can, for example, be the graph Dual_Graph, G, and the vertices vert of G, and the output can, for example, be any vertices reachable from the verts labeled as found. The depth-first search technique can, for example, be applied to the digital tooth without having one or more tunnels. In some embodiments, the computer-implemented method can determine a conventionally known depth-first search as follows: The procedure Depth_First_Search(Dual_Graph, vert) is as follows. Label a vert as discovered For every directed edge from a vertex in Dual_Graph.adjacentEdges(vert) to y, do the following: If vert y has not been labeled as discovered, then Call Depth_First_Search(Dual_Graph,y) recursively

[0155] In some embodiments, the computer-implemented method may, for example, perform a depth-first search on the dual graph as follows: The procedure Depth_First_Search_Iterative(Dual_Graph, vert) is as follows. Let S be a stack S.push(vert) While S is non-empty, do the following vert = S.pop() If the vert has not been labeled as discovered, then Label a vert as discovered For every edge from vert to y in Dual_graph.adjacentEdges(vert), do the following: S.push(y)

[0156] In some embodiments, for example, if the discovered dual graph is within the bounded loop and / or the root vertex of the starting dual graph was within the bounded loop, then the computer-implemented method will determine that the discovered dual graph is in the domain of the inner dual graph and therefore belongs to the digital tooth. The computer-implemented method can determine all other dual graph vertices to belong to the outer dual graph and therefore to the digital gingiva and other surrounding digital teeth. In some embodiments, for example, if the discovered dual graph is outside the bounded loop and / or the root vertex of the starting dual graph was outside the bounded loop, then the computer-implemented method can determine that the discovered dual graph is in the domain of the outer dual graph and therefore belongs to the digital gingiva and other surrounding digital teeth. The computer-implemented method can determine that all other dual graph vertices to belong to the domain of the inner dual graph and therefore to the digital tooth. Based on the vertices of the discovered dual graph, the computer-implemented method can, for example, in some embodiments, determine the digital surfaces of the digital teeth and the digital surfaces that belong to the surrounding teeth and gum regions. The final labels can, in some embodiments, include the tooth number, e.g., tooth 1, tooth 2, tooth 3, etc., and the gums. In some embodiments, each tooth can be differentiated from the other teeth and gum regions. Figure 24 shows an example of a fully segmented and labeled set of digital teeth and gum regions, such as digital teeth 2202 and gum region 2204 in the digital model 2200.

[0157] In some embodiments, the received labeled 3D digital model may optionally be a segmented 3D digital model. In some embodiments, the segmented 3D digital model may have been segmented using curvature-based segmentation. In some embodiments, the segmented digital model may be segmented using any technique known in the art.

[0158] In some embodiments, the curvature-based segmentation model may be made more precise, for example, by performing two-point segmentation in some embodiments. One advantage of two-point segmentation may include, for example, improved accuracy.

[0159] One or more advantages of the one or more features may include, for example, providing a digital model with digital teeth that are also segmented and labeled, along with non-tooth regions that are labeled and segmented, such as the gingiva. One or more advantages of using a panoramic image may include, for example, more accurate boundaries. One or more advantages of using a panoramic image may include, for example, preserving the features of teeth from both buccal and lingual sides, as well as the boundaries with the gingiva, whereas on a standard view, such as a 2D depth map, for example, the buccal and / or lingual sides are fully or partially occluded, and thus the neural network cannot see and segment them. On a standard view, such as a 2D depth map, for example, the boundaries between the teeth and the gingiva are not visible for all teeth and from all sides. This may occur because the teeth occlude (hide) the boundaries. Moreover, the problem is more general: whatever the simple view that is acquired (whether with a perspective projection or with an orthographic camera), some boundaries will not be visible. One or more advantages of using a panoramic image may include, for example, preserving the characteristics of the teeth from both the buccal and lingual sides, as well as the gingival border.

[0160] FIG. 25(a) illustrates an example of a computer-implemented method for digitally segmenting teeth in a digital model. The method may include receiving a 3D digital model of patient scan data of at least a portion of the patient's dentition at 2302, generating (or alternatively receiving) a panoramic image from the 3D digital model at 2304, labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image at 2306, mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model at 2308, and segmenting the labeled 3D digital model to provide a segmented 3D digital model at 2310. In some embodiments, the computer-implemented method may alternatively generate or receive a regular 2D depth map image from the 3D digital model at 2304. An example of a regular 2D depth map image is illustrated in FIG. 8.

[0161] The computer-implemented method of FIG. 25(a) may include one or more of the following features, alone or in combination, in various embodiments. The method of claim 1, further comprising: training a first neural network to label one or more regions of the panoramic image and provide a labeled panoramic image. The labeled 3D digital model may include one or more uncertain regions. The segmented 3D digital model may include one or more individual dental regions and non-dental regions. The first trained neural network may include a first convolutional neural network. The first convolutional neural network may include a U-Net neural network. The first convolutional neural network may include a YOLO neural network. Segmenting the 3D digital model may include geometrically segmenting the labeled 3D digital model. The geometric segmentation may include curvature-based segmentation. The geometric segmentation may include two-point segmentation. Generating the panoramic image can include using a second trained neural network to determine one or more digital tooth bounding regions corresponding to one or more digital teeth, connecting the one or more digital tooth bounding regions with a spline, sampling one or more spline points on the spline, determining one or more sampled digital surface points from the one or more sampled spline points, and determining one or more associated digital surface points corresponding to each sampled digital surface point. The method can further include determining a digital tooth bounding region center for each digital tooth bounding region, and connecting the one or more digital tooth bounding regions can include connecting the digital tooth bounding region centers. The second trained neural network can include a YOLO neural network. The second trained neural network can include a U-net convolutional neural network.Each sampled digital surface point and its corresponding associated digital surface point can be a row in the panoramic image. Determining the one or more associated digital surface points can include determining a sample depth point for each sampled digital surface point, and determining, from the sample depth point, one or more digital surface points that intersect with one or more respective rays that extend in a sample plane at an oblique angle from the occlusal direction, where the sample plane is orthogonal to the spline and the occlusal direction. The one or more oblique angles can be within an oblique angle range, and the oblique angle range can be from -45 degrees to +45 degrees.

[0162] 25(b) illustrates an example of a computer-implemented method for generating a panoramic image. The method may include receiving 2350 a 2D depth map of patient scan data of at least a portion of the patient's dentition, determining 2352 using a trained neural network one or more digital tooth bounding regions corresponding to one or more digital teeth, connecting 2354 the one or more digital tooth bounding regions by a spline, sampling 2356 one or more spline points on the spline, determining 2358 one or more sampled digital surface points from the one or more sampled spline points, and determining 2360 one or more associated digital surface points corresponding to each sampled digital surface point.

[0163] The method in FIG. 25(b) may include one or more of the following optional features, either alone or in combination: The method may further include determining a digital tooth bounding area center for each digital tooth bounding area, and connecting the one or more digital tooth bounding areas may include connecting the digital tooth bounding area centers. The second trained neural network may include a YOLO neural network. The second trained neural network may include a U-net convolutional neural network. Each sampled digital surface point and its corresponding associated digital surface point may be a column in the panoramic image. Determining the one or more associated digital surface points may include determining a sample depth point for each sampled digital surface point, and determining, from the sample depth point, one or more digital surface points that intersect with one or more respective rays extending in a sample plane at an oblique angle from the occlusal direction, where the sample plane is orthogonal to the spline and the occlusal direction. The one or more oblique angles may be within an oblique angle range, and the oblique angle range may be from -45 degrees to +45 degrees.

[0164] Some embodiments may include a processing system. The processing system may include a processor and a computer-readable storage medium including instructions executable by the processor capable of performing steps including receiving a 3D digital model of patient scan data of at least a portion of the patient's dentition, generating a panoramic image from the 3D digital model, labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image, mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model, and segmenting the labeled 3D digital model to provide a segmented 3D digital model. Alternatively, in some embodiments, the instructions executable by the processor steps can include receiving a panoramic image; labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image; mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in a 3D digital model to provide a labeled 3D digital model; and segmenting the labeled 3D digital model to provide a segmented 3D digital model.

[0165] One or more features described in this disclosure may be part of and / or performed on a processing system.

[0166] 26 illustrates a processing system 14000 in some embodiments. The system 14000 can include a processor 14030 and a computer-readable storage medium 14034 having instructions executable by the processor to perform one or more steps described in this disclosure.

[0167] In some embodiments, one or more steps may be performed, for example, by a user. In some embodiments, one or more features may be performed by a user, for example, using an input device while viewing the digital model on a display. In some embodiments, the computer-implemented method may allow the input device to manipulate the digital model displayed on the display. For example, in some embodiments, the computer-implemented method may rotate, scale, move, and / or otherwise manipulate the digital model in any manner as known in the art. In some embodiments, one or more features may be performed by a user, using an input device. In some embodiments, one or more steps may be initiated using techniques known in the art, such as, for example, a user selecting another button. In some embodiments, one or more features may be performed automatically, without user intervention.

[0168] In some embodiments, the computer-implemented method can display the digital model on a display and accept input from an input device, such as, for example, a mouse or a touch screen on the display. For example, the computer-implemented method can accept a start command. Upon accepting the start command, the computer-implemented method can perform one or more steps using one or more features described in this disclosure. Upon accepting a manipulation command, the computer-implemented method can rotate, scale, move, and / or otherwise manipulate the digital model in any manner as known in the art.

[0169] One or more of the features disclosed herein may be performed and / or accomplished automatically, without manual or user intervention. One or more of the features disclosed herein may be performed by a computer-implemented method. The features disclosed, including but not limited to any method and system, may be implemented in a computing system. For example, the computing environment 14042 used to perform these functions may be any of a variety of computing devices (e.g., desktop computers, laptop computers, server computers, tablet computers, gaming systems, mobile devices, programmable automation controllers, video cards, etc.) that may be incorporated into a computing system that includes one or more computing devices. In some embodiments, the computing system may be a cloud-based computing system.

[0170] For example, the computing environment 14042 may include one or more processing units 14030 and memory 14032. The processing unit executes computer-executable instructions. The processing unit 14030 may be a central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In some embodiments, the one or more processing units 14030 may execute multiple computer-executable instructions in parallel, for example. In a multiprocessing system, multiple processing units execute computer-executable instructions to increase processing power. For example, a representative computing environment may include a central processing unit as well as a graphics processing unit or co-processing unit. The tangible memory 14032 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both, accessible by the processing unit. The memory stores software implementing one or more innovations described herein in the form of computer-executable instructions suitable for execution by the processing unit.

[0171] A computing system may have additional features. For example, in some embodiments, a computing environment includes storage 14034, one or more input devices 14036, one or more output devices 14038, and one or more communication connections 14037. An interconnection mechanism, such as a bus, controller, or network, interconnects the components of a computing environment. Typically, operating system software provides an operating environment for other software executing within the computing environment and coordinates the activities of the components of the computing environment.

[0172] Tangible storage 14034 may be removable or non-removable and includes magnetic or optical media such as magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within a computing environment. Storage 14034 stores instructions for software that implements one or more of the innovations described herein.

[0173] An input device may be, for example, a touch input device such as a keyboard, a mouse, a pen, or a trackball, a voice input device, a scanning device, any of a variety of sensors, another device that provides input to the computing environment, or a combination of these. For video encoding, an input device may be a camera, a video card, a TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into the computing environment. An output device may be a display, a printer, a speaker, a CD writer, or another device that provides output from the computing environment.

[0174] A communication connection enables communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may be electrical, optical, RF, or other carrier waves.

[0175] Any of the methods of the present disclosure may be implemented as computer-executable instructions (e.g., the computer-executable instructions cause one or more processors of a computer system to perform the method) stored on one or more computer-readable storage media 14034 (e.g., one or more optical media disks, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as flash memory or hard drives)) and executed on a computer (e.g., any commercially available computer, including a smartphone, other mobile device including computing hardware, or a programmable automation controller). The term computer-readable storage medium does not include communication connections, such as signals and carrier waves. Any of the computer-executable instructions for implementing the techniques of the present disclosure, and any data created and used during the implementation of the embodiments of the present disclosure, may be stored on one or more computer-readable storage media 14034. The computer-executable instructions may be, for example, a dedicated software application or part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a networked environment using one or more networked computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network).

[0176] For clarity, only certain selected aspects of the software-based implementation are described. Other details well known in the art are omitted. For example, it should be understood that the technology of the present disclosure is not limited to any particular computer language or program. For example, the technology of the present disclosure may be implemented by software written in C++, Java, Perl, Python, JavaScript, Adobe Flash, or any other suitable programming language. Similarly, the technology of the present disclosure is not limited to any particular computer or type of hardware. Specific details of suitable computers and hardware are well known and need not be described in detail in this disclosure.

[0177] It should also be appreciated that any functionality described herein may be performed, at least in part, by one or more hardware logic components instead of software. For example, and without limitation, exemplary types of hardware logic components that may be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[0178] Additionally, any of the software-based embodiments (e.g., including computer-executable instructions for causing a computer to perform any of the methods of the present disclosure) may be uploaded, downloaded, or accessed remotely through suitable communication means including, for example, the Internet, the World Wide Web, an intranet, a software application, cable (including fiber optic cable), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication means.

[0179] In view of the many possible embodiments to which the principles of the present disclosure may be applied, it should be recognized that the illustrated embodiments are merely examples and should not be construed as limiting the scope of the present disclosure. [Explanation of symbols]

[0180] 100 Digital Models 202 First Convolutional Layer 203 Input images 204 Kernel 206 Convolution output, first convolution image 207 Feature Channel 208 First kernel output 212 First Pooling Layer 214 First Pooling Filter 216 First Pooling Image 217 Feature Channel 218 Second Convolutional Layer 219 Feature Channel 220 Second convolution image 222 Second Pooling Layer 224 Second Pooling Image 224 Output forecast 230 fully connected layer 302 layers 304 layers 306 layers 307 Residual 308 layers 310 layers 311 Residual 312 layers 314 layers 315 Residual 316 layers 318 layers 319 Residual 320 layers 322 layers 323 Residual 401 Input Image 402 Contraction Path 404 Expansion Path 406 First Convolutional Layer 408 First Pooling Operation 410 First Convolution 411 First feature map 412 Final feature map 413 Second Convolution 414 Convolutional Layer Input 416 Second Convolutional Layer 418 Third Convolutional Layer 422 Final pooling operation 424 Upsampling 425 upsampling layers 426 Convolutional Layer 427 Fourth convolutional layer, final contraction path convolutional layer 428 Feature Map 429 Concatenation 430 Feature Map 432 Final upsampling layer 434 Last Convolution 435 Output Segment Map 500 Digital Model 502 Boundary Loop 504 Digital Dental Area 506 Digital Non-dental Area 508 Digital Surface Triangles 512 Vertex 514 Digital Surface Triangles 516 sides 518 sides 520 sides 522 Interior Dual Graph Domains 524 Exterior Dual Graph Domains 700 2D Depth Maps 702 Labeled Digital Tooth Bounding Region 800 Digital Model 802 Digital Bounding Box 804 Digital Teeth 806 Occlusal direction 808 Digital bounding box center, first bounding box center 810 2nd bounding box center, 2nd digital bounding box center 812 First Spline 814 Second Spline 816 Third Digital Bounding Box Center 820 Continuous Spline 822 Last bounding box center 824 straight line 826 Last Spline 828 straight line 902 sampling points 904 Continuous Smoothing Spline 906 Digital Model 1002 sampling digital surface points 1004 Continuous Smoothing Spline 1006 3D Digital Models 1008 Sample Plane 1010 Occlusal direction 1012 depth rays, sample point depth rays 1014 sample depth points 1016 First positive inclination ray 1018 First positive tilt angle 1020 First negative tilt ray 1022 First Digital Surface Points 1024 second digital surface points 1050 Digital Model 1052 sample points 1054 Spline 1056 Ray of light 1102 Center row 1103 Sampling Digital Surface Points Line 1104 Column 1105 Line 1106 1110 1st Positive Row 1112 First negative row 1202 Darker Areas 1204 Brighter Areas 1300 Panorama Images 1302 Labeled digital tooth bounding area 1402 Digital Bounding Area 1404 Digital Tooth Bounding Region 1500 3D Digital Models 1502 Digital Tooth Bounding Box 1504 First Digital Tooth Bounding Box Corner 1506 First Ray 1508 Second Digital Tooth Bounding Box Corner 1510 Second Ray 1512 depth points 1513 Third Digital Tooth Bounding Box Corner 1516 The Third Ray 1518 Fourth digital tooth bounding box corner 1520 The Fourth Ray 1602 Digital Tooth Boundary Region 1603 distance 1604 Certainty Area 1605 distance 1606 Uncertain Area 1607 Uncertain Area 1614 First Labeled Digital Surface Area 1616 Unlabeled Region 1618 Second Labeled Digital Surface Area 1620 3rd Labeled Digital Surface Area 1786 sides 1788 triangle 1790 triangle 1792 dihedral angle 1794 The Third Plane 2000 labeled 3D digital models 2002 Labeled Regions 2004 center point 2010 Parabola 2012 Occlusal direction 2020 Separation Area 2030 plane 2032 Lingual point 2034 Buccal point 2036 First Segment Path 2050 Digital Teeth 2051 center point 2052 Lingual point 2053 1st route, 1st segment route 2054 First Side 2056 Second Path, Second Segment Path 2057 Separation area 2058 Second Side 2059 Parabola 2062 Cheek point, face point 2064 sides 2066 sides 2068 inner loop region 2070 Segmented Digital Teeth 2072 Second Segment Path 2074 Boundary Loop 2200 Digital Model 2202 Digital Teeth 2204 Gingival area 2402 Specific Triangle 2404 Adjacent triangles, nearby triangles 2405 First Side 2406 Adjacent Triangles, Nearby Triangles 2407 Second Side 2408 Adjacent triangles, nearby triangles 2409 Third Side 2010 Best Fit Parabola 2411 Segments 2412 Segments 2414 Segmented Digital Surface Mesh 2416 Segments 4000 U-net Neural Network 4002 input images 4004 convolutional layers, first convolutional layer 4005 Second convolution layer 4006 Contraction Path 4007 Third Convolutional Layer 4008 First feature map 4009 4th convolution layer 4010 Second feature map 4011 5th convolution layer 4012 Third feature map 4013 6th convolution layer 4014 4th feature map 4015 7th convolution layer 4016 5th feature map 4018 6th feature map 4020 7th feature map 4021 Expansion path 4031 3rd deconvolution layer 4032 First deconvolution layer 4033 Second deconvolution layer 4034 Third Deconvolution Feature Map 4035 6th deconvolution layer 4036 Second Deconvolution Feature Map 4037 5th deconvolution layer 4038 First deconvolution feature map 4039 4th deconvolution layer 4040 5th deconvolution feature map 4042 4th deconvolution feature map 4044 7th deconvolution layer 4048 6th deconvolution feature map 4050 7th deconvolution feature map 5001 input images 5002 encoder 5004 Conv-1 5008 Res-2 5010 Res-3 5012 Res-4 5014 Res-5 5016 GAU-3 5018 GAU-2 5020 GAU-1 5022 FPA 5030 FPA 5032 Resnet 5034 Feature Map 5036 Global Pooling Branch 5038 Downsampling Branch 5040 Upsampling Branch 5042 1st upsample branch stage 5044 Second Upsample Branch Stage 5046 3rd upsample branch stage 5048 First Downsample Convolution 5050 second downsample convolution 5052 third downsample convolution 5054 3x3 convolution 5056 5x5 convolution 5058 7x7 convolution 5060 Global Pooling 5062 1x1 convolution 5064 upsampling 5070 per-pixel multiplier 5072 Convolutional Layer 5074 Decoder 5075 Segmented Images 5076 First Decoder Stage 5078 Second Decoder Stage 5080 Third Decoder Stage 5102 rows Column 5104 6002 Input images 6004 First ResNet convolutional layer Conv-1 6006 Pooling Step 6010 Res-2 6012 Res-3 6014 Res-4 6016 Res-5 6018 1st Block 6020 First Res-2 Convolution 6022 Second Res-2 Convolution 6024 First Res-3 Convolution 6026 First Res-4 Convolution 6027 First Res-5 Convolution 6028 Bypass 6029 Pooling Stage 6040 Next Block 6050 1st Block 6052 1st block input feature map, input image 6054 First Convolution 6056 First convolution output 6058 Second Convolution 6060 1st block output 6062 Bypass 6064 Second block input feature map 6066 Second Block 8502 Center point 8504 1st x-axis 8506 1st y-axis 8508 Parabola 8510 x-axis 8512 y-axis 8514 Parabola 14000 Processing System 14030 Processor, Processing Unit 14032 Memory, Tangible Memory 14034 Computer-readable storage medium, storage, tangible storage 14036 Input Devices 14037 Communication Connection 14038 Output Device 14042 Computing Environment 50100 GAU 50101 Low-level features 50102 Convolution, Convolutional Layer 50104 High level features 50106 Global Pooling 50108 Feature Map 50110 Convolutional Layer 50112 Multiplier 50114 Weighted low-level features 50116 Total

Claims

1. 1. A computer-implemented method for digitally segmenting teeth in a digital model, comprising: receiving a 3D digital model of at least a portion of the patient's dentition from patient scan data; generating a panoramic image from the 3D digital model; and labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image; mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model; segmenting the labelled 3D digital model to provide a segmented 3D digital model; The method includes:

2. The method of claim 1 , further comprising training the first trained neural network to label one or more regions of the panoramic image and provide a labeled panoramic image.

3. The method of claim 1 , wherein the labeled 3D digital model includes one or more regions of uncertainty.

4. The method of claim 1 , wherein the segmented 3D digital model includes one or more individual dental regions and non-dental regions.

5. 2. The method of claim 1 , wherein the first trained neural network comprises a first convolutional neural network.

6. The method of claim 1 , wherein the first trained neural network comprises a semantic segmentation neural network.

7. The method of claim 1 , wherein the first trained neural network comprises a YOLO neural network.

8. The method of claim 1 , wherein segmenting the 3D digital model comprises geometrically segmenting the labeled 3D digital model.

9. The method of claim 8 , wherein the geometrically segmenting comprises curvature-based segmentation.

10. The method of claim 8 , wherein the geometrically segmenting comprises two-point segmentation.

11. generating the panoramic image, determining one or more digital tooth bounding regions corresponding to the one or more digital teeth with a second trained neural network; connecting said one or more digital tooth bounding regions by a spline; Sampling one or more spline points on the spline; determining one or more sampled digital surface points from the one or more sampled spline points; determining one or more associated digital surface points corresponding to each sampled digital surface point; The method of claim 1 , comprising:

12. 12. The method of claim 11, further comprising determining a digital tooth bounding area center for each digital tooth bounding area, and wherein connecting the one or more digital tooth bounding areas comprises connecting the digital tooth bounding area centers.

13. The method of claim 11 , wherein the second trained neural network comprises a YOLO neural network.

14. The method of claim 11 , wherein the second trained neural network comprises a semantic segmentation neural network.

15. The method of claim 11 , wherein each of the sampled digital surface points and its corresponding associated digital surface point comprises a column in the panoramic image.

16. Determining one or more relevant digital surface points determining a sample depth point for each sampled digital surface point; determining, from the sample depth points, one or more digital surface points that intersect with one or more respective rays that extend in a sample plane at one or more oblique angles from an occlusion direction, the sample plane being orthogonal to the spline and the occlusion direction; The method of claim 11 , comprising:

17. The method of claim 11 , wherein the one or more tilt angles are within a tilt angle range, the tilt angle range being from −45 degrees to +45 degrees.

18. 1. A system for digitally segmenting teeth in a digital model, comprising: A processor; a computer-readable storage medium containing instructions executable by said processor to perform steps; wherein the steps include: receiving a 3D digital model of at least a portion of the patient's dentition from patient scan data; generating a panoramic image from the 3D digital model; and labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image; mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model; segmenting the labelled 3D digital model to provide a segmented 3D digital model; Including, the system.

19. generating the panoramic image, determining one or more digital tooth bounding regions corresponding to the one or more digital teeth with a second trained neural network; connecting said one or more digital tooth bounding regions by a spline; Sampling one or more spline points on the spline; determining one or more sampled digital surface points from the one or more sampled spline points; determining one or more associated digital surface points corresponding to each sampled digital surface point; The system of claim 18 , comprising:

20. 1. A non-transitory computer readable medium storing executable computer program instructions for segmenting a digital model, the computer program instructions comprising: receiving a 3D digital model of at least a portion of the patient's dentition from patient scan data; generating a panoramic image from the 3D digital model; and labeling one or more regions of the panoramic image using a first trained neural network to provide a labeled panoramic image; mapping one or more regions of the labeled panoramic image to one or more corresponding coarse digital surface triangle labels in the 3D digital model to provide a labeled 3D digital model; segmenting the labelled 3D digital model to provide a segmented 3D digital model; A non-transitory computer readable medium comprising:

Images