Automatic Semantic Segmentation of Non-Euclidean 3D Datasets Using Deep Learning

Abstract

The present invention relates to a computer-implemented method for semantic segmentation of point clouds, comprising the steps of receiving a point cloud, the point cloud comprising points representing vectors in 3D space, the points representing a predetermined object, preferably a portion of a dentofacial structure, the dentofacial structure comprising a dentition including teeth; and determining, using a non-uniform resampling algorithm, one or more subsets of the point cloud, each of the one or more subsets comprising a first number of points arranged within a predetermined spatial distance of a selected point of the point cloud and a second number of points arranged at a spatial distance greater than the predetermined spatial distance, the first number of points representing one or more fine features of the object around the selected point. the second number of points represent one or more global features of the object; providing each of the one or more subsets of points to an input of a deep neural network DNN, the deep neural network being trained to semantically segment each of the points of the one or more subsets provided to the input of the DNN according to a plurality of classes associated with the object; and receiving, for each point of the subset provided to the input of the DNN, a multi-element vector at the output of the DNN, each element of the vector representing a probability that the point belongs to one of the plurality of classes of object.

Description

[Technical field]

[0001] The present invention relates to automatic semantic segmentation of non-Euclidean 3D datasets using deep learning, and in particular, but not exclusively, to systems and methods for semantic segmentation of non-Euclidean 3D datasets using deep learning, methods for training such deep learning neural networks, and computer program products for using such methods. [Background technology]

[0002] The advent of digital equipment for extraoral imaging (e.g., X-ray panoramic, cephalometric and cone-beam computed tomography) and intraoral imaging (e.g., laser or structured light projection scanners) has been the driving force to develop computer-aided dental treatment systems that can analyze different sources of patient image data, typically 3D image data, and extract useful clinical information therefrom. Such systems may support one or more clinical workflows, for example for implants and / or orthodontics, so that the system can determine or assist in determining a dental treatment plan. Although 3D CT scans provide very useful information, the resolution of these scans cannot provide the precision and detail required for optimal dental planning in areas where accurate and detailed information about the patient's dentition and more general dentofacial structure is essential. It is therefore desirable in that case to rely on more accurate data sources, such as high-resolution 3D data representations of the scanned surfaces of the teeth and gingiva (gums) determined by intraoral scanners.

[0003] Accurate automatic semantic segmentation of such 3D data representations is an essential feature for dental treatment systems, and involves the assignment of a label based on the Federation de Dentist International (FDI) standard to each data point, e.g. labeling all points that belong to a particular tooth crown. FDI specifies 32 labels, referring to, e.g., 16 teeth in each upper and lower jaw as may be present in a healthy adult. Thus, the problem involves finding multiple classes (e.g., 16 or 32 classes plus one for the gingiva) in a large 3D dataset. When building an accurate 3D dental model based on a 3D dataset representing a dentition, several problems may arise. One problem concerns dentition (arrangement of teeth on a dental arch) where two adjacent crowns (e.g., two molars) are very similar in appearance. Thus, additional information is needed to assign the correct label to a point in the 3D dataset, such as the relative position of one tooth with respect to one or more other teeth on the dental arch. Furthermore, the presence of anomalies in the dentition (e.g., one or more missing teeth) and / or shape deformations makes the automatic semantic segmentation of 3D data representing the dentition a computationally challenging task. Further problems are related to data acquisition, such as partially missing data (e.g., due to occlusion during scanning), lack of a universal coordinate system, the presence of noise, outliers, etc.

[0004] Extraoral imaging data, such as 3D x-ray data, are typically provided in a Euclidean format, e.g., a voxel representation in which each voxel is associated with a predefined location in a 3D grid. Systems are known in the art that can use deep learning to perform semantic segmentation (classification) on voxelized CBCT data. For example, the paper by Miki et al., “Classification of teeth in cone-beam CT using deep convolutional neural network”, Computers in Biology and Medicine 80 (2017) pp. 24-29, describes a 2D deep convolutional neural network system trained to classify 2D CBCT images of teeth into seven different tooth types. Euclidean 3D datasets, such as voxelized 3D data representations, are particularly suitable for processing by trained deep neural networks, since the voxel representation provides both local and global information about features in the dataset (e.g., information about the voxel relative to a particular voxel representation).

[0005] However, the 3D datasets generated by intraoral scanning devices, such as point clouds or surface meshes, are non-Euclidean. For example, in a point cloud representation, each point may be represented by a 3D vector in a non-universal 3D Cartesian coordinate system (i.e., the Cartesian coordinate system may differ between two IOS datasets). Similarly, in a surface mesh representation, three points may define a triangular surface that constructs a 3D surface mesh. A normal vector associated with the triangular surface defines an orientation in 3D space. Non-Euclidean datasets, such as point clouds and surface meshes, are irregular, permutation invariant, and have a variable number of points per sample. For that reason, non-Euclidean datasets, such as raw point cloud data, cannot be directly applied to traditional deep learning models (e.g., CNNs) for image analysis, such as segmentation.

[0006] Various approaches have been proposed to deal with non-Euclidean datasets. For example, US 2017 / 0169562 describes a system for automatic tooth shape recognition based on intraoral optical 3D scanning. A 3D surface mesh of each tooth crown is generated and used to determine aggregate features of each tooth. These features are then used as training data for training classifiers that utilize traditional machine learning methods such as support vector machines or decision trees. Similar feature-based approaches have been created by (Guo et al., 2015; Fang et al., 2015), where a set of standard shape features (e.g., based on computer graphic algorithms) are extracted from the point cloud, and then a neural network (e.g., CNN) is used to classify the features. The performance of such feature-based approaches is limited to the discriminative properties of the handcrafted features (Qi et al., 2017). In volumetric approaches, the object represented by the point cloud is voxelized and used to form a 3D model based on a 3D CNN (Wu et al., 2015; Qi et al., 2016). However, the process of spatially quantizing point cloud information into voxels limits the performance of the model, especially when fine high-frequency details of the point cloud representation need to be preserved in the shape curvature for accurate prediction.

[0007] Point cloud deep learning models are configured to directly receive and process raw point clouds. Each point may have some attributes, e.g. 3D coordinates, and possibly other attributes, such as surface normals, colors, etc. Point cloud deep learning does not exhibit some drawbacks related to handcrafted features, quantization errors, or high processing demands, associated with the aforementioned approaches. Several studies are published on point cloud analysis with deep neural networks (DNNs) for object classification and / or segmentation tasks. PointNet (Qi et al., Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2):4, 2017.) and DeepSets (Ravanbakhsh et al. Deep learning with sets and point clouds. arXiv preprint arXiv:1611.04500, 2016) are deep neural networks based on the so-called multi-layer perceptron (MLP) network. The performance of these MLP networks is improved by Le and Duan. (Pointgrid: A deep network for 3d shape understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9204-9214, 2018) based on the techniques for handling data irregularities and permutation invariance mentioned above. In addition, a resampling technique was used to fix the number of points provided to the input of the neural network. Such an approach, while suitable for many applications, is not suitable for classifying point clouds that represent fine high-resolution details (e.g., classifying points close to the tooth-gingival border). [Prior art documents] [Patent documents]

[0008]

Patent Document 1

Non-licensed literature

[0009] [Non-licensed document 1] Miki, "Classification of teeth in cone-beam CT using deep convolutional neural network", Computers in Biology and Medicine 80 (2017) pp. 24~29 [Non-licensed document 2] PointNet (Others, Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2):4, 2017.) [Non-licensed document 3] DeepSets (Ravanbakhsh Deep learning with sets and point clouds. arXiv preprint arXiv:1611.04500, 2016)

Non-licensed Document 4

Non-licensed Document 5

[0010] The above problems make accurate semantic segmentation of large high-resolution point clouds based on deep learning very challenging. Therefore, there is a need in the art for improved methods for semantic segmentation of non-Euclidean 3D datasets, such as point clouds, using deep learning. [Means for solving the problem]

[0011] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon, e.g., stored thereon.

[0012] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this specification, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0013] A computer-readable signal medium may include a propagated data signal in which computer-readable program code is embodied, for example in baseband or as part of a carrier wave. Such a propagated signal may be in any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0014] Program code embodied on a computer readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wireline, fiber optic, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including functional or object-oriented programming languages, such as Java™, Scala, C++, Python, and traditional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer, server, or virtualization server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0015] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU), or a graphics processing unit (GPU), of a general-purpose computer, special-purpose computer, or other programmable data processing device, to generate a machine, such that the instructions, executed via the processor of the computer, other programmable data processing device, or other device, create means for implementing the functions / acts specified in the blocks of the flowchart illustrations and / or block diagrams.

[0016] These computer program instructions may also be stored on a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored on the computer-readable medium produce an article of manufacture that includes instructions that implement the functions / acts specified in the flowchart and / or block diagram blocks.

[0017] The computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device such that a series of operational steps are performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process such that the instructions, executing on the computer or other programmable apparatus, provide a process for implementing the functions / acts specified in the flowchart and / or block diagram blocks.

[0018] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, including one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or a combination of dedicated hardware and computer instructions.

[0019] In a first aspect, the present invention relates to a computer-implemented method for semantic segmentation of a point cloud, the method comprising the steps of receiving a point cloud, the point cloud comprising points representing vectors in a 3D space, the points representing a given object; and applying a non-uniform resampling algorithm. determining one or more subsets of the point cloud using a deep neural network (DNN) that is trained to semantically segment the points of each of the one or more subsets provided to the input of the DNN according to a plurality of classes associated with the object; and receiving, for each point of the subset provided to the input of the DNN, a multi-element vector at an output of the DNN, each element of the vector representing a probability that the point belongs to one of a plurality of classes of the object.

[0020] In one embodiment, the first number of points is greater than the second number of points.

[0021] A point cloud typically contains more points than can be processed at once by a neural network. To obtain a complete segmentation of all points in the point cloud, the point cloud may be sampled into multiple subsets of points. These subsets can be segmented individually and their resulting segmentations can be combined to form a complete point cloud segmentation. The division into subsets causes the loss of contextual information about global features in the point cloud.

[0022] To address this issue, non-uniform resampling can be used to resample a subset of points from the point cloud, and then the subset can be directly fed to the input of the deep neural network of the segmentation module. Non-uniform resampling is required to address the size of the point cloud set. Additionally, it is required to address data irregularities, permutation invariance, and variable numbers of points in different point cloud sets. The non-uniform resampling algorithm generates locally dense and globally sparse subsets of points. In this way, the subsets of points define a fixed-size resampled point cloud that contains different levels of spatial resolution with both local fine details and global shape structure. These fine details and global shape information may be used to train the deep neural network. Non-uniform resampling may facilitate the training and deployment of deep learning approaches for semantic segmentation of 3D point clouds IOS, for example, of networks on fixed-size resampled point clouds that contain different levels of spatial resolution with both local fine details and global shape structure. In one embodiment, the object may be a part of a dentofacial structure, the dentofacial structure comprising a dentition including teeth.The invention may be used for semantic segmentation of irregular high resolution point cloud representations.

[0023] In one embodiment, the non-uniform resampling algorithm may include randomly selecting a point from the point cloud, determining a first number of points by densely sampling points of the point cloud that are within a predetermined distance from the randomly selected point, determining a second number of points by sparsely sampling points of the point cloud that are greater than the predetermined distance from the randomly selected point, and combining the first and second points into a subset of points.

[0024] In one embodiment, the predetermined spatial distance may be determined based on random numbers, preferably Monte Carlo techniques.

[0025] In one embodiment, the spatial distance may be based on a weighted distance metric. In one embodiment, the weighted distance metric may be based on an exponential function. In a further embodiment, the weighted distance metric may define a radial basis kernel (RBK) K according to the following formula:

[0026]

number

[0027] where the parameter σ is a free parameter that may be used to control the bandwidth of the kernel, and x i is the position of point i, and x fovea is the location of the randomly selected point.

[0028] In one embodiment, the DNN is trained based on a training set including one or more point clouds, each of the one or more point clouds representing a given object, and each point of the one or more point clouds being associated with a target label, and preferably training the DNN includes determining a subset of the one or more point clouds using a non-uniform resampling algorithm, and training parameters of the DNN by minimizing a loss function representing the deviation between a label predicted by the DNN and the target label.

[0029] In one embodiment, the loss function is a weighted loss function (L p ), where the loss function defines a weight for each point based on the distance metric. In one embodiment, the weighted distance metric may be based on an exponential function, and more preferably, the weighted distance metric defines a radial basis kernel (RBK) K according to the following formula:

[0030]

number

[0031] where the parameter σ is a free parameter that may be used to control the bandwidth of the kernel, and x i is the position of point i, and x fovea is the location of the randomly selected point.

[0032] In one embodiment, the DNN may be trained based on a discriminator network, which is configured to distinguish between a target label and a label generated by the DNN, or between at least one feature associated with the target label and at least one feature associated with a label generated by the DNN. Pointwise cross-entropy loss for semantic segmentation of point clouds is an ill-posed problem, since the relative geometric structure between instances (e.g., teeth) is not formulated. Therefore, a heavy post-processing stage (e.g., iterative energy minimization of a graph constructed for relational modeling) is usually applied. Results may be significantly improved by training a second-order (deep neural) network as a discriminator in an adversarial setting and penalizing unrealistic arrangements of labels assigned to teeth on the dental arch. Experiments show that the performance of the baseline network is improved and surpasses that of the state-of-the-art by achieving an IOU score of 0.94.

[0033] In one embodiment, the feedback module randomly provides the target label or a label generated by the DNN to an input of the discriminator network, and in response, the discriminator network generates an indication of whether the discriminator input is associated with the target label or a label generated by the DNN.

[0034] In a further aspect, the invention may relate to a computer implemented method for training a deep neural network DNN to semantically segment point clouds, the method comprising the steps of receiving training data, the training data comprising one or more point clouds, each of the one or more point clouds representing a predefined object, each point of the one or more point clouds being associated with a target label, preferably the predefined object being part of a dentofacial structure, the dentofacial structure comprising a dentition including teeth, and determining a subset of the one or more point clouds of the training data using a non-uniform resampling algorithm, each of the one or more subsets being associated with a target label, and determining a subset of the one or more point clouds of the training data using a non-uniform resampling algorithm, each of the one or more subsets being associated with a target label, providing one of the one or more subsets to an input of the DNN and receiving from an output of the DNN, for each point of the subset provided to the input of the DNN, receiving at the output of the DNN a multi-element vector, each element of the vector representing a probability that the point belongs to one of a plurality of classes of the object, and training parameters of the DNN by minimizing a loss function representing a deviation between a label predicted by the DNN and a target label.

[0035] In one embodiment, the non-uniform resampling algorithm may include randomly selecting a point from the point cloud, determining a first number of points by densely sampling points of the point cloud that are within a predetermined distance from the randomly selected point, determining a second number of points by sparsely sampling points of the point cloud that are greater than the predetermined distance from the randomly selected point, and combining the first and second points into a subset of points.

[0036] In one embodiment, the loss function is a weighted loss function (L p ), where the loss function defines a weight for each point based on a distance metric, preferably the weighted distance metric is based on an exponential function, and more preferably the weighted distance metric defines a radial basis kernel (RBK) K according to the following formula:

[0037]

number

[0038] where the parameter σ is a free parameter that may be used to control the bandwidth of the kernel, and x i is the position of point i, and x fovea is the location of the randomly selected point.

[0039] In one embodiment, the DNN may be trained based on a discriminator network configured to distinguish between a target label and a label generated by the DNN, or between at least one feature associated with the target label and at least one feature associated with a label generated by the DNN.

[0040] In one embodiment, the method may include a feedback module randomly providing a target label or a label generated by the DNN to a feature extraction module; the feature extraction module determining one or more features based on the received target label or a label generated by the DNN and providing the one or more features to an input of a discriminator network; and in response, the discriminator network generating an indication of whether the one or more features are associated with the target label or a label generated by the DNN and providing the indication to an input of the DNN.

[0041] In addition to point-wise classification losses, adversarial losses may be used to train the DNN in an adversarial setting. In this way, the segmentation network is empowered to improve point classification by learning a realistic layout of the labeling space, which entails high-level semantics and preserves valid ordering of tooth labels on the dental arch.

[0042] In a further aspect, the invention may relate to a computer system, preferably a server system, adapted for semantically segmenting point clouds, the computer system comprising: a computer readable storage medium having computer readable program code embodied therein, the program code comprising a non-uniform sampling module and at least a trained deep neural network DNN; the computer readable program code; and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein in response to executing a first computer readable program code, the processor is: receiving a point cloud, the point cloud comprising points representing vectors in a 3D space, the points representing a portion of a predetermined object, preferably a dentition including teeth, the dentition-maxillofacial structure comprising a dentition including teeth; and determining one or more subsets of the point cloud using a non-uniform resampling algorithm, the one or more subsets being selected from the set of points. determining that each of the one or more subsets of points includes a first number of points densely arranged within a predetermined spatial distance of a selected point of the point cloud and a second number of points sparsely arranged at a spatial distance greater than the predetermined spatial distance, the first number of points representing one or more fine features of the object around the selected point and the second number of points representing one or more global features of the object; providing each of the one or more subsets of points to an input of a deep neural network DNN, the deep neural network being trained to semantically segment the points of each of the one or more subsets provided to the input of the DNN according to a plurality of classes associated with the object; and for each point of the subset provided to the input of the DNN, receiving at an output of the DNN a multi-element vector, each element of the vector representing a probability that the point belongs to one of a plurality of classes of objects.

[0043] In yet a further aspect, the invention may relate to a computer system, preferably a server system, adapted for training a deep neural network DNN for semantically segmenting point clouds, the computer system comprising: a computer readable storage medium having computer readable program code embodied thereon, the program code comprising a non-uniform sampling module and at least a trained deep neural network DNN; the computer readable program code; and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein in response to executing the first computer readable program code, the processor is adapted to receive training data, the training data comprising one or more point clouds, each of the one or more point clouds representing a predefined object, each point of the one or more point clouds being associated with a target label, preferably the predefined object being a part of a dentofacial structure, the dentofacial structure comprising a dentition including teeth. determining, using a non-uniform resampling algorithm, a subset of one or more point clouds of the training data, each of the one or more subsets associated with a target label, each of the one or more subsets including a first number of points densely arranged within a predetermined spatial distance of a selected point of the point cloud and a second number of points sparsely arranged at a spatial distance greater than the predetermined spatial distance, the first number of points representing one or more fine features of the object around the selected point, and the second number of points representing one or more global features of the object; providing one of the one or more subsets to an input of the DNN and receiving from an output of the DNN, and for each point of the subset provided to the input of the DNN, receiving at the output of the DNN a multi-element vector, each element of the vector representing a probability that the point belongs to one of a plurality of classes of the object, and training parameters of the DNN by minimizing a loss function representing a deviation between a label predicted by the DNN and the target label.

[0044] In one embodiment, the DNN described above may include one or more multi-layer perceptron (MLP) networks and / or one or more χ-Conv operator layers of a pointCNN network. In a further embodiment, the discriminator network described above may include one or more multi-layer perceptron (MLP) networks. The deep neural network may be configured to directly process the "raw" point cloud data. In one embodiment, the deep neural network may include a multi-layer perceptron (MLP) network. In a further embodiment, the deep neural network may be configured as a pointCNN network as described in the paper by Li et al. PointCNN: convolution on χ-transformed points, arXiv:1801.07791v5 November 5, 2018 (to be published at Neural Information Processing Systems (NIPS) 2018). In one embodiment, the pointCNN network may be trained in an adversarial setting using a discriminator network that provides accurate semantic segmentation of IOS data representing dentition.

[0045] The present invention relates to an end-to-end deep learning framework for semantic segmentation of individual teeth and gingiva from point clouds derived from IOS point clouds. By introducing non-uniform resampling techniques, models may be trained and deployed at the highest available spatial resolution, learning local fine details along with the global coarse structure of the IOS.

[0046] The invention may also relate to a computer program product comprising software code portions configured to carry out any of the methods described above, when executed in the memory of a computer.

[0047] The invention will be further explained with reference to the attached drawings, which show, in a schematic manner, embodiments according to the invention, it being understood that the invention is in no way limited to these particular embodiments. [Brief description of the drawings]

[0048] [Figure 1A] FIG. 1 illustrates a system according to one embodiment of the present invention. [Figure 1B] FIG. 1 illustrates a system according to one embodiment of the present invention. [Diagram 2] 4 is a flow diagram of a method for determining a non-uniform sample of a point cloud according to an embodiment of the present invention. [Diagram 3] FIG. 2 illustrates a non-uniform resampling method according to one embodiment of the present invention. [Figure 4] FIG. 1 illustrates a method for training a deep neural network according to an embodiment of the present invention. [Diagram 5] FIG. 1 illustrates sampling implications that may be utilized in one embodiment of the present invention. [Figure 6] FIG. 1 shows a schematic diagram of a computer system for classification and segmentation of 3D dentofacial structures according to an embodiment of the present invention. [Figure 7] FIG. 2 illustrates a neural network architecture for segmentation that may be utilized in one embodiment of the present invention. [Figure 8] FIG. 2 illustrates a discriminator network that may be utilized in one embodiment of the present invention. [Figure 9] FIG. 2 illustrates an example of segmented point cloud data by a system in accordance with an embodiment of the present invention. [Figure 10] FIG. 1 shows an example of segmented point cloud data for accuracy purposes. [Figure 11] FIG. 1 is a block diagram illustrating an example data computing system that may be used to execute the methods and software products described in this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0049] In this disclosure, embodiments of a computer system and computer-implemented method using one or more deep neural networks (DNNs) for accurate semantic segmentation of non-Euclidean 3D datasets are described. Here, a non-Euclidean 3D dataset refers to a dataset that defines a 3D representation of an object, where the dataset includes a collection of points, where each point may be represented by a vector in a non-universal 3D Cartesian coordinate system (i.e., the Cartesian coordinate system may be different between two IOS datasets). Examples of such non-Euclidean 3D datasets include point clouds and 3D surface meshes, where the 3D surface mesh includes points that define triangular surfaces that collectively describe the surface mesh in 3D space. The normal vectors associated with the triangular surfaces define the orientation in 3D space. In contrast to Euclidean 3D datasets such as voxel representations, non-Euclidean datasets such as point clouds and surface mesh representations may be irregular, permutation invariant, and have a variable number of points per sample, where a sample here is intended to mean a representation of a surface, where the number of points is variable across multiple instances of recording (sampling) the same surface. In this application, the term point cloud is used to refer to any non-Euclidean 3D data set for representing objects in 3D space based on points that define vectors in the 3D space.

[0050] The computer system according to the invention may include at least a non-uniform resampling module for generating non-uniform samples of the point cloud (subsets of the original point cloud), each sample (subset) having a fixed number of points, the non-uniform resampling including dense local information around specific points of the point cloud and sparse global information associated with the global shape of the object, e.g. dentition, represented by the point cloud. Furthermore, the computer system according to the invention may include at least a segmentation module including a deep neural network that receives at its input a resampled subset of the point cloud data and is trained to label each point of the resampled subset according to the labels of a training set. The deep neural network may be configured to process the "raw" point cloud data directly. In one embodiment, the deep neural network may include a multi-layer perceptron (MLP) network. In a further embodiment, the deep neural network may be configured as a pointCNN network, as described in the paper by Li et al. PointCNN: convolution on χ-transformed points, arXiv:1801.07791v5 November 5, 2018 (to be published at Neural Information Processing Systems (NIPS) 2018). In one embodiment, the pointCNN network may be trained in an adversarial setting using a discriminator network that provides accurate semantic segmentation of IOS data representing dentition. An example of such a network is described in more detail with reference to Figures 7 and 8. A non-uniformly resampled subset of the labeled training data of the 3D point cloud may result in a trained segmentation deep neural network that can accurately segment a high-resolution point cloud dataset, such as an IOS point cloud dataset representing dentition.The computer system, the non-uniform resampling module, the segmentation module, and the training of the neural network of the segmentation module are described in more detail below.

[0051] 1A and 1B show a system according to an embodiment of the present invention. In particular, FIGS. 1A and 1B show a system 100 for automatic semantic segmentation of non-Euclidean 3D datasets, e.g., point clouds and / or 3D surface meshes. The system may be implemented on one or more computers including one or more processing units 101 connected to a memory 103. The system may be configured to execute multiple modules, including a 3D deep neural network that may be trained to receive (parts of) one or more non-Euclidean 3D datasets as input and process the data according to a trained model. The system may be implemented as a standalone system (e.g., a server system or a network application such as a cloud application) connected to a data storage and retrieval system, e.g., a database system, including one or more storage units, e.g., databases. As shown in Figures 1A and 1B, the system may include a non-uniform resampling module 104, a storage 106 (e.g., a buffer or cache) for (temporarily) storing resampled data generated by the non-uniform resampling module, e.g., non-uniform resampled point cloud or 3D surface mesh data, and a segmentation module 108 including a deep neural network (DNN) for semantic segmentation of the resampled data.

[0052] A point cloud typically contains more points than can be processed at once by a neural network. To obtain a complete segmentation of all points in the point cloud, the point cloud may be sampled into multiple subsets of points. These subsets can be individually segmented and the resulting segmentations can be combined to form a complete point cloud segmentation. The division into subsets may result in the loss of contextual information about global features in the point cloud.

[0053] To address this issue, the non-uniform resampling module is configured to resample a subset of points from the point cloud, the subset of points being of a size that can be directly fed to the input of the deep neural network of the segmentation module. Non-uniform resampling is required to address point cloud size, data irregularity, permutation invariance, and variable numbers of points in different point clouds. The non-uniform resampling algorithm may be configured to generate locally dense and globally sparse subsets of points. In this manner, the subsets of points may define a fixed-size resampled point cloud that includes different levels of spatial resolution with both local fine details and global shape structure. These fine details and global shape information may be used to train the deep neural network. Non-uniform resampling is described in more detail below with reference to Figures 2 and 3.

[0054] The system may include a training modus and an inference modus. Figure 1A shows the system in the training modus. In this modus, the system is configured to train the DNN of the segmentation module using the labeled point cloud training data 102. Once the deep neural network is trained, the inference modus of the system may be used to apply the trained 3D model to new point cloud data as shown in Figure 1B.

[0055] As shown in FIG. 1A, in the training modus, a so-called discriminator network 110, i.e. a second-order deep neural network, may be used to help train the deep neural network (generator network) of the segmentation module 108. In this way, the trained neural network can accurately predict the labels 114 from the resampled point cloud data 106 provided to the input of the segmentation module. The training data 102 may include multiple point cloud datasets, each set representing a dentition. Such data may be generated by an intraoral scanner (IOS) capable of generating high-resolution point cloud data. The points of each point cloud dataset of the training data may be labeled based on a tooth class, a so-called target label. For example, the target labels may be based on the International Dental Federation (FDI) standard, which defines 32 labels for the 16 teeth of each upper and lower jaw for a healthy adult dentition.

[0056] The training data may be augmented by various operations including random 3D rotation, permutation, and / or adding artificial noise (jitter) to the position of each point. Additionally, instance dropout may be applied to the dataset. Such instance dropout here is intended to mean randomly removing all points, or at least a significant portion of points belonging to a particular tooth, from the point cloud in each batch of training data. Augmentation may help the deep neural network learn labels that are missing and do not occur in the training set. In one embodiment, a preprocessing step may be applied to the point cloud dataset such that the data is normalized, more specifically, normalization of the vectors representing the points, such that within the point set the data has a mean close to 0 and unit variance (standard deviation as close to 1 as possible).

[0057] Thus, the DNN of the segmentation module 108 is trained to predict the appropriate class (e.g., 16 or 32 classes plus one for gingiva) for each point in the point cloud dataset. The training data is used to iteratively update the model (represented by the network parameters, e.g., weights, of the neural network) multiple times. Known stochastic gradient descent optimization methods may be used to learn the most beneficial values ​​of the network parameters of the DNN by reducing the error between the output of the neural network and the correct label. The training phase of the network may be considered complete when the error rate stops decreasing. A predefined loss function may be used to determine the error used in the stochastic gradient descent optimization method.

[0058] During training, the generator network (DNN of the segmentation module) may receive a non-uniform subsample of the point cloud training data. In addition, the generator network may receive feedback from the discriminator network 110. The main task of the discriminator network is to distinguish whether the predicted label it receives as input is the label generated by the generator network from its output or the target label that is part of the training data. Thus, in that case, the target label is used as the ground truth for training the generator network. The discriminator receives feedback whether the output it gave was correct, i.e., whether the point labels were indeed "predicted" or "real" as the discriminator indicated. This information helps the generator network to effectively learn to label the points of the non-uniform subsample, since both the generator and the discriminator learn from the given feedback.

[0059] The discriminator network is configured to distinguish between the target label and the label predicted by the generator network. During training, the feedback module 112 presents either the target label or the output label of the generator network (randomly) to the discriminator network. In response, the discriminator network generates an indication of whether the input of the discriminator is associated with the "real" label generated by the generator network or with the predicted label. The discriminator network then generates a "score" indicating whether the information it receives is real or correct. For example, in one embodiment, the discriminator may have an output between zero and one, with "1" indicating that the information is related to the target label and "0" indicating that the information is associated with the label predicted by the generator network. During training, if the generator network generates a predicted label that the discriminator can distinguish from the target label, this is a signal to the system that the predicted label is not "realistic" enough. Thus, in this way, the generator network is stimulated to generate predicted labels that are similar to the target labels of the training data.

[0060] The more realistic the predictions generated by the generator network, the more difficult it is for the discriminator to distinguish between the two. Thus, the ability of the discriminator to distinguish between the two is a measure of the quality of the labels generated by the generator. This information may be fed back to the discriminator as well as the generator network via backpropagation. In this way, the generator is trained to generate accurate labels for semantically classifying the points of the resampled subset into different dental classes. The loss functions used in the backpropagation method may include point-wise classification losses and adversarial losses to improve classification of points by enabling the segmentation network to learn a realistic layout of the labeling space, with high-level semantics, and by preserving a valid arrangement of tooth labels on the dental arch. The training modus and loss functions of the system are described in more detail below with reference to Figures 4 and 5.

[0061] In the inference modus, the system is configured to provide new point cloud data 120, e.g., an IOS point cloud representing a dentition, to an input of a non-uniform resampling module 104. The resampled data, e.g., a resampled point cloud dataset 106 (representing a non-uniformly resampled subset of the point cloud provided to the input), may be provided to an input of a segmentation module 108, where a trained deep neural network predicts a label for each point of the subset of points. The labeling process may be applied to different subsets of the point cloud until all points of the input point cloud are labeled by the trained neural network. The interference process and an example segmentation output of the system are described below with reference to Figures 6 and 9.

[0062] As already briefly mentioned above, training a deep learning model based on all or at least a significant portion of the points of an IOS point cloud may introduce problems related to the nature of large high-resolution point cloud datasets. For example, due to the fact that the number of points in different IOS datasets can vary considerably (e.g., from 100,000 points to 500,000 points or even more), in principle, it is necessary to process a matrix with varying rankings. In addition, hardware limitations may require a maximum value for the size of the point cloud that can be processed. Such problems may be solved using a patch classification technique, which is common for large image datasets. This technique relies on determining several patches (local subsets of a large dataset) by resampling the point cloud and classifying each data point of each subset. However, the use of such techniques may degrade the classification results, since the extracted patches (local subsets of the point cloud) do not provide sufficient information about the global structure of the object that the point cloud represents.

[0063] Thus, conventional uniform resampling of high-resolution point clouds does not result in an accurate analysis of the point cloud at its highest available resolution. Various non-uniform resampling methods are proposed to preserve high-frequency components (Chen et al., 2018; Huang et al., 2013) or local directional density (Skrodzki et al., 2018). However, the effectiveness of such data abstraction methods (i.e., reducing the data while retaining the most relevant information) on the performance of deep neural networks is unclear. Moreover, such approaches are not suitable for designing end-to-end learning schemes applied directly to the raw data by having such abstraction levels performed by the network itself with respect to its objective function. It may be considered beneficial to be able to infer relevant abstractions directly based on the desired results produced by such networks.

[0064] To solve the above problems, a non-uniform resampling method based on Monte Carlo sampling techniques may be used. As will be described in more detail below, such Monte Carlo sampling techniques enable the application of non-uniform resampling methods to high-resolution point clouds, resulting in a locally dense and globally sparse subset of points for efficiently training a deep neural network capable of processing such data. The point cloud may be represented as a matrix X = [x 1 , x 2 ,..., x N having N points, where each point may be associated with D attributes, x i ∈ R D and the point cloud X ∈ R N×D . Here, for 3D geometric points, D = 3. Further, a radial basis kernel function (RBF) denoted by K may be defined, and the RBF may be placed on randomly selected points. This point may be called a fovea, and x fovea ∈ X. Then, the geometric similarity (spatial distance) to the fovea may be determined based on a weighted distance metric. In one embodiment, the RBF kernel may be defined as follows.

[0065]

Number

[0066] Here, the parameter σ is a free parameter that may be used to control the bandwidth (compactness) of the kernel. By resampling, a subset Y from X having M1 points may be selected (M1 < N) such that dense sampling around the fovea and sparse sampling at positions further away from the fovea can be achieved. Based on Monte Carlo sampling, by randomly (with replacement) selecting a point x i from the set X, the condition K(x i , x fovea ) > r δA selected point may be inserted into the subset Y if and only if the condition is satisfied. If the condition is not satisfied, the point is rejected. δ may be a random number selected from a uniform distribution in the unit interval as defined by applicable Monte Carlo techniques. This resampling process may continue until M1-1 unique points are accepted. Table 1 shows an example of a non-uniform resampling algorithm.

[0067] [Table 1]

[0068] This algorithm may be used to determine a non-uniform sample Y from a point cloud X. FIG. 2 shows a block scheme for determining a non-uniform sample Y comprising M points according to an embodiment of the present invention. The algorithm may be implemented as a module comprising a memory containing code portions and a processor for executing the code portions. As shown in FIG. 2, in a first step 202, the input of the module may consist of a non-Euclidean data set X, such as a point cloud or a 3D surface mesh. In a further step 203, a point (called the fovea) from the non-Euclidean data set X may be randomly selected. The foveal point is added to the set Y as its first member. Then, in 204, another point (x iA point Y may be randomly selected and compared to the fovea according to an RBF kernel, as described with respect to Equation 1. The value of the RBF kernel may be compared at 205 to a random value drawn from a uniform distribution in the range of 0 and 1. If the kernel value is greater than the random value, the selected point may be added to set Y, otherwise it may be rejected. This procedure may continue at 206 until set Y has M points. Depending on the bandwidth of such RBF kernel, the compactness of the resampled data may vary. For example, if a bandwidth is selected that has a value significantly smaller than 1 (which is the variance of the normalized point cloud), the resampling may be very compact and thus produce a local patch. Conversely, if a bandwidth is selected that is significantly larger than a value of 1, the non-uniform resampling may closely approximate uniform resampling. In this sense, patch extraction and uniform resampling may be considered as two special cases of such non-uniform resampling algorithms.

[0069] FIG. 3 illustrates the resampling of a point cloud using non-uniform resampling, according to an embodiment of the present invention. As shown in this figure, a high-resolution IOS point cloud 302 representation of a dentition including 80,000 points may be resampled based on a non-uniform resampling scheme 304, for example as described with reference to FIG. 2. The resampling may result in a first resampled subset 306 of 30,000 points, where the resampling was for example based on selected points 308. For the purposes of this visualization, the figure also illustrates a second resampled subset 312 of 30,000 points that may be obtained by applying a uniform resampling 310 to the point cloud. When comparing the first and second resampled subsets, the first resampled subset includes different levels of granularity, i.e. a relatively dense region of points around the fovea providing high-resolution local information about the portion of the dentition at the location of the fovea, and a relatively sparse region of points away from the fovea providing global information about the dentition. The resampled subsets based on non-uniform resampling on multiple foves may be used to train a deep neural network to learn both detailed local and global features of the dentition.

[0070] It should be noted that the above-described resampling scheme may also be applied to other non-Euclidean 3D data sets, for example 3D surface meshes. As an example, when faces constituting a surface are defined, such a face may be represented by its most representative point (e.g., a point at the average position of all points describing such a face). Effectively, this reduces the set of faces to a representative set of points to which the described method may be applied.

[0071] 4 illustrates a method for training a deep neural network according to an embodiment of the present invention. In particular, for this exemplary embodiment, the figure illustrates a flow diagram of training a deep neural network for classification and segmentation of dentofacial 3D image data according to an embodiment of the present invention. As shown in the figure, a discriminator 3D deep learning neural network 420 may be used in an adversarial training setting with the aim of improving the training quality of a segmentation 3D deep learning neural network of a segmentation module 422 used for semantic segmentation of point cloud data as described with reference to FIG. 6.

[0072] For the training procedure, the training data is used to train a 3D deep learning neural network so that it can automatically perform semantic segmentation from the 3D optical scanning data of the dentofacial structures. The computer may extract point clouds and / or surface normals, i.e. elements associated with coordinate values ​​in 3D space, from the 3D surface mesh of the dentofacial structures based on the optical scanning data. For all 3D image data in the training data 402, a corresponding ground truth class label 410 is assigned for each point in the 3D image data so that it can be determined by a dental professional.

[0073] During training, labeled 3D image data belonging to a single dentofacial structure 404 may be randomly selected from the training data stack of labeled dentofacial 3D image data 402. Then, a point x fovea 406 may be randomly selected from the cloud of points represented by 404, and a set of M elements is generated based on the points x foveaThis set of M elements may then be fed forward directly to a feature extractor 418 along with their ground truth labels 410 or fed to the input of a semantic segmentation 3D deep learning neural network 422 that outputs predictions 414 of the labels. Whether the ground truth or predicted labels are selected (via selector 416) may be determined with equal probability at each training iteration. The set of M elements and their (ground truth or predicted) labels are then fed to a feature extractor 418, which groups the elements based on their labels and extrapolates features from each group of elements. These features may include, for example, the mean and variance of each class (group of elements with the same label) and, in particular, the mean and variance of the location of each class (group of elements with the same label) in 3D space. Features of certain classes, for example gingiva, may be omitted from the calculation due to their detrimental contribution to the desired performance.

[0074] The set of all features for all classes may then be provided as input to a discriminator network 420. The function of such a discriminator network may be to determine whether the input features are associated with the ground truth (target label) or with the label predicted by the segmentation network 414. Feedback (loss) from the discriminator performance is then used to improve both the discriminator network 420 and the segmentation network of the segmentation module 422. For example, if the discriminator network receives negative feedback about failing its classification task, the segmentation network may receive an equal amount of, but positive, feedback.

[0075] Training a segmentation network by computing a traditional equally weighted loss for each point in the input non-uniformly resampled data is inefficient: Because the resampled point set contains various levels of granularity, penalizing the output error for dense and sparse regions equally would prevent the model from optimally adapting its convolution kernels to capture the fine details in the data, since the error for the sparse points would increase relatively equally.

[0076] This effect is illustrated in FIG. 5, where the certainty of the label classification is predicted by the network using an equally weighted loss function and presented for each point. The points 502 shown in dark colors are points where the label prediction was uncertain according to the trained deep neural network. As expected, sparse regions with low sampling rate (at the top side of the image) have a lower certainty of classification (i.e., are often misclassified by the trained network) because the lack of data in sparse regions makes it difficult for the model to perform as accurately as it does in dense regions. To optimize the performance of the learning algorithm, a trade-off may need to be found, for example by parameter tuning, between preserving sparse points (including the global dental arch structure) and utilizing the fine curvature in the IOS data. To that end, in one embodiment, a weight may be applied to each point. This weight may be calculated based on the distance metric of the RBF kernel, as defined by equation (1) above.

[0077] The transfer function of S and its parameter θ S The posterior probability vector (P i ) by assuming that the weighted loss value (L p ) may be formulated as follows:

[0078]

number

[0079] where y i is the x coordinate in 3D. i represents the one-hot encoded target label for point i with L = 17 and M = 3 10 4 is the number of.

[0080] Training a segmentation network solely by applying a traditional point-wise cross-entropy loss function has an important drawback. The label of each point in the point cloud has a high dependency on the labels of its neighboring points. For example, if a point belongs to an incisor, its neighboring points can only belong to the same or another incisor, canine, or gingiva, but cannot certainly belong to a molar. Such strong structural constraints are present in the data, but are ignored when the optimization problem is formulated based on Eq. (2). Since semantic segmentation is not, intrinsically, a pixel-based (point-wise) classification problem, the formulation of the optimization problem based on Eq. (2) is therefore inappropriate.

[0081] To address this issue, a discriminator network may be used to discriminate between the labels generated by the segmentation network and the (actual) target labels. In addition, an embedding loss (the distance between hidden layer features in the discriminator network) is proposed to increase the stability of training. Furthermore, instead of training the discriminator network directly based on the input space (i.e., point clouds and associated labels) and defining the embedding loss based on the predicted output, which requires a time- and processing-intensive training scheme, one or more statistical parameters may be calculated based on the actual labels and the predicted by the feature extractor. These parameters may be used to train the discriminator network quickly and efficiently.

[0082] The statistical parameters may include the mean and variance of the coordinates of all points (or at least a substantial portion thereof) that have the same label, as given by the segmentation network or ground truth labels. These parameters may be expressed as follows:

[0083]

number

[0084] Here, the symbol "||" denotes the vertical vector concatenation (stacking) operation, and L denotes the number of labels in the data.

[0085]

number

[0086] is the predicted label (p ij ) may represent a soft calculation of the tooth center positions in the dentition in 3D space and their variances (i.e., their soft bounding boxes). The statistical mean and variance may be calculated only for the L-1 classes, such as those belonging to teeth. p in Eq. (3) ij The value is the ground truth label (y i ) with the one-hot encoded value of (u),

[0087]

number

[0088] A corresponding feature set of u may be obtained. For labels not present in the point cloud, a vector of zeros may be inserted. The feature set (u) represents a realistic statistical measurement of the labeled data.

[0089] Fig. 6 shows a schematic of a computer system for classification and segmentation of 3D dentofacial structures according to an embodiment of the present invention. In particular, the computer system may be configured to receive 3D image data of dentofacial structures at 602. The structures may include gingiva and individual crown structures. The 3D image data may include a point cloud and / or surface normals, i.e. elements associated with coordinate values ​​in 3D space. Preferably, the 3D image data may include IOS image data in a predefined format, for example a triangle mesh format or a derivative thereof. The computer system may include a preprocessor for preprocessing the 3D image data into a point cloud.

[0090] A prediction cycle may then be started, which continues until all elements of the point cloud have been assigned at least one predicted label, and if there are still unlabeled elements, one random unlabeled element x is assigned 606. fovea is selected and, at 608, x is calculated according to non-uniform resampling as described with respect to FIG. fovea A set of M elements is taken from around the point cloud. The combination of 606 and 608 may be considered a non-uniform resampling module 611. The set of M elements is fed to the input of a first 3D deep learning neural network 610, which is trained to perform semantic segmentation, i.e., generate label probabilities for each of the M elements belonging to each possible class, at 612 (as described with respect to FIG. 4). If all elements of the point cloud have been assigned at least one predicted label, the labeled data may be obtained by taking the average of all label probabilities for each element of the point cloud, at 614, and selecting the most probable class for each element, at 616.

[0091] The systems shown in Figures 1 and 4 may include at least two deep learning networks: a segmentation network and a discriminator network. The segmentation network may be based on any deep learning network architecture capable of analyzing a point cloud, taking into account spatial correlation between different points of the point cloud. In one embodiment, the semantic segmentation network may be based on the PointCNN model.

[0092] A schematic diagram of such a deep neural network is shown in Figures 7A and 7B. As shown in Figure 7A, the PointCNN model 702 uses a χ-Conv operator 704 that weights and permutes the input points and their corresponding features before processing them with a traditional convolution operator. 1~9 The structure of the χ-Conv operator is shown in more detail in FIG. 7B. The χ-Conv operator may be controlled by parameters N, c, K, and D, where N defines the number of points of the resampled point cloud fed to the input of the segmentation module. The field of view of each χ-Conv operator includes a fixed set of K-nearest neighbor (KNN) points 709. In addition, the χ-Conv operator may include two MLP networks 710, 712. These two MLP networks learn a K×K χ-transform on the coordinates of the K input points. The result of the χ-Conv operation is the aggregation and projection of the KNN point features onto a representative set of points, after which a representative convolution is applied to them.

[0093] The PointCNN model, shown in FIG. 7, can learn local correlations from data represented by point clouds. In addition, the PointCNN model has a significantly smaller amount of learning parameters compared to other deep learning models for point clouds. This is beneficial for our case study, as it is less prone to severe overfitting on small datasets. The input to the segmentation network may be the resampled points, and the output is a multi-element vector for each point, with each element of the vector representing a class probability. For example, in one embodiment, the multi-vector elements may include 16 tooth classes and 17 elements representing the gingiva. The discriminator network divides the feature set u into

[0094]

number

[0095] In one embodiment, the discriminator network may be configured as a deep convolutional neural network.

[0096] An example of a discriminator network is shown in Figure 8. As shown in this figure, the network may include two cascaded sections 802, 804. The first section 802 estimates an affine transformation that is applied to an input 96-element vector. The second section 804 includes an MLP network, in this example a three-layer MLP network, that maps the transformed input vector to a scalar value by a sigmoid activation function at its output nodes. The network may be trained to produce a scalar 1 at its output when the network is applied to (u), while if the network is

[0097]

number

[0098] may produce the scalar 0.

[0099] In the adversarial setting for training the discriminator network D and the segmentation network S, the parameters θ D The discriminator loss (L D ) and the adversarial loss of network S may be defined by the following equation:

[0100]

number

[0101] Therefore, the total loss of the segmentation network is the loss L defined by equation (2). p and L defined by equation (5) Adv is the contribution of. To avoid hyperparameter tuning for the contribution weight (λ) between the two loss terms, an adaptive loss weighting may be used. λ = [λ 1 , λ 2 After initializing [ with a vector of unit values, a regularization term R(λ) may be added to the total loss function of the segmentation network (S) as defined by the following equation:

[0102]

number

[0103] Once the segmentation network has been trained in the above manner, the system may be configured in its inference mode as described with reference to FIG. 1B above. The segmentation network is trained on non-uniformly resampled data, and therefore, to predict the label of the entire point cloud, it is necessary to form several subsets of points based on the non-uniform resampling algorithm as described above with reference to FIGS. 2 and 3. A prediction for all points in the point cloud may be obtained by aggregating the predicted labels for each of the subsets. The pseudocode for the algorithm is shown in Table 2.

[0104] [Table 2]

[0105] Experiments were conducted to test the performance of the system. The training data included 120 optical scans of the dentition from 60 adult subjects, each including one maxillary and one mandibular scan. The dataset includes scans from healthy dentition and different types of abnormalities between subjects. IOS data were generated using a 3Shape d500 optical scanner (3Shape AS, Copenhagen, Denmark). On average, each IOS point cloud contains 180,000 points (varying between 100,000 and 310,000). All optical scan meshes were manually segmented and their respective points classified into one of 32+1 classes by a dental expert, reviewed and refined by one dental expert (DAM) using Meshmixer 3.4 (Autodesk Inc, San Rafael CA, USA). Labeling of tooth categories was done according to the international tooth numbering standard, Federation Internationale de Dentaire (FDI). Segmentation of each optical scan took, on average, about 45 minutes, which represents a very labor-intensive task for humans.

[0106] The performance of the models is evaluated by 5-fold cross-validation and the results are compared by the average Jaccard Index, also known as intersection over union (IoU). In addition to IoU, precision and recall for the multi-class segmentation problem are reported. To calculate precision and recall, each class is treated separately (one vs. all) as a binary problem and finally the average score is reported. We divided the experiments into three parts: (1) to benchmark the performance of the PointCNN network against two other state-of-the-art deep learning models for IOS segmentation. These models include PointNet (Qi et al., 2017) and PointGrid (Le and Duan, 2018); (2) to evaluate the impact of applying non-uniform resampling versus using simple uniform resampling. For a fair comparison, the number of resampled points is kept equal (M = 30,000); and (3) to evaluate the effectiveness with adversarial loss.

[0107] All models are trained by stochastic gradient descent and Adam learning adaptation techniques for 1000 epochs. The initial learning rate is equal to 5e-3 and is decreased by a factor of 0.9 every 20K iterations. The point clouds are normalized to have unit variance (by adjusting σ to 0.4 in equation (1)).

[0108] [Table 3]

[0109] Table 3 shows the results obtained under different experimental settings. Examples of predicted labels for IOS point clouds generated by the model are shown in Figures 9-11. As can be seen from Table 3, PointCNN performs better than the two other state-of-the-art models when simple uniform sampling is used. This is mainly because the χ-Conv operator in PointCNN can use the spatial correlation information of points in the point cloud. In addition, PointCNN uses a relatively small number of parameters, and as a result, the system is less prone to overfitting. PointGrid samples the points of the point cloud in a predefined grid and predicts the labels using a convolution operator. However, in this case, its performance is limited to the spatial resolution of the spatial quantization grid. Furthermore, the PointNet model ignores the spatial correlation between points in the point cloud. The results for PointCNN in Table 3 show the effectiveness of using non-uniform sampling and / or adversarial loss. Both techniques improve the performance of PointCNN individually. The combination of non-uniform sampling and adversarial loss provides the best performance. The accuracy of the system is demonstrated by the semantically segmented IOS point cloud data shown in Figures 8-11.

[0110] FIG. 9 illustrates 3D dentofacial image data, more specifically data that may be derived from an intraoral scan labeled as predicted by an embodiment of the present invention, and more specifically an embodiment incorporating a discriminator network as described with respect to FIGS. 4 and 8 during training. All four dentofacial images are held out samples (samples not used during training). For the purposes of these visualizations, the labeling as applied to the point cloud representing the mesh was applied inversely to a representative face of the surface mesh of the intraoral scan. Detailed sections 902-908 show the accuracy of the segmentation, especially in areas that may be considered difficult, such as the boundaries between the teeth and between the teeth and the gums.

[0111] FIG. 10A shows an example of 3D dentofacial image data labeled as predicted by a segmentation network trained without the support of a discriminator network. As shown in the highlighted circles 1002, the segmentation network trained without such a discriminator inaccurately predicted the labels of four different tooth objects across three teeth. For comparison, in FIG. 10B the same dentofacial 3D image data is shown with ground truth labeling. Results as seen with respect to FIG. 9 utilizing training using a discriminator as described with respect to FIG. 4 and FIG. 8 may be considered very accurate when compared to the ground truth labels, especially in the boundary regions 1004.

[0112] 11 is a block diagram illustrating an exemplary data processing system described in the present disclosure. The data processing system 1100 may include at least one processor 1102 coupled to a memory element 1104 via a system bus 1106. Thus, the data processing system may store program code in the memory element 1104. Furthermore, the processor 1102 may execute program code accessed from the memory element 1104 via the system bus 1106. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be understood that the data processing system 1100 may be implemented in the form of any system including a processor and memory capable of performing the functions described herein.

[0113] The memory elements 1104 may include one or more physical memory devices, such as, for example, a local memory 1108 and one or more mass storage devices 1110. The local memory may refer to a random access memory or other non-persistent memory device that is typically used during the actual execution of the program code. The mass storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1100 may also include one or more cache memories (not shown) that provide temporary storage of at least some of the program code to reduce the number of times the program code must be retrieved from the mass storage device 1110 during execution.

[0114] Input / output (I / O) devices, depicted as input device(s) 1112 and output device(s) 1114, may optionally be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, and the like. Examples of output devices may include, but are not limited to, a monitor or display, speakers, and the like. The input and / or output devices may be coupled to the data processing system directly or through an intervening I / O controller. A network adapter 1116 may also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. A network adapter may include a data receiver for receiving data transmitted to the system, device, and / or network by the system, device, and / or network, and a data transmitter for transmitting data to the system, device, and / or network. A modem, a cable modem, and an Ethernet card are examples of different types of network adapters that may be used with the data processing system 1100.

[0115] As depicted in FIG. 11, the memory element 1104 may store an application 1118. It should be understood that the data processing system 1100 may further execute an operating system (not shown) that may facilitate execution of the application. The application may be implemented in the form of executable program code and executed by the data processing system 1100, for example by the processor 1102. In response to execution of the application, the data processing system may be configured to perform one or more operations described in further detail herein. In one aspect, for example, the data processing system 1100 may represent a client data processing system. In that case, the application 1118 may represent a client application that, when executed, configures the data processing system 1100 to perform various functions described herein with reference to a "client." Examples of clients may include, but are not limited to, a personal computer, a portable computer, a mobile phone, and the like. In another aspect, the data processing system may represent a server. For example, the data processing system may represent an (HTTP) server, in which case the application 1118, when executed, may configure the data processing system to perform (HTTP) server operations. In another aspect, a data processing system may represent a module, unit, or function referred to herein.

[0116] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprise" and / or "comprising", as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0117] Corresponding structures, materials, acts, and equivalents of all means or step-plus-function elements in the following claims are intended to include any structure, material, or act for performing a function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been selected and described in order to best explain the principles and practical applications of the invention, and to enable those skilled in the art to understand the invention in various embodiments with various modifications suitable for the particular use contemplated. [Explanation of symbols]

[0118] 100 Systems 101 Processing Unit 102 labeled point cloud training data 103 Memory 104 Non-uniform resampling module 106 Resampled Point Cloud Datasets 106 Storage 108 Segmentation Module 110 Discriminator Network 112 Feedback Module 114 Label 120 point cloud data 302 high resolution IOS point cloud 304 Non-uniform resampling schemes 306 First resampled subset 308 Selected Points 310 Uniform Resampling 312 Second resampled subset 402 Training Data 404 Single tooth-maxillofacial structure 406 points xfovea 408 Non-uniform resampling schemes 410 Ground Truth Class Labels 414 Label Prediction 416 Selector 418 Feature Extractor 420 Discriminator 3D Deep Learning Neural Network 422 Segmentation Module 610 The First 3D Deep Learning Neural Network 611 Non-uniform resampling module 702 PointCNN Model 709 K nearest neighbor (KNN) points 710 MLP Network 712 MLP Network 802 Cascaded Part 804 Cascaded Part 1002 Highlighted Circle 1004 Boundary area 1100 Data Processing System 1102 Processor 1104 Memory elements 1106 System Bus 1108 Local Memory 1110 Mass Storage Device 1112 Input Devices 1114 Output Device 1116 Network Adapter 1118 Application

Claims

1. 1. A computer-implemented method for semantic segmentation of point clouds, comprising: receiving a point cloud, the point cloud comprising points in 3D space, the points representing an object, the object representing a dental-maxillofacial structure, the dental-maxillofacial structure comprising a dentition including teeth; determining a subset of points based on the point cloud using a non-uniform resampling algorithm, where each of the subsets of points comprises randomly selecting a point from the point cloud and determining a first number of points arranged within a predetermined spatial distance of the randomly selected point of the point cloud and a second number of points arranged at a spatial distance greater than the predetermined spatial distance, where the first number of points represent one or more local features of the object around the selected point and the second number of points represent one or more global features of the object; providing points of the subset of points to an input of a deep neural network (DNN), the DNN being trained to semantically segment the points of the subset of points provided to the input of the DNN according to a plurality of classes associated with the object; For each point of the subset of points provided to the input of the DNN, receiving a multi-element vector at an output of the DNN, each element of the multi-element vector representing a probability that the point belongs to one of the classes of the object; A method comprising:

2. Determining each of the subset of points comprises: randomly selecting points from the point cloud; determining the first number of points by sampling points of the point cloud that are within the predetermined spatial distance from the randomly selected point, and determining the second number of points by sampling points of the point cloud that are at a distance greater than the predetermined spatial distance from the randomly selected point; combining the first number of points and the second number of points into a subset of points; 2. The method of claim 1, comprising:

3. The method of claim 1 or 2, wherein the predetermined spatial distance is determined based on a random number.

4. The predetermined spatial distance between the point of the point cloud and the selected point is a weighted distance function, [0010] The method according to any one of claims 1 to 3, wherein the method is determined based on:

5. 5. The method of claim 1, wherein the DNN is trained based on a training set including one or more point clouds, each of the one or more point clouds representing the predetermined object, and each point of the one or more point clouds is associated with a target label.

6. The loss function is a weighted loss function (L p ), the loss function defines a weight for each point based on a weighted distance function, the weighted distance function being expressed as [0025] Define a radial basis kernel (RBK) K according to i is the position of point i, and x fovea The method of claim 5 , wherein: x is the location of the selected point, and the selected point is selected randomly.

7. 7. The method of claim 1, wherein the DNN is trained based on a discriminator network, the discriminator network being configured to distinguish between a target label and a label generated by the DNN.

8. The method of any one of claims 1 to 7, wherein the DNN comprises one or more multi-layer perceptron (MLP) networks and / or one or more χ-Conv operator layers of a PointCNN network.

9. 1. A computer-implemented method for training a deep neural network (DNN) to semantically segment a point cloud, comprising: a computer receiving training data, the training data comprising one or more point clouds, each of the one or more point clouds representing an object, the object representing a dentition-maxillofacial structure, the dentition-maxillofacial structure comprising a dentition including teeth, each point of the one or more point clouds being associated with a target label; determining a subset of points based on the point cloud of the training data using a non-uniform resampling algorithm, each of the subsets being associated with a target label, determining each of the subsets comprising: randomly selecting points from the point cloud; and determining a first number of points arranged within a predetermined spatial distance of the randomly selected point of the point cloud and a second number of points arranged at a spatial distance greater than the predetermined spatial distance, the first number of points representing one or more local features of the object around the randomly selected point, and the second number of points representing one or more global features of the object; providing points of the subset of points to an input of the DNN and, for each point provided to the input of the DNN, receiving a multi-element vector at an output of the DNN, each element of the multi-element vector representing a probability that the point belongs to one of a plurality of classes associated with the object; training the DNN by minimizing a loss function that represents the deviation between a label predicted by the DNN and the target label; A method comprising:

10. The non-uniform resampling algorithm comprises: randomly selecting points from the point cloud; determining the first number of points by sampling points of the point cloud that are within the predetermined spatial distance from the randomly selected point, and determining the second number of points by sampling points of the point cloud that are at a distance greater than the predetermined spatial distance from the randomly selected point; combining the first number of points and the second number of points into a subset of points; 10. The method of claim 9, comprising:

11. The loss function is a weighted loss function (L p ), the loss function defines a weight for each point based on a weighted distance function, the weighted distance function being expressed as [0030] Define a radial basis kernel (RBK) K according to The parameter σ is a free parameter that may be used to control the bandwidth of the kernel, x i is the position of point i, and x fovea The method of claim 9 or 10, wherein: x is the location of the selected point, and the selected point is selected randomly.

12. 12. The method of claim 9, wherein the DNN is trained based on a discriminator network, the discriminator network being configured to distinguish between a target label and a label generated by the DNN.

13. A feedback module randomly provides a target label or a label generated by the DNN to a feature extraction module; the feature extraction module determining one or more features based on the received target label or a label generated by the DNN and providing the one or more features to the input of the discriminator network; In response, the discriminator network generates an indication of whether the one or more features are associated with a target label or a label generated by the DNN; providing the instructions to the input of the DNN; 13. The method of claim 12, comprising:

14. 1. A computer system configured to semantically segment a point cloud, the computer system comprising: A computer-readable storage medium having computer-readable program code embodied therein, the program code including a non-uniform sampling module and at least a trained deep neural network (DNN); and a processor coupled to the computer-readable program code, wherein in response to executing the computer-readable program code, the processor: receiving a point cloud, the point cloud including points represented by vectors in 3D space, the points representing objects, the objects representing dental-maxillofacial structures, the dental-maxillofacial structures including a dentition including teeth; determining a subset of points based on the point cloud using a non-uniform resampling algorithm, where each of the subsets of points determining comprises randomly selecting a point from the point cloud and determining a first number of points arranged within a predetermined spatial distance of the randomly selected point of the point cloud and a second number of points arranged at a spatial distance greater than the predetermined spatial distance, where the first number of points represent one or more local features of the object around the selected point and the second number of points represent one or more global features of the object; providing a subset of points to an input of the DNN, the DNN being trained to semantically segment each of the points provided to the input of the DNN according to a plurality of classes associated with the object; receiving, for each point of the subset of points provided to the input of the DNN, a multi-element vector at an output of the DNN, each element of the multi-element vector representing a probability that the point belongs to one of the classes of the object; 23. A computer system configured to perform executable operations including:

Images