Deep Learning-Based Object Detection and Instance Segmentation of 3D Point Clouds

Abstract

1. A method for object detection in a point cloud, preferably a point cloud generated by a 3D optical scanner such as an intraoral scanning (IOS) point cloud, comprising the steps of: determining, by a deep neural network of a first type, first features associated with points of the point cloud, the point cloud including points representing one or more objects in at least a 3D space of the point cloud, the first features defining geometric information for each point of the point cloud, the first type of deep neural network being configured to receive the points of the point cloud as input; and determining, by a deep neural network of a second type based on the first features, second point cloud features, the second features defining local geometric information for the point cloud at the locations of the nodes of a uniform 3D grid. wherein the nodes are uniformly distributed in the 3D space of the point cloud; generating one or more object proposals based on second features, the object proposals defining 3D bounding boxes centered on nodes of the 3D grid, the 3D bounding boxes containing points of the point cloud that may define the object, the 3D bounding boxes defining 3D anchors; and determining a score for the 3D anchors by a third type of deep neural network, the score indicating a probability that the 3D anchor contains points that define the object or part of the object, the determining being based on the second features located within the 3D anchors.

Description

[Technical field]

[0001] The present invention relates to deep learning based object detection and instance segmentation of 3D point clouds, particularly but not exclusively to methods and systems for object detection in 3D point clouds using deep learning, methods and systems for instance segmentation of 3D point clouds using deep learning, deep neural network systems for object detection in 3D point clouds, deep neural network systems for instance segmentation of 3D point clouds, and computer program products for performing such methods. [Background technology]

[0002] In image processing, instance segmentation refers to the process of object detection, where specific objects in an image are detected (typically by determining a bounding box that contains each of the detected objects and creating a pixel mask for each identified object). Instance segmentation can be considered as object detection where the output is a pixel mask instead of just a bounding box. Thus, unlike semantic segmentation, which aims to categorize each pixel in an image, instance segmentation aims to label the pixels within the determined bounding box. Recently, fast and reliable instance segmentation for 2D camera images based on the so-called Mask R-CNN deep learning method is gaining increasing application in solving real-world problems. However, in many applications, such as autonomous driving, robotics and certain medical applications, the sensor information that needs to be analyzed represents a 3D scene, not a 2D scene. These 3D applications generally utilize optical scanners, e.g. laser scanners such as LiDAR used in surveying applications and intraoral scanners used in dentistry, which generate heterogeneous 3D volumetric data in the form of point clouds. These data are not organized in the form of a homogenous grid of data, such as pixels or, in the case of non-optical 3D scanners, e.g., CT scanners, voxels.

[0003] Data collection methods based on optical scanners typically generate 3D volume data in the form of a point cloud dataset, or point cloud for short. The data points of a point cloud may represent the surface of an object. In general, a point cloud includes a large number of points that are unevenly distributed in 3D space. The 3D space may include areas of densely distributed data points, areas of sparsely distributed data points, and areas that have no data points at all, e.g., voids "inside" the object. The term point cloud may refer to any type of 3D dataset in which each point can be represented as a vector in 3D space. These points may be associated with further attributes, e.g., color, etc. Special types of point clouds include 3D surface definitions, such as triangular or polygonal meshes.

[0004] Although point cloud-based 3D analysis is a fast-growing technology area, methods for 3D object detection and instance segmentation are still in their infancy when compared to their 2D counterparts. Currently, only a few sources are known to address 3D instance segmentation. Qi et al., in their paper "Frustum pointnets for 3D object detection from RGB-D data," IEEE Conference on Computer Vision and Pattern Recognition, pp. 918-927 (2018), describe a high-grid framework that requires two stages: in the first stage, 2D bounding boxes of objects in 2D images are detected, and in the second stage, the 3D point cloud is processed in a 3D search space, partially bounded by the 2D bounding boxes. Similarly, Hou et al. in their paper "3D-SIS: 3D semantic instance segmentation of RGB-D scans", arXiv preprint arXiv:1812.07003 (2018) describe a model in which a first 2D image is processed by a 2D convolutional network. The learned features are then backprojected onto the voxelized point cloud data, where the extracted 2D features and geometric information are combined to obtain object proposals and voxel-wise mask predictions. The dependency of the above mentioned models on 2D images and voxelization limits the performance of such methods.

[0005] In a different approach, Yi et al., in their paper "Generative shape proposal network for 3D instance segmentation in point cloud," arXiv preprint arXiv:1812.03320 (2018), describe an analysis-by-synthesis strategy in which a conditional variational auto-encoder (CVAE) is used instead of directly determining object bounding boxes in the point cloud. However, GSPN training requires a fairly complex separate two-stage training of a CVAE part- and region-based network (which performs classification, regression, and mask generation on the proposals).

[0006] In yet another approach, object proposals are determined based on a clustering scheme. Wang et al., in their paper "Similarity group proposal network for 3d point cloud instance segmentation," IEEE Conference on Computer Vision and Pattern Recognition. pp. 2569-2578 (2018), describe the use of a similarity matrix between features of each pair of points in the embedded feature space to indicate whether a given pair of points belongs to the same object instance. However, such pairwise distance calculations are impractical for large point clouds. Similarly, Liu et al., in their paper "Masc: Multi-scale affinity with sparse convolution for 3D instance segmentation," arXiv preprint arXiv:1902.04478 (2019), describe a voxelization of point clouds for processing volumetric data with a 3D U-Net model and a clustering scheme to find similarities between each pair of points by comparing their extracted features in several hidden layers of the trained U-Net. Voxelization and similarity calculations on large amounts of fine-detailed point clouds severely limit the performance of such methods. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "Frustum pointnets for 3D object detection from RGB-D data" IEEE Conference on Computer Vision and Pattern Recognition. pp. 918-927 (2018) [Non-Patent Document 2] "3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans", arXiv preprint arXiv:1812.07003 (2018) [Non-Patent Document 3] "Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud", arXiv preprint arXiv:1812.03320 (2018) [Non-Patent Document 4] "Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation", IEEE Conference on Computer Vision and Pattern Recognition. pp. 2569 - 2578 (2018) [Non-Patent Document 5] "Masc: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation", arXiv preprint arXiv:1902.04478 (2019) [Non-Patent Document 6] Paper by Li et al., "PointCNN: Convolution on χ-transformed Points", arXiv:1801.07791v5 on November 5, 2018 [Non-Patent Document 7] PointNet (Qi, C.R. et al.: Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE 1(2), 4 (2017)) [Non-Patent Document 8] Qi, Charles Ruizhongtai et al.;PointNet++:Deep hierarchical feature learning on point sets in a metric space; Advances in Neural Information Processing Systems.2017 [Non-Patent Document 9] PointGrid (Le, T. et al.: Pointgrid: A deep network for 3d shape understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 9204-9214 (2018)) [Non-Patent Document 10] MCCNet (Hermosilla, P. et al.: Monte carlo convolution for learning on non-uniformly sampled point clouds. In: SIGGRAPH Asia 2018 Technical Papers. 235 pages ACM (2018)) [Non-Patent Document 11] PointCNN (Li, Y. et al.: Pointcnn. arXiv preprint arXiv:1801.07791(2018)) [Non-Patent Document 12] SpiderCNN (Xu, Y et al., SpiderCNN: Deep learning on point sets with parameterized convolutional filters, ECCV 2018) [Non-Patent Document 13] Hermosilla et al., "Monte Carlo convolution for learning on non-uniformly sampled point clouds," ACM Transactions on Graphics, Volume 37, No. 6, Paper 235, November 2018. [Non-Patent Document 14] Shaoqing He, K et al., "Mask r-cnn," Proceedings of the IEEE international conference on computer vision, pp. 2961-2969 (2017) Summary of the Invention [Problem to be solved by the invention]

[0008] It follows from the above that a need exists in the art for improved instance segmentation of 3D point clouds. In particular, a need exists for a method and system that enables accurate, fast, and computationally efficient instance segmentation of 3D point clouds. [Means for solving the problem]

[0009] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Thus, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may generally be referred to herein as a "circuit," "module," or "system." Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon, e.g., stored thereon.

[0010] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that may contain or store a program for use by or in combination with an instruction execution system, apparatus, or device.

[0011] A computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but may be any computer-readable medium that can communicate, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device.

[0012] Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wireline, fiber optic, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including functional or object-oriented programming languages, such as Java™, Scala, C++, Python, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may run completely on the user's computer, as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or completely on a remote computer, server, or virtualization server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider).

[0013] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, specifically a microprocessor or central processing unit (CPU), or a graphics processing unit (GPU), to generate a machine such that the instructions, which execute via a processor of the computer, other programmable data processing device, or other device, produce means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0014] These computer program instructions may be stored in a computer-readable medium, where the instructions stored in the computer-readable medium product may direct a computer, other programmable data processing apparatus, or other device to function in a particular manner to produce an article of manufacture including instructions that implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. The computer program instructions may be loaded onto the computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process such that the instructions executing on the computer or other programmable apparatus provide a process for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0015] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions mentioned in the blocks may occur out of the order mentioned in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, depending on the functionality involved, or these blocks may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or a combination of dedicated hardware and computer instructions.

[0016] This application describes a method and system for object detection and instance segmentation schemes for 3D point clouds. A point cloud represents a data set that defines at least the geometric location of points in the point cloud (such as the Cartesian coordinates of the points). In addition, in some embodiments, the points may be associated with other attributes as well, such as color and normal vectors. Thus, the points of the point cloud may define a vector of information, including at least a location in 3D space. The point cloud may be generated by scanning a given object using a 3D optical scanner. Such an object may be, for example, a dental-maxillofacial structure, which may include a dentition including teeth.

[0017] In one aspect, the invention relates to a method of object detection in a point cloud, preferably a point cloud generated by a 3D optical scanner such as an intraoral scanning (IOS) point cloud, comprising the steps of determining, by a deep neural network of a first type, a first feature associated with the points of the point cloud, the point cloud comprising points representing one or more objects in at least a 3D space of the point cloud, the first feature defining geometric information for each point of the point cloud, the first type of deep neural network being configured to receive as input the points of the point cloud, and determining, by a deep neural network of a second type based on the first feature, a second feature defining local features for the point cloud at the locations of the nodes of a uniform 3D grid. The method may include determining a feature set of a 3D grid of 3D anchors based on the second features, the feature set defining geometric information of the 3D grid of 3D anchors, the nodes being uniformly distributed in the 3D space of the point cloud; generating one or more object proposals based on the second features, the object proposals defining 3D bounding boxes centered on the nodes of the 3D grid, the 3D bounding boxes containing points of the point cloud that may define the object, the 3D bounding boxes defining 3D anchors; and determining a score for the 3D anchors by a third type of deep neural network, the score indicating a probability that the 3D anchor contains a point that defines the object or a portion of the object, the determining being based on the second features located within the 3D anchors.

[0018] This method provides an accurate and efficient way to detect objects in a point cloud. The process is applied directly to the point cloud (input data) so that all geometric information embedded in the point cloud can be used for object detection. Furthermore, the process determines objects in the point cloud by evaluation of features in a new domain, i.e., a grid domain, without the need to classify each point in the point cloud. In this way, the process provides a very efficient way to detect (predict) whether a point representing a given object is present in the point cloud.

[0019] In one embodiment, a second type of deep neural network may also be trained to determine a score for a node of a uniform 3D grid, the score indicating the probability that the 3D anchor contains a point that defines an object or part of an object.

[0020] In one embodiment, the first point cloud features may include first feature vectors, each first feature vector associated with a point of the point cloud, and / or the second point cloud features may include second feature vectors, each second feature vector associated with a node of the uniform 3D grid.

[0021] In one embodiment, the first point cloud features may be determined by a first type of deep neural network, where the first type of deep neural network defines a feature extraction network.

[0022] In one embodiment, a first type of deep neural network may be configured to receive points of the point cloud and generate first features, preferably first feature vectors, associated with the points of the point cloud.

[0023] In one embodiment, the first type of deep neural network may include multiple convolutional layers including a multi-layer perceptron (MLP), and the feature extraction network is configured to receive the points of the point cloud at its input and generate a feature vector for each point of the point cloud at its output.

[0024] In one embodiment, the feature extraction network may include one or more χ-Conv layers, each configured to weight and permute points and corresponding features provided to the input of the χ-Conv layer, and then apply a convolution kernel to the permuted points and features, and preferably, the feature extraction network is configured as a PointCNN including a χ-Conv layer.

[0025] In one embodiment, the second type of deep neural network may represent an object proposal network, the object proposal network including a plurality of convolutional layers, each of the plurality of convolutional layers including a multi-layer perceptron (MLP) including one or more convolutional kernels.

[0026] In one embodiment, at least one of the multiple convolutional layers may be configured to receive a first feature and nodes of the uniform 3D grid and determine a second feature based on the first feature.

[0027] In one embodiment, the uniform 3D grid may be determined based on the spatial distribution of points in the 3D space of the point cloud.

[0028] In one embodiment, the object proposal network is configured as a Monte Carlo Convolutional Network (MCCNet) including multiple Monte Carlo (MC) spatial convolutional layers, preferably including a convolution kernel configured to determine a convolution at a location of a node x located in the 3D space of the point cloud.

[0029] In one embodiment, determining the convolutions includes determining neighboring points y within a receptive field r, where the receptive field defines a field of view (FOV) of the convolution kernel; determining a probability density function p(x,y) for each neighboring point y; and determining the convolutions at the nodes based on Monte Carlo estimation using the neighboring points y and the probability density values ​​p(x,y) for each neighboring point.

[0030] In one embodiment, the third type deep neural network may represent an object classification network, where the third type deep neural network includes a plurality of fully connected (FC) multi-layer perceptron (MLP) layers, and the second type deep neural network is configured to receive features associated with the 3D anchors and use those features to determine a score associated with the 3D anchors, where the score indicates a probability that the 3D anchors include a point that defines an object or a portion of an object.

[0031] In one aspect, the present invention provides a computer system adapted for object detection in a point cloud, preferably a point cloud generated by a 3D optical scanner, such as an intraoral scanning (IOS) point cloud, comprising: A method and apparatus for processing a point cloud comprising: determining, by a first type of deep neural network, a first feature associated with a point of the point cloud, the point cloud including points representing one or more objects in a 3D space of the point cloud, the first feature defining geometric information for each point of the point cloud, the first type of deep neural network being configured to receive as input the points of the point cloud; and determining, by a second type of deep neural network, a second feature based on the first feature. the second features define local geometric information about the point cloud at locations of nodes of the uniform 3D grid, the nodes being uniformly distributed within the 3D space of the point cloud; generating one or more object proposals based on the second features, the object proposals defining 3D bounding boxes centered on nodes of the uniform 3D grid, the 3D bounding boxes containing points of the point cloud that may define an object, the 3D bounding boxes defining 3D anchors; and determining, by a third type of deep neural network, a score for the 3D anchor, the score indicating a probability that the 3D anchor contains a point that defines an object or a portion of an object, the determining being based on the second features located within the 3D anchor.

[0032] In one aspect, the invention provides a computer system adapted for instance segmentation of a point cloud, preferably a point cloud generated by a 3D optical scanner, such as an intraoral scanning (IOS) point cloud, comprising: a computer readable storage medium having embodied thereon computer readable program code including a pre-processing algorithm and at least a trained first 3D deep neural network; and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein in response to execution of the first computer readable program code, the processor determines, by the first type of deep neural network, first features associated with points of the point cloud, wherein the point cloud includes points representing one or more objects in a 3D space of the point cloud, the first features defining geometric information for each point of the point cloud, and the first type of deep neural network is configured to receive as input the points of the point cloud. determining, by a second type of deep neural network, second features based on the first features, the second features defining local geometric information about the point cloud at locations of nodes of a uniform 3D grid, the nodes being uniformly distributed in the 3D space of the point cloud; generating an object proposal based on the second features, the object proposal defining a 3D volume including points that may define an object, the 3D volume of the object proposal defining 3D anchors centered on the nodes of the 3D grid; determining, by a third type of deep neural network, classified 3D anchors, the determining being based on a second feature set, the second feature set being a subset of the second features located within the 3D anchors; determining, by a fourth type of deep neural network (object location predictor network), object volume (object location predictor network). determining a center location of the object volume that matches the center location of the object instance and the dimensions of the object volume match the outer dimensions of the object instance; The present invention may relate to a computer system configured to perform executable operations including determining, based on a set of features, and determining classification points including first classification points belonging to the object instance and second classification points not belonging to the object instance based on the set of points and the set of first point cloud features located within the object volume by a fifth type deep neural network (mask predictor network).

[0033] In another aspect, the invention provides a method for training a deep neural network system for object detection in a point cloud, preferably an intra-oral scan (IOS) point cloud, comprising the steps of providing training point cloud samples including one or more labeled object instances to an input of a deep neural network system, the deep neural network system including at least a feature extraction network, an object proposal network, and an object classification network; and computing an object proposal based on second features, the second features defining local geometric information about the point cloud at locations of nodes of a uniform 3D grid, the nodes being uniformly distributed in the 3D space of the point cloud, the second features being determined by the object proposal network, the object proposal network being configured to determine the second features based on the first features generated by the feature extraction network receiving the training point cloud samples at its input, and computing an object proposal based on second features generated by the object proposal network. determining an overlap between the 3D anchors and 3D bounding boxes of labeled object instances in the training point cloud samples, and labeling the 3D anchors as positive if the overlap is above a predetermined threshold and as negative if the overlap is below a predetermined threshold; determining one or more object predictions for the positively and / or negatively labeled 3D anchors by the object classification network using point cloud features in the 3D anchors, and determining a first loss value based on the one or more object predictions, the positively and / or negatively labeled 3D anchors, and a first loss function; and using the first loss value to simultaneously train the feature extraction network, the object proposal network, and the object classification network using a back-propagation method.

[0034] In yet another aspect, the invention provides a method for training a deep neural network system for instance segmentation of a point cloud, preferably a point cloud generated by a 3D optical scanner such as an intraoral scanning (IOS) point cloud, comprising the steps of providing training point cloud samples including one or more labeled object instances to an input of a deep neural network system, the deep neural network system including at least a feature extraction network, an object proposal network, an object classification network, an object location predictor network, and a mask predictor network; and computing an object proposal based on second features, the second features defining local geometric information about the point cloud at locations of nodes of a uniform 3D grid, the nodes being uniformly distributed in the 3D space of the point cloud, the second features being determined by the object proposal network, the object proposal network being computed by the feature extraction network that receives at its input the training point cloud samples. determining an overlap between the 3D anchors and the 3D bounding boxes of the labeled object instances in the training point cloud samples, and determining a positive 3D anchor if the overlap is above a predetermined threshold, and determining a negative 3D anchor if the overlap is below a predetermined threshold; determining one or more object predictions for the positively and negatively labeled 3D anchors by an object classification network using the point cloud features in the 3D anchors, and determining a first loss value based on the one or more object predictions and a first loss function; determining by an object location predictor network a location and size prediction of an object volume based on the features in the 3D anchors, and using the location and size predictions and the second loss function to determine a second loss contribution; determining by a mask predictor network a location and size prediction of an object volume based on the features in the 3D anchors, and using the location and size predictions and the second loss function to determine a second loss contribution;using the classification points and a third loss function to determine classification points, including a first classification point belonging to the object instance and a second classification point not belonging to the object instance, and to determine a third loss contribution; and using the first, second, and third loss contributions to train a feature extraction network, an object proposal network, an object classification network, an object location predictor network, and a mask predictor network using a backpropagation method, preferably simultaneously. The first point cloud feature and the second 3D grid feature may be learned implicitly simultaneously through a process of training the system of neural networks.

[0035] The method and system described in this application are based on an object detection and instance segmentation model (scheme) sometimes called Mask-MCNet. Mask-MCNet enables accurate and efficient object detection and instance segmentation in 3D point clouds, e.g., intraoral scan (IOS) data generated by an intraoral scanner. In contrast to known deep learning models, this model does not require a voxelization step to process the point cloud. As a result, the data can be processed while preserving the fine geometric information of the point cloud, which is important for successful segmentation of detailed structures. Furthermore, by extracting and transforming first point cloud features into second point cloud features on a uniform 3D grid, Mask-MCNet can efficiently handle the processing of highly heterogeneous point cloud data, allowing fast generation of object proposals in the point cloud. Such properties are important for the scalability of the scheme to large point cloud data structures (e.g., more than 100,000 points). Experimental results show that Mask-MCNet achieves an IoU score of 98% on the test data, thereby outperforming state-of-the-art networks in point cloud segmentation tasks. Mask-MCNet's performance is close to human level, and complete point cloud object detection and segmentation can be obtained with just a few seconds of processing time, whereas a full point cloud object detection and segmentation is a lengthy and labor-intensive task for humans.

[0036] In some embodiments, the system may include three modules, each of which may include one or more sub-networks. The first module (feature extraction) may include a deep neural network trained to transform the geometric location of the input points set in a high-dimensional feature space. The high-dimensional feature vectors so obtained for each point may be transferred to a 3D grid spanning the entire 3D space. Such a transformation is performed by a second module (object proposal), which may include a Monte Carlo convolutional network or another suitable network architecture. This network may be trained to distribute and transform the information contained in the feature vectors of each point of the irregular point cloud (obtained from the first module) into a regular grid area. A third module, which may include two sub-networks, may be employed to detect whether an object (e.g., a tooth) is present within each candidate bounding box (also called an anchor). All features for the nodes of the grid encompassed by the candidate anchors are examined by a classification sub-network to detect anchors that have a high degree of overlap with objects in the 3D space. For anchors correctly detected by the classification sub-network, another sub-network (Object Location Predictor Network) is used to estimate the difference values ​​of the candidate anchor bounding boxes and the centers of objects in the candidate anchor bounding boxes compared to their ground truth. After localizing each object, a mask generator network, which may consist of an MLP-based cascade network, is employed for binary classification of all points from the input point cloud inside the detected bounding boxes. Such a classification task aims to find the points belonging to each object that are completely encompassed by the detected bounding box. Thus, the model is capable of firstly detecting all objects in the input point cloud by fitting a 3D bounding box centered on the center of each object and secondly indicating all points belonging to each individual tooth inside each detected bounding box.

[0037] Although the embodiments of the present application identify and describe (functionally) separate network components, such as separate deep learning networks, it is alternatively proposed that in other embodiments, a combination of these separate network components may be considered as a single combined network. The present invention will be further described with reference to the accompanying drawings, which show, in a schematic manner, embodiments according to the present invention. It will be understood that the present invention is not limited in any way to these particular embodiments. [Brief description of the drawings]

[0038] [Figure 1] FIG. 1 is a diagram of a deep learning system for object detection and instance segmentation of 3D point clouds, in accordance with an embodiment of the present invention. [Figure 2A] FIG. 2 illustrates a scheme for object detection and instance segmentation of 3D point clouds according to one embodiment of the present invention. [Figure 2B] FIG. 2 illustrates a scheme for object detection and instance segmentation of 3D point clouds according to one embodiment of the present invention. [Figure 2C] FIG. 2 illustrates a scheme for object detection and instance segmentation of 3D point clouds according to one embodiment of the present invention. [Figure 2D] FIG. 2 illustrates a scheme for object detection and instance segmentation of 3D point clouds according to one embodiment of the present invention. [Diagram 3] FIG. 1 illustrates a deep learning system for object detection and instance segmentation of 3D point clouds in accordance with another embodiment of the present invention. [Figure 4A] 1 is a flow diagram of a process for object detection and instance segmentation of point clouds in accordance with various embodiments of the present invention. [Figure 4B] 1 is a flow diagram of a process for object detection and instance segmentation of point clouds in accordance with various embodiments of the present invention. [Diagram 5] FIG. 2 is a schematic diagram of a portion of a feature extraction network, according to one embodiment of the present invention. [Figure 6]FIG. 4 is a schematic diagram of a portion of a feature extraction network according to another embodiment of the present invention. [Figure 7] 1 is a schematic diagram of an object proposal network, according to one embodiment of the present invention; [Figure 8A] FIG. 2 is a schematic diagram of a portion of a Monte Carlo convolutional network, according to one embodiment of the present invention. [Figure 8B] FIG. 2 is a schematic diagram of a portion of a Monte Carlo convolutional network, according to one embodiment of the present invention. [Figure 9] FIG. 2 is a schematic diagram of an object classification network, according to one embodiment of the present invention. [Figure 10] 1 is a schematic diagram of an object location predictor network according to one embodiment of the present invention; [Figure 11] FIG. 2 is a schematic diagram of a mask predictor network, according to one embodiment of the present invention. [Figure 12A] 1 is a flow diagram for training a deep learning system in accordance with various embodiments of the present invention. [Figure 12B] 1 is a flow diagram for training a deep learning system in accordance with various embodiments of the present invention. [Figure 13] FIG. 1 illustrates a visualization of correctly classified 3D anchors determined by Mask-MCNet, in accordance with an embodiment of the present invention. [Figure 14A] FIG. 2 illustrates an example of segmented IOS data generated by a deep learning system, according to one embodiment of the present invention. [Figure 14B] FIG. 2 illustrates an example of segmented IOS data generated by a deep learning system, according to one embodiment of the present invention. [Figure 14C] FIG. 2 illustrates an example of segmented IOS data generated by a deep learning system, according to one embodiment of the present invention. [Figure 14D] FIG. 2 illustrates an example of segmented IOS data generated by a deep learning system, according to one embodiment of the present invention. [Figure 14E]FIG. 2 illustrates an example of segmented IOS data generated by a deep learning system, according to one embodiment of the present invention. [Figure 14F] FIG. 2 illustrates an example of segmented IOS data generated by a deep learning system, according to one embodiment of the present invention. [Figure 14G] FIG. 2 illustrates an example of segmented IOS data generated by a deep learning system, according to one embodiment of the present invention. [Figure 14H] FIG. 2 illustrates an example of segmented IOS data generated by a deep learning system, according to one embodiment of the present invention. [Figure 15] 4 is a flow diagram for post-processing a segmented point cloud according to one embodiment of the present invention. [Figure 16] FIG. 1 is a block diagram illustrating an exemplary data processing system that may be used to execute the methods and software products described in the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] The embodiments described in this disclosure include systems and computer-implemented methods for accurate and efficient object detection and instance segmentation of irregular (heterogeneous) 3D point clouds based on one or more deep neural networks (DNNs). A point cloud may refer to a set of sampling points that define a 3D representation of one or more objects or a scene containing one or more objects. Each sampling point (short, point) may be represented by a vector in a 3D Cartesian coordinate system that is not universal (i.e., the Cartesian coordinate system may differ between the two IOS datasets). A point cloud may be constructed as a 3D surface mesh, where the points in the 3D space may define a triangular or polygonal mesh that collectively describes the representation mesh in the 3D space. Normal vectors associated with these meshes may define an orientation in the 3D space. A normal vector associated with each point in the point cloud may represent a vector perpendicular to the surface consisting of each point and its neighbors. In contrast to Euclidean 2D and 3D datasets, such as pixel and voxel representations, point clouds are irregular, permutation-invariant, and may have a variable number of points per scan. In this application, the term point cloud is used to refer to any non-Euclidean 3D dataset for representing an object in 3D space based on points in the 3D space.

[0040] FIG. 1 shows a schematic diagram of a deep learning system for object detection and instance segmentation of point clouds according to an embodiment of the present invention. Specifically, FIG. 1 shows a system 100 for instance segmentation of 3D point clouds. The process of object detection and instance segmentation of 3D point clouds by the system is shown diagrammatically in FIGS. 2A-2D. The system may be implemented on one or more computers with one or more processing units 101 connected to a memory 103. The system may be configured to execute multiple units, including 3D deep neural networks that may be trained to receive (parts of) the point cloud and possibly spatial information related to the point cloud at their input and process the data according to a trained model. The system may be implemented as a standalone system (e.g., a server system or a network application such as a cloud application) connected to one or more storage units, e.g., a data storage and retrieval system, e.g., a database system, including a database.

[0041] As shown in Fig. 1, a deep learning system may include multiple units, including at least a feature extractor 104, an object detector 106, and a mask generator 108. The feature extractor may include a first deep neural network configured to receive at its input an irregular 3D point cloud 102. A schematic diagram of a point cloud representation is shown in Fig. 2A. The point cloud may be represented by one or more (distinct) objects, e.g. a first object 204 representing a sphere. 1~3 and / or a second object 206 representing a cylinder 1、2The first deep neural network may include an irregular set of points representing at least a portion of a surface of a 3D object. In general, the points may be represented as coordinates in 3D space 202, e.g., Cartesian coordinates x, y, z. The first deep neural network may be configured to determine a feature vector for each point of the point cloud. In one embodiment, the features may be formatted as a feature vector, i.e., a multi-element vector (256 elements in this example). The feature vector associated with a point may describe the spatial correlation between that point and other points in the 3D point cloud, typically other points located within a certain distance from that point. The feature vectors may be used by the object detector and mask generator to efficiently perform instance segmentation of objects in the point cloud.

[0042] The object detector 106 may include a deep neural network system configured to generate an object proposal. The object proposal may include a volume in the 3D space of the point cloud that includes a set of points with a high probability of representing an object. To that end, the deep neural network system may determine a 3D bounding box of a predetermined size and location in the 3D space of the point cloud that has a high probability of containing points that represent the object. The deep neural network system of the object detector may be configured to receive at its input a point cloud (or a subset (e.g., a patch) of the point cloud) and a 3D grid 207. As shown in FIG. 2B, the 3D grid may define a certain density of nodes 207 in the 3D space of the point cloud. The nodes of the 3D grid may define centers for the object proposal in the 3D space. In this manner, the deep neural network system of the object detector may define an object proposal network configured to determine an object proposal, e.g., a 3D bounding box, located on the nodes of the 3D grid, the 3D grid including a plurality of nodes spanning the 3D space of the point cloud. Each of the 3D bounding boxes (sometimes called anchors) is associated with a feature determined based on the feature vector computed by the feature extractor. Thus, given a point cloud as shown in FIG. 2A and a 3D grid of nodes as shown in FIG. 2B, the object detector's deep neural network system may determine 3D bounding boxes, each having a center located on one of the nodes of the 3D grid, each containing features associated with a point in the 3D bounding box. This is shown diagrammatically in FIG. 2C. As shown in this figure, the object detector may determine anchors associated with different objects, e.g., first object 208, second object 209, third object 210, and fourth object 211. 1~3 and a second object 210 associated with 1、2 Generates an anchor related to.

[0043] The object detector may include a further deep neural network configured to classify features located within each of the anchors. This network may define an object classification network trained to receive as its input features associated with points located within the anchors and use these features to determine whether the anchor contains a point representing a given object.

[0044] The mask generator 108 may process points within the anchors that have been classified by the object classification network as containing a given object. The centers and dimensions of the anchors input into the object proposal network are determined by a 3D grid. Thus, the anchors do not necessarily provide the exact location of the center of the object and its dimensions. To generate the exact location of the object, the mask generator may include a deep neural network trained to determine the center and dimensions of a 3D bounding box that contains the points that represent the given object. This neural network may be referred to as an object location predictor network.

[0045] Furthermore, based on the accurately positioned and dimensioned 3D bounding box, the mask generator may select points that are located within the 3D bounding box.

[0046] The points within the 3D bounding box may include points that represent points that are part of the object and background. Classification of the points may be performed using a deep neural network, sometimes referred to as a mask predictor network. The mask predictor network may, for example, classify points as belonging to a predefined object, e.g., one or more different objects. In some embodiments, the mask predictor network may classify points that belong to an unknown object inside the 3D bounding box as background. The points correctly classified by the mask generator network inside each bounding box may be referred to as a "mask." The object classification network, the object location predictor network, and the mask predictor network are described in more detail below.

[0047] FIG. 2D illustrates three segmented sets of points 212, where each set of points represents an instance of a first object (e.g., a spherical object). 1~3 , and two segmented sets of points 214, where each set of points represents an instance of a second object (e.g., a cylindrical object). 1~2 The results of this process are shown below, including:

[0048] The deep learning system described with reference to Figures 1 and 2 is capable of accurate and efficient object detection and instance segmentation of 3D point clouds by using a deep neural network that can process the points of the point cloud directly, without the need for voxelization of the point cloud as known from prior art solutions. The deep neural network may be based on a deep multi-layer perceptron (MLP) network.

[0049] Figure 3 shows a schematic diagram of a deep learning system for object detection and instance segmentation of 3D point clouds according to an embodiment of the present invention. The deep learning system of Figure 3 is configured to apply instance segmentation to a point cloud generated by a 3D optical scanner. Here, instance segmentation refers to the process of assigning a unique label to all points that belong to an instance of an object, e.g., a tooth, using a computational model. In general, the computational model includes one or more trained neural networks.

[0050] In one embodiment, the 3D optical scanner may be an intraoral scanner (IOS) for scanning teeth. The point cloud generated by the IOS may be referred to as an IOS point cloud. Automatic instance segmentation of the IOS point cloud into a set of points representing individual teeth is highly desirable for many applications in dentistry, implantology, and orthodontics. The IOS point cloud may include a large set of points, typically hundreds of thousands of points or more, including high-resolution information about the anatomical structures of the crown and gums (gingiva). In some embodiments, the point cloud may also be converted into a mesh data structure (e.g., after applying a triangulation algorithm to the points).

[0051] As shown in FIG. 3, input data 302, such as an IOS point cloud or a patch of an IOS point cloud, may be provided to a feature extractor 304. A patch of an IOS point cloud may include n points, where n may vary between different patches. The input data may include coordinates 312 that define the points of the point cloud, which may be defined based on a Cartesian coordinate system x, y, z. Additionally, in an embodiment where the points define a 3D surface mesh, the input data may also include normal vectors 314 of the surface defined by the points. The mesh may define a graph that includes several vertices at locations defined by the points of the point cloud. Adjacent vertices may be connected to each other by edges as a result of a triangulation algorithm. Each triangle includes three adjacent vertices in the mesh that may define a surface (sometimes called a "face"). In this way, it is possible to calculate a normal vector oriented perpendicular to such a face. To assign a normal vector to a vertex (point), the average of all normal vectors of the faces (i.e., triangles) to which each vertex contributes may be calculated. Thus, for a patch of n points of a 3D surface mesh, the input data to the feature extraction network may be represented as an n by 6 matrix containing the n point coordinates and the n associated normal vectors. Further generated input data may include the 3D positions of the points and one or more additional attributes (normal vectors, colors, etc.).

[0052] The feature extractor may include a feature extraction network 316 (sometimes referred to as a backbone network). The feature extraction network is a deep neural network that may be trained to receive a point cloud or a patch of a point cloud at its input and determine feature vectors 324 for the points of the point cloud. In one embodiment, the feature extraction network may be trained to determine a feature vector for each point of the point cloud. The feature vector may define multiple features that describe geometric information around each point of the point cloud. For example, in an exemplary implementation, the feature extraction network may generate n feature vectors (where n indicates the number of input points), and each feature vector may include multiple feature elements, e.g., 256 elements.

[0053] In one embodiment, the feature extraction network may be implemented as a deep multi-layer perceptron (MLP), based on a network applied to the entire point cloud (or depending on hardware limitations - patches of the point cloud). In one embodiment, the MLP-based network may be configured as a so-called PointCNN network. The PointCNN network is described in the paper "PointCNN: convolution on χ-transformed points" by Li et al., arXiv:1801.07791v5, November 5, 2018, Neural Information Processing Systems (NIPS) 2018. Many choices for the feature extractor network (backbone network) are possible, but the advantage of the PointCNN network architecture, which will be described in more detail with reference to Figures 5 and 6, is that this architecture allows for the processing of fine details of the point cloud and has a small model size. The output of the feature extraction network may be an n x 256 matrix of features (where n indicates the number of input points). The feature vector associated with a point of the point cloud may contain rich geometric information within the volume around that point.In other embodiments, the following can be used: PointNet (Qi, CR et al.: Pointnet: Deep learning on point sets for 3d classication and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE 1(2), 4 (2017)), Qi, Charles Ruizhongtai et al.; PointNet++: Deep hierarchical feature learning on point sets in a metric space; Advances in Neural Information Processing Systems. 2017, PointGrid (Le, T. et al.: Pointgrid: A deep network for 3d shape understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9204-9214 (2018)), MCCNet (Hermosilla, P. et al.: Monte carlo convolution for learning on non-uniformly sampled point clouds. In: SIGGRAPH Asia 2018 Technical Papers. pp. 235 Other deep neural networks capable of extracting features from point cloud datasets can be used for the feature extraction network, including PointCNN (Li, Y. et al.: Pointcnn. arXiv preprint arXiv:1801.07791 (2018)), and SpiderCNN (Xu, Y. et al., SpiderCNN: Deep learning on point sets with parameterized convolutional filters, ECCV 2018).

[0054] The feature vectors determined by the feature extraction network may be provided to the input of an object detector 306, which includes a so-called object proposal network 318. The object proposal network may be trained to receive the feature vectors 324 from the feature extraction network and a 3D grid of nodes that define so-called anchors in the 3D space of the point cloud (or patches thereof). The 3D grid defines a 3D array of nodes that span the 3D space of the point cloud. The anchors are defined by a central position [x a ,y a ,z a ] and dimensions [w,d,h]. The centers of the anchors may be determined by the nodes of the 3D grid.

[0055] The points of the point cloud generally define "surface points", i.e., samples of the surface of an object as determined by a 3D optical scanner. An example of such a scanner is the IOS scanner. The features computed by the feature extraction network generally contain local geometric information on the manifold of the 3D space. In other words, such rich geometric information may encode the local surface of the object as the local curvature of the surface in 3D space. However, for regression tasks such as accurate localization of a 3D bounding box encompassing an object, the model requires the aggregation of 3D geometric information from the 3D surface mesh into voids in the 3D space of the point cloud, such as voids inside the object. For example, for accurate object determination, the model requires 3D spatial information on different parts (e.g., sides) of the collection of points of the point cloud representing the object. Voxelization of the point cloud and application of 3D CNN to process the voxelized volume data would be a conventional approach to solve this problem. However, as explained earlier, voxelization of non-uniform point clouds severely limits the performance of neural networks because quantization errors from converting point clouds into volumetric data degrade the geometric details.

[0056] To address the shortcomings in the prior art, the object proposal network may be configured to distribute and transfer geometrically rich information from the surface of the object (encoded in the feature vector determined by the feature extraction network) into the entire 3D space (e.g., into the internal voids of the object, such as teeth). In one embodiment, a so-called Monte Carlo Convolutional Network (MCCNet) architecture may be used to implement the object proposal network 318. The MCCNet architecture, which will be described in more detail with reference to Figs. 7 and 8, may include one or more multi-layer perceptron (MLP)-based networks. Specifically, the MCCNet may include several modular MLP sub-networks, including at least two hidden layers with functions similar to a set of convolution kernels. These layers may be referred to as the convolutional layers of the MCCNet. In this way, the MCCNet is capable of computing convolutions for any (new) set of points in the field of view (FOV) of the kernel, regardless of its presence in the set of input points. In other words, a new set of points of location is generated by the output of the convolutional layers. Those locations can be any arbitrary location on the point cloud, even among locations where there are no samples in the point cloud. In this way, MCCNet allows the computation of convolutions of non-uniform distributions of points.

[0057] The MCCNet is configured to transfer the features computed by the feature extraction network from the point cloud domain (features associated with points in the 3D space of the point cloud) to a new domain based on a 3D grid of nodes spanning the 3D space of the point cloud. This new domain may be referred to as a grid domain, and the features are associated with nodes of the 3D grid spanning the 3D space of the point cloud (the input space). In one embodiment, the 3D grid of nodes may be provided to the network as input data. In one embodiment, the grid generator 319 may generate a 3D grid 326 with a constant node density and provide the 3D grid to the input of the object proposal network. The grid generator may be configured to generate the 3D grid based on information of the points of the point cloud. For example, the grid generator may determine the density of the nodes of the 3D grid based on the point cloud provided to the input of the system of FIG. 3. Determining the nodes of the 3D grid based on the point cloud ensures that sufficient details of the feature vectors associated with the points of the point cloud are transferred to the 3D grid domain by the MCCNet.

[0058] Thus, as shown in Figure 3, the object proposal network may receive feature vectors from the feature extraction network and a 3D grid 326 spanning the entire 3D space of the point cloud or at least a substantial portion thereof. Based on this input, the object proposal network may be configured to determine 3D anchors that may be used as object proposals, i.e., volumes of a certain shape, e.g., a 3D bounding box, that contain the points representing the object. The 3D bounding box may have any suitable shape, e.g., rectangular, spherical, cylindrical, etc.

[0059] The object proposal network makes no assumptions about the possible positions of objects in 3D space. In one embodiment, the 3D grid can be a uniform grid that defines nodes uniformly distributed in the 3D space of the point cloud. The spatial resolution of the 3D grid can then determine the performance of the model in terms of accuracy and computational power. Choosing a low-resolution 3D grid can result in placing too few anchors (e.g., incisors in the case of IOS data) inside small objects, while a high-resolution grid can result in computational inefficiencies.

[0060] To improve the accuracy and efficiency of the model, in one embodiment, a non-uniform grid may be provided to the network instead of a uniform grid. The non-uniform 3D grid may include a dense distribution of nodes close to the surface of the object and a sparse distribution of nodes at distances further away from the surface of the object. The non-uniform 3D grid may be determined by a grid generator based on the point cloud fed to the input of the network and an initially dense uniform 3D grid. The non-uniform 3D grid may be obtained by filtering out nodes of the initially dense 3D grid using the distance between the node and the point in the point cloud closest to the node. Additionally, a predefined lower bound on the grid resolution may be used. Nodes from the 3D grid that have a nearest neighbor from the point cloud compared to the threshold are removed if there are any points in this grid that are too far from the threshold.

[0061] The above-mentioned properties of the object proposal network allow the transfer of features from the point cloud domain (features related to the position of points in the 3D space of the cloud) to the above-mentioned grid domain (features related to the position of nodes of a 3D grid spanning the 3D space of the point cloud). In one embodiment, the object proposal network may generate m 3D anchors (e.g., m 3D bounding boxes related to the m nodes of the 3D grid), each having a center that coincides with a node of the 3D grid fed to the input of the object proposal network. Furthermore, for each node of the 3D grid, the object proposal network may determine the geometric information (features) belonging to the 3D anchor that has this node as its center coordinate. The object classification network 322 may determine whether the anchor contains points that represent an object based on the features in the anchor. The evaluation of the features in the anchor by the object classification network may result in classified 3D anchors, e.g., 3D bounding boxes, where each classified 3D anchor contains points that belong to an object. The classified anchors that do not contain an object are discarded.

[0062] The first MLP network from the input of MCCNet may act as a convolutional layer to transfer the features of the point cloud domain to the grid domain. The data is further processed by the hidden layer of MCCNet. Based on the FOV of each convolution kernel, the geometric information of the "surface points" will be distributed across the nodes of the 3D grid. In one embodiment, each node of the 3D grid may represent the location (e.g., center) of one anchor (k=1). In another embodiment, each node of the 3D grid may represent the location of multiple anchors (k>1), and the k anchors associated with the node may have different aspect ratios. Thus, the total number of 3D anchors generated by the object proposal network 318 may be k×m.

[0063] Based on the features within each 3D anchor, the object classification network 322 may predict whether the 3D anchor contains an object. To do so, the object classification network may determine a score for each 3D anchor, which may represent the probability that the 3D anchor does or does not contain an object instance. A threshold may be used to determine whether the score indicates that the 3D anchor contains a point that defines an object.

[0064] The object classification network 322 may be implemented as a fully connected MLP with fixed length input. In that case, the feature set inside each anchor needs to be of fixed length. Thus, an interpolation module 320 between the object proposal network 318 and the object classification network 322 may be used to determine a fixed set of features for each 3D anchor. For example, in one embodiment, the interpolation module may determine (interpolate) that each 3D anchor has s×s×s nodes. The value of s may be between 2 and 12. In this way, each 3D anchor may have a fixed length feature set. For example, an interpolation using the nearest nodes of the grid (e.g., the 3 nearest neighbors) and a weighting of their feature vectors based on their distance to the new node in 3D space may be used. Thus, for k×m 3D anchors determined by the object proposal network, the interpolation module may determine (interpolate) k×m×s nodes. 3 An output matrix of features may be determined.

[0065] If the object detector determines that the point cloud includes one or more 3D anchors that include an object, the mask generator 308 may be triggered to classify points within each of the one or more 3D anchors. The classification process may determine which points within the 3D anchors may represent an object. The object location predictor network may then receive information 340 regarding which 3D anchors are classified as including an object by the object classification network. The classified 3D anchors 332 may be provided to an object location predictor network 334 configured to determine a volume that precisely contains the object. This volume may be referred to as an object volume. The object volume may have a center location that precisely matches the center location of the object. Furthermore, the dimensions of the object volume may precisely match the external dimensions of the object. In one embodiment, the object location predictor network may generate information for relocating and rescaling the classified 3D anchors within the object volume.

[0066] The location and dimensions of the object volumes may be provided to a 3D cropping module 336, which may use the location and dimensions to crop object patches from the point cloud. Each object patch may include points of the point cloud that are located within the object volume. In this manner, each object patch may define a set of points that includes points that represent the object as predicted (detected) by the object detector 306. In one embodiment, the 3D cropping module may use each of the m×k object volumes to crop p points from the point cloud. Furthermore, each of the p points may be associated with one of the n feature vectors generated by the feature extraction network.

[0067] Thus, for each object volume, the 3D cropping module can extract p points from the point cloud, each point associated with a feature vector 324 resulting in an mxkxp feature vector, each of which may have multiple, e.g., 256, feature elements.

[0068] The object patch and associated feature vectors may be provided to a mask predictor network 338 configured to classify points in the object patch into classification points, i.e., points that belong to the object and points that do not belong to the object (e.g., background). This classification process may yield segmented instances of points that represent the object, e.g., segmented tooth instances in the IOS patch 310 as shown in FIG.

[0069] Thus, from the above, three individual MLP-based networks are employed to process the output of the object proposal network. The first deep neural network (Object Classification Network) is responsible for predicting whether each point in the grid region is positive or negative. A node on the grid is considered positive if it belongs to at least one positive anchor. A positive anchor is an anchor that has a high IoU (overlap) with an object in the data. Thus, the first deep neural network is employed for classification of the grid nodes. All points within the positive anchors are processed by two other networks, namely, Object Location Predictor Network and Mask Predictor Network. One of these networks is trained to estimate the spatial displacement between the center of the positive anchor (candidate) and its assigned ground truth center. The second deep neural network (Object Location Predictor Network) is responsible for predicting the difference value between the size of the positive anchor (e.g., width, height, and depth of a 3D cube) and its equivalent value in the ground truth. In total, one of the networks performs classification on the grid nodes, while the next two networks are employed to solve the regression problem. A sub-network is employed to segment all the points that belong to the objects enclosed by a 3D box. This network, called the Mask Predictor network, is responsible for classifying the points inside each 3D bounding box.

[0070] 4A and 4B show process flow diagrams for point cloud detection and instance segmentation according to various embodiments of the present invention. These processes may be performed by modules of a deep learning system as described above with reference to FIGS. 1-3. FIG. 4A shows a process flow diagram for detecting objects in a point cloud by a system of trained neural networks. As shown, the process may include a first step 402 of receiving a point cloud, the point cloud including points representing one or more objects in the 3D space of the point cloud. In a next step (step 404), a first point cloud feature of the point cloud may be determined. In one embodiment, the first point cloud feature may be determined by a first type of deep neural network based on the points of the point cloud. The first point cloud feature may define local geometric information about the point cloud at the location of each point of the point cloud. In one embodiment, the first point cloud feature may define a first feature vector. Each feature vector associated with a point may define geometric information about the space around that point, e.g., the presence or absence of points within a certain volume around that point, the relative distance between those points and the local curvature defined by those points.

[0071] A third step 406 may include transforming the first point cloud features into second point cloud features by the object proposal network, where the second point cloud features may define local geometric information about the point cloud at the location of a node of a 3D grid spanning the 3D space of the point cloud. A further fourth step (step 408) may include generating one or more object proposals based on the second features. The object proposals may define one (or more) 3D bounding boxes centered on a node of the 3D grid, where the 3D bounding boxes may include several points of the point cloud that may define the object. Such 3D bounding boxes generated by the object proposal network may also be referred to as 3D anchors.

[0072] Each of the 3D anchors may be centered on a node of the 3D grid, and a second feature vector associated with each of the 3D anchors may define geometric information within the space defined by the 3D anchors. Determining the second feature vector may include providing the first feature vector and the nodes of the 3D grid to an input of a second deep neural network, and the second deep neural network providing the second feature vector for each of the 3D anchors at its output.

[0073] A fifth step 410 may include selecting a feature set from the second features located in the 3D anchor and determining a score for the 3D anchor based on the feature set by the object classification network. The score may indicate the probability that the 3D anchor contains a point that defines an object or a part of an object.

[0074] The process shown in FIG. 4A provides an accurate and efficient way to predict objects in a point cloud. The determination of the first feature vector may include providing the points of the point cloud to the input of a first deep neural network, and the first deep neural network providing at its output the first feature vector for the points of the point cloud. The process is applied directly to the point cloud (input data) without the need for voxelization or the like, so that all geometric information embedded within the point cloud can be used for object detection. Furthermore, the process determines objects in the point cloud by evaluation of features in a new region, i.e., a grid region, without the need to classify each point in the point cloud. In this way, the process provides a very efficient way to detect (predict) whether a point representing a given object is present in the point cloud.

[0075] FIG. 4B shows a flow diagram of a process for instance segmentation of a point cloud. This process may be performed after the object detection process described with reference to FIG. 4A. The instance segmentation process may begin at step 412 of receiving a set of features contained within a 3D anchor that has been correctly classified by an object classification network to include an object instance. A further step (step 414) may include determining an object volume by an object location predictor network. The object location predictor network may generate a center location of the object instance. Alternatively, the object location predictor network may generate an external dimension of the object instance. In one embodiment, the center location of the object volume may coincide with the center location of the object instance. In another embodiment, the dimensions of the object volume may match the external dimensions of the object instance.

[0076] The location and dimensions of the object volume may be used to crop a set of points from the point cloud and crop a set of point cloud features from the first point cloud features (step 416). Classification points may be determined by a mask predictor network (step 418). Classification points may be determined based on the set of first point cloud features, and may include first classification points that belong to the object instance and second classification points that do not belong to the object instance. The cropped set of points and the matching point features may be input into a mask predictor network that classifies each point as being part of the object instance or not (background) (step 418).

[0077] The process steps illustrated in Figure 4B thus enable efficient instance segmentation of the points of a point cloud by accurately determining a volume in 3D space of the point cloud that contains the points that represent the object. In this way, the process enables segmentation of the points without the need to classify each point of the point cloud as is known from conventional semantic segmentation processes.

[0078] 5 and 6 show schematic diagrams of parts of a feature extraction network according to an embodiment of the present invention. In particular, these figures show diagrams of a deep neural network architecture of a feature extraction network 504 capable of directly processing points of a point cloud 502 that may be represented by n points, each point being represented by a Cartesian coordinate and possibly a normal vector. The deep learning network may be configured as an MLP-based network. In one embodiment, the MLP-based network may have a PointCNN network architecture (already briefly mentioned in the description of FIG. 3). As shown in FIG. 5, the MLP-based network model 504 includes a so-called χ-Conv layer 506. 1~9 Each χ-Conv layer may be configured to weight and permute the input points and their corresponding features prior to processing them through a convolution operation.

[0079] The structure of the χ-Conv layer is explained in more detail in Figure 6. The χ-Conv layer is characterized by parameters N, c, K, and D. Each χ-Conv layer returns N representative points with c channels, where the representative points are the points (representing the locations in the 3D point cloud space, i.e., the input space) where the convolution is computed. Furthermore, the channels represent the dimension of the feature vector. For example, a point associated with a 256-dimensional feature vector has 256 channels. The constant K is the number of neighbors of each representative point, and D is the dilation rate of the χ-Conv layer, i.e., the ratio between the points that are input to the layer and the points generated as output.

[0080] The field of view (FOV) of each χ-Conv layer may include a fixed set of K nearest neighbor (KNN) points 609, i.e., points closest to the location of the point for which the convolution is computed. In addition, a χ-Conv layer may include two or more MLP networks 610, 612. These MLP networks learn a K×K χ-transform on the coordinates of the K input points. The result of the χ-Conv layer is an aggregation and projection of the KNN point features onto a representative set of points, after which the convolution is applied.

[0081] The PointCNN model shown in Figures 5 and 6 is capable of learning local geometric correlations between points in a point cloud. This model has a fairly small number of learning parameters compared to other known deep learning models for point clouds. This is beneficial because it is less prone to significant overfitting to small database sets. The input to the network may be the points of the point cloud 506 (e.g., n points, x, y, z coordinates of a normal vector and each point as an element (possibly) thereby defining an nx6 matrix). The output of the network 510 may be a set of feature vectors (e.g., n vectors, each vector having multiple feature elements, e.g., 256 feature elements), each of which may be associated with a point of the point cloud. The feature vector may be a multi-element vector (in this example, 256 elements), with each element of the vector representing a class probability.

[0082] 7 shows a schematic diagram of an object proposal network according to one embodiment of the present invention. As shown, the architecture of the object proposal network 704 includes a multi-layer stack of MLP layers. The network may be referred to as a Monte Carlo Convolutional Network (MCCNet), which includes a Monte Carlo (MC) spatial convolutional layer 712. 1~4 ,Batch normalization (BN) layer714 1~6 , and a 1x1 convolutional layer 716 1~3 The object proposal network may be configured to receive feature vectors defined within the point cloud domain 702 (i.e., feature vectors associated with a point cloud, each feature vector defining geometric information within a neighborhood of a point of the point cloud) and a 3D grid of nodes 707. In one embodiment, the 3D grid of nodes may be generated by a 3D grid generator 708. In some embodiments, a non-uniform 3D grid may be used as the 3D grid. In that case, the 3D grid generator may be determined by filtering nodes from a dense uniform 3D grid 710. The 1×1 convolutional layer may be inInput Channel and C out These layers can be used to reduce or increase the number of point features.

[0083] As shown in the figure, the first MC spatial convolutional layer 712 1 may be configured to receive feature vectors of the point cloud domain and m nodes of the 3D grid. Based on this input, the convolutional layer converts the feature vectors of the point cloud domain into feature vectors of the grid domain. The input data will be further processed by the hidden layer of the network. Based on the FOV of each convolution kernel, the geometric information of the "surface points" will be distributed through the nodes of the 3D grid, resulting in m feature vectors in the grid domain 706.

[0084] A Monte Carlo convolutional layer is a convolutional layer capable of computing convolutions on unstructured data, such as points in a point cloud. The MC convolutional layer is described in the paper by Hermosilla et al., "Monte Carlo convolution for learning on non-uniformly sampled point clouds," ACM Transactions on Graphics, Vol. 37, No. 6, Paper 235, November 2018. As shown in the figure, an MC convolutional layer may be characterized by four parameters, where the notation A|B|C|D indicates that the layer is configured to map points from level B to level C using a field of view (FOV) equal to D. The constant A determines the dimension of the feature vector at the output of the layer.

[0085] Figure 8 illustrates the operation of a Monte Carlo convolutional layer configured to transfer features f from the point cloud domain to the grid domain. The convolution operation can be defined as the integral of the product of two functions f and g:

[0086]

number

[0087] where function f is the function to be convoluted with R 3 is a scalar function of (in this case the feature vector determined by the feature extraction network), and the function g is 3 , where f is a convolution kernel that is a scalar function over . Specifically, the function f defines a feature function for a set S of discrete samples xi ∈ S (given data points). If no information is provided other than the spatial coordinates of each point, f may represent a binary function that has value "1" at the sample surface and value "0" otherwise if the point is represented by its spatial coordinates. In other variants, the function f may also take into account other types of input information, e.g., color, normals, etc.

[0088] In the case of a hierarchical network structure (e.g., a multi-layer network), the output from the first convolutional layer represents the features that form the input for the subsequent convolutional layers. The value of g depends only on the relative position, so the convolution integral is translation invariant. Evaluating the convolution integral over the entire point cloud can be problematic for large datasets, and therefore the domain of g can be restricted to a sphere of radius 1 centered at 0. To support multiple radii, the input of g can be normalized by dividing it by the receptive field r, i.e., the radius of the spherical field of view (FOV) of the convolution kernel. Specifically, r can be chosen to be a fraction of the scene bounding box diameter b, e.g., r=0.1·b. Since the point cloud data is normalized to have a variance equal to 1, the diameter of the scene boundary is considered to be equal to 1 (b=1). This results in scale invariance. Such relative kernel sizes would render the processing ineffective for input scans with different sizes (diameters). This construction can result in compactly supported kernels that are quick to evaluate.

[0089] A multi-layer perceptron (MLP) network may be used for kernel g similar to the work of Hermosilla et al. cited above. In one embodiment, the MLP network may include three normalized coordinates and divide them by the receptive field r to determine the spatial offset δ=(xy) / r. The output of the MLP network is a single scalar. To balance accuracy and performance, a number of hidden layers, e.g., 2 or 3, may be used, and each hidden layer may include multiple neurons. In one embodiment, the hidden layers may include 6-12 neurons, preferably 8 neurons. The number of parameters in the MLP kernel grows with the dimensions of the input and output vectors. Thus, in one embodiment, a single MLP with multiple outputs (representing different g kernels) may be used to reduce the number of trainable parameters (e.g., by a factor of 8 if the output of the MLP produces 8 different g). Because the output of the MLP network is distinguishable relative to its inputs, a backpropagation algorithm may be used to update the kernel's parameters. In one embodiment, a steepest descent optimization scheme may be used to train the network.

[0090] To calculate the convolution for each sampling point, the convolution integral needs to be evaluated. Since only one set of samples of the feature function f is available, so-called Monte Carlo integration may be used to determine an estimate of the convolution at a point x in the 3D point cloud space. In the Monte Carlo integration approach, a set of random samples may be used to calculate the value of the integral. Based on this approach, an estimate of the convolution for a point x may be given by the following formula:

[0091]

number

[0092] where N(x) is the set of neighborhood indices, i.e., the set of indices belonging to the neighboring points in a sphere of radius r (the receptive field), and plx is the value of the probability density function (PDF) at point yi when point x is fixed (i.e., the convolution is calculated at point x). Here, x is any output point in the 3D space of the point cloud that does not necessarily match a point in the set of input points yi. In one embodiment, the output of the first convolutional layer of MCCNet can be a 3D grid of points (in this application, the points of the 3D grid also refer to nodes to distinguish them from the points of the point cloud). In deeper layers (second and subsequent layers), the output points of each layer can be a subset of the points of the initial 3D grid. These output points are called representative points. This property of Monte Carlo convolution allows for resampling to other levels or other regular regions (e.g., uniform 3D grid of nodes) or non-regular regions (non-uniform 3D grid of nodes).

[0093] If the nodes of the 3D grid are distributed non-uniformly for each point yi, the value of plx will be different and may depend not only on the sample position yi, but also on the radius r of the receptive field, i.e., FOV, and the distribution of other points within the receptive field. Since the sample density is not known for a given point cloud, it may be approximated by a technique called kernel density estimation. The estimation function has a high value if the samples are dense and a low value if the samples are sparse. This function may be calculated based on the following formula:

[0094]

number

[0095] where δ is the bandwidth that determines the smoothing of the obtained sample density function, h is the density estimation kernel, i.e., a non-negative function whose integral is equal to 1 (e.g., a Gaussian function), and d is the R 3where σ is one of the three dimensions of the Gaussian window, and σ is the radius of the Gaussian window. Instead of a Gaussian kernel density estimator, other types of functions may be used, including but not limited to Epanechnikov, Quartic, or tri-cube kernel estimators.

[0096] An example of the application of MC convolution to a point cloud sample at node location x800 in a 3D grid 808 is shown in Figures 8A and 8B. Figure 8A shows a 3D grid of nodes in a 3D input space (input space) of point clouds. For clarity, the points of the point cloud are not shown.

[0097] FIG. 8B shows a more detailed view of the area around node X. The view shows node x and points yi of the point cloud around node x. For clarity, in FIG. 8B, nodes of the 3D grid 808 other than node x are not shown. To calculate the convolution, features f(yi) related to points yi in a spherical field of view (FOV) 802 (with a radius equal to r) are processed by an MLP kernel. For each point yi 804 of the point cloud in the FOV, a point density function (PDF) with a Gaussian window 806 with a radius equal to σ can be calculated locally. It should be noted that instead of a regular 3D grid of nodes, an irregular grid of nodes can be used, for example as described with reference to FIG. 7.

[0098] The PDF of a point yi for a given node x is always relative to all other points in the receptive field. Therefore, we cannot pre-compute the density for a point yi, as its value will be different for each receptive field defined by x and radius r. In case of uniform sampling (e.g., uniform distribution of points, such as voxelization, or a uniform grid), the density function p can be set to a constant value for all given yi points. The output of the convolution computed by the MC convolution layer is distinguishable with respect to the input, and the kernel learning parameters.

[0099] 9-11 show example network architectures for the object classification network, the object location predictor network, and the mask predictor network. As shown in FIGS. 9 and 10, both the classification and location predictor networks are implemented as fully connected (FC) MLP networks configured to receive feature vectors from the object proposal network. Each network includes a batch normalization layer 904. 1~3 , 1004 1~3 FC MLP layer 902, which is separated by 1~4 , 1002 1~4 where the parameter C denotes the number of neurons in each layer of the fully connected network.

[0100] The object classification network of FIG. 9 can receive feature sets 900 from the object proposal network, where each feature set defines features in a 3D anchor. In one embodiment, the feature sets can represent feature matrices. As explained above, each feature set can include a fixed number of features. Based on the feature sets, the object classification network can determine classified 3D anchors 906, i.e., 3D bounding boxes, each centered on a node of the 3D grid. The classified 3D anchors can be associated with a score indicating the probability that the 3D anchor does or does not contain an object instance. In one embodiment, the object classification network can be trained to perform binary classification. In that case, the network can determine whether the 3D anchor contains a point that defines an object instance (e.g., a tooth) or not (background). In another embodiment, the object classification network can be trained to determine different classes of object instances, e.g., an incisor or a molar.

[0101] The location predictor network of FIG. 10 may receive feature sets 900 from the object proposal network of positively classified 3D anchors (i.e., 3D anchors that contain an object) and generate offsets that define the object's center location relative to the 3D anchor's node and dimension offsets (e.g., scaling values) that define the dimensions of a 3D bounding box that accurately contains the object relative to the dimensions of the 3D anchor. Thus, the location predictor network may determine the difference (delta) between the 3D anchor dimensions and the object dimensions and the difference between the 3D anchor's center location and the center of the corresponding object's location. In another embodiment, the location predictor network may determine absolute values ​​of the location and dimensions.

[0102] Figure 11 shows a schematic diagram of a mask predictor network according to an embodiment of the present invention. As described with reference to Figure 3, the location and dimensions of each of the object volumes may be used by the 3D cropping module to crop object patches from the point cloud, each object patch may define a set of points including points representing an object predicted (detected) by the object detector. The 3D cropping module may use each of the mxk object volumes to crop p points from the point cloud, each point being associated with a feature vector, resulting in an mxkxp feature vector (each feature vector having multiple, e.g., 256, feature elements).

[0103] As shown, each of the mxkxq masks, which are constructed by assigning a value of 1 to all points inside the detected bounding box and zero to points outside the box, is passed to the mask generator. Additionally, a first feature vector for all points with a size of nx256 is provided to the mask generator, where q is the number of detected objects (i.e., teeth). The output of the mask generator is a binary classification of the points inside each bounding box. Thus, each of the mxkxpx3 points 1100 and the associated feature vector 1101 of mxkxpx256 may be provided to the input of the network. The mask predictor network may include an MLP-based network. In one embodiment, the MLP-based network may have a PointCNN network architecture (e.g., as described with reference to Figures 5 and 6). Alternatively, the MLP-based network may be based on other neural networks capable of processing point clouds, including (but not limited to) PointNet, PointGrid, MCCNet, PointCNN, PointNet++, SpiderCNN, etc. As shown in FIG. 11, the MLP-based network model 1102 includes a χ-Conv layer 1104. 1~3 , where each χ-Conv layer may be configured to weight and permute the input points and their corresponding features prior to processing by the convolution operator.

[0104] The mask predictor network may perform classification of points within each object patch. In one embodiment, the mask predictor network may perform a binary classification that classifies the points of an object patch into two classes. For example, for IOS data, the binary classification may classify points into a first point, e.g., foreground points that may belong to an instance of a tooth, and a second point, e.g., background points that may belong to other teeth or gums.

[0105] Thus, the deep learning system for object detection and instance segmentation of point clouds described above with reference to Figures 1-11 may provide an accurate and high-performance system for segmenting point clouds without the need for voxelization. At a high level, the deep learning system can be considered as a 3D point cloud analogue of the well-known Mask R-CNN system for instance segmentation of 2D pixel-based images described in the paper by Shaoqing He, K et al., "Mask r-cnn," Proceedings of the IEEE international conference on computer vision, pp. 2961-2969 (2017). Thus, the deep learning system according to the present invention may be referred to as Mask MCNet, having a feature extraction network as a backbone network, an object proposal network as a region proposal network (RPN), and three predictor networks for classification, regression, and mask generation.

[0106] Training of the deep learning system described with reference to Figures 1-11 may be performed using an end-to-end training method, e.g., steepest descent and Adam learning adaptation techniques. For example, 1000 epochs with a batch size of 32 (equally balanced between positive and negative anchors) may be used. Pre-processing of the input may include normalization of the point cloud to obtain zero mean and unit variance.

[0107] The training set may include a set of labeled point cloud samples. For example, in the case of IOS data, the training set may include optical scans of dentition, e.g., 100 or more optical scans of dentition from different adult subjects, each including one maxillary scan and one mandibular scan (e.g., as shown in FIG. 3). The IOS data may be recorded using a 3D scanner, e.g., a 3Shape d500 optical scanner (3Shape AS, Copenhagen, Denmark), including 180,000 points on average (varying in terms of the range interval [100,000, 310,000]). The data set may include scans from healthy dentition with various abnormalities among subjects.

[0108] In one embodiment, the optical scans may be manually segmented, with each of their points labeled and classified into one of 32 classes by a dental professional according to the FDI standard, and reviewed and refined by a dental expert (DAM). Segmenting an optical scan can take an average of 45 minutes, illustrating that point cloud segmentation is a human-intensive and challenging task.

[0109] Loss functions may be used to train the networks of the embodiments described in this application. The loss function of Mask-MCNet may include loss contributions from different networks of Mask-MCNet. In one embodiment, the loss contributions may include a first loss contribution associated with the output of the object classification network, a second loss contribution associated with the object location predictor network, and a third loss contribution associated with the mask predictor network. In one embodiment, the loss contributions may be similar to the loss function used by Shaoqing He, K. et al. (above) for Mask R-CNN, where three loss contributions were used, with equal contributions of each of the three loss contributions.

[0110] In one embodiment, the first loss contribution (classification loss) may define a cross-entropy loss value for the classification branch (object classification network) above its output layer, e.g., a softmax layer. In one embodiment, the first loss contribution may be calculated as follows:

[0111]

number

[0112] In the formula, p i may define the probability that a node of the 3D grid is the center of a 3D anchor with an object (having a high IoU), and p i * is the ground truth selected from the set {0,1}. The parameter N a defines the number of nodes in the 3D grid.

[0113] In one embodiment, the second loss contribution (regression loss) may define the mean squared error at the output layer (preferably, the linear output layer) of the regression branch (object location predictor network). In one embodiment, the second loss contribution may be calculated as follows:

[0114]

number

[0115] In the formula, t i may define a three element vector representing the delta value or dimension of the center position of the 3D anchor, i * is the ground truth. Parameter N p may define the number of positive anchors in the input IOS.

[0116] In one embodiment, the third loss contribution (mask loss) may define a binary cross-entropy loss for classification of points (preferably all points) within a positive anchor (preferably each positive anchor) in the output layer of the mask branch (mask predictor network). The regression loss and mask loss may be taken into account if the 3D anchor is labeled as positive. The mask loss may be calculated based on the following formula:

[0117]

number

[0118] In the formula, p i is the probability that a point belongs to the corresponding object encompassed by the 3D anchor, where p i * is the ground truth value from the set {0,1}. m may define the number of points inside the i-th 3D anchor. During training, the regression loss and mask loss may be used to modify the weights of the object location predictor network and the weights of the mask predictor network, respectively.

[0119] In an exemplary embodiment of the system's training process, a 3D anchor may be labeled as positive if it has an overlap with any tooth instance in the labeled point cloud patch above a first threshold. In one embodiment, the first threshold may be 0.4 IoU, where IoU defines the average Jaccard Index. In another embodiment, a 3D anchor may be labeled as negative if it has an overlap with any tooth instance in the labeled point cloud patch below a second threshold.

[0120] In one embodiment, the second threshold may be 0.2. Since the number of positive and negative 3D anchors is highly unbalanced, about 50% of each training batch may be selected from positive 3D anchors and 25% may be selected from negative 3D anchors. The rest of the 25% sampling 3D anchors in the training batch may be selected from marginal 3D anchors, which are also considered as negative samples, e.g., (0.2 <IoU<0.4)である。

[0121] Furthermore, in one embodiment, during the training phase, the input to Mask-MCNet may be randomly cropped patches of the point cloud. Each cropped patch may contain several object instances, e.g., 2-4 tooth instances. In one embodiment, a uniform grid region may be constructed. In another embodiment, a non-uniform grid region may be constructed by filtering out nodes of a dense regular grid with 0.04 (lower bound) spatial resolution in each dimension. An upper bound on the grid resolution may be set equal to 0.12. 3D anchors of different dimensions may be used to create sufficient overlap between the 3D anchors and both small and large objects (e.g., incisors and molars, respectively). For example, in one embodiment, two types (k=2) of 3D anchor boxes (with sizes of [0.3,0.3,0.2] and [0.15,0.2,0.2]) may be employed.

[0122] 12A and 12B show flow diagrams of training deep learning systems according to various embodiments of the present invention. FIG. 12A shows a flow diagram of training a deep neural network system for object detection in a point cloud. The deep neural network may include a feature extraction network, an object proposal network, and an object classification network according to any of the embodiments described in this application. As shown in the figure, in a first step (step 1202), a training point cloud sample including one or more labeled object instances may be provided to an input of a deep neural network system. The deep neural network system may include at least a feature extraction network, an object proposal network, and an object classification network configured for object detection in a point cloud. A further step (step 1204) may include a step of calculating an object proposal by the object proposal network based on the point cloud features determined by the feature extraction network. Here, the object proposal may define a 3D bounding box that may include the points that define the object. Such a 3D bounding box may be referred to as a 3D anchor. The training process may further include determining an overlap between the 3D anchors and 3D bounding boxes of labeled object instances in the training point cloud samples (step 1206). The 3D anchors may be labeled as positive if the overlap is above a predetermined threshold and may be labeled as negative if the overlap is below a predetermined threshold. Features in the 3D anchors may then be used to determine an object prediction for the 3D anchor by an object classification network (step 1208).

[0123] Thus, both the positively labeled 3D anchors and the negatively labeled 3D anchors may be used for training the object classification network. The object classification network may predict whether the 3D anchor should be labeled positive (containing an object) or negative (containing no object). During training, a first loss function is used in each case to determine the contribution (i.e., if the prediction was correct, the loss will be of value 0 (no correction), but if the prediction was incorrect (positive but negative prediction, or vice versa), the loss value will be positive (and correction of the network will be performed)). The object prediction and the loss function may be used to obtain a loss value. Finally, the loss value may be used to train the feature extraction network, the object proposal network, and the object classification network using the backpropagation method (step 1210).

[0124] 12B shows a flow diagram of training a deep neural network system for point cloud instance segmentation according to an embodiment of the present invention. In step 1222, a training point cloud sample including one or more labeled object instances may be provided to an input of a deep neural network system, where the deep neural network system may be configured for point cloud instance segmentation. Such networks may include at least a feature extraction network, an object proposal network, an object classification network, an object location predictor network, and a mask predictor network.

[0125] The training process may include computing an object proposal by an object proposal network based on the point cloud features determined by the feature extraction network (step 1224), where the object proposal may define a 3D bounding box that may contain the points that define the object. Such a 3D bounding box may define a 3D anchor. In a further step (step 1226), an overlap between the 3D anchor and the 3D bounding box of a labeled object instance in the training point cloud sample may be determined, where the 3D anchor may be labeled as positive if the overlap is above a predetermined threshold and may be labeled as negative if the overlap is below a predetermined threshold.

[0126] An object prediction may be determined for 3D anchors labeled positive or negative by an object classification network based on features in the 3D anchor (step 1228). The object classification network predicts whether the 3D anchor should be labeled as positive (containing an object) or negative (not containing a (complete) object). During training, a first loss function is used in each case to determine the contribution (i.e., if the prediction was correct, the loss will be of value 0 (no correction), but if the prediction was incorrect (it was positive but a negative prediction, or vice versa), the loss value will be positive (and a correction of the network will be made)). In this way, the difference between the object prediction and the ground truth (labeled object) may be used to determine a first loss contribution based on the first loss function.

[0127] A location and size prediction of the object volume may be determined by the object location predictor network based on features in the 3D anchors (step 1230). Here, the object volume may be a 3D bounding box with a center that should match the center of the positively classified 3D anchor and with dimensions that should closely match the dimensions of the object instance. The difference between the predicted location and size and the ground truth (labeled location and size) may be used to determine a second loss contribution based on a second loss function. The larger the difference, the larger the loss contribution.

[0128] Classification points may be predicted by a mask predictor network based on a first point cloud feature in the object volume, and the classification points may include a first classification point that belongs to the object instance and a second classification point that does not belong to the object instance (step 1232). The difference between the predicted point classification and the ground truth (classification points to be labeled) may be used to determine a third loss contribution based on a third function. The first, second, and third loss contributions may be used to train a feature extraction network, an object proposal network, an object classification network, an object location predictor network, and a mask predictor network using a backpropagation method (step 1234).

[0129] Inference on a new point cloud, e.g., a new IOS dataset, can be performed by applying Mask-MCNet to several cropped overlapping patches. Given a 3D patch and applying a uniform 3D grid of nodes with the highest specified resolution (e.g., 0.04), 3D anchors located on the nodes of the 3D grid are classified as object-present / object-absent by the classification branch. The dimensions and (center) locations of the positively classified 3D anchors (i.e., detected objects) are updated according to the estimates by the regression branch (i.e., object location predictor network). Figure 13 shows the positively classified 3D anchors 1302 determined by Mask-MCNet. 1、2and a 3D bounding box representing the ground truth,1302 1、2 Figure 1 shows a visualization of the two 3D anchors. As shown, the two 3D anchors have different scales and center locations. A positively classified 3D anchor has an overlap with the ground truth 3D bounding box that is higher than a first threshold (in IoU) for labeling the 3D anchor as positive (i.e., containing (part of) an object).

[0130] Similar to Fast RCNN, for each object, multiple 3D anchors may be detected, and each 3D anchor is associated with a probability score higher than a first threshold. Therefore, a non-maximum suppression algorithm may be adopted according to the maximum objectivity score (from the classification probability). Therefore, based on this algorithm, the 3D anchor with the highest probability score may be determined. Non-maximum suppression also handles repeated points by overlapping input patches. After predicting the bounding boxes of all tooth instances, it is easy to search for masks for all points inside the bounding boxes from the mask generation branch.

[0131] Examples of IOS instance segmentation by Mask_MCNet are visualized in Fig. 14A-H, where Fig. 14A-B show segmentation of a normal dentition; Fig. 14C shows a segmented dentition with a segmented tooth with missing data; Fig. 14D-F shows a segmented dentition with a segmented tooth with anomalies and artifacts; and Fig. 14G-H show two examples of failed segmentation.

[0132] The performance of Mask-MCNet compared to state-of-the-art classification networks is evaluated by five-fold cross-validation. The average Jaccard Index (also called IoU) is used as the segmentation metric. Besides IoU, precision and recall for the multi-class segmentation problem are reported by treating each class individually (one-vs-all) as a binary problem. In addition, the average scores are reported as well. The performance of Mask-MCNet is shown in Table 1. As shown in Table 1, the proposed Mask-MCNet significantly outperforms state-of-the-art networks capable of segmenting point cloud data.

[0133] [Table 1]

[0134] FIG. 15 shows a flow diagram of post-processing of segmented point clouds according to an embodiment of the present invention. For clinical purposes, for consistency of tooth label assignment, a post-processing method can be used to convert the instance labels predicted by Mask-MCNet to FDI standard labels (by a look-up table). By measuring the average center position and size of the FDI standard labels in the training data, a combinatorial search algorithm can determine the most likely label assignments that satisfy predefined constraints (pre-measurements on the training data) in the context of constraint satisfaction problems (CSP). Constraint satisfaction problems for finite domains are generally solved in the form of search. Such search techniques can be variants of backtracking, constraint propagation, or local search.

[0135] 16 is a block diagram illustrating an exemplary data processing system that may be used in the embodiments described in this disclosure. The data processing system 1600 may include at least one processor 1602 coupled to memory elements 1604 through a system bus 1606. Thus, the data processing system may store program code in the memory elements 1604. Additionally, the processor 1602 may execute program code accessed from the memory elements 1604 via the system bus 1606. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be appreciated that the data processing system 1600 may be implemented in any form of system including a processor and memory capable of performing the functions described herein.

[0136] The memory elements 1604 may include one or more physical memory devices, such as, for example, a local memory 1608 and one or more mass storage devices 1610. Local memory may refer to random access memory or generally other non-persistent memory devices used during the actual execution of the program code. The mass storage devices may be implemented as hard drives or other persistent data storage devices. The processing system 1600 may also include one or more cache memories (not shown) that provide temporary storage of at least some of the program code to reduce the number of times the program code must be retrieved from the mass storage device 1610 during execution.

[0137] Input / output (I / O) devices, depicted as input device(s) 1612 and output device(s) 1614, may be coupled to the data processing system in some cases. Examples of input devices may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, etc. Examples of output devices may include, but are not limited to, for example, a monitor or display, speakers, etc. The input and / or output devices may be coupled to the data processing system either directly or through an intervening I / O controller. A network adapter 1616 may also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through an intervening private or public network. A network adapter may comprise a data receiver for receiving data transmitted by the system, device, and / or network, and a data transmitter for transmitting data to the system, device, and / or network. A modem, a cable modem, and an Ethernet card are examples of different types of network adapters that may be used with the data processing system 1600.

[0138] 16, memory element 1604 may store application 1618. It should be appreciated that data processing system 1600 may further execute an operating system (not shown), which may facilitate execution of the application. Applications implemented in the form of executable program code may be executed by data processing system 1600, for example, by processor 1602. Depending on the executing application, the data processing system may be configured to perform one or more operations that will be described in further detail herein.

[0139] In one aspect, for example, data processing system 1600 may represent a client data processing system. In that case, application 1618 may represent a client application that, when executed, configures data processing system 1600 to perform various functions described herein with reference to a "client." Examples of clients may include, but are not limited to, personal computers, portable computers, mobile phones, and the like. In another aspect, the data processing system may represent a server. For example, the data processing system may represent a server, a cloud server, or a system of (cloud) servers.

[0140] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0141] The corresponding structure, material, acts, and equivalents of all means or step-plus-function elements of the following claims are intended to include any structure, material, or acts for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiment has been chosen and described in order to best explain the principles and practical application of the invention, and to enable others skilled in the art to understand the invention in terms of various embodiments, with various modifications suitable to the particular use contemplated. [Explanation of symbols]

[0142] 100 Systems 101 Processing Unit 102 Irregular 3D Point Clouds 103 Memory 104 Feature Extractor 106 Object Detector 108 Mask Generator 202 3D space 204 1~3 First Object 206 1、2 Second Object 207 3D Grid, Nodes 208 1~3 First Object 210 1,2 Second Object 212 1~3 Three segmented sets of points 302 Input Data 306 Object Detector 308 Mask Generator 310 IOS patch 312 Coordinates 314 Normal Vector 316 Feature Extraction Network 318 Object Proposal Network 319 Grid Generator 320 Interpolation Module 322 Object Classification Network 324 feature vectors 326 3D Grid 332 Classified 3D Anchors 334 Object Location Predictor Network 336 3D Cropping Module 338 Mask Predictor Network 340 Information 502 point cloud 504 Feature Extraction Network, MLP-Based Network Model 506 1~9 χ-Conv layer 510 Network 609 K-Nearest Neighbor (KNN) points 610 MLP Network 612 MLP Network 702 Point cloud region 704 Object Proposal Network 706 Grid Area 707 nodes 708 3D Grid Generator 710 Dense Uniform 3D Grid 712 1~4 Monte Carlo (MC) spatial convolution layer 712 1 First MC special convolution layer 714 1~6 Batch Normalization (BN) Layer 716 1~3 1x1 convolutional layer 800 Node Position x 802 Spherical field of view (FOV) 804 points yi 806 Gaussian Window 808 3D Grid 900 feature set 902 1~4 FC MLP layer 904 1~3 Batch normalization layer 906 3D Anchor 1002 1~4 FC MLP layer 1004 1~3 Batch normalization layer 1100 points 1101 related feature vectors 1102 MLP-Based Network Model 1104 1~3 χ-Conv layer 1600 Data Processing System 1602 processor 1604 Memory Elements 1606 System Bus 1608 Local Memory 1610 Mass Storage Device 1612 Input Devices 1614 Output Device 1616 Network Adapter 1618 Applications

Claims

1. A method for object detection in an intraoral scanning (IOS) point cloud generated by a 3D optical scanner, comprising: determining first features associated with points of the IOS point cloud by a first deep neural network representing a feature extraction network, the IOS point cloud includes points representing one or more objects representing one or more teeth in at least a 3D space of the IOS point cloud, the first features define geometric information for each point of the IOS point cloud, and the first deep neural network is configured to receive the points of the IOS point cloud as input; determining, by a second deep neural network representing an object proposal network, second features based on the first features and a 3D grid of nodes in the 3D space of the IOS point cloud; the second features defining local geometric information about the IOS point cloud at the location of the node of the 3D grid in the 3D space of the IOS point cloud; generating one or more object proposals based on the second features, an object proposal defining a 3D bounding box centered on a node of the 3D grid, the 3D bounding box containing points of the IOS point cloud that may define teeth, the 3D bounding box defining 3D anchors; determining a score for the 3D anchor by a third deep neural network representing an object classification network, the score indicating a probability that the 3D anchor includes a point defining a tooth or a portion of a tooth, and the determining step is based on a second feature located within the 3D anchor; A method comprising:

2. The first features include first feature vectors, each first feature vector being associated with a point of the IOS point cloud; and / or The method of claim 1 , wherein the second features include second feature vectors, each second feature vector being associated with a node of the 3D grid.

3. The first deep neural network: receiving points of said IOS point cloud; The method of claim 1 or 2, further comprising defining a feature extraction network configured to generate the first features.

4. the first deep neural network includes a plurality of convolutional layers including a multi-layer perceptron (MLP); The feature extraction network: receiving at its input the points of said IOS point cloud; The method of claim 3 , configured to generate at its output a feature vector for each point of the IOS point cloud.

5. the feature extraction network includes one or more χ-Conv layers; Each χ-Conv layer is weighting and permuting the points and corresponding features provided at the input of the χ-Conv layer; The method of claim 3 or 4, further comprising subsequently applying a convolution kernel to the permuted points and features.

6. the second deep neural network represents an object proposal network; the object proposal network includes a plurality of convolutional layers; 6. The method of claim 1, wherein each of the plurality of convolutional layers comprises a multi-layer perceptron (MLP) including one or more convolution kernels.

7. 7. The method of claim 6, wherein the object proposal network is configured as a Monte Carlo Convolutional Network (MCCNet) including multiple Monte Carlo (MC) spatial convolutional layers.

8. the third deep neural network represents an object classification network; the third deep neural network includes a plurality of fully connected (FC) multi-layer perceptron (MLP) layers; the second deep neural network: receiving features associated with the 3D anchor; configured to use the features to determine a score associated with the 3D anchor; 8. The method of claim 1, wherein the score indicates the probability that the 3D anchor contains a point that defines a tooth or a part of the tooth.

9. 1. A method for instance segmentation of an intraoral scan (IOS) point cloud, comprising: determining first features associated with points of the IOS point cloud by a first deep neural network representing a feature extraction network, the IOS point cloud includes points representing one or more objects representing one or more teeth in a 3D space of the IOS point cloud, the first features define geometric information for each point of the IOS point cloud, and the first deep neural network is configured to receive the points of the IOS point cloud as input; determining, by a second deep neural network representing an object proposal network, second features based on the first features and a 3D grid of nodes in the 3D space of the IOS point cloud; the second features defining local geometric information about the IOS point cloud at the location of the node of the 3D grid in the 3D space of the IOS point cloud; generating an object proposal based on the second feature, an object proposal defining a 3D volume containing points that may define teeth, the 3D volume of the object proposal defining 3D anchors centered on nodes of the 3D grid; determining classified 3D anchors by a third deep neural network representing an object classification network, the determining is based on a second set of features, the second set of features being a subset of the second features located within the 3D anchor; determining an object volume by a fourth deep neural network representing an object location predictor network, a center location of the object volume corresponds to a center location of an object instance, and a dimension of the object volume matches an outer dimension of the object instance, the determining being based on the second set of features; determining, by a fifth deep neural network representing a mask predictor network, a classification point based on a set of points located within the object volume and a set of first point cloud features; said classification points including first classification points belonging to teeth and second classification points not belonging to teeth; A method comprising:

10. the first deep neural network defines a feature extraction network; the feature extraction network includes a plurality of convolutional layers including a multi-layer perceptron (MLP); The feature extraction network: receiving at its input the points of the point cloud; and / or configured to generate at its output a feature vector for each point of the IOS point cloud; the second deep neural network represents an object proposal network; the object proposal network is configured as a Monte Carlo Convolutional Network (MCCNet) including multiple Monte Carlo (MC) spatial convolutional layers; and / or the third deep neural network represents an object classification network; the third deep neural network includes a plurality of fully connected (FC) multi-layer perceptron (MLP) layers; and / or the fourth deep neural network represents an object location predictor network; the fourth deep neural network includes a plurality of fully connected (FC) multi-layer perceptron (MLP) layers; and / or the fifth deep neural network represents a mask predictor network; the fifth deep neural network includes one or more χ-Conv layers; Each χ-Conv layer is weighting and permuting the points and corresponding features provided at the input of the χ-Conv layer; and then applying a convolution kernel to the permuted points and features. The method of claim 9.

11. 1. A method for training a deep neural network system for object detection in an intraoral scan (IOS) point cloud, comprising: providing a training point cloud sample comprising one or more labeled object instances to an input of the deep neural network system, the deep neural network system including at least a feature extraction network, an object proposal network, and an object classification network; calculating an object proposal based on the second features, the second feature defines local geometric information about the IOS point cloud at the location of a node of a 3D grid in the 3D space of the IOS point cloud; the second feature is determined by the object proposal network; the object proposal network is configured to determine the second features based on first features generated by the feature extraction network receiving at its input the training point cloud samples; the object proposal defines a 3D bounding box centered on a node of the 3D grid that may contain points defining a tooth; the 3D bounding box defining a 3D anchor; determining an overlap between the 3D anchor and a 3D bounding box of a labeled object instance in the training point cloud sample, and labeling the 3D anchor as positive if the overlap is above a predefined threshold and as negative if the overlap is below a predefined threshold; determining one or more object predictions for the 3D anchors positively and / or negatively labeled by the object classification network using point cloud features in the 3D anchors, and determining a first loss value based on the one or more object predictions, the positively and / or negatively labeled 3D anchors, and a first loss function; using the first loss value to train the feature extraction network, the object proposal network, and the object classification network using a backpropagation method; A method comprising:

12. 1. A method for training a deep neural network system for instance segmentation of intraoral scan (IOS) point clouds, comprising: providing a training point cloud sample comprising one or more labeled object instances representing one or more teeth to an input of the deep neural network system; the deep neural network system includes at least a feature extraction network, an object proposal network, an object classification network, an object location predictor network, and a mask predictor network; calculating an object proposal based on the second features, the second feature defines local geometric information about the IOS point cloud at the location of a node of a 3D grid in the 3D space of the IOS point cloud; the second feature is determined by the object proposal network; the object proposal network is configured to determine the second features based on first features generated by the feature extraction network which receives at its input the 3D grid of nodes and the training point cloud samples; the object proposal defines a 3D bounding box centered on a node of the 3D grid that may contain points defining a tooth; the 3D bounding box defining a 3D anchor; determining an overlap between the 3D anchor and a 3D bounding box of a labeled object instance in the training point cloud samples, and determining a positive 3D anchor if the overlap is above a predefined threshold, and determining a negative 3D anchor if the overlap is below a predefined threshold; determining one or more object predictions for positively labeled 3D anchors and negatively labeled 3D anchors by the object classification network using point cloud features in the 3D anchors, and determining a first loss value based on the one or more object predictions and a first loss function; determining, by the object location predictor network, a location and size prediction of an object volume based on features in the 3D anchors, and using the location and size prediction and a second loss function to determine a second loss contribution; determining, by the mask predictor network, classification points based on point cloud features in the object volume, the classification points including first classification points belonging to teeth and second classification points not belonging to teeth, and using the classification points and a third loss function to determine a third loss contribution; using the first, second, and third loss contributions to train the feature extraction network, the object proposal network, the object classification network, the object location predictor network, and the mask predictor network using a backpropagation method; A method comprising:

13. 1. A computer system adapted for tooth detection in an intraoral scan (IOS) point cloud, comprising: A computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code including a pre-processing algorithm and at least a trained first 3D deep neural network; and a processor coupled to the computer-readable storage medium. and in response to execution of the computer readable program code, the processor: determining, by a first deep neural network representing a feature extraction network, first features associated with the points of the IOS point cloud; determining that the IOS point cloud includes points representing one or more teeth in a 3D space of the IOS point cloud, the first features defining geometric information for each point of the IOS point cloud, and the first deep neural network configured to receive the points of the IOS point cloud as input; determining, by a second deep neural network representing an object proposal network, second features based on the first features; determining that the second feature of the point cloud defines local geometric information about the IOS point cloud at the location of a node of a 3D grid in the 3D space of the IOS point cloud; generating one or more object proposals based on the second features, generating an object proposal defining a 3D bounding box centered on a node of the 3D grid, the 3D bounding box containing points of the IOS point cloud that may define teeth, the 3D bounding box defining 3D anchors; determining a score for the 3D anchor by a third type of deep neural network representing an object location predictor network; determining, the score indicating a probability that the 3D anchor includes a point defining a tooth or a portion of a tooth, the determining being based on a second feature located within the 3D anchor; 16. A computer system configured to perform executable operations including:

14. 1. A computer system adapted for instance segmentation of an intraoral scan (IOS) point cloud, comprising: A computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code including a pre-processing algorithm and at least a trained first 3D deep neural network; and a processor coupled to the computer-readable storage medium. and in response to execution of the computer readable program code, the processor: determining first features associated with the points of the IOS point cloud by a first type of deep neural network representing a feature extraction network; determining that the IOS point cloud includes points representing one or more teeth in a 3D space of the IOS point cloud, and the first features define geometric information for each point of the IOS point cloud; determining second features based on the first features by a second type of deep neural network representing an object proposal network; determining that the second features define local geometric information about the IOS point cloud at the locations of nodes of a 3D grid spanning the 3D space of the IOS point cloud, and the first type of deep neural network is configured to receive as input points of the IOS point cloud; generating an object proposal based on the second feature, generating an object proposal defining a 3D volume containing points that may define teeth, the 3D volume of the object proposal defining 3D anchors centered on nodes of the 3D grid in the 3D space of the IOS point cloud; determining classified 3D anchors by a third type of deep neural network representing an object classification network, determining, the determining being based on a second set of features, the second set of features being a subset of the second features located within the 3D anchor; determining an object volume by a fourth type of deep neural network representing an object location predictor network, determining that a center location of the object volume corresponds to a central location of an object instance and that dimensions of the object volume match exterior dimensions of the object instance, said determining being based on the second set of features; determining a classification point based on the set of points located within the object volume and the set of first point cloud features by a fifth type of deep neural network representing a mask predictor network; determining, the classification points including first classification points belonging to teeth and second classification points not belonging to teeth; 16. A computer system configured to perform executable operations including:

Images