Object Detection and Instance Segmentation in 3D Point Clouds Based on Deep Learning

Abstract

A method for object detection in a point cloud is described, the method may include: determining a first feature of a point cloud by a first neural network type, the point cloud including points representing one or more objects, the first feature defining geometric information of each point, the first network type being configured to receive the point cloud points; determining a second point cloud feature by a second dnn type using the first feature, the second feature defining local geometric information about the point cloud at locations of nodes of a uniform 3D grid, the nodes being uniformly distributed in the 3D space of the point cloud; generating one or more object proposals using the second feature, the object proposal defining a 3D bounding box positioned around a node of the 3D grid, the 3D bounding box defining a 3D anchor point, and a third dnn type determining a score for the 3D anchor point.

Description

Technical Field

[0001] The present invention relates to object detection and instance segmentation of 3D point clouds based on deep learning, and specifically but not limited to, to methods and systems for object detection in 3D point clouds using deep learning, methods and systems for instance segmentation of 3D point clouds using deep learning, deep neural network systems for object detection in 3D point clouds, deep neural network systems for instance segmentation of 3D point clouds, and computer program products for executing these methods. Background Art

[0002] In image processing, instance segmentation refers to the process of object detection, where specific objects in an image are detected (usually by determining a bounding box that contains each detected object and creating a pixel mask for each identified object). Instance segmentation can be thought of as object detection where the output is a pixel mask, rather than just a bounding box. Therefore, unlike semantic segmentation, which aims to classify each pixel in an image, instance segmentation aims to label the pixels in the determined bounding box. Recently, fast and reliable instance segmentation of 2D camera images based on the so-called Mask R-CNN deep learning scheme has found increasing application in solving real-world problems. However, in many applications such as autonomous driving, robotics, and certain medical applications, the sensor information to be analyzed needs to represent a 3D scene, rather than a 2D scene. These 3D applications rely on information generated by optical scanners (e.g., laser scanners, such as LiDAR used in measurement applications, and intraoral scanners used in dentistry), which typically generate inhomogeneous 3D volume data in the form of point clouds. These data are not structured in the form of a homogeneous grid of data (e.g., pixels or - in the case of non-optical 3D scanners, such as CT scanners - voxels).

[0003] Data acquisition schemes based on optical scanners typically generate 3D volume data in the form of a point cloud dataset, or simply a point cloud. The data points of a point cloud can represent the surface of an object. Typically, a point cloud includes a large number of points that are unevenly distributed in a 3D space. The 3D space can include areas where data points are densely distributed, areas where data points are sparsely distributed, and areas where there are no data points at all (for example, empty space "inside" an object). The term point cloud can refer to any type of 3D dataset in which each point can be represented as a vector in 3D space. These points can be associated with other attributes (e.g., color, etc.). Special types of point clouds include 3D surface definitions, such as triangular meshes or polygonal meshes.

[0004] Although point cloud-based 3D analysis is a rapidly developing field of technology, solutions for 3D object detection and instance segmentation are still in their infancy compared to their 2D counterparts. Currently, there are few known sources for solving 3D instance segmentation. Qi et al. in their article "Frustum pointnets for 3D object detection from RGB-D data", IEEE Conference on Computer Vision and Pattern Recognition, pp. 918–927 (2018) describe a hybrid framework involving two stages, where in the first stage, 2D bounding boxes of objects are detected in 2D images, and in the second stage, 3D point clouds are processed in a 3D search space partially defined by the 2D bounding boxes. Similarly, Hou et al. in their article “3D-SIS: 3D semantic instance segmentation of RGB-D scans”, arXiv preprint arXiv:1812.07003 (2018) describe a model where the 2D image is first processed by a 2D convolutional network. Thereafter, the learned features are back-projected onto voxelized point cloud data, where the extracted 2D features and geometric information are combined to obtain object proposals and mask predictions for each voxel. The above model’s reliance on 2D images and voxelization limits the performance of this approach.

[0005] In another approach, Yi et al. describe a synthetic analysis strategy in their article "Generative shape proposal network for 3d instance segmentation in point cloud", arXiv preprint arXiv:1812.03320 (2018), where instead of directly determining the bounding boxes of objects in the point cloud, a conditional variational autoencoder (CVAE) is used. However, GSPN training requires a fairly complex two-stage separate training of the CVAE part and the region-based network (which performs classification, regression, and mask generation on the proposals).

[0006] In yet another approach, object proposals are determined based on a clustering scheme. Wang et al., in their article “Similarity group proposal network for 3d point cloud instance segmentation”, IEEE Conference on Computer Vision and Pattern Recognition, pp. 2569–2578 (2018), describe using a similarity matrix between the features of each pair of points in the embedded feature space to indicate whether a given pair of points belongs to the same object instance. However, computing such pairwise distances is impractical for large point clouds. Similarly, Liu et al. in their article "Masc: Multi-scale affinity with sparse convolution for 3D instance segmentation" arXiv preprint arXiv:1902.04478 (2019) described voxelization of point clouds to process volume data through a 3D U-Net model and a clustering scheme to find the similarity between each pair of points by comparing their features extracted in several hidden layers of the trained U-Net. Voxelization and similarity calculation of large refined point clouds greatly limit the performance of this approach.

[0007] Therefore, from the above, it can be seen that there is a need in the art for improved instance segmentation of 3D point clouds. In particular, there is a need for methods and systems that can perform accurate, fast and computationally efficient instance segmentation of 3D point clouds. Summary of the invention

[0008] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Therefore, aspects of the present invention may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or an embodiment in combination with software and hardware aspects (which may be collectively referred to herein as "circuit", "module" or "system"). The functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. In addition, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media, which has a computer-readable program code embodied therein, such as stored therein.

[0009] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media will include the following: electrical connections with one or more wires, portable computer floppy disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this article, a computer-readable storage medium can be any tangible medium that can contain or store programs for use by or in conjunction with an instruction execution system, device or apparatus.

[0010] Computer readable signal media may include, for example, a propagated data signal in baseband or as part of a carrier wave, having computer readable program code embodied therein. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. Computer readable signal media may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0011] The program code embodied in the computer readable medium can use any suitable medium to transmit, including but not limited to wireless, wired, optical fiber, cable, RF etc., or any suitable combination mentioned above. The computer program code for performing the multifaceted operation of the present invention can be written with any combination of one or more programming languages, and the programming language includes functions such as Java (TM), Scala, C++, Python etc. or object-oriented programming languages, and conventional program programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly on the remote computer, or completely on the remote computer, server or virtual server. In the latter case, the remote computer can be connected to the user's computer by any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet).

[0012] Aspects of the present invention are described below with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to an embodiment of the present invention. It will be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, particularly a microprocessor, or a central processing unit (CPU) or a graphics processing unit (GPU) to produce a machine, so that the instructions are executed via the processor of the computer, other programmable data processing devices or other devices to create a component for implementing the function / action specified in one or more frames of the flowchart and / or block diagram. For example, but not limited to, the illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0013] These computer program instructions may also be stored in a computer-readable medium, which may direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture, which includes instructions for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. The computer program instructions may also be loaded onto a computer, other programmable data processing device, or other apparatus to cause a series of operational steps to be performed on the computer, other programmable device, or other apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0014] The flow chart and block diagram in the accompanying drawings illustrate the architecture, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of a code, which includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the function indicated in the frame may not occur in the order indicated in the figure. For example, depending on the function involved, the two frames shown in succession can actually be performed substantially simultaneously, or sometimes these frames can be performed in reverse order. It should also be noted that each frame of the block diagram and / or flow chart and the combination of the frames in the block diagram and / or flow chart can be realized by a system based on special-purpose hardware or a combination of special-purpose hardware and computer instructions that performs a specified function or action.

[0015] In the present application, methods and systems for object detection and instance segmentation schemes for 3D point clouds are described. A point cloud represents a data set that defines at least the geometric positions of points in the point cloud (e.g., the Cartesian coordinates of the points). In addition, in some embodiments, the points may also be associated with other attributes (e.g., color or normal vectors). Thus, the points of the point cloud may define a vector of information, including at least a position in 3D space. A point cloud may be generated by scanning a predetermined object using a 3D optical scanner. Such an object may, for example, be a dentofacial structure, wherein the dentofacial structure may include a dentition, which includes teeth.

[0016] In one aspect, the present invention relates to a method for object detection in a point cloud (preferably, a point cloud generated by a 3D optical scanner, such as an intraoral scan (IOS) point cloud), wherein the method may include: determining, by a first type of neural network, a first feature associated with a point of the point cloud, the point cloud including points representing one or more objects in at least a 3D space of the point cloud, the first feature defining geometric information for each point of the point cloud, the first type of network being configured to receive points of the point cloud as input; determining, by a second type of deep neural network based on the first feature, a second feature defining local geometric information about the point cloud at locations of nodes of a uniform 3D grid, the nodes being uniformly distributed in the 3D space of the point cloud; determining, based on the second feature, one or more object proposals, the object proposals defining a 3D bounding box positioned around the nodes of the 3D grid, the 3D bounding box containing points of the point cloud that can define an object, the 3D bounding box defining a 3D anchor point; and determining, by a third type of deep neural network, a score for the 3D anchor point, the score indicating a probability that the 3D anchor point includes a point defining an object or a part of an object, the determination being based on a second feature located in the 3D anchor point.

[0017] The method provides an accurate and efficient way to detect objects in point clouds. The process is applied directly to the point cloud (input data) so that all the geometric information embedded in the point cloud can be used for object detection. In addition, the process determines the objects in the point cloud by evaluating features in a new domain (the mesh domain) without classifying every point in the point cloud. In this way, the process provides a very efficient way to detect (predict) whether a point representing a predetermined object is present in a point cloud.

[0018] In an embodiment, a second type of deep neural network may also be trained to determine a fraction of the nodes of the uniform 3D grid that indicates the probability that the 3D anchor point includes a point defining an object or a portion of an object.

[0019] In an embodiment, the first point cloud feature may include first feature vectors, each first feature vector is associated with a point of the point cloud; and / or, the second point cloud feature may include second feature vectors, each second feature vector is associated with a node of the uniform 3D grid.

[0020] In an embodiment, the first point cloud feature may be determined by a first type of deep neural network, the first type of deep neural network defining a feature extraction network.

[0021] In an embodiment, the first type of network may be configured to receive points of the point cloud and to generate first features, preferably first feature vectors associated with the points of the point cloud.

[0022] In an embodiment, the first type of deep neural network may include a plurality of convolutional layers including a multi-layer perceptron (MLP), and the feature extraction network is configured to receive points of a point cloud at its input and generate a feature vector for each point of the point cloud at its output.

[0023] In an embodiment, the feature extraction network may include one or more χ-Conv layers, each χ-Conv layer is configured to weight and permute the input points and corresponding features provided to the χ-Conv layer, and then subject the permuted points and features to a convolution kernel. Preferably, the feature extraction network is configured to include a PointCNN of the χ-Conv layer.

[0024] In an embodiment, the second type of deep neural network may represent an object proposal network, which includes a plurality of convolutional layers, each of which includes a multi-layer perceptron (MLP), which includes one or more convolutional kernels.

[0025] In an embodiment, at least one of the plurality of convolutional layers may be configured to receive the first feature and the nodes of the uniform 3D grid and determine the second feature based on the first feature.

[0026] In an embodiment, a uniform 3D grid may be determined based on the spatial distribution of points in the 3D space of the point cloud.

[0027] In an embodiment, the object proposal network can be configured as a Monte Carlo Convolutional Network (MCCNet), which includes multiple Monte Carlo (MC) spatial convolutional layers, preferably, including a MC spatial convolutional layer configured to determine a convolution kernel of a convolution at a position of a node x in the 3D space of a point cloud.

[0028] In an embodiment, determining the convolution includes: determining a neighboring point y within a receptive field r, which defines a field of view (FOV) of a convolution kernel; determining a probability density function p(x, y) for each neighboring point y; and determining the convolution at a node based on Monte Carlo estimation using the neighboring point y and the probability density value p(x, y) of each neighboring point.

[0029] In an embodiment, the third type of deep neural network may represent an object classification network, the third type of deep neural network comprising a plurality of fully connected (FC) multi-layer perceptron (MLP) layers, the second type of deep neural network being configured to receive features associated with the 3D anchor point and use the features to determine a score associated with the 3D anchor point, the score indicating a probability that the 3D anchor point comprises a point defining an object or a portion of an object.

[0030] In one aspect, the present invention may be directed to a computer system adapted for object detection in a point cloud, preferably a point cloud generated by a 3D optical scanner, such as an intraoral scan (IOS) point cloud, the computer system comprising:

[0031] A computer-readable storage medium containing a computer-readable program code, the program code comprising a preprocessing algorithm and at least a trained first 3D deep neural network, the computer-readable program code; and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein in response to executing the first computer-readable program code, the processor is configured to perform executable operations, including: determining, by a first type of neural network, a first feature associated with a point of a point cloud, the point cloud comprising points representing one or more objects in a 3D space of the point cloud, the first feature defining geometric information for each point of the point cloud, the first type of network being configured to receive a point of the point cloud as input; determining, by a second type of deep neural network based on the first feature, a second feature, the second feature defining local geometric information about the point cloud at the locations of nodes of a uniform 3D grid, the nodes being uniformly distributed in the 3D space of the point cloud; determining, based on the second feature, one or more object proposals, the object proposal defining a 3D bounding box positioned around the nodes of the uniform 3D grid, the 3D bounding box containing points of the point cloud that can define an object, the 3D bounding box defining a 3D anchor point; and determining, by a third deep neural network, a score for the 3D anchor point, the score indicating the probability that the 3D anchor point includes a point defining an object or a part of an object, the determination being based on the second feature located in the 3D anchor point.

[0032] In one aspect, the present invention may relate to a computer system suitable for instance segmentation of a point cloud (preferably, a point cloud generated by a 3D optical scanner, such as an intraoral scan (IOS) point cloud), the computer system comprising: a computer-readable storage medium containing a computer-readable program code, the program code comprising a pre-processing algorithm and at least a trained first 3D deep neural network, the computer-readable program code; and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein, in response to executing the first computer-readable program code, the processor is configured to perform executable operations, including: determining, by a first type of deep neural network, a first feature associated with a point of a point cloud, the point cloud comprising points representing one or more objects in a 3D space of the point cloud, the first feature defining geometric information for each point of the point cloud, the first type of network being configured to receive a point of the point cloud as input; determining, by a second type of deep neural network, a second feature, the second feature The feature defines local geometric information about the point cloud at the location of the nodes of the uniform 3D grid, and the nodes are uniformly distributed in the 3D space of the point cloud; based on the second feature, an object proposal is determined, the object proposal defines a 3D volume containing points that can define the object, and the 3D volume of the object proposal defines 3D anchor points positioned around the nodes of the 3D grid; classified 3D anchor points are determined by a third type of deep neural network, and the determination is based on a second feature set, which is a subset of the second features located in the 3D anchor points; the object volume is determined by a fourth type of deep neural network (object position predictor network), the center position of the object volume coincides with the center position of the object instance, and the size of the object volume matches the external size of the object instance, and the determination is based on the second feature set; and classified points are determined by a fifth type of deep neural network (mask predictor network) based on the set of points located in the object volume and the first point cloud feature set, and the classified points include first classified points belonging to the object instance and second classified points that do not belong to the object instance.

[0033] In another aspect, the present invention may relate to a method for training a deep neural network system for object detection in a point cloud (preferably, an intraoral scan (IOS) point cloud), the method comprising: providing a training point cloud sample to an input of the deep neural network system, the training point cloud sample comprising one or more labeled object instances, the deep neural network system comprising at least a feature extraction network, an object proposal network, and an object classification network; calculating an object proposal based on a second feature, the second feature defining local geometric information about the point cloud at the location of a node of a uniform 3D grid, the nodes being uniformly distributed in the 3D space of the point cloud, the second feature being determined by the object proposal network, wherein the object proposal network is configured to determine the second feature based on a first feature generated by a feature extraction network, the feature extraction network receiving at its input A training point cloud sample, the object proposal is defined as a 3D bounding box around a node of a uniform 3D grid, the 3D bounding box may include points defining the object, the 3D bounding box defines a 3D anchor point; determining an overlap between the 3D anchor point and a 3D bounding box of a labeled object instance in the training point cloud sample, and marking the 3D anchor point as positive if the overlap is above a predetermined threshold, and marking it as negative if the overlap is below a predetermined threshold; determining, by an object classification network, one or more object predictions for the positively and / or negatively labeled 3D anchor points using point cloud features in the 3D anchor points, and determining a first loss value based on the one or more object predictions, the positively and / or negatively labeled 3D anchor points, and a first loss function; and using the first loss value, simultaneously training a feature extraction network, an object proposal network, and an object classification network using a back-propagation method.

[0034] In yet another aspect, the present invention may relate to a method for training a deep neural network system for instance segmentation of a point cloud (preferably, a point cloud generated by a 3D optical scanner, such as an intraoral scan (IOS) point cloud), the method comprising: providing a training point cloud sample to an input of the deep neural network system, the training point cloud sample comprising one or more labeled object instances, the deep neural network system comprising at least a feature extraction network, an object proposal network, an object classification network, an object position predictor network, and a mask predictor network; calculating an object proposal based on a second feature, the second feature defining local geometric information about the point cloud at locations of nodes of a uniform 3D grid, the nodes being uniformly distributed in the 3D space of the point cloud, the second feature being determined by the object proposal network, wherein the object proposal network is configured to determine the second feature based on a first feature generated by a feature extraction network, the feature extraction network receiving at its input the training point cloud sample, the object proposal defining a 3D bounding box, the 3D bounding box defining a 3D anchor point; determining a 3D alignment of the 3D anchor point with the labeled object instance in the training point cloud sample; The method comprises the steps of: determining an overlap between the 3D bounding boxes and determining a positive 3D anchor if the overlap is above a predetermined threshold and determining a negative 3D anchor if the overlap is below a predetermined threshold; determining one or more object predictions for the positive and negative labeled 3D anchors by an object classification network using point cloud features in the 3D anchors, and determining a first loss value based on the one or more object predictions and a first loss function; determining a position and size prediction of an object volume by an object position predictor network based on features in the 3D anchors, and determining a second loss contribution using the position and size predictions and a second loss function; determining classified points by a mask predictor network based on point cloud features in the object volume, the classified points including first classified points belonging to object instances and second classified points not belonging to object instances, and determining a third loss contribution using the classified points and a third function; and using the first loss contribution, the second loss contribution and the third loss contribution, and using, preferably simultaneously using, a back-propagation method to train a feature extraction network, an object proposal network, an object classification network, an object position predictor network and a mask predictor network. The first point cloud features and the second 3D mesh features can be implicitly learned simultaneously by a training process of a system of neural networks.

[0035] The methods and systems described in this application are based on an object detection and instance segmentation model (scheme), which may be referred to as a mask-MC network (Mask-MCNet). Mask-MCNet allows accurate and efficient object detection and instance segmentation in 3D point clouds (e.g., intraoral scan (IOS) data generated by an intraoral scanner). Compared to known deep learning models, the model does not require a voxelization step for processing point clouds. Therefore, the data can be processed while retaining the fine geometric information of the point cloud, which is important for the successful segmentation of detailed structures. In addition, by extracting first point cloud features and converting them into second point cloud features on a uniform 3D grid, Mask-MCNet can effectively handle the processing of highly non-uniform point cloud data, thereby quickly generating object proposals in the point cloud. This property is important for the scalability of the scheme to large point cloud data structures (e.g., more than 100k points). Experimental results show that Mask-MCNet achieves an IoU (intersection of union) score of 98% on the test data, thereby outperforming the state-of-the-art networks in the point cloud segmentation task. Mask-MCNet achieves near-human performance and achieves full point cloud object detection and segmentation in just a few seconds of processing time, which is a time-consuming and labor-intensive task for humans.

[0036] In some embodiments, the system may include three modules, each of which may include one or more subnetworks. The first module (feature extraction) may include a deep neural network trained to transform the geometric position of the input point set into a high-dimensional feature space. The high-dimensional feature vector of each point thus obtained can be transferred to a 3D grid spanning the entire 3D space. This transformation is performed by the second module (object proposal), which may include a Monte Carlo convolutional network or other suitable network architecture. The network can be trained to distribute and transform the information contained in the feature vector of each point of the irregular point cloud (obtained from the first module) to a regular grid domain. In order to detect whether an object (e.g., a tooth) exists within each candidate bounding box (also called an anchor point), a third module may be employed, which includes two subnetworks. All features on the grid nodes surrounded by the candidate anchor points are checked by the classification subnetwork to detect anchor points that have a high degree of overlap with the object in the 3D space. In the case where the classification subnetwork positively detects the anchor point, another subnetwork (object location predictor network) is used to estimate the difference between the candidate anchor point bounding box and the center of the object in the candidate anchor point bounding box and compare it with their ground truth. After localizing each object, a mask generator network, which can consist of a cascade of MLP-based networks, is employed to perform binary classification on all points within the detected bounding boxes from the input point cloud. This classification task aims to find points belonging to each object that is completely surrounded by the detected bounding box. Therefore, the model is able to first detect all objects in the input point cloud by fitting a 3D bounding box centered on the center of each object, and then indicates all points belonging to each individual tooth within each detected bounding box.

[0037] It is considered that although the embodiments in the present application identify and describe (functionally) separate network components, for example, separate deep learning networks, alternatively, in other embodiments, a combination of these separate network components can be considered as a single connected network. The present invention will be further described with reference to the accompanying drawings, which will schematically illustrate embodiments according to the present invention. It should be understood that the present invention is not limited in any way to these specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a diagram of a deep learning system for object detection and instance segmentation of 3D point clouds according to an embodiment of the present invention.

[0039] Figures 2A-2D A scheme for object detection and instance segmentation for 3D point clouds according to an embodiment of the present invention is depicted.

[0040] Figure 3 A deep learning system for object detection and instance segmentation of 3D point clouds according to another embodiment of the present invention is depicted.

[0041] Figure 4A and Figure 4B Depicted is a flow diagram of a process for object detection and instance segmentation of point clouds, according to various embodiments of the invention.

[0042] Figure 5 A schematic diagram depicting a portion of a feature extraction network according to an embodiment of the present invention is depicted.

[0043] Figure 6 A schematic diagram depicting a portion of a feature extraction network according to another embodiment of the present invention is shown.

[0044] Figure 7 A schematic diagram of an object proposal network according to an embodiment of the present invention is depicted.

[0045] Fig. 8A and Figure 8B A schematic diagram depicting a portion of a Monte Carlo convolutional network according to an embodiment of the present invention is depicted.

[0046] Fig. 9 Depicted is a schematic diagram of an object classification network according to an embodiment of the present invention.

[0047] Fig.10 Depicted is a schematic diagram of an object position predictor network in accordance with an embodiment of the present invention.

[0048] Fig.11 Depicted is a schematic diagram of a mask predictor network according to an embodiment of the present invention.

[0049] Fig. 12A and Fig. 12B A flow chart for training a deep learning system according to various embodiments of the invention is depicted.

[0050] Fig.13 Depicted is a visualization of positively classified 3D anchors determined by Mask-MCNet according to an embodiment of the present invention.

[0051] FIG. 14A to FIG. 14H Depicted are examples of segmented IOS data generated by a deep learning system according to an embodiment of the present invention.

[0052] Fig.15 Depicted is a flow chart for post-processing a segmented point cloud in accordance with an embodiment of the present invention.

[0053] Fig.16 is a block diagram illustrating an exemplary data processing system that may be used to execute the methods and software products described in this application. DETAILED DESCRIPTION

[0054] Embodiments described in the present disclosure include systems and computer-implemented methods for accurate and efficient object detection and instance segmentation of irregular (non-uniform) 3D point clouds based on one or more deep neural networks (DNNs). A point cloud may refer to a set of sampled points defining a 3D representation of a scene that defines one or more objects or includes one or more objects. Each sampled point (referred to as a point) may be represented by a vector in a 3D Cartesian coordinate system that is not universal (i.e., the Cartesian coordinate system may be different between two IOS data sets). A point cloud may be structured as a 3D surface mesh, wherein a point in 3D space may define a triangular or polygonal mesh that together describes a surface mesh in 3D space. A normal vector associated with a mesh may define an orientation in 3D space. A normal vector associated with each point in a point cloud may represent a vector perpendicular to a surface consisting of each point and its neighboring points. Unlike Euclidean 2D and 3D data sets (e.g., pixel and voxel representations), a point cloud is irregular, permutation-invariant, and may have a variable number of points per scan. In this application, the term point cloud is used to refer to any non-Euclidean 3D data set used to represent objects in 3D space based on points in the 3D space.

[0055] Figure 1 A schematic diagram of a deep learning system for object detection and instance segmentation of point clouds according to an embodiment of the present invention is depicted. In particular, Figure 1 A system 100 for instance segmentation of 3D point clouds is depicted. FIG. 2A to FIG. 2D The process of object detection and instance segmentation of a 3D point cloud performed by the system is schematically depicted in . The system can be implemented on one or more computers, which include one or more processing units 101 connected to a memory 103. The system can be configured to execute multiple units, which include a 3D deep neural network, which can be trained to receive a point cloud (one or more parts) and optionally spatial information associated with the point cloud at its input, and process the data according to the trained model. The system can be implemented as a stand-alone system (e.g., a server system or a network application, such as a cloud application), which is connected to a data storage and retrieval system, such as a database system, etc., and the system includes one or more storage units, such as a database.

[0056] like Figure 1 As shown, the deep learning system may include a plurality of units, including at least a feature extractor 104, an object detector 106, and a mask generator 108. The feature extractor may include a first deep neural network configured to receive an irregular 3D point cloud 102 at its input. Figure 2AA schematic diagram of a point cloud representation is depicted in FIG. A point cloud may include a representation of one or more (different) objects (eg, a representation of a sphere 204 1-3 A first object and / or a second object representing a cylinder 206 1,2 ) is an irregular collection of points on at least a portion of a surface of a 3D point cloud. Typically, points can be represented as coordinates, such as Cartesian coordinates x, y, z in 3D space 202. The first deep neural network can be configured to determine a feature vector for each point of the point cloud. In an embodiment, the features can be formatted as a feature vector, i.e., a multi-element vector (256 elements in the example). The feature vector associated with a point can describe the spatial correlation between the point and other points in the 3D point cloud (typically other points located within a certain distance from the point). The feature vector can be used by object detectors and mask generators to efficiently perform instance segmentation of objects in the point cloud.

[0057] The object detector 106 may include a deep neural network system configured to generate object proposals. An object proposal may include a volume in the 3D space of the point cloud that includes a set of points with a high probability of representing an object. To this end, the deep neural network system may determine a 3D bounding box of a predetermined size and position in the 3D space of the point cloud that is likely to contain points representing the object. The deep neural network system of the object detector may be configured to receive at its input a point cloud (or a subset of the point cloud (e.g., a patch)) and a 3D mesh 216. As shown in FIG. Figure 2B As depicted, the 3D grid may define a certain density of nodes 207 in the 3D space of the point cloud. The nodes of the 3D grid may define the centers of object proposals in the 3D space. Thus, a deep neural network system of an object detector may define an object proposal network configured to determine object proposals, such as 3D bounding boxes, located at the nodes of the 3D grid, wherein the 3D grid includes a plurality of nodes spanning the 3D space of the point cloud. Each 3D bounding box (which may be referred to as an anchor point) is associated with a feature determined based on a feature vector calculated by a feature extractor. Thus, given a Figure 2A The point cloud depicted in Figure 2B The deep neural network system of the object detector can determine 3D bounding boxes, each centered at one of the nodes in the 3D grid, and each 3D bounding box containing features associated with the points in the 3D bounding box. Figure 2C As shown in the figure, the object detector generates anchor points associated with different objects, for example, with the first object 208 1-3 The associated anchor point and the second object 206 1,2 The associated anchor point.

[0058] The object detector may include another deep neural network configured to classify features located in each anchor point. The network may define an object classification network trained to receive features associated with points located in the anchor point as its input and use these features to determine whether the anchor point contains a point representing a predetermined object.

[0059] The mask generator 108 may process points within anchors that have been classified by the object classification network as containing a predetermined object. The center and size of the anchors input in the object proposal network are determined by the 3D mesh. Therefore, the anchors do not necessarily provide the exact location of the center of the object and its size. In order to generate the accurate location of the object, the mask generator may include a deep neural network that is trained to determine the center and size of a 3D bounding box containing points representing the predetermined object. The neural network may be referred to as an object location predictor network.

[0060] Furthermore, based on the accurately positioned and sized 3D bounding box, the mask generator may select points that lie within the 3D bounding box.

[0061] The points in the 3D bounding box may include points representing objects and points that are part of the background. Classification of the points may be performed using a deep neural network, which may be referred to as a mask predictor network. The mask predictor network may, for example, classify points as belonging to a predetermined object (e.g., one or more different objects). In some embodiments, it may classify points belonging to unknown objects within the 3D bounding box as background. The points that are positively classified by the mask generator network within each bounding box may be referred to as a "mask." The object classification network, object position predictor network, and mask predictor network are described in more detail below.

[0062] Figure 2D The result of this process is depicted, including three segmented point sets 212 where each point set represents an instance of a first object (eg, a spherical object). 1-3 And each point set 214 1-3 Two sets of segmentation points representing instances of a second object (eg, a cylindrical object).

[0063] refer to Figure 1 The deep learning system described in Figure 2 is capable of accurate and efficient object detection and instance segmentation of 3D point clouds by using a deep neural network that can directly process the points of the point cloud, thereby eliminating the need for voxelization of the point cloud as known from prior art solutions. The deep neural network can be based on a deep multi-layer perceptron (MLP) network.

[0064] Figure 3 A schematic diagram of a deep learning system for object detection and instance segmentation of 3D point clouds according to an embodiment of the present invention is depicted. Figure 3The deep learning system in is configured to apply instance segmentation on a point cloud generated by a 3D optical scanner. Here, instance segmentation refers to the process of assigning unique labels to all points belonging to an instance of an object (e.g., a tooth) using a computational model. Typically, the computational model includes one or more trained neural networks.

[0065] In an embodiment, the 3D optical scanner may be an intraoral scanner (IOS) for scanning teeth. The point cloud generated by the IOS may be referred to as an IOS point cloud. Automatic instance segmentation of an IOS point cloud in a point set representing individual teeth is highly desirable for many applications in dentistry, implantology, and orthodontics. An IOS point cloud may include large point sets, typically hundreds of thousands or more points, including high-resolution information about the anatomical structure of the crowns and gums (gingiva). In some embodiments, the point cloud may also be converted to a mesh data structure (e.g., after applying a triangulation algorithm to the points).

[0066] like Figure 3 As shown, input data 302 such as an iOS point cloud or a patch of an iOS point cloud can be provided to a feature extractor 304. The iOS point cloud block may include n points, where n can vary between different blocks. The input data may include coordinates 312 of points defining the point cloud, where the coordinates may be defined based on a Cartesian coordinate system x, y, and z. In addition, in an embodiment where the points define a 3D surface mesh, the input data may also include a normal vector 314 of a surface defined by a plurality of points. The mesh may define a graph including a plurality of vertices at positions defined by points in the point cloud. As a result of the triangulation algorithm, adjacent vertices may be connected to each other by edges. Each triangle includes three adjacent vertices in the mesh, which may define a surface (sometimes referred to as a "face"). In this way, a normal vector oriented perpendicular to such a face may be calculated. In order to assign a normal vector to a vertex (point), the average of all normal vectors of the face (i.e., triangle) to which each vertex contributes may be calculated. Thus, in the case of a patch of n points of a 3D surface mesh, the input data to the feature extraction network can be represented as an n×6 matrix, including the n point coordinates and n associated normal vectors. More generated input data can include the 3D position of the point and one or more other attributes (normal vector, color, etc.).

[0067] The feature extractor may include a feature extraction network 316 (also referred to as a backbone network). The feature extraction network is a deep neural network that can be trained to receive a point cloud or a block of point clouds at its input and determine a feature vector 324 for a point of the point cloud. In an embodiment, the feature extraction network can be trained to determine a feature vector for each point of the point cloud. The feature vector can define a plurality of features that describe geometric information around each point of the point cloud. For example, in an exemplary implementation, the feature extraction network can generate n feature vectors (where n represents the number of input points), where each feature vector can include a plurality of feature elements, for example, 256 elements.

[0068] In an embodiment, the feature extraction network can be implemented as a deep multi-layer perceptron (MLP) based network that is applied to the entire point cloud (or, depending on hardware limitations, to blocks of the point cloud). In an embodiment, the MLP based network can be configured as a so-called PointCNN network. The PointCNN network is described in the following paper: Li et al., "PointCNN: convolution on χ-transformed points", Neural Information Processing Systems (NIPS) 2018, arXiv:1801.07791v5, November 5, 2018. Although many choices can be made for the feature extractor network (backbone network), the advantages of the PointCNN network architecture (see Figure 5 and Figure 6The feature extraction network described in more detail in the figure allows processing fine details of point clouds and has a small model size. The output of the feature extraction network can be an n×256 feature matrix (where n represents the number of input points). The feature vector associated with a point of the point cloud can contain rich geometric information in the volume around the point.In other embodiments, other deep neural networks capable of extracting features from point cloud datasets may also be used for the feature extraction network, including but not limited to PointNet (Qi, CR et al.: Pointnet: Deep learning on pointsets for 3d classication and segmentation, Proc. Computer Vision and Pattern Recognition (CVPR), IEEE 1 (2), 4 (2017)), Qi, Charles Ruizhongtai et al.: PointNet++: Deep hierarchical feature learning on point sets in a metric space, Advances in Neural Information Processing Systems, 2017), PointGrid (Le, T., et al.: Pointgrid: A deep network for 3D shape understanding, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017). Recognition.pp.9204{9214(2018)), MCCNet(Hermosilla,P.,et al.: Montecarlo convolution for learning on non-uniformly sampled point clouds,SIGGRAPH Asia 2018Technical Papers.p.235.ACM(2018)), PointCNN(Li,Y.,et al.: Pointcnn,arXiv preprint arXiv:1801.07791(2018)), SpiderCNN(Xu,Y et al.: SpiderCNN: Deep learning on point sets with parameterizedconvolutional filters,ECCV2018.

[0069] The feature vector determined by the feature extraction network can be provided to the input of the object detector 306, which includes a so-called object proposal network 318. The object proposal network can be trained to receive the feature vector 324 from the feature extraction network and receive a 3D grid of nodes, which define so-called anchor points in the 3D space of the point cloud (or point cloud block). The 3D grid defines a 3D arrangement of nodes across the 3D space of the point cloud. The anchor point can define a 3D bounding box with a center position [xa, ya, za] and a size [w, d, h]. The center of the anchor point can be determined by the node of the 3D grid.

[0070] The points of the point cloud usually define "surface points", that is, samples of the surface of the object determined by a 3D optical scanner. An example of such a scanner is an IOS scanner. The features calculated by the feature extraction network usually include local geometric information about the manifold of the 3D space. In other words, this rich geometric information can encode the local surface of the object as the local curvature of the surface in the 3D space. However, for regression tasks, such as accurately locating the 3D bounding box surrounding the object, the model needs to aggregate the 3D geometric information from the 3D surface mesh into the empty space of the 3D space of the point cloud (e.g., the empty space inside the object). For example, in order to accurately determine the object, the model needs 3D spatial information about different parts (e.g., sides) of the set of points representing the object of the point cloud. Voxelization of the point cloud and applying 3D CNN to process voxelized volume data will be a conventional approach to solve this problem. However, as mentioned above, the voxelization of non-uniform point clouds will severely limit the performance of the neural network in terms of quantization error because converting the point cloud into volume data will degrade the fine details of the geometric information.

[0071] To address the shortcomings of the prior art, the object proposal network can be configured to distribute and transfer rich geometric information from the surface of the object (as encoded in the feature vector determined by the feature extraction network) into the entire 3D space (e.g., the empty space inside the object (e.g., tooth)). In an embodiment, a so-called Monte Carlo Convolutional Network (MCCNet) architecture can be used to implement the object proposal network 318. Figure 7The MCCNet architecture, described in more detail in FIG. 8 , may include one or more multilayer perceptron (MLP) based networks. In particular, MCCNet may include several modular MLP subnetworks including at least two hidden layers having functions similar to a set of convolutional kernels. These layers may be referred to as convolutional layers of MCCNet. In this way, MCCNet is able to compute convolutions for any (new) set of points within the field of view (FOV) of the kernel, regardless of whether it exists in the input point set. In other words, the output of the convolutional layer generates a new set of points at locations. These locations may be arbitrary locations on the point cloud, even locations where there are no samples in the point cloud. In this way, MCCNet allows for the computation of convolutions for non-uniform distributions of points.

[0072] MCCNet is configured to transfer features computed by the feature extraction network from a point cloud domain (features are associated with points in the 3D space of the point cloud) to a new domain based on a 3D grid of nodes, where the 3D grid of nodes spans the 3D space of the point cloud. The new domain may be referred to as a mesh domain, where features are associated with nodes of a 3D grid spanning the 3D space of the point cloud (input space). In an embodiment, the 3D grid of nodes may be provided as input data to the network. In an embodiment, a mesh generator 319 may generate a 3D mesh 326 of a specific node density and provide the 3D mesh to the input of the object proposal network. The mesh generator may be configured to generate a 3D mesh based on information about points of the point cloud. For example, the mesh generator may be based on information provided to Figure 3 The input point cloud of the system determines the node density of the 3D mesh. Determining the nodes of the 3D mesh based on the point cloud ensures that sufficient details of the feature vectors associated with the points of the point cloud are transferred to the 3D mesh domain by MCCNet.

[0073] Therefore, if Figure 3 As shown, the object proposal network can receive the feature vector from the feature extraction network and receive a 3D grid 326 that spans the entire 3D space of the point cloud or at least a large part of it. Based on this input, the object proposal network can be configured to determine 3D anchor points that can be used as object proposals, i.e., volumes of a specific shape that include points representing an object, such as a 3D bounding box. The 3D bounding box can have any suitable shape, such as a rectangle, a sphere, a cylinder, etc.

[0074] The object proposal network does not make any assumptions about the possible locations of objects in 3D space. In an embodiment, the 3D grid can be a uniform grid that defines nodes that are evenly distributed in the 3D space of the point cloud. In this case, the spatial resolution of the 3D grid can determine the performance of the model in terms of accuracy and computational power. Choosing a low-resolution 3D grid may result in too few anchor points being positioned within small objects (e.g., front teeth in the case of IOS data), while a high-resolution grid may result in inefficient computation.

[0075] In order to improve the accuracy and efficiency of the model, in an embodiment, a non-uniform grid may be provided to the network instead of a uniform grid. The non-uniform 3D grid may include a dense distribution of nodes close to the surface of the object and a sparse distribution of nodes at a distance away from the surface of the object. The grid generator may determine the non-uniform 3D grid based on the point cloud fed to the network input and the initial dense uniform 3D grid. The non-uniform 3D grid may be obtained by filtering out the nodes of the initial dense 3D grid using the distance between the node and the point closest to the node in the point cloud. In addition, a predetermined lower bound on the grid resolution may be used. Nodes from the 3D grid whose nearest neighbor points to the point cloud are farther than a threshold are removed if there are any points in the grid that are closer to these nodes than the threshold.

[0076] The above characteristics of the object proposal network enable the transfer of features from the point cloud domain (features are associated with the positions of points in the 3D space of the point cloud) to the above-mentioned mesh domain (features are associated with the positions of nodes of the 3D mesh spanning the 3D space of the point cloud). In an embodiment, the object proposal network can generate m 3D anchors (e.g., m 3D bounding boxes associated with m nodes of the 3D mesh), and the center of each anchor coincides with a node of the 3D mesh fed to the input of the object proposal network. In addition, for each node of the 3D mesh, the object proposal network can determine the geometric information (features) belonging to the 3D anchor with the node as its center coordinate. The object classification network 322 can determine whether the anchor includes a point representing an object based on the features in the anchor. The evaluation of the features in the anchor by the object classification network can produce classified 3D anchors (e.g., 3D bounding boxes), where each classified 3D anchor contains points belonging to a certain object. Anchors that are classified and do not contain objects are discarded.

[0077] The first MLP network from the MCCNet input can be used as a convolutional layer for transferring features from the point cloud domain to the mesh domain. The data is further processed by the hidden layer of the MCCNet. Based on the FOV of each convolution kernel, the geometric information of the "surface point" will be distributed on the nodes of the 3D mesh. In an embodiment, each node of the 3D mesh may represent the location (e.g., center) of one anchor point (k=1). In another embodiment, each node of the 3D mesh may represent the location of multiple anchor points (k>1), where the k anchor points associated with the node may have different aspect ratios. Therefore, the total amount of 3D anchor points generated by the object proposal network 318 may be k×m.

[0078] Based on the features inside each 3D anchor point, the object classification network 322 can predict whether the 3D anchor point contains an object. To this end, the object classification network can determine a score for each 3D anchor point, wherein the score can represent the probability that the 3D anchor point includes or does not include an object instance. A threshold can be used to determine whether the score indicates that the 3D anchor point contains a point that defines an object.

[0079] The object classification network 322 can be implemented as a fully connected MLP with a fixed-length input. In this case, the feature set inside each anchor point needs to be of fixed length. Therefore, the interpolation module 320 between the object proposal network 318 and the object classification network 322 can be used to determine a fixed feature set for each 3D anchor point. For example, in an embodiment, the interpolation module can determine (interpolate) each 3D anchor point to have s×s×s nodes. The value of s can be between 2 and 12. In this way, each 3D anchor point can have a feature set of fixed length. For example, a weighted interpolation of the feature vectors of the nearest neighbor nodes of the grid (e.g., three nearest neighbors) and these nearest neighbor nodes based on their distance to the new node in the 3D space can be used. Therefore, the interpolation module can determine k×m×s for the k×m 3D anchor points determined by the object proposal network. 3 The output matrix of features.

[0080] If the object detector has determined that the point cloud includes one or more 3D anchors that include an object, the mask generator 308 can be triggered to classify the points in each of the one or more 3D anchors. The classification process can determine which points in the 3D anchors can represent objects. In this case, the object position predictor network can receive information 340 about which 3D anchors are classified by the object classification network as containing objects. The classified 3D anchors 332 can be provided to the object position predictor network 334, which is configured to determine a volume that accurately surrounds the object. The volume can be referred to as the object volume. The object volume can have a center position that accurately matches the center position of the object. In addition, the size of the object volume can accurately match the external size of the object. In an embodiment, the object position predictor network can generate information for repositioning and rescaling the classified 3D anchors into the object volume.

[0081] The location and size of the object volume can be provided to the 3D cropping module 336, which can use the location and size to crop object patches from the point cloud. Each object patch can include points of the point cloud located within the object volume. In this way, each object patch can define a set of points that include points representing the objects predicted (detected) by the object detector 306. In an embodiment, the 3D cropping module can use each of the m×k object volumes to crop p points from the point cloud. In addition, each of the p points can be associated with one of the n feature vectors generated by the feature extraction network.

[0082] Thus, for each object volume, the 3D cropping module may extract p points from the point cloud, where each point is associated with a feature vector 324 , resulting in m×k×p feature vectors, where each feature vector may have multiple (eg, 256) feature elements.

[0083] The object block and the associated feature vector can be provided to a mask predictor network 338, which is configured to classify the points in the object block into classified points, i.e., points belonging to the object and points not belonging to the object (e.g., background). The classification process can produce segmented instances of points representing the object, e.g., the segmented tooth instances in the IOS block 310, as shown in FIG. Figure 3 Depicted.

[0084] Thus, from the above, it can be seen that three separate MLP-based networks can be sampled to process the output of the object proposal network. The first network (the object classification network) is responsible for predicting whether each point in the grid domain is positive or negative. A node on the grid is considered positive if it belongs to at least one positive anchor. Positive anchors are those that have a high IoU (overlap) with the objects in the data. Therefore, the first network is adopted for classification of grid nodes. All points within the positive anchors are processed by two other networks (the object location predictor network and the mask predictor network). One of these networks is trained to estimate the spatial displacement between the center of the positive anchor (candidate) and the center of its assigned ground truth. The second network (the object location predictor network) is responsible for predicting the difference between the size of the positive anchor (e.g., the width, height, and depth of a 3D cube) and its equivalent value of the ground truth. In general, one of the networks performs classification on the grid nodes and the next two networks have been used to solve the regression problem. In order to segment all points belonging to objects enclosed by a 3D box, a sub-network is adopted. This network (called the mask predictor network) is responsible for classifying the points within each 3D bounding box.

[0085] Figure 4A and Figure 4B Describes a flow chart of the process for detection and instance segmentation of point clouds according to various embodiments of the present invention. These processes can be referred to above Figures 1 to 3 Describes the modules of the deep learning system to perform. Figure 4AA flow chart depicting a process for detecting an object in a point cloud by a system of trained neural networks. As shown, the process may include a first step 402 of receiving a point cloud including points representing one or more objects in a 3D space of the point cloud. In the next step (step 404), a first point cloud feature of the point cloud may be determined. In an embodiment, the first point cloud feature may be determined by a deep neural network of a first type based on the points of the point cloud. The first point cloud feature may define local geometric information about the point cloud at the location of each point of the point cloud. In an embodiment, the first point cloud feature may define a first feature vector. Each feature vector associated with a point may define geometric information about the space surrounding the point, for example, geometric information about the presence or absence of points within a particular volume surrounding the point, relative distances between the points, and local curvatures defined by the points.

[0086] The third step 406 may include converting the first point cloud features into second point cloud features by an object proposal network. Here, the second point cloud features may define local geometric information about the point cloud at the location of the nodes of the 3D grid spanning the 3D space of the point cloud. A further fourth step (step 408) may include generating one or more object proposals based on the second features. The object proposal may define one (or more) 3D bounding boxes positioned around the nodes of the 3D grid, wherein the 3D bounding box may contain points of the point cloud that may define an object. Such a 3D bounding box generated by the object proposal network may also be referred to as a 3D anchor point.

[0087] Each 3D anchor point can be positioned around a node of the 3D grid, and a second feature vector associated with each 3D anchor point can define geometric information within a space defined by the 3D anchor point. Determination of the second feature vector can include providing the first feature vector and the node of the 3D grid to an input of a second deep neural network, and the second deep neural network provides a second feature vector for each 3D anchor point at its output.

[0088] The fifth step 410 may include selecting a feature set from the second features located in the 3D anchor point; and determining, by the object classification network, a score for the 3D anchor point based on the feature set. The score may indicate the probability that the 3D anchor point includes a point defining an object or a portion of an object.

[0089] Figure 4AThe process depicted in provides an accurate and efficient way to predict objects in a point cloud. The determination of a first feature vector may include providing the points of the point cloud to the input of a first deep neural network, and the first deep neural network provides a first feature vector for the points of the point cloud at its output. The process is applied directly to the point cloud (input data) without the need for voxelization or the like, so all geometric information embedded in the point cloud can be used for object detection. In addition, the process determines objects in the point cloud by evaluating features in a new domain (the grid domain) without the need to classify each point in the point cloud. In this way, the process provides a very efficient way to detect (predict) whether a point representing a predetermined object is present in a point cloud.

[0090] Figure 4B A flowchart depicting the process for instance segmentation of a point cloud. The process can be found in reference Figure 4A The instance segmentation process may begin with step 412 of receiving a feature set contained in a 3D anchor that is positively classified by the object classification network as containing an object instance. A further step (step 414) may include determining the object volume by an object position predictor network. The object position predictor network may generate a center position of the object instance. Alternatively, the object position predictor network may generate an outer dimension of the object instance. In an embodiment, the center position of the object volume may coincide with the center position of the object instance. In another embodiment, the size of the object volume may match the outer dimension of the object instance.

[0091] The location and size of the object volume may be used to crop a point set from the point cloud and a point cloud feature set from the first point cloud feature (step 416). Classified points may be determined by a mask prediction network (step 418). Classified points may be determined based on the first point cloud feature set, wherein the classified points may include first classified points belonging to the object instance and second classified points that do not belong to the object instance. The cropped point set and the matched point features may be input into a mask prediction network (step 418), which classifies each point as being part of the object instance or not part of the object instance (background).

[0092] therefore, Figure 4B The process steps depicted in allow for efficient instance segmentation of points of a point cloud by accurately determining the volume in 3D space of the point cloud containing the points representing the object. In this way, the process allows for segmentation of points without the need to classify each point of the point cloud as is known in conventional semantic segmentation processes.

[0093] Figure 5 and Figure 6Schematic diagrams of a portion of a feature extraction network according to an embodiment of the present invention are depicted. In particular, these figures depict schematic diagrams of a deep neural network architecture of a feature extraction network 504 that is capable of directly processing points of a point cloud 502 that can be represented by n points, where each point can be represented by a Cartesian coordinate (and optionally, a normal vector). The deep learning network can be configured as an MLP-based network. In an embodiment, the MLP-based network can have a PointCNN network architecture (such as in Figure 3 , as briefly mentioned in the description of Figure 5 As shown, the MLP-based network model 504 may include a so-called χ-Conv layer 506 1-9 Each χ-Conv layer can be configured to weight and permute the input points and their corresponding features before being processed by the convolution operator.

[0094] exist Figure 6 The structure of the χ-Conv layer is described in more detail in . The χ-Conv layer is characterized by parameters N, c, K, and D. Each χ-Conv layer returns N representative points with c channels, where a representative point is a point at which the convolution is calculated (representing a location in the 3D point cloud space (i.e., the input space)). In addition, the channel represents the dimensionality of the feature vector. For example, a point associated with a 256-dimensional feature vector has 256 channels. The constant K is the number of neighbors for each representative point, and D is the dilation rate of the χ-Conv layer, that is, the ratio between the points input to the layer and the points generated as output.

[0095] The field of view (FOV) of each χ-Conv layer can include a fixed set of K nearest neighbor (KNN) points 609 (i.e., the points closest to the location of the point at which the convolution is calculated). In addition, the χ-Conv layer can include two or more MLP networks 610, 612. These MLP networks learn a K×K χ-transformation for the coordinates of the K input points. The result of the χ-Conv layer is to aggregate and project the KNN point features to a representative point set, and then apply convolution.

[0096] Figure 5 and Figure 6The PointCNN model depicted in is capable of learning local geometric correlations between points in a point cloud. The model has much fewer learning parameters than other known deep learning models for point clouds. This is beneficial because it is less prone to severe overfitting on small data sets. The input to the network can be points of the point cloud 506 (e.g., n points, each of which is an element of an x, y, z coordinate and (optionally) a normal vector, thereby defining an nx 6 matrix). The output of the network 510 can be a set of feature vectors (e.g., n vectors, each of which has multiple feature elements, e.g., 256 feature elements), where each feature vector is associated with a point of the point cloud. The feature vector can be a multi-element vector (256 elements in the example), where each element of the vector represents a class probability.

[0097] Figure 7 Schematic diagram of an object proposal network according to an embodiment of the present invention is depicted. As shown, the architecture of the object proposal network 704 includes a multi-layer stack of MLP layers. The network can be referred to as a Monte Carlo Convolutional Network (MCCNet), which includes a Monte Carlo (MC) spatial convolution layer 712 1-4 , Batch Normalization (BN) layer 714 1-6 and 1x1 convolutional layer 716 1-3 . The object proposal network can be configured to receive feature vectors defined in the point cloud domain 702 (i.e., feature vectors associated with a point cloud, where each feature vector defines geometric information near a point of the point cloud) and a 3D mesh of nodes 707. In an embodiment, the 3D mesh of nodes can be generated by a 3D mesh generator 708. In some embodiments, a non-uniform 3D mesh can be used as the 3D mesh. In this case, the 3D mesh generator can be determined by filtering nodes from a dense uniform 3D mesh 710. The 1×1 convolutional layer has C in input channels and C out These layers can be used to reduce or increase the number of point features.

[0098] As shown, the first MC spatial convolution layer 7121 can be configured to receive a feature vector of the point cloud domain and m nodes of the 3D mesh. Based on the input, the convolution layer converts the feature vector of the point cloud domain into a feature vector of the mesh domain. The input data will be further processed by the hidden layer of the network. Based on the FOV of each convolution kernel, the geometric information of the "surface point" will be distributed on the nodes of the 3D mesh, resulting in m feature vectors in the mesh domain 706.

[0099] A Monte Carlo convolution layer is a convolution layer that is capable of computing convolutions for unstructured data (e.g., points of a point cloud). The MC convolution layer is described in the following article: Hermosilla et al., "Monte Carlo convolution for learning on non-uniformly sampled point clouds", ACM Transactions on Graphics, Vol. 37, No. 6, No. 235, November 2018. As shown in the figure, the MC convolution layer can be characterized by four parameters, where the notation A|B|C|D indicates that the layer is configured to map points from level B to level C using a field of view (FOV) equal to D. The constant A determines the dimension of the feature vector at the output side of the layer.

[0100] Figure 8 schematically illustrates the working of a Monte Carlo convolutional layer, which is configured to transform the feature f from the point cloud domain to the mesh domain. The convolution operation can be defined as the integral of the product of two functions f and g:

[0101] (f*g)(x)=∫f(y)g(xy)dy

[0102] Among them, the function f is the R to be convolved 3 (in this case, the feature vector determined by the feature extraction network), the function g is the convolution kernel, and is R 3 In particular, the function f can be defined as i ∈S (given data points). If no other information is provided besides the spatial coordinates of each point, f can represent a binary function that has a value of "1" on the sample surface if the point is represented only by its spatial coordinates, and a value of "0" otherwise. In other variations, the function f can also take into account other types of input information, including, for example, color, normal, etc.

[0103] For a hierarchical network structure (e.g., a multi-layer network), the output from the first convolutional layer represents features that form the input to subsequent convolutional layers. Since the value of g depends only on the relative position, the convolution integral is translation invariant. For large datasets, evaluating the convolution integral over the entire point cloud can be problematic, so the domain of g can be restricted to a sphere centered at 0 and with a radius of 1. To support multiple radii, the input to g can be normalized by dividing it by the receptive field r, i.e., the radius of the spherical field of view (FOV) of the convolution kernel. In particular, r can be chosen as a fraction of the scene bounding box diameter b, e.g., r = 0.1·b. Since the point cloud data is normalized to have a variance equal to 1, the diameter of the scene boundary is considered to be equal to 1 (b = 1). This will result in scale invariance. This relative kernel size will make the processing effective for input scans of different sizes (diameters). This construction can result in a compactly supported kernel that is quickly evaluated.

[0104] A multilayer perceptron (MLP) network may be used for the kernel g, similar to the work of Hermosilla et al. cited above. In an embodiment, the MLP network may determine a spatial offset δ=(xy) / r, including three normalized coordinates, and divide them by the receptive field r. The output of the MLP network is a single scalar. In order to balance accuracy and performance, many hidden layers (e.g., two or three) may be used, wherein each hidden layer may include multiple neurons. In an embodiment, the hidden layer may include 6-12 neurons, preferably 8 neurons. The number of parameters in the MLP kernel increases with the dimensions of the input and output vectors. Therefore, in an embodiment, an MLP with multiple outputs (which represent different g kernels) may be used to reduce the number of trainable parameters (e.g., if the output of the MLP produces 8 different g, then it is reduced by 8 times). Since the output of the MLP network is differentiable with respect to its input, the parameters of the kernel may be updated using a backpropagation algorithm. In an embodiment, the network may be trained using a gradient descent optimization scheme.

[0105] In order to calculate the convolution in each sample point, the convolution integral needs to be evaluated. Since only a sample set of the characteristic function f is available, the so-called Monte Carlo integration can be used to determine the estimate of the convolution at a point x in the 3D point cloud space. In the Monte Carlo integration method, a random sample set can be used to calculate the integral value. Based on this method, the estimate of the convolution at a point x can be given by the following expression:

[0106]

[0107] where N(x) is the set of neighborhood indices, i.e., the set of indices of neighboring points that belong to a sphere of radius r (receptive field), and plx is the value of the point x when it is fixed (i.e., the convolution is calculated at point x). Here, x is any output point in the 3D space of the point cloud, which does not necessarily need to be the same as the input point The points in the set coincide. In an embodiment, the output of the first convolutional layer of MCCNet can be points of a 3D grid (in this application, the points of the 3D grid are also referred to as nodes to distinguish them from the points of the point cloud). In deeper layers (layer 2 and subsequent layers), the output points of each layer can be a subset of the points of the initial 3D grid. These output points are called representative points. This feature of Monte Carlo convolution allows resampling to other levels or other regular domains (for example, a uniform 3D grid of nodes) or irregular domains (non-uniform 3D grids of nodes).

[0108] In the case of uneven node distribution of 3D grid, for each point , the value of plx will be different and may depend not only on the sample position , also depends on the radius r of the receptive field (i.e., FOV) and the distribution of other points in the receptive field. Since the sample density of a given point cloud is unknown, it can be approximated by a technique called kernel density estimation. The estimated function has high values ​​when the samples are dense and low values ​​when the samples are sparse. The function can be calculated based on the following expression:

[0109]

[0110] Where δ is the bandwidth that determines the smoothness of the resulting sample density function, h is the density estimation kernel, which is a non-negative function whose integral is equal to 1 (e.g., a Gaussian function), and d is the R 3 is one of the three dimensions of , and σ is the radius of the Gaussian window. Other types of functions can be used instead of the Gaussian kernel density estimator, including but not limited to Epanechnikov, Quartic, or tri-cube kernel estimators.

[0111] Application of MC convolution on a point cloud sample at node position x800 in 3D grid 808 Fig. 8A and Figure 8B Shown in. Fig. 8A A 3D grid of nodes in a 3D input space (Input Space) of a point cloud is depicted. For clarity, the points of the point cloud are not shown.

[0112] Figure 8B A more detailed illustration of the area around node X is depicted. This figure depicts the points of node x and the point cloud surrounding node x. For clarity, Figure 8B In FIG. 8 , the nodes of the 3D grid 808 other than the node x are not shown. To compute the convolution, the points within the spherical field of view (FOV) 802 (radius equal to r) are Associated Features Processed by the MLP kernel. For each point in the point cloud in the FOV 804, a point density function (PDF) may be locally calculated with a Gaussian window 806 having a radius equal to σ. Note that instead of a regular 3D grid of nodes, an irregular grid of nodes may be used, e.g., as described with reference to Figure 7 as described.

[0113] Points about a given node x The PDF of is always relative to all other points in the receptive field. Therefore, it is not possible to precompute the point The density of , because its value is different for each receptive field defined by x and radius r. In the case of uniform sampling (e.g., a uniform distribution of voxels or points (e.g., a uniform grid)), for all given The density function p can be set to a constant value. The output of the convolution computed by the MC convolution layer is differentiable with respect to the kernel input and the learned parameters.

[0114] Figures 9 to 11 Depicted are exemplary network architectures for an object classification network, an object location predictor network, and a mask predictor network. Fig. 9 and Fig.10 As shown, both the classification and location predictor networks are implemented as fully connected (FC) MLP networks, which are configured to receive feature vectors from the object proposal network. Each network includes an FC MLP layer 902 1-4 ,1002 1-4 A stack of layers, each of which is batch normalized by layer 904 1-3 ,1004 1-3 Here, the parameter C represents the number of neurons in each layer of the fully connected network.

[0115] Fig. 9The object classification network may receive feature sets 900 from the object proposal network, wherein each feature set defines features in a 3D anchor point. In an embodiment, the feature set may represent a feature matrix. As described above, each feature set may include a fixed number of features. Based on the feature set, the object classification network may determine classified 3D anchor points 906, i.e., 3D bounding boxes, each 3D anchor point being centered on a node of a 3D grid. The classified 3D anchor points may be associated with scores indicating the probability that the 3D anchor point includes or does not include an object instance. In an embodiment, the object classification network may be trained to perform binary classification. In this case, the network may determine whether the 3D anchor point includes a defined object instance (e.g., a tooth) or does not include a point that defines an object instance (background). In another embodiment, the object classification network may be trained to determine object instances of different categories, e.g., incisors and molars.

[0116] Fig.10 The position predictor network of the object proposal network can receive a feature set 900 of a positively classified 3D anchor (i.e., a 3D anchor including an object) and generate an offset defining the center position of the node of the object relative to the 3D anchor and a size offset (e.g., a scaling value) defining the size of a 3D bounding box that accurately contains the object relative to the size of the 3D anchor. Thus, the position predictor network can determine the difference (delta) between the size of the 3D anchor and the size of the object and the difference between the center position of the 3D anchor and the center of the position of the corresponding object. In another embodiment, the position predictor network can determine the absolute values ​​of the position and size.

[0117] Fig.11 Depicts a schematic diagram of a mask predictor network according to an embodiment of the present invention. Figure 3 As described, the 3D cropping module uses the position and size of each object volume to crop an object block from the point cloud, wherein each object block may define a point set including points representing an object predicted (detected) by the object detector. The 3D cropping module may use each of the m×k object volumes to crop p points from the point cloud, wherein each point is associated with a feature vector, resulting in m×k×p feature vectors (each feature vector having a plurality of (e.g., 256) feature elements).

[0118] As shown, each of the m×k×q masks constructed by assigning a value of 1 to all points within the detected bounding box and 0 to points outside the box is passed to the mask generator. In addition, the first feature vector of all points of size n×256 is provided to the mask generator. Here, q is the number of detected objects (i.e., teeth). The output of the mask generator is a binary classification of the points within each bounding box. Therefore, each of the m×k×p×3 points 1100 and the m×k×p×256 associated feature vectors 1101 can be provided to the input of the network. The mask predictor network may include an MLP-based network. In an embodiment, the MLP-based network may have a PointCNN network architecture (e.g., reference Figure 5 and Figure 6 Alternatively, the MLP-based network can be based on other neural networks capable of processing point clouds, including (but not limited to) PointNet, PointGrid, MCCNet, PointCNN, PointNet++, SpiderCNN, etc. Fig.11 As shown, the MLP-based network model 1102 may include a χ-Conv layer 1104 1-3 , where each χ-Conv layer can be configured to weight and permute the input points and their corresponding features before being processed by the convolution operator.

[0119] The mask predictor network may classify points within each object block. In an embodiment, it may perform a binary classification that classifies the points of the object block into two categories. For example, in the case of IOS data, the binary classification may classify the points into a first point (e.g., a foreground point that may belong to a tooth instance) and a second point (e.g., a background point that may belong to other teeth or gums).

[0120] Therefore, as mentioned above Figures 1 to 11The deep learning system for object detection and instance segmentation of point clouds can provide an accurate and high-performance system for segmenting point clouds without voxelization. At a high level, the deep learning system can be viewed as a 3D point cloud analog of the well-known Mask R-CNN system for instance segmentation of 2D pixel-based images, as described in the article by Shaoqing He, K. et al.: "Mask r-cnn", proceedings of the IEEE international conference on computer vision, pages 2961-2969 (2017). Therefore, the deep learning system according to the present invention can be called Mask MCNet, which has a feature extraction network as a backbone network, an object proposal network as a region proposal network (RPN), and three predictor networks for classification, regression, and mask generation.

[0121] refer to Figures 1 to 11 The training of the described deep learning system can be performed using an end-to-end training scheme (e.g., gradient descent and Adam learning adaptation techniques). For example, 1000 epochs with a batch size of 32 (balanced between positive and negative anchors) can be used. Preprocessing of the input can include normalization of the point cloud to obtain zero mean and unit variance.

[0122] The training set may include a set of labeled point cloud samples. For example, in the case of IOS data, the training set may include optical scans of dentition, such as hundreds of optical scans of dentition from different adult subjects or more, each scan comprising one maxillary scan and one mandibular scan (e.g., Figure 3 ). IOS data can be recorded using a 3D scanner (e.g., a 3Shaped500 optical scanner (3Shape AS, Copenhagen, Denmark)) and include on average 180k points (varying in the range of [100k, 310k]). The dataset may include scans from healthy dentition with various anomalies between subjects.

[0123] In an embodiment, the optical scan may be manually segmented and its corresponding points may be labeled by a dental professional and classified into one of 32 categories according to the FDI criteria and reviewed and adjusted by a dental expert (DAM). The segmentation of the optical scan may take an average of 45 minutes, indicating that the segmentation of the point cloud is a difficult task for humans.

[0124] The loss function may be used to train the networks of the embodiments described in the present application. The loss function of Mask-MCNet may include loss contributions from different networks of Mask-MCNet. In an embodiment, the loss contribution may include a first loss contribution associated with the output of the object classification network, a second loss contribution associated with the object location predictor network, and a third loss contribution associated with the mask predictor network. In an embodiment, the loss contribution may be similar to the loss function used for Mask R-CNN by Shaoqing He, K. et al. (cited above), in which three loss contributions are used, wherein the contribution of each of the three loss contributions is equal.

[0125] In an embodiment, the first loss contribution (classification loss) may define the cross entropy loss value of the classification branch (object classification network) at its output layer (e.g., softmax layer, etc.). In an embodiment, the first loss contribution may be calculated as follows:

[0126]

[0127] Among them, p i It is possible to define the probability that a node of the 3D mesh is the center of a 3D anchor with an object (with high IoU), and p i * is its ground truth value selected from the set {0,1}. Parameter N a Limits the number of nodes in the 3D grid.

[0128] In an embodiment, the second loss contribution (regression loss) may define the mean squared error at the output layer (preferably, the linear output layer) of the regression branch (object position predictor network). In an embodiment, the second loss contribution may be calculated as follows:

[0129]

[0130] Among them, t i A vector of three elements can be defined, which represent the delta value of the center position or size of the 3D anchor point, and t i * is its ground truth value. Parameter N p The number of positive anchor points in the input IOS can be limited.

[0131] In an embodiment, the third loss contribution (mask loss) may define a binary cross entropy loss for classification of points (preferably all points) in the positive anchors (preferably each positive anchor) at the output layer of the mask branch (mask predictor network). The regression loss and the mask loss may be considered only when the 3D anchor is labeled as positive. The mask loss may be calculated according to the following expression:

[0132]

[0133] where p i is the probability that a point belongs to the corresponding object surrounded by the 3D anchor. Here, p i * is the ground truth value from the set {0, 1}. The parameter N m can limit the number of points within the i-th 3D anchor. During the training process, regression loss and quality loss can be used to change the weights of the target position predictor network and the mask predictor network, respectively.

[0134] In an exemplary embodiment of the training process of the system, if the overlap of a 3D anchor with any tooth instance in the labeled point cloud patch is higher than a first threshold, the 3D anchor can be labeled as positive. In an embodiment, the first threshold can be 0.4 IoU, where IoU defines the average Jaccard index. In another embodiment, if the overlap of a 3D anchor with any tooth instance in the labeled point cloud patch is lower than a second threshold, it can be labeled as negative.

[0135] In an embodiment, the second threshold can be 0.2. Due to the highly imbalanced number of positive and negative 3D anchors, approximately 50% of each training batch can be selected from positive 3D anchors, and approximately 25% of each training batch can be selected from negative 3D anchors. The remaining 25% of the sampled 3D anchors in the training batch can be selected from marginal 3D anchors (e.g., (0.2 < IoU < 0.4), which are also considered negative samples).

[0136] Furthermore, in an embodiment, during the training phase, the input to Mask-MCNet can be a randomly cropped patch of the point cloud. Each cropped patch can include multiple object instances, e.g., 2 to 4 tooth instances. In an embodiment, a uniform grid domain can be constructed. In another embodiment, a non-uniform grid domain can be constructed by filtering out the nodes of a dense regular grid with a spatial resolution of 0.04 (lower limit) in each dimension. The upper limit of the grid resolution can be set to be equal to 0.12. To generate sufficient overlap between 3D anchors and both small and large objects (e.g., incisors and molars, respectively), 3D anchors of different sizes can be used. For example, in an embodiment, two types (k = 2) of 3D anchor boxes (sizes [0.3, 0.3, 0.2] and [0.15, 0.2, 0.2]) can be employed.

[0137] Fig. 12A and Fig. 12B depicts a flowchart for training a deep learning system according to various embodiments of the present invention. Fig. 12AA flow chart for training a deep neural network system for object detection in a point cloud is depicted. The deep neural network may include a feature extraction network, an object proposal network, and an object classification network according to any embodiment described in the present application. As shown, in a first step (step 1202), a training point cloud sample including one or more labeled object instances may be provided to an input of a deep neural network system. The deep neural network system may include at least a feature extraction network, an object proposal network, and an object classification network configured for object detection in a point cloud. A further step (step 1204) may include calculating, by the object proposal network, an object proposal based on point cloud features determined by the feature extraction network. Here, the object proposal may define a 3D bounding box, which may include points defining an object. Such a 3D bounding box may be referred to as a 3D anchor point. The training process may also include determining an overlap between the 3D anchor point and the 3D bounding box of the labeled object instance in the training point cloud sample (step 1206). If the overlap is above a predetermined threshold, the 3D anchor point may be marked as positive, and if the overlap is below a predetermined threshold, the 3D anchor point may be marked as negative. Next, features in the 3D anchor point may be used to determine an object prediction for the 3D anchor point by an object classification network (step 1208).

[0138] Therefore, both positively labeled 3D anchors and negatively labeled 3D anchors are used for training the object classification network. The object classification network can predict whether the 3D anchor should be labeled as positive (contains an object) or negative (does not contain an object). During training, in both cases, the first loss function is used to determine the contribution (i.e., if the prediction is correct, the value of the loss is 0 (no correction), but if the prediction is wrong (although it is positive but negative prediction, or vice versa), then the loss value will be positive (and the network will be corrected)). The object prediction and loss function can be used to obtain the loss value. Finally, the loss value can be used to train the feature extraction network, the object proposal network, and the object classification network using the back-propagation method.

[0139] Fig. 12B A flow chart for training a deep neural network system for instance segmentation of a point cloud according to an embodiment of the present invention is depicted. In step 1222, a training point cloud sample including one or more labeled object instances may be provided to an input of the deep neural network system. Here, the deep neural network system may be configured for instance segmentation of a point cloud. Such a network may include at least a feature extraction network, an object proposal network, an object classification network, an object location predictor network, and a mask predictor network.

[0140] The training process may include computing, by the object proposal network, object proposals based on the point cloud features determined by the feature extraction network (step 1224), wherein the object proposals may define a 3D bounding box that may include points defining the object. Such a 3D bounding box may define a 3D anchor point. In a further step (step 1226), an overlap between the 3D anchor point and the 3D bounding box of the labeled object instance in the training point cloud sample may be determined, wherein the 3D anchor point may be labeled as positive if the overlap is above a predetermined threshold, and the 3D anchor point may be labeled as negative if the overlap is below a predetermined threshold.

[0141] The object classification network can determine an object prediction for a positively labeled 3D anchor or a negatively labeled 3D anchor based on the features in the 3D anchor (step 1228). The object classification network predicts whether the 3D anchor should be labeled as positive (contains the object) or negative (does not contain the (complete) object). During training, in both cases, a first loss function is used to determine the contribution (i.e., if the prediction is correct, the value of the loss is 0 (no correction), but if the prediction is wrong (although it is positive, it is a negative prediction, or vice versa), the loss value will be positive (and a correction of the network will be performed). In this way, the difference between the object prediction and the ground truth (labeled object) can be used to determine the first loss contribution based on the first loss function.

[0142] The position and size prediction of the object volume can be determined by the object position predictor network based on the features in the 3D anchor points (step 1230). Here, the object volume can be a 3D bounding box whose center should coincide with the center of the positively classified 3D anchor point and the 3D bounding box has a size that should closely match the size of the object instance. The difference between the predicted position and size and the ground truth (the position and size of the marker) can be used to determine a second loss contribution based on a second loss function. The larger the difference, the larger the loss contribution.

[0143] Classified points may be predicted by the mask predictor network based on the first point cloud features in the object volume, and the classified points may include first classified points belonging to the object instance and second classified points that do not belong to the object instance. The difference between the predicted point classification and the ground truth (labeled classified points) may be used to determine a third loss contribution based on a third function (step 1234). The first loss contribution, the second loss contribution, and the third loss contribution may be used to train the feature extraction network, the object proposal network, the object classification network, the object location predictor network, and the mask predictor network using a back-propagation method (step 1234).

[0144] Inference on new point clouds (e.g., new IOS datasets) can be performed by applying Mask-MCNet on several cropped overlapping patches. Given a 3D patch and applying a uniform 3D grid of nodes with the highest defined resolution (e.g., 0.04), the classification branch classifies 3D anchors located on the nodes of the 3D grid as objects / no objects. The size and (center) position of the positively classified 3D anchors (i.e., detected objects) are updated according to the values ​​estimated by the regression branch (i.e., the object position predictor network). Fig.13 Depicts the 3D anchors 1302 of the positive classification determined by Mask-MCNet 1,2 and a 3D bounding box 1302 representing the ground truth 1,2 Visualization of . As shown in the figure, the two 3D anchors have different scales and center locations. The overlap of the positively classified 3D anchor with the ground truth 3D bounding box is above a first threshold (in IoU units), which is used to label the 3D anchor as positive (i.e., containing (part of) the object).

[0145] Similar to Faster-RCNN, for each object, multiple 3D anchors can be detected, where each 3D anchor is associated with a probability score above a first threshold. Therefore, the non-maximum suppression algorithm can be employed based on the highest objectiveness score (from the classification probability). Therefore, based on the algorithm, the 3D anchor with the highest probability score can be determined. Non-maximum suppression also handles duplicate points by overlapping the input patches. After predicting the bounding boxes for all tooth instances, it is very simple to obtain the masks of all points within the bounding boxes from the mask generation branch.

[0146] exist FIG. 14A to FIG. 14H An example of IOS instance segmentation performed by Mask-MCNet is visualized in . Here, Fig.14A and Fig. 14B The segmentation of the normal dentition is depicted. Fig. 14C The segmented dentition including segmented teeth with missing data is depicted. FIG. 14D to FIG. 14F The segmented dentition including segmented teeth with anomalies and artifacts is depicted. Figure 14G and Fig.14H Two examples of segmentation failures are shown.

[0147] The performance of Mask-MCNet is evaluated by five-fold cross validation compared to the state-of-the-art classification networks. The average Jaccard index (also known as IoU) is used as the segmentation metric. In addition to IoU, the precision and recall of the multi-class segmentation problem are reported by treating each class separately (one-versus-all, one-versus-many) as a binary problem. In addition, the average score is also reported. The performance of Mask-MCNet is shown in Table 1. As shown in Table 1, the proposed Mask-MCNet significantly outperforms the state-of-the-art networks that can segment point cloud data.

[0148]

[0149] NVIDIA Titan-X GPU

[0150] Table 1

[0151] Fig.15 A flow chart for post-processing a segmented point cloud according to an embodiment of the present invention is depicted. A post-processing method can be used to convert instance labels predicted by Mask-MCNet (via a lookup table) into FDI standard labels for clinical purposes and consistency of tooth landmark assignments. By measuring the average center position and size of FDI labels within the training data, a combinatorial search algorithm can determine the most likely label assignment that satisfies predetermined constraints (prior measurements on the training data) in the context of a constraint satisfaction problem (CSP). Constraint satisfaction problems over finite fields are typically solved using a search formalism. The techniques used for such searches can be variations of backtracking, constraint propagation, or local search.

[0152] Fig.16 1600 is a block diagram illustrating an exemplary data processing system that can be used in the embodiments described in the present disclosure. Data processing system 1600 may include at least one processor 1602 coupled to memory element 1604 via system bus 1606. In this way, the data processing system can store program codes in memory element 1604. In addition, processor 1602 can execute program codes accessed from memory element 1604 via system bus 1606. On the one hand, the data processing system can be implemented as a computer suitable for storing and / or executing program codes. However, it should be understood that data processing system 1600 can be implemented in the form of any system including a processor and a memory capable of performing the functions described in this specification.

[0153] Memory element 1604 may include one or more physical memory devices, such as, for example, local memory 1608 and one or more mass storage devices 1610. Local memory may refer to random access memory or other non-persistent memory devices commonly used during the actual execution of program code. Mass storage devices may be implemented as hard drives or other persistent data storage devices. Processing system 1300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code to reduce the number of times program code must be retrieved from mass storage device 1610 during execution.

[0154] Input / output (I / O) devices, depicted as input device 1612 and output device 1614, may optionally be coupled to the data processing system. Examples of input devices may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, and the like. Examples of output devices may include, but are not limited to, for example, a monitor or display, a speaker, and the like. The input device and / or the output device may be coupled to the data processing system directly or through an intermediate I / O controller. A network adapter 1616 may also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through an intermediate private or public network. The network adapter may include: a data receiver for receiving data transmitted to the data by the system, device, and / or network; and a data transmitter for transmitting data to the system, device, and / or network. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that may be used with the data processing system 1650.

[0155] like Fig.16 As shown, memory element 1604 can store application 1618. It should be understood that data processing system 1600 can also execute an operating system (not shown) that can facilitate the execution of application programs. Application programs implemented in the form of executable program code can be executed by data processing system 1300, for example, by processor 1302. In response to executing the application program, the data processing system can be configured to perform one or more operations that will be further described in detail herein.

[0156] In one aspect, for example, data processing system 1600 may represent a client data processing system. In this case, application 1618 may represent a client application that, when executed, configures data processing system 1600 to perform various functions described herein with reference to a "client." Examples of clients may include, but are not limited to, personal computers, portable computers, mobile phones, and the like. In another aspect, data processing system may represent a server. For example, data processing system may represent a server, a cloud server, or a system of (cloud) servers.

[0157] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "one (kind)" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will be further understood that when the terms "include" and / or "comprise" are used in this specification, it is indicated that there are described features, wholes, steps, operations, elements and / or components, but it does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0158] All parts or steps in the following claims plus the corresponding structures, materials, functions and equivalents of the functional elements are intended to include any structure, material or function for performing a function in combination with other claimed elements specifically claimed. The description of the present invention has been given for the purpose of illustration and description, but is not intended to be exhaustive or to limit the present invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The embodiments are selected and described in order to best explain the principles and practical applications of the present invention and to enable other persons of ordinary skill in the art to understand the various embodiments of the present invention with various modifications suitable for the intended specific use.

Claims

1. A method for detecting an object in an intraoral scan IOS point cloud generated by a 3D optical scanner, the method comprising: determining, by a first deep neural network representing a feature extraction network, a first feature associated with a point of the IOS point cloud, the IOS point cloud comprising points representing one or more objects in at least a 3D space of the IOS point cloud, the one or more objects representing one or more teeth, the first feature defining geometric information for each point of the IOS point cloud, the first deep neural network being configured to receive as input the points of the IOS point cloud; determining, by a second deep neural network representing an object proposal network, a second feature based on the first feature and a uniform or non-uniform 3D grid of nodes in the 3D space of the iOS point cloud, the non-uniform 3D grid comprising a dense distribution of nodes close to an object surface and a sparse distribution of nodes at a distance away from the object surface, the second feature defining local geometric information about the iOS point cloud at locations of the nodes of the 3D grid in the 3D space of the iOS point cloud; generating one or more object proposals based on the second feature, the object proposals defining a 3D bounding box positioned around a node of the 3D mesh, the 3D bounding box containing points of the IOS point cloud capable of defining a tooth, the 3D bounding box defining a 3D anchor point; as well as A score for the 3D anchor point is determined by a third deep neural network representing an object classification network, the score indicating a probability that the 3D anchor point includes a point defining a tooth or a portion of a tooth, the determination being based on a second feature located in the 3D anchor point.

2. The method according to claim 1, wherein: The first features include first feature vectors, each first feature vector is associated with a point of the IOS point cloud; and / or the second features include second feature vectors, each second feature vector is associated with a node of the 3D grid.

3. The method according to claim 1 or 2, wherein: The first deep neural network is configured to receive points of the IOS point cloud and generate the first features.

4. The method according to claim 3, wherein: The first deep neural network includes multiple convolutional layers, which include a multi-layer perceptron (MLP). The feature extraction network is configured to receive points of the IOS point cloud at its input and generate a feature vector for each point of the IOS point cloud at its output.

5. The method according to claim 3, wherein: The feature extraction network includes one or more χ-Conv layers, each χ-Conv layer is configured to weight and permute points and corresponding features provided to the input of the χ-Conv layer, and then subject the permuted points and features to a convolution kernel.

6. The method according to claim 1, wherein: The object proposal network includes a plurality of convolutional layers, each of the plurality of convolutional layers includes a multi-layer perceptron (MLP), and the multi-layer perceptron (MLP) includes one or more convolutional kernels.

7. The method according to claim 6, wherein: The object proposal network is configured as a Monte Carlo convolutional network MCCNet, the Monte Carlo convolutional network MCCNet comprising a plurality of Monte Carlo MC spatial convolutional layers, the Monte Carlo MC spatial convolutional layers comprising a convolution kernel configured to determine a convolution at a position of a node x in the 3D space of the IOS point cloud, determining the convolution comprising: - determining neighboring points y within a receptive field r, the receptive field defining the field of view FOV of the convolution kernel; - Determine the probability density function p(x,y) for each neighboring point y; - Determine the convolution at the node based on Monte Carlo estimation using the neighboring point y and the probability density value p(x,y) of each neighboring point.

8. The method according to claim 1, wherein: The third deep neural network comprises a plurality of fully connected FC multi-layer perceptron (MLP) layers, and the second deep neural network is configured to receive features associated with the 3D anchor point and use the features to determine a score associated with the 3D anchor point, the score indicating a probability that the 3D anchor point comprises a point defining a tooth or a portion of a tooth.

9. A method for instance segmentation of an intraoral scan IOS point cloud generated by a 3D optical scanner, the method comprising: determining, by a first deep neural network representing a feature extraction network, a first feature associated with a point of the IOS point cloud, the IOS point cloud comprising points representing one or more objects in a 3D space of the IOS point cloud, the one or more objects representing one or more teeth, the first feature defining geometric information for each point of the point cloud, the first deep neural network being configured to receive as input the points of the IOS point cloud; determining, by a second deep neural network representing an object proposal network, a second feature based on the first feature and a uniform or non-uniform 3D grid of nodes in the 3D space of the iOS point cloud, the second feature defining local geometric information about the iOS point cloud at locations of the nodes of the 3D grid in the 3D space of the iOS point cloud, the non-uniform 3D grid comprising a dense distribution of nodes close to an object surface and a sparse distribution of nodes at a distance away from the object surface; generating an object proposal based on the second feature, the object proposal defining a 3D volume containing points capable of defining a tooth, the 3D volume of the object proposal defining 3D anchor points positioned around nodes of the 3D mesh; determining, by a third deep neural network representing an object classification network, a classified 3D anchor point, the determination being based on a second set of features that is a subset of second features located in the 3D anchor point; determining, by a fourth deep neural network representing an object position predictor network, an object volume, wherein a center position of the object volume coincides with a center position of the object instance and a size of the object volume matches an outer size of the object instance, the determination being based on the second feature set; Classified points are determined by a fifth deep neural network representing a mask predictor network based on the set of points located in the object volume and the set of first features, the classified points including points belonging to a first classification of teeth and points not belonging to a second classification of teeth.

10. The method according to claim 9, in, The feature extraction network comprises a plurality of convolutional layers, the plurality of convolutional layers comprising a multi-layer perceptron (MLP), the feature extraction network being configured to receive points of a point cloud at its input and to generate a feature vector for each point of the IOS point cloud at its output; and / or Wherein, the object proposal network is configured as a Monte Carlo convolutional network MCCNet, and the Monte Carlo convolutional network MCCNet includes a plurality of Monte Carlo MC spatial convolutional layers; and / or Wherein, the object classification network comprises a plurality of fully connected FC multi-layer perceptron MLP layers; and / or wherein the object position predictor network comprises a plurality of fully connected FC multilayer perceptron MLP layers; and / or The mask predictor network comprises one or more χ-Conv layers, each χ-Conv layer being configured to weight and permute the input points and corresponding features provided to the χ-Conv layer, and then subjecting the permuted points and features to a convolution kernel.

11. A method for training a deep neural network system for object detection in an intra-oral scan IOS point cloud generated by a 3D optical scanner, the method comprising: Providing a training point cloud sample to an input of the deep neural network system, the training point cloud sample comprising one or more labeled object instances, the deep neural network system comprising at least a feature extraction network, an object proposal network, and an object classification network; computing an object proposal based on a second feature, the second feature defining local geometric information about the IOS point cloud at locations of nodes of a uniform or non-uniform 3D grid in the 3D space of the IOS point cloud, the non-uniform 3D grid comprising a dense distribution of nodes close to the object surface and a sparse distribution of nodes at a distance away from the object surface, the second feature being determined by the object proposal network, wherein the object proposal network is configured to determine the second feature based on the first feature generated by the feature extraction network, the feature extraction network receiving the training point cloud samples at its input, the object proposal defining a 3D bounding box around the nodes of the 3D grid, the 3D bounding box being able to include points defining a tooth, the 3D bounding box defining a 3D anchor point; determining an overlap between the 3D anchor point and a 3D bounding box of a labeled object instance in the training point cloud sample, and marking the 3D anchor point as positive if the overlap is above a predetermined threshold, and marking the 3D anchor point as negative if the overlap is below a predetermined threshold; determining, by the object classification network, one or more object predictions for the positively labeled and / or negatively labeled 3D anchor points using point cloud features in the 3D anchor points, and determining a first loss value based on the one or more object predictions, the positively labeled and / or negatively labeled 3D anchor points, and a first loss function; and Using the first loss value, the feature extraction network, the object proposal network, and the object classification network are trained using a back-propagation method.

12. A method for training a deep neural network system for instance segmentation of an intra-oral scan IOS point cloud generated by a 3D optical scanner, the method comprising: providing a training point cloud sample comprising one or more labeled object instances representing one or more teeth to an input of a deep neural network system, the deep neural network system comprising at least a feature extraction network, an object proposal network, an object classification network, an object location predictor network, and a mask predictor network; computing an object proposal based on a second feature, the second feature defining local geometric information about the IOS point cloud at locations of nodes of a uniform or non-uniform 3D grid in the 3D space of the IOS point cloud, the second feature being determined by the object proposal network, the non-uniform 3D grid comprising a dense distribution of nodes close to an object surface and a sparse distribution of nodes at a distance away from the object surface, wherein the object proposal network is configured to determine the second feature based on the 3D grid of nodes and a first feature generated by the feature extraction network, the feature extraction network receiving at its input a training point cloud sample, the object proposal defining a 3D bounding box around a node of the 3D grid, the 3D bounding box being able to include points defining a tooth, the 3D bounding box defining a 3D anchor point; determining an overlap between the 3D anchor point and a 3D bounding box of an object instance labeled in the training point cloud sample, and determining a positive 3D anchor point if the overlap is above a predetermined threshold, and determining a negative 3D anchor point if the overlap is below a predetermined threshold; determining, by the object classification network, one or more object predictions for the positively labeled and negatively labeled 3D anchor points using point cloud features in the 3D anchor points, and determining a first loss value based on the one or more object predictions and a first loss function; determining, by the object position predictor network, a position and size prediction of an object volume based on features in the 3D anchor point, and determining a second loss contribution using the position and size prediction and a second loss function; determining, by the mask predictor network, classified points based on point cloud features in the object volume, the classified points including points belonging to a first classification of teeth and points not belonging to a second classification of teeth, and determining a third loss contribution using the classified points and a third function; and The feature extraction network, the object proposal network, the object classification network, the object location predictor network, and the mask predictor network are trained using a back-propagation method using the first loss contribution, the second loss contribution, and the third loss contribution.

13. A computer system suitable for detecting teeth in an intraoral scan IOS point cloud generated by a 3D optical scanner, the system comprising: A computer-readable storage medium containing a computer-readable program code, wherein the program code includes a preprocessing algorithm and at least a trained first 3D deep neural network, a computer-readable program code; and a processor coupled to the computer-readable storage medium, wherein, in response to executing the first computer-readable program code, the processor is configured to perform executable operations including: determining, by a first deep neural network representing a feature extraction network, a first feature associated with a point of the IOS point cloud, the IOS point cloud comprising points representing one or more teeth in a 3D space of the IOS point cloud, the first feature defining geometric information for each point of the IOS point cloud, the first deep neural network being configured to receive as input the points of the IOS point cloud; determining, by a second deep neural network representing an object proposal network, a second feature based on the first feature, the second feature defining local geometric information about the iOS point cloud at locations of nodes of a uniform or non-uniform 3D grid in the 3D space of the iOS point cloud, the non-uniform 3D grid comprising a dense distribution of nodes close to an object surface and a sparse distribution of nodes at a distance away from the object surface; generating one or more object proposals based on the second feature, the object proposals defining a 3D bounding box positioned around a node of the 3D mesh, the 3D bounding box containing points of the IOS point cloud capable of defining a tooth, the 3D bounding box defining a 3D anchor point; and A score for the 3D anchor point is determined by a third deep neural network representing an object classification network, the score indicating a probability that the 3D anchor point includes a point defining a tooth or a portion of a tooth, the determination being based on a second feature located in the 3D anchor point.

14. A computer system adapted for instance segmentation of an intraoral scan IOS point cloud generated by a 3D optical scanner, the system comprising: A computer-readable storage medium containing a computer-readable program code, the program code comprising a preprocessing algorithm and at least a trained first 3D deep neural network, the computer-readable program code; and a processor coupled to the computer-readable storage medium, wherein, in response to executing the first computer-readable program code, the processor is configured to perform executable operations including: determining, by a deep neural network of a first type representing a feature extraction network, first features associated with points of the IOS point cloud, the IOS point cloud comprising points representing one or more teeth in a 3D space of the IOS point cloud, the first features defining geometric information for each point of the IOS point cloud; determining, by a second type of deep neural network representing an object proposal network, a second feature based on the first feature, the second feature defining local geometric information about the iOS point cloud at locations of nodes of a uniform or non-uniform 3D grid spanning the 3D space of the iOS point cloud, the non-uniform 3D grid comprising a dense distribution of nodes close to an object surface and a sparse distribution of nodes at a distance away from the object surface, the first type of network being configured to receive points of the iOS point cloud as input; generating an object proposal based on the second feature, the object proposal defining a 3D volume containing points capable of defining a tooth, the 3D volume of the object proposal defining 3D anchor points positioned around nodes of the 3D grid in the 3D space of the IOS point cloud; determining, by a deep neural network of a third type representing an object classification network, a classified 3D anchor point, the determination being based on a second set of features being a subset of second features located in the 3D anchor point; determining, by a fourth type of deep neural network representing an object position predictor network, an object volume, a center position of the object volume coinciding with a center position of the object instance and a size of the object volume matching an outer size of the object instance, the determination being based on the second feature set; and Classified points are determined by a fifth type of deep neural network representing a mask predictor network based on a set of points located in the object volume and a set of first features, the classified points including points belonging to a first class of teeth and points not belonging to a second class of teeth.

15. A computer program product comprising software code portions configured to, when run in a memory of a computer, perform the method steps according to any one of claims 1 to 12.

Images