Method to generate an object using hourglass predictor

Abstract

A computer-implemented method for generating 2D or 3D objects is disclosed, comprising training an autoencoder on a first set of training data to identify a first set of latent variables and generate a first set of output data; training an hourglass predictor on a second set of training data, wherein the hourglass predictor encoder converts a related but different set of training input data into a second set of latent variables that decode into a second set of output data of the same type as the first set of output data; and using the hourglass predictor to predict a 2D or 3D object of the same type as the first set of output data based on the 2D or 3D object of the same type as the second set of input data.

Description

Technical Field

[0001] The present disclosure is generally directed to a system and method for generating two-dimensional or three-dimensional objects by an hourglass predictor using information from similar types of objects and related objects. Various embodiments of the present disclosure are directed to applying these methods to dental, orthodontic, and ear-related objects. Background Art

[0002] Machine learning can be used to derive information from large sets of data and use that data to generate new objects. However, many methods for generating new objects only use objects of similar types. These methods do not consider related data that does not come from the objects themselves, and therefore may miss important considerations about the environment and other factors. Summary of the invention

[0003] A computer-implemented method configured to generate output data is disclosed. The method comprises:

[0004] training an autoencoder on a first set of training input data to identify a first set of latent variables and to generate a first set of output data, wherein the autoencoder comprises a first encoder and a first decoder, wherein the first encoder converts the first set of input data into a first set of latent variables, wherein the first decoder converts the first set of latent variables into a first set of output data, wherein the first set of output data is at least substantially the same as the first set of training input data;

[0005] training an hourglass predictor to return a second set of latent variables, wherein the hourglass predictor includes a second encoder and a first decoder, wherein the second encoder converts a second set of training input data into a second set of latent variables, wherein the second set of latent variables has a data format comparable to the first set of latent variables, and the first decoder converts the second set of latent variables into a second set of output data that is at least substantially the same as a set of training target data, and the second set of training input data is different from the first set of training input data; and

[0006] The hourglass predictor is used on a third set of input data to generate a third set of output data, wherein the third set of output data is in a data format comparable to the data format of the first set of output data.

[0007] Typical machine learning methods generate new objects based on existing objects of the same type. For example, a dental crown can be generated by training a machine learning method on a large data set of existing dental crowns. However, the dental crowns generated in this way do not take into account the unique circumstances of the dental environment, such as available space. The hourglass predictor adds information from the dental environment to generate a dental crown that is similar to a crown that a dental professional might make for a unique set of dental environments. This embodiment is a single example of the flexibility of the hourglass predictor, as the hourglass predictor can be used with any relevant input data set.

[0008] The hourglass predictor can be configured to generate objects based not only on similar objects but also on related objects. For example, one embodiment uses the hourglass predictor to generate a crown based on a dental environment from a dental patient. A crown is a dental restoration that covers or covers a tooth to restore it to normal shape, size, or function. The dental environment is a collection of objects surrounding a crown or crown preparation, including but not limited to: adjacent teeth, opposing teeth, gums, jaws, and / or preparations. Currently, crowns are typically handmade or manually designed using CAD software based on the judgment of a dental professional. This is time consuming and highly dependent on the personal judgment of the dental professional.

[0009] The first step in creating an hourglass predictor is to train an autoencoder. An autoencoder takes input data, encodes the input data into a set of latent variables, and then decodes the latent variables into output data, where the output data is at least substantially the same as the input data, as described below. They are often used in denoising applications, such as removing artifacts from digital photographs. The objects themselves may be difficult to use as input data, so a representation of the object may be used instead. For example, a 2D image may represent a tooth.

[0010] One embodiment uses a variational autoencoder. A variational autoencoder is an autoencoder in which the latent variables are generated from a probability distribution rather than directly from the encoder. Using a probability distribution allows the data to be interpolated to make reasonable estimates when similar input data is not available. For example, the probability distribution can be generated from the mean and standard deviation vector or the logarithm of the variance.

[0011] To train the autoencoder, a first encoder converts a first set of training input data into a first set of latent variables, and then a first decoder converts the first set of latent variables into a first set of output data. The encoder and decoder may be neural networks. Examples of neural networks include, but are not limited to, convolutional neural networks and dense neural networks.

[0012] In embodiments involving generating a crown, the crown may be represented by a sampling matrix. In the process discussed below, the sampling matrix represents a 3D object. The sampling matrix enables the neural network to be applied to three-dimensional objects and to treat multiple three-dimensional objects consistently so that they can be compared.

[0013] The autoencoder can use multiple sampling matrices as the first set of training input data. Each sampling matrix can be represented as a first set of latent variables. In this embodiment, there can be at least three latent variables, each of which can be a scalar number. The first decoder then decodes each of the first set of latent variables back into a sampling matrix.

[0014] Training is complete when the first set of output data is at least substantially identical to the first set of input data. In an embodiment where the autoencoder is used to generate crowns, this can be measured by the average distance between points on the surface of the generated crowns and the original crowns, and a threshold for sufficiently similarity can be an average distance equal to or less than 0.1 mm.

[0015] Training the autoencoder enables the first set of latent variables to parameterize a type of object and allows the latent variables to be decoded back into a sampled matrix that corresponds to the representation of the underlying object.

[0016] However, an autoencoder by itself only generates output data of the same type as the input data, without considering other constraints. For example, an autoencoder that creates a crown can only generate the crown without considering additional information, such as the adjacent teeth of the crown. In the second step, an hourglass predictor is trained, which allows the incorporation of additional information when generating new objects.

[0017] The hourglass predictor keeps the first decoder from the autoencoder unchanged. However, it uses a second set of training data, where the second set of training input data has different but related underlying objects than the first set of training data. Although these data sets may not be the same type of objects, they can be related to each other because each data set has some information about the other data set. The training target data has the same type of underlying objects as the first set of training input data.

[0018] For example, in the embodiment of making crowns, the second set of training input data is a set of sample matrices derived from the dental surroundings of the crown or preparation, and the training target data is crowns that fit these surroundings. Having one set of data will tell us something about the other set of data, for example, the size of a crown designed for a particular set of surroundings will reveal something about the distance between adjacent teeth.

[0019] To train the hourglass predictor, a second set of training input data is encoded by a second encoder to return a second set of latent variables. The second set of latent variables has the same data format as the first set of latent variables. In an embodiment of generating a crown, the data format of the latent variable group is a vector of scalar numbers. The second set of latent variables is then decoded by the first decoder to return a second set of output data, which has the same type of underlying objects as the first set of output data. The second set of output data can at least substantially match the training target data. Note that these steps train the second encoder to return a second set of latent variables that can be decoded into the same type of objects as the first set of output data.

[0020] In embodiments where a crown is generated, a representation of the crown's surroundings may be used to train a second encoder to return latent variables that decode to the crown. Training may be complete when the generated crown is at least substantially identical to the initial crown, e.g., by the same metric as described above for training the autoencoder.

[0021] Once the hourglass predictor is trained, it can be used to generate a third set of output data that has the same type of underlying objects as the first set of training input data, where the third set of input data has the same type of underlying objects as the second set of training data.

[0022] For the embodiment of generating a crown, this means that given a set of dental surroundings, a crown can be generated that predicts what a dental professional would create for those dental surroundings. The hourglass predictor is flexible in application; as described below, it can be trained for different situations, generating different types of objects.

[0023] In one embodiment, the hourglass predictor may be used to generate a proposed digital 2D image of a desired dental setting based on the pre-treatment digital 2D image.

[0024] The desired dental setup represents a dental setup proposed for a patient and can be used to design a dental restoration or plan an orthodontic treatment. The desired dental setup may have a proposed dental treatment and may be based on an actual case designed by a dental professional. The proposed digital 2D image is a 2D image of the desired dental setup.

[0025] A pre-treatment digital 2D image is a digital image of a patient before treatment. It can be a 2D image showing the patient's smile, where a portion of the upper front teeth is visible.

[0026] In dental treatment, it is important to be able to give the patient an aesthetically pleasing smile while maintaining functionality. There are some planning tools in which the expected treatment results are presented to the patient so that the patient can see the final smile. The hourglass predictor in this embodiment can provide a means to do this. For example, a dental professional can use a proposed 2D digital image of a desired dental setting to show a patient how her teeth will look after a proposed course of treatment.

[0027] To train the autoencoder, the first set of input training data will be a set of proposed digital 2D images of the desired dental setting. The 2D images can be direct 2D images, such as photographs and / or renderings of 3D models as 2D images, such as 2D images based on 3D scans. Similarity can be measured by a common metric for comparing digital images, such as the structural similarity index (SSIM) (Wang, Zhou, Eero P. Simoncelli, and Alan C. Bovik, "Multiscale structural similarity for image quality assessment," 37th Asilomar Conference on Signals, Systems, and Computers, 2003, Vol. 2, IEEE, 2003.)

[0028] For training the hourglass predictor, the second set of input training data will be a set of pre-treatment digital 2D images, e.g., digital photographs of patients before treatment. The set of training target data may include a set of corresponding proposed digital 2D images of desired dental settings, e.g., proposed or actually treated digital 2D images of cases from pre-treatment digital 2D images. The similarity metric is similar to the metric used to train the autoencoder.

[0029] Thus, the trained hourglass predictor can generate proposed digital 2D images of desired dental settings for new cases represented by pre-treatment digital 2D images, similar to those designed by dental professionals. This automates work previously done manually by dental professionals, which can save time in treating dental patients. If necessary, the dental professional can further modify the proposed digital 2D images of the desired dental settings.

[0030] Embodiments also include cases where the first set of latent variables and / or the second set of latent variables are used as parameterization of the input data and / or the output data.

[0031] Parameters can be used to define and change the objects they represent. For example, an object may be defined as having a height of 5 mm, but the height may be changed to 10 mm. Objects may be parameterized in Euclidean space, where they may be translated, scaled, rotated, sheared, reflected, mapped, and / or otherwise parameterized by various functions.

[0032] In addition to traditional Euclidean parameters, machine learning methods can allow for new and meaningful parameters based on information about the object. The latent variables of the hourglass predictor are examples of such parameters. A set of latent variables defines the shape of the object; at least one latent variable can also be used to change the shape of the object.

[0033] For example, a latent variable that is a parameter can be used to change the three-dimensional shape of a crown for a better occlusion by changing multiple parts of the tooth at once. If the change is unsuccessful, the parameter value can simply be set back to its original value. Parameters also allow for optimization. When an object changes according to a parameter value, the parameter can be changed until some aspect is optimized.

[0034] Embodiments also include situations where the first set of training input data, the second set of training input data, the set of training target data and / or the third set of input data are based on a three-dimensional object, such as a three-dimensional scan of the 3D object.

[0035] A three-dimensional object is an object that exists in three dimensions. In various embodiments of the present disclosure, three-dimensional objects include, but are not limited to, physical objects such as dental restorations and dentition, digital objects such as individual tooth models, and scans of physical objects such as dental restorations and ears.

[0036] Embodiments also include wherein the first set of training input data, the second set of training input data, the third set of input data, the set of training target data, the first set of output data, the second set of output data and / or the third set of output data include one or more of: a 2D image, a depth map, a pseudo image, a point cloud, a 3D mesh and / or volumetric data of an object.

[0037] Three-dimensional objects can be represented using both two-dimensional and three-dimensional representations.

[0038] The two-dimensional representation of a three-dimensional object includes, but is not limited to, a depth map, a pseudo image based on surface information, and other multi-angle images.

[0039] Two-dimensional representations can be relatively easy to use in an hourglass predictor, since they are typically images with a structure that an encoder / decoder neural network can use directly. For example, a two-dimensional image can consist of an array of pixels, where pixels have a defined position relative to other pixels, and RGB or grayscale values ​​that can be read as scalar numbers. Scalar numbers can be normalized and subjected to other processes that are useful in neural network preprocessing.

[0040] The three-dimensional representation of the three-dimensional object includes, but is not limited to: volume representation, point cloud, primitive-based model, mesh.

[0041] Three-dimensional representations can also be used by the hourglass predictor, although their data format may require some preprocessing before being used by the neural network. Three-dimensional machine learning can be a challenging problem in the field because three-dimensional data formats may be too large to process efficiently and commonly used machine learning operations may not exist for three-dimensional data formats.

[0042] Volumetric representations, which are essentially three-dimensional pixels, can be represented in a matrix format where each element has three dimensions - x, y, and z coordinates. Point clouds are collections of points in three-dimensional space and can be used by systems such as PointNet (Qi, Charles R. et al., "Pointnet: Deep learning on point sets for 3D classification and segmentation," Proceedings of Computer Vision and Pattern Recognition (CVPR), IEEE 1.2 (2017): 4). Primitives can also be used as neural networks, for example, by at least one depth image and a number containing information about the configuration of the primitive (Zou, Chuhang et al., "3d-prnn: Generating shape originals with recurrent neural networks," 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, 2017).

[0043] A mesh is a collection of vertices, edges, and faces. A vertex is a single point representing a surface, edges are lines connecting vertices, and faces are continuous regions enclosed by vertices and edges. Meshes can also be used in neural networks, either in a similar way to point clouds, or through the process discussed below.

[0044] The embodiment also includes a method, wherein the first set of training input data, the second set of training input data, the third set of input data, the set of training target data, the first set of output data, the second set of output data, and / or the third set of output data are corresponding 3D grids corresponding to a sampling matrix, wherein the sampling matrix is ​​generated by a method comprising:

[0045] An initial three-dimensional mesh is transformed into a planar mesh, the initial three-dimensional mesh comprising a first set of vertices and edges, and the planar mesh comprising a second set of vertices and edges, wherein:

[0046] - each vertex in the second set of vertices is a transformation of a vertex from the first set of vertices and includes the value of the vertex from the first set of vertices, and

[0047] - each edge in the second set of edges is a transformation of an edge from the first set of edges and includes the value of the edge from the first set of edges;

[0048] A planar grid is sampled to generate a plurality of samples such that each sample from the plurality of samples includes:

[0049] - three-dimensional coordinates comprising three values ​​representing a point in three-dimensional space, wherein the three values ​​are directly derived and / or obtained from the initial three-dimensional mesh, and

[0050] - coordinates comprising numerical values ​​representing the position of the sample relative to other samples in the plurality of samples; and

[0051] Generate a sampling matrix based on multiple samples;

[0052] Represent the sampling matrix as a corresponding 3D grid.

[0053] While 3D meshes are a popular way to represent 3D objects, they are difficult to use directly with many machine learning algorithms. 3D meshes are often non-uniform and have non-deterministic features, making it difficult to directly compare them via machine learning. Commonly used machine learning operations, such as convolutions, may not exist for 3D meshes. Additionally, computer-readable formats of 3D meshes can be very large, requiring a lot of processing power to analyze.

[0054] The embodiment uses an hourglass predictor on a three-dimensional grid, through a 3D grid corresponding to a sampling matrix. The method flattens the three-dimensional grid into a two-dimensional grid, representing them as a matrix. In addition, the matrix can be a unified data format for representing different objects.

[0055] In various embodiments, the method may be particularly useful in generating dental, orthodontic, and ear-related objects, as discussed below.

[0056] Although some embodiments in the present disclosure use three-dimensional meshes from three-dimensional scans, any three-dimensional mesh that can be embedded in a plane may be used.

[0057] The initial 3D mesh may be from an open surface or a closed surface. Examples of open surfaces include, but are not limited to: the outer surface of a crown, a 3D scan of the tooth to the gum margin, a 3D scan of a jaw. Examples of closed surfaces include, but are not limited to: the entire surface of a crown, a 3D model of a tooth, a 3D scan of an implant guide.

[0058] In order to flatten the initial three-dimensional mesh into a planar mesh, a boundary should be set. In the embodiment of a three-dimensional object with an open surface, the boundary can be a boundary of the open surface, a cut formed along the surface, and / or a combination of the two.

[0059] In embodiments of three-dimensional objects with zero-generic closed surfaces, the boundary may be a single facet, or at least one cut along the surface. For closed surfaces, there may be obvious locations to make cuts, such as the edge between the top and bottom. In a three-dimensional mesh of teeth, locations to make cuts include, but are not limited to: anatomical features such as the gum margin, the long axis of the tooth; geometric features such as a plane along a hemisphere; and / or any axis orthogonal to these.

[0060] Cutting a 3D mesh can produce multiple planar meshes. This can be useful for capturing more detail of a 3D object.

[0061] Then the surface boundary of the three-dimensional object is fixed to the boundary of the planar object. The planar object can be of different shapes, including but not limited to: triangle, quadrilateral, circle, curved quadrilateral including square and circle, and the shape based on the three-dimensional object itself.

[0062] Once the boundaries are fixed, the remaining vertices and edges are mapped to the planar objects, thereby flattening the initial three-dimensional mesh into a planar mesh. Various embodiments may use different methods for this mapping. Initially, a matrix of the connectivity of each vertex to other vertices can be created, and the matrix system is solved to map the coordinates of each vertex to the plane (Tutte, William Thomas, "How to draw a graph," Journal of the London Mathematical Society 3.1 (1963): 743-767). This connectivity can be weighted in different ways, resulting in different mappings. In various embodiments, these methods include, but are not limited to: uniform weights (Tutte, 1963), angle-based weights (Floater, Michael S. and Ming-Jun Lai, "Polygonal spline space and the numeric solution of the Poisson equation," SIAM Journal of Numerical Analysis 54.2 (2016): 797-824), cotangent-based weights (Meyer, Mark et al., "Discrete differential-geometry operators for triangulated 2-manifolds," Visualization and Mathematics III, Springer, Berlin, Heidelberg, 2003. 35-57).

[0063] See below for an example of flattening an initial 3D scan into a planar mesh.

[0064] Flattening the initial 3D mesh in this way allows the initial 3D mesh and the flat mesh to be bijective, meaning that each vertex can be mapped back and forth between the two meshes.

[0065] The plane grid is then sampled, where each sample is a point on the plane grid. The plane grid can be sampled randomly, using lattice sampling, or a combination of the two.

[0066] Lattice sampling samples based on a lattice on a planar grid. In various embodiments, the lattice can be based on polar or rectangular coordinates, regular or irregular. A regular lattice has evenly spaced lines, such as a regular grid of squares or triangles. An irregular lattice does not have evenly spaced lines and may be useful in situations where information is not evenly distributed. For example, a planar grid of a tooth surface with its boundary set at the gum margin has more information in the center of the grid than at the edge. Therefore, an irregular lattice that collects more samples from the center has more useful information about the tooth. See below for more examples.

[0067] Lattice-based samples can be taken from line intersections, different positions along a line, or even from cells of the lattice. Lattice sampling can be combined with random sampling, for example, by selecting a random point from each lattice cell.

[0068] Each sample on the planar grid has a corresponding 3D Euclidean coordinate based on the initial 3D grid. If the sample is on a vertex, the Euclidean coordinate is the same as the vertex. If the sample is on an edge or face, the Euclidean coordinate is interpolated based on the vertices of the edge or face.

[0069] Each sample also has a position on the lattice. The lattice can be the basis of a sampling matrix, where each element of the matrix is ​​a sample and the position of each element corresponds to the lattice position of the sample. The sampling matrix can be thought of as a two-dimensional array, each with three dimensions of elements; three two-dimensional arrays, each with one dimension of elements; or any other format that can express both the Euclidean coordinates of the samples and the lattice positions.

[0070] The sampling matrix can be visually represented as a two-dimensional red, green, and blue matrix (2D RGB matrix). In a two-dimensional RGB matrix, the position of each matrix element is the lattice position of the sample, and the RGB value of each matrix element is the Euclidean coordinate of the sample. The 2D RGB matrix is ​​equivalent to the sampling matrix, and the two can be converted back and forth.

[0071] The sampling matrix can be analyzed using machine learning methods for two-dimensional data. This requires less computational power than similar methods for three-dimensional data. In addition, there is more work on two-dimensional machine learning methods, which may allow new methods that could not be applied to three-dimensional objects before.

[0072] Since each element of the sampling matrix has a specific position relative to its neighbors, machine learning methods that require known neighbors can be applied. For example, convolutions pool several neighboring elements to better capture the underlying information about the object and reduce the risk of changing the overall result. These are used in convolutional neural networks.

[0073] The sampling matrix also allows other improvements. The sampling matrix can represent objects with fewer data points, which requires less processing power. In addition, the sampling matrix allows a uniform data format for non-uniform objects, thereby allowing direct comparison between 3D objects.

[0074] The present disclosure also includes representing the sampling matrix as a corresponding 3D grid. Since each element of the sampling matrix includes a three-dimensional point in Euclidean space, it can also be represented as a point cloud. Analysis of the sampling matrix will also be an analysis of the point cloud, and manipulation of the sampling matrix will have a corresponding effect on the point cloud.

[0075] Point clouds can be reconnected into 3D meshes. Although point clouds can be reconnected in any way, triangular meshes are typical for computer-aided design and manufacturing.

[0076] Embodiments also include situations where the third set of output data is used to estimate unscanned and / or erroneously scanned areas.

[0077] The hourglass predictor can be used to estimate unscanned and / or incorrectly scanned areas. Often, obtaining a complete scan of an object is difficult due to location, blocking objects, or the shape of the object itself. In this case, having a 3D model mesh that matches the scan allows the underlying object to be modeled anyway, as described below.

[0078] In an embodiment of estimating unscanned and / or erroneously scanned areas, to train the autoencoder, the first set of training input data may be a 3D mesh of a model object. To train the hourglass predictor, the second set of training input data may use incomplete and / or erroneous scans of similar types of objects, and the set of training target data may be 3D meshes corresponding to incomplete and / or erroneous scans of similar types of objects. Thus, the third set of output data may be a 3D mesh of a model object fitted to the incomplete and / or erroneous scans. A dental example is discussed below.

[0079] Embodiments also include a method of estimating an unscanned and / or erroneously scanned area, wherein the unscanned and / or erroneously scanned area is a tooth area or an ear area.

[0080] In dental scanning, the patient's teeth and jaws may be scanned, for example, by using an intraoral scanner on the patient, or by scanning a plaster model of the patient's oral condition. However, certain areas of individual teeth may not be fully scanned, including but not limited to: interproximal areas, where adjacent teeth block the scanner; areas under brackets and other appliances, where the appliances block the scanner; and subgingival areas, where the gums block the scanner. For facial scans, the tooth area includes areas behind facial features, where the lips may block the scanner. In these cases, using a known tooth or library model as an initial model, the incomplete area can be estimated with the help of an hourglass predictor.

[0081] For example, for an incomplete scan of an incisor, at least one unscanned and / or incorrectly scanned region may be estimated using an hourglass predictor. When training an autoencoder, a first set of training input data may be a set of 3D representations of complete incisors, possibly taken from a scan or digital model of the complete incisors. The 3D representations of the complete incisors may also be parameterized by latent variables of the autoencoder.

[0082] When training the hourglass predictor, the second set of input data may be a set of 3D representations of incomplete and / or erroneous scans of the incisors. The training target data set for this second set of input training data will be a corresponding set of 3D representations of complete incisors. The training data set may be created artificially, for example, by cutting away portions of the incisor surface that are not scanned in a typical scan.

[0083] Training an hourglass predictor on this training data set results in an hourglass predictor that outputs an estimate of a 3D representation of a complete incisor based on an input of an incomplete and / or erroneously scanned 3D representation of the incisor.

[0084] The third set of output data from the hourglass predictor can be used directly, or portions of it can be spliced ​​together with existing scans.

[0085] The estimation of unscanned and / or incorrectly scanned areas can be used to simulate orthodontic treatment. Orthodontic treatment may move or rotate teeth, and tooth surfaces that are visible in the final result may not be scanned initially. The desired surfaces include, but are not limited to: interproximal areas, subgingival areas, tooth roots, and / or any portion of a combination of the above. Therefore, the estimation of unscanned and / or incorrectly scanned areas of the teeth can be used to simulate treatment.

[0086] Embodiments also include estimating at least one unscanned and / or incorrectly scanned ear region. Ear regions include, but are not limited to, the first bend, the second bend, the anterior notch, the antihelix, the antitragus, the concha, the ear canal, the ear canal including the hole, the helix, the intertragic notch, and / or the tragus.

[0087] Scanning the ear can be difficult. The size and curvature of the ear canal limits the size and reach of the scanner. Portions of the ear may be obstructed by hair, cerumen, and / or other parts of the ear. The ear is not rigid, so the scanner or other objects may deform the ear and affect the scan. Therefore, it is helpful to estimate unscanned and / or incorrectly scanned areas of the ear to model what the ear scan should be, even if the original scan is incomplete or otherwise erroneous.

[0088] For estimating the hourglass predictor of the ear scans, complete ear scans can be used as the first set of input data for training the autoencoder. For training the hourglass predictor, the second set of input training data can be incomplete and / or erroneous ear scans, and the training target data set can be the corresponding complete ear scans. These complete ear scans can come from laboratory scans of ear casts or a set of good scans. The process will be similar to the hourglass predictor described above for estimating the incisors.

[0089] Embodiments also include using the hourglass predictor to design a dental restoration, an orthodontic appliance, or an ear-related device using at least one of the third set of output data, any portion thereof, and / or a combination thereof.

[0090] Dental restorations include one or more of the following: bridges, dentures, crowns, implants, implant guides, inlays, onlays, posts and cores, and / or veneers. Orthodontic appliances include one or more of the following: bracket guides, clear aligners, expanders, lingual wires, and / or quad helices. Ear-related devices include one or more of the following: worn hearing aids, in-ear monitors, and / or noise protection devices.

[0091] By changing the training data, different types of objects can be generated. For example, in an embodiment of generating a crown, the first set of training input data can be a plurality of crowns and / or teeth as veneer models, the second set of training input data can be a plurality of surroundings of the crown, and the set of training target data can be crowns corresponding to these surroundings. A detailed example of generating a crown using an hourglass predictor is discussed below.

[0092] Embodiments also include using an hourglass predictor to identify immovable objects in a three-dimensional scan, thereby allowing movable objects to be excluded from the three-dimensional scan.

[0093] When scanning an immovable object to obtain a virtual 3D model of the immovable object, a movable object may be captured in a sub-scan. For example, when scanning a dental patient's oral condition using a dental scanner, movable objects such as the patient's cheeks, tongue, or instruments or fingers of a dental professional may be included.

[0094] The goal is to use only surfaces from immovable objects when generating a virtual 3D model by distinguishing between surfaces from movable objects and surfaces from immovable objects.

[0095] An hourglass predictor for estimating an immovable object may train an autoencoder using scans of known immovable objects as a first set of input data. To train the hourglass predictor, the second set of input training data may be scans of the immovable object along with a movable object, and the set of training target data may be corresponding scans of the immovable object. The corresponding scan of the immovable object may be, for example, a scan of the jaw and dentition of a dental patient, and the scan of the immovable object along with the movable object may be a corresponding scan of the jaw and dentition of a dental patient including a movable object (e.g., tongue, finger, cotton ball, etc.). The process will be similar to the hourglass predictor described above for estimating incisors.

[0096] In some embodiments, the non-removable object is a patient's oral condition. Removable objects include, but are not limited to: soft tissues of portions of a patient's oral cavity, such as the inside of a cheek, tongue, lips, gums, and / or loose gums; dental appliances or remedies temporarily present in a patient's oral cavity, such as dental suction devices, cotton rolls, and / or cotton pads; fingers, such as those of a dental professional.

[0097] Embodiments also include using an hourglass predictor to identify objects, object categories.

[0098] As mentioned above, the latent variables of the hourglass predictor can be parameterized by a type of object. Therefore, the latent variables from the encoded object may be unique enough to distinguish the object from other objects of the same type.

[0099] In an embodiment, an hourglass predictor may be used to identify a class of teeth based on incomplete and / or erroneous scans. To train the autoencoder, a first set of input training data may be 3D representations of complete teeth. To train the hourglass predictor, a second set of input training data may be 3D representations of incomplete and / or erroneous scans of teeth, and the set of training target data may be 3D representations of corresponding 3D representations of complete teeth.

[0100] A set of 3D representations of complete teeth with labels of tooth type (incisor, bicuspid, molar, premolar, etc.) can then be run through the first encoder. Latent variables are retained and annotated with the labels. Based on the value, range, and / or other characteristics of the latent variables, categories can be divided. When new incomplete and / or erroneous scans are encoded as latent variables using the second encoder, these latent variables can be compared to the categories and the type of tooth identified.

[0101] In an embodiment, this can also be used to identify similar objects, for example, in previous scans from the same dental patient in a patient monitoring system for dental care. Only an autoencoder would be required for this. The first set of training input data could be a set of jaw and dentition scans. The first encoder would be trained to encode these into a first set of latent variables. New input data, such as a new scan of the jaw and dentition, could then be encoded by the first encoder into a set of latent variables. The latent variables could be unique enough to identify similar scans based on a new scan of the jaw and dentition, especially compared to multiple latent variables for existing scans of the jaw and dentition. The multiple latent variables for existing scans of the jaw and dentition could be based on a set of existing records from a dental clinic or laboratory.

[0102] Also disclosed is a computer-implemented method for transforming an object, comprising:

[0103] training an autoencoder on a set of training input data to identify latent variables and generate a set of training output data, wherein the autoencoder comprises an encoder and a decoder, wherein the encoder converts the training input data into latent variables, wherein the decoder converts the latent variables into training output data, wherein the training output data is at least substantially the same as the training input data;

[0104] identifying a transformation vector having at least one data pair, wherein each of the at least one data pair includes an initial object in an initial state and a transformed object in a transformed state, by: generating an initial state latent variable for each initial object by using a representation of each initial object as input data to an encoder; generating a transformed state latent variable for each transformed object by using a representation of each transformed object as input data to the encoder;

[0105] identifying a transformation function based on a difference between the initial state latent variable and the transformed state latent variable; and

[0106] A new object is transformed using a transformation function by: generating new latent variables by using a representation of the new object as input data for an encoder; generating transformed new latent variables by applying a transformation vector and / or a function based on the transformation vector to the new latent variables; and generating transformed output data by decoding the transformed new latent variables using a decoder.

[0107] In an embodiment, an object may be transformed into a modified version of itself by generating a transformation vector of latent variables. New objects may be transformed in a similar manner by generating a transformation function from a data pair of the original object and the transformed object. This may be used, for example, to predict the aging of teeth, as described below. The teeth will "age" with a transformation function derived from the data pair of the original tooth and the aged tooth. Furthermore, an embodiment may use a vector as a transformation function, allowing the transformation function to act as a "slider" to age the teeth at different intervals, allowing a time-lapse view of what the teeth will look like as they age.

[0108] An embodiment of implementing this first trains an autoencoder to recognize latent variables. The autoencoder can be trained by encoding training input data into latent variables, decoding the latent variables into training output data, and adjusting the weights of the encoder and decoder accordingly so that the training output data is at least substantially the same as the training input data. This process is discussed in more detail above.

[0109] After training the autoencoder, at least one data pair is used as input data for the encoder. The data pair includes an initial object in an initial state and a transformed object in a transformed state. Examples of data pairs include: an initial tooth and an aged tooth, an initial tooth and a tooth with a convex cusp, an initial tooth and a tooth with increased angular symmetry.

[0110] The initial object and the transformed object may need to be converted into a representation that can be used as input data for the encoder. For example, if the initial object is a tooth, it may need to be converted into a 3D mesh, and the 3D mesh converted into a numerical matrix before encoding. Examples of such representations are discussed above.

[0111] Next, the transformation function can be identified by comparing the latent variables generated from the initial object and the transformed object. The initial state latent variables can be generated by inputting the representation of the initial object into the encoder. The transformed state latent variables can be generated by inputting the representation of the transformed object into the encoder. The transformation function is generated to transform the initial state latent variables into the transformed state latent variables.

[0112] The transformation function can be, for example, a vector of average changes between latent variables based on data pairs. The vector can be embodied as a list of numbers of the same length as the number of latent variables. The vector measures the difference between each initial state latent variable and the corresponding latent variable in the transformed state latent variable. In the case where there is more than one data pair, these distances can be averaged for each latent variable, where the averages form a vector. This vector can then be added to a new set of latent variables, thereby producing a transformed new latent variable. An embodiment of such a vector is discussed below.

[0113] The transformation function may also be, for example, a linear or polynomial function that transforms the initial state latent variable into the transformed state latent variable. Examples include, where x represents the initial state latent variable, f(x) represents the transformed state latent variable, c represents a constant, and m represents the slope:

[0114] f(x)=mx+c

[0115] f(x)=mx c

[0116] Once the transformation function is identified, it can be used to transform the new object. The representation of the new object is used as input data to the encoder and encoded into new latent variables. The new latent variables are then transformed by the transformation function, and the transformed new latent variables are run through the decoder. The decoder then outputs transformed new output data, which can be converted into the transformed new object.

[0117] Embodiments also include where the transformation function is a vector, and changing the vector controls the degree of the transformation.

[0118] As mentioned above, a vector can be a list of numbers. Therefore, a vector can have simple operations performed on it that change the value. For example, if a vector is halved and then applied to the initial state latent variable, the resulting transformed new object will only be partially transformed. If the vector is doubled, the resulting transformed new object will be transformed beyond the original. An embodiment where this is used to age a tooth is discussed below.

[0119] Embodiments also include where the initial object is an original tooth and the transformed object is an aged version of the original tooth.

[0120] In an embodiment, the transformation function can be used to age teeth. Teeth and dental objects wear out over time. Predicting how they age can help design better dental devices from the beginning. For example, a dental crown is typically expected to last 10 to 30 years. When designing a crown, it can be useful to predict how the crown and adjacent teeth change over time.

[0121] In an embodiment, the initial object may be an original tooth and the transformed object may be an aged version of the original tooth. A representation of the original tooth / teeth is used as input data in an encoder, generating an initial state latent variable representing the original tooth / teeth. Additionally, a representation of the aged tooth / teeth is used as input data in an encoder, generating a transformed state latent variable.

[0122] The transformation function can then be identified by comparing the difference between the transformed state latent variable and the initial state latent variable. As described above, this can be a vector, a linear function, and / or a polynomial function.

[0123] Now a new object (e.g. another tooth) can be aged using an autoencoder and a transformation function. Here, the representation of the new tooth or crown will be used as input data in the encoder, encoded into a set of latent variables, transformed by the transformation function, and then decoded back into output data, which can be used to generate a digital or physical object.

[0124] In embodiments where the transformation function is a vector, the vector can be used as a "slider" to control the amount of aging. If each data pair consists of an original tooth and an aged version of the original tooth, and the original tooth is considered to be at year 0 and the aged tooth at year 20, then a vector based on the mean difference can be considered to age the tooth by 20 years. A vector of half the value would age the tooth by 10 years; a vector of twice the value would age the tooth by 40 years; and so on, assuming a linear relationship between the vector and aging. Non-linear relationships can also be identified.

[0125] Embodiments also include wherein the initial state is that of the original tooth and the transformed state is a tooth having increased angular symmetry compared to the original tooth.

[0126] In an embodiment, the object is a tooth, the initial state of the tooth is an original tooth, and the transformed state is a tooth with increased angular symmetry. Increasing the angular symmetry makes the tooth more rounded and the major protrusions less prominent. This enables the tooth to better fit adjacent teeth and optimize chewing.

[0127] Properties like these are often difficult to quantify; while the angular symmetry of a right circular cylinder is easy to compute, the generated teeth need to be realistic and tooth-like. This preserves the features that make the object tooth-like (such as ridges and crevices), while functionally allowing control over the roundness of the teeth. Furthermore, this is done without carefully calculating rotation angles and surface changes; the data pairs used provide a range of examples, and the transformation function captures what the autoencoder has learned about the changes.

[0128] Embodiments also include outputting to a data format configured to fabricate a physical object from at least one of the third set of output data, any portion thereof, and / or any combination of the foregoing.

[0129] Once the corresponding 3D mesh of the object or the transformed 3D mesh is generated, the mesh or a portion thereof can be converted into a data format suitable for manufacturing. In various embodiments, the generated 3D mesh can be a standard file type for 3D objects, including but not limited to: Collaborative Design Activity (COLLADA), Initial Graphics Exchange Specification (IGES), ISO 10303 (STEP), STL, Virtual Reality Modeling Language (VRML).

[0130] Embodiments also include generating a physical object from a corresponding 3D mesh, a transformed mesh, a 3D model mesh, any portion of the foregoing meshes, and / or any combination of the foregoing by 3D printing or milling.

[0131] The methods described in this invention are intended to be applied to the physical world. Milling and 3D printing are methods of turning a 3D mesh or 3D file format into a physical object. Milling is common in dentistry because it allows the use of durable, well-tested materials. Dental milling machines carve out 3D objects that may require post-processing (such as curing). 3D printing is another method of generating physical objects and is improving rapidly. It can enable the manufacture of customized objects with less waste than traditional milling.

[0132] In this disclosure, the term data format has been used as a term commonly used in mathematics and computer science as a type of format for storing related data. Examples of data formats include: scalar, vector, scalar vector, matrix, character, string, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] The above, additional objects and / or features of the present invention will be further described by the following illustrative and non-limiting detailed description of embodiments of the present invention, with reference to the accompanying drawings, in which:

[0134] Figure 1 A schematic diagram of a system according to an embodiment of the present disclosure is shown;

[0135] Figure 2 A method of creating an hourglass predictor according to an embodiment is shown;

[0136] Figure 3 An example architecture of a suitable convolutional neural network according to an embodiment is shown;

[0137] Figure 4 An hourglass predictor for generating a crown according to an embodiment is shown;

[0138] Figure 5 An hourglass predictor according to an embodiment is shown generating a proposed digital 2D image of a desired dental setting based on a pre-treatment digital 2D image;

[0139] Figures 6A-6E Possible representations of 3D objects in various embodiments are shown;

[0140] Fig. 6A shows a depth view of a molar in an embodiment;

[0141] Figure 6B A pseudo image of a molar in an embodiment is shown;

[0142] Figure 6C shows a voxel representation of a molar in an embodiment;

[0143] Fig.6D shows a point cloud representation of a molar surface in an embodiment;

[0144] Fig. 6Eshows a mesh representation of a molar surface in an embodiment;

[0145] Figure 7 A method of generating a corresponding 3D mesh from a scan or digital object according to an embodiment is shown;

[0146] Figures 8A-8C shows the correspondence between a 2D RGB matrix, a sampling matrix, and a corresponding 3D grid according to an embodiment;

[0147] Fig. 8A Grayscale images of red, green and blue matrices according to an embodiment are shown respectively;

[0148] Figure 8B shows a subset of sampling matrices according to an embodiment;

[0149] Figure 8C The embodiment according to the present invention is shown Figure 8B The sampling matrix and Fig. 8A The 3D grid corresponding to the image in;

[0150] Fig. 9 An embodiment of a method of transforming an object using a transformation function is shown;

[0151] Fig.10 A simple example of a vector-based transformation function is shown;

[0152] Fig.11 An embodiment of using a vector as a transformation function to age teeth is shown;

[0153] FIG12 shows an embodiment in which the angular symmetry of the lingual side of the crown is increased by a transformation function;

[0154] Fig. 12A The original crown is shown; and

[0155] Fig. 12B A transformation crown is shown. DETAILED DESCRIPTION

[0156] In the following description, reference is made to the accompanying drawings which show, by way of illustration, how the invention may be practiced.

[0157] Although some embodiments have been described and shown in detail, the present invention is not limited to them, but can also be implemented in other ways within the scope of the subject matter defined in the appended claims. In particular, it should be understood that other embodiments can be utilized and structural and functional modifications can be made without departing from the scope of the present invention.

[0158] In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims or described in different embodiments does not indicate that a combination of these measures cannot be used.

[0159] A claim may refer to any of the preceding claims, and "any" is understood to mean "any one or more" of the preceding claims.

[0160] The term "obtaining" as used in this specification may refer to physically acquiring, for example, a medical image using a medical imaging device, but it may also refer to, for example, loading a previously acquired image or digital representation into a computer.

[0161] It should be emphasized that the term "include / comprises" used in this specification is used to specify the presence of stated features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0162] The features of the methods described above and below may be implemented in software and executed on a data processing system or other processing device caused by the execution of computer-executable instructions. The instructions may be program code means loaded from a storage medium or from another computer via a computer network into a memory such as RAM. Alternatively, the features described may be implemented by hard-wired circuits instead of software or in combination with software.

[0163] Figure 1 A schematic diagram of a system according to an embodiment of the present disclosure is shown. The system 100 comprises a computer device 102 comprising a computer readable medium 104 and a microprocessor 103. The system further comprises a visual display unit 107, an input unit such as a computer keyboard 105 and a computer mouse 106 for inputting data and activating virtual buttons visualized on the visual display unit 107. The visual display unit 107 may be, for example, a computer screen.

[0164] The computer device 102 can obtain a digital representation of at least a portion of a patient's jaw, including the jaw bone, from, for example, a CBCT scanner 101 b. The obtained digital representation can be stored in a computer readable medium 104 and provided to a processor 103.

[0165] Additionally or alternatively, the computer device 102 can also receive a digital 3D representation of a surface of, for example, a patient's teeth and gums from an image acquisition device 101a, such as a 3D scanning device, such as the TRIOS intraoral scanner manufactured by 3shape TRIOS A / S, or can receive scan data from such a 3D scanning device and form a digital 3D representation of the patient's teeth and / or gums based on such scan data. The received or formed digital 3D representation can be stored in a computer-readable medium 104 and provided to the microprocessor 103.

[0166] The system 100 is configured to allow an operator to design a customized dental restoration using information obtained from a bone scan and / or a surface scan, with constraints set based on predetermined design criteria. This can be achieved, for example, by displaying a digital representation of the patient's jaw on a visual display unit 107, on which the operator can then visualize his / her restoration design in relation to the surface of the patient's bone.

[0167] The system comprises a unit 108 for transmitting the digital design as output data to a manufacturing machine for generating a dental appliance, such as a customized dental restoration, to, for example, a computer-aided manufacturing (CAM) device 109 for manufacturing the customized dental restoration, or to another computer system, for example located at a milling or printing center for manufacturing the customized dental restoration. The unit for transmitting can be a wired or wireless connection, and the transmission can be performed, for example, using the Internet or a file transfer protocol (FTP).

[0168] A 3D scan of the patient's teeth and / or gums using a 3D scanning device 101a and / or a bone scan using a CBCT scanner 101b can be performed at the dentist, while the design of the custom dental restoration can be performed in a dental laboratory. In this case, a digital 3D representation of the patient's teeth and / or a 3D representation of the patient's jaw acquired from a CBCT scanner and / or scanning device can be provided via an Internet connection between the dentist and the dental laboratory.

[0169] The system 100 shown is an illustrative example. For example, the computer device 102 may include one or more microprocessors 103 and / or one or more computer-readable media 104, the visual display unit 107 may be integrated into the computer device 102 or separate from the computer device 102, and so on.

[0170] Figure 2 A method of creating an hourglass predictor according to an embodiment is shown.The hourglass predictor predicts what an object should be based on different but related data.

[0171] The first step is 200, training an autoencoder. In one embodiment, a first set of input training data 201 is a sampling matrix. A first encoder 202 encodes the first set of input training data 201 into a first set of latent variables 203. In one embodiment, the first encoder 202 is a neural network, such as a convolutional neural network or a dense neural network.

[0172] In one embodiment, the first set of latent variables 203 may have, for example, 3-50 latent variables, although there may be more or fewer latent variables. The latent variables may act as parameterizations of the tooth itself, both for embodying the object and for transforming the object. In one embodiment, the latent variables are scalar numbers.

[0173] In one embodiment, the first decoder 204 is a neural network, including but not limited to: a convolutional neural network, a dense neural network. The first decoder 204 decodes the first set of latent variables 203 into a first set of output data 205. The first set of output data 205 has the same data format as the first set of training input data 201. In one embodiment, 201 and 205 are both sampling matrices.

[0174] The neural network is trained on the target data, and the similarity between the output data and the target data determines the weights assigned to the neurons. Measuring the match between the target data and the output data is discussed above in the Summary of the Invention. In an autoencoder, the training target data is the input data itself. Thus, the target data for the first set of output data 205 is the first set of training input data 206. Once the first set of output data 205 at least substantially matches the target data, i.e., the first set of training input data 201 / 206, the autoencoder is trained. The specific measures used to determine a substantial match depend on the purpose of the hourglass predictor, but Figure 4 An example is given in .

[0175] The second step is 207, training the hourglass predictor. The second set of input data 208 has different underlying objects than the first set of input data. However, there should be an underlying connection between the objects represented by the first set of input data and the second set of input data, so that each object provides some information about the other object. For example, a crown designed for a set of tooth surroundings provides some information about the position of adjacent teeth and opposing teeth, while the tooth surroundings of a set of crowns provide some information about the possible size and shape of the crown.

[0176] The data format of the second set of input data 208 may also be different from the first set of input data 201. Examples of underlying objects are shown in Figure 4 middle.

[0177] The second encoder 209 encodes the second set of input training data 208 into a second set of latent variables 210. In one embodiment, the second encoder 209 is a neural network, such as a convolutional neural network or a dense neural network. The second set of latent variables 210 must have the same data format as the first set of latent variables 203, including having the same number of latent variables and the same data type. However, since they come from different data sets, they will have different values ​​of the latent variables.

[0178] The first decoder 211 is the same as the first decoder 204 of the first step, and decodes the second set of latent variables 210 into the second set of output data 212. Because the first decoder 204 / 211 is used here, the second set of output data 212 has the same type of underlying objects and the same data format as the first set of output data.

[0179] The target training data set typically has the same type of underlying objects as the first set of training input data. The underlying objects represented in the training data set are typically similar enough that they can be parameterized by the same set of latent variables, although they need not be identical. For example, the outer surface of a crown and a portion of the surface of a tooth may be similar enough to be parameterized by the same set of latent variables, but not be the same underlying object.

[0180] When training the hourglass predictor, each element of the target training data set is matched with an element in the second set of training input data. The second set of output data 212 at least substantially matches the training target data 213, and the second encoder can be weighted accordingly. Once the second set of output data 212 at least substantially matches the training target data 213, the hourglass predictor is trained. As described above, the exact measure of this match depends on the application for which the hourglass predictor is used.

[0181] The third step is 214, using the hourglass predictor. The third set of input data 215 and the second set of input data 208 have the same type of underlying objects and the same data format. The second encoder 216 is the same as the second encoder 209 of the second step, and encodes the third set of input data 215 into a third set of latent variables 217.

[0182] The third set of latent variables 217 has the same data format as the first and second sets of latent variables 203 and 210 .

[0183] The first decoder 218 is the same as the first decoders 204 and 211 of the first and second steps, and decodes the third group of latent variables 217 into the third group of output data 219. Because the first decoder 204 / 211 is used here, the third group of output data 219 has the same type of underlying objects and the same data format as the first and second groups of output data 205 and 212.

[0184] The third set of output data 219 is a prediction of what the underlying object should be based on the third set of input data 215 .

[0185] Figure 3 An example architecture of a suitable convolutional neural network according to an embodiment is shown. This particular convolutional neural network can be used as an encoder. Figure 3 Based on tensorflow graphs, often used to draw neural network diagrams.

[0186] Input data 301 is processed by convolution node 302. In this embodiment, the input data may be a sampling matrix of various crowns and / or teeth.

[0187] The convolution node 302 may be composed of a series of steps, for example: convolution 303 , bias addition 304 , rectified linear unit 305 .

[0188] Convolution 303 convolves input data 301 with kernel 303a. Kernel 303a is a series of filters. Kernel 303a is a learnable parameter that is changed by back propagation. The filter can be any size of at least 2x2, but preferably the size is 4x4. There is at least one filter, but preferably ranges from 16 to 512, and more preferably 64 for this first convolution. The filter can be randomly initialized. The stride used by convolution 303 is at least 1, but preferably at least 2.

[0189] Convolution 303 outputs the filter activation of the input data 301, which is then input to the BiasAdd operation 304.

[0190] The BiasAdd operation 304 takes the output of the convolution 303 and adds a bias 304a to each element. The bias 1006 is a set of scalar numbers. The bias 304a is a learnable parameter that is changed by backpropagation. The bias 304a may be randomly initialized or may be initialized at all zeros. The BiasAdd operation 304 is optional for the convolution node, but may help optimize the prediction results, depending on the embodiment.

[0191] The output of the convolution 303 or the output of the biasAdd operation 304 is then input to a rectified linear unit (ReLU) 305. ReLU 305 is an activation function that sets the input to zero if it is negative and does not change the input otherwise. The rectified linear unit is a popular activation function, but other activation functions may also be used, including but not limited to: sigmoid function, hyperbolic tangent.

[0192] The output of ReLU 305 is a set of filter activation matrices, which can then be input into another convolution node. Convolution node 306 is a series of convolution nodes with the same format as convolution node 302, although their filter size, number of filters, filter initialization, stride size, bias, and activation function may be different. Although convolution node 302 may be sufficient for an embodiment of the network, more convolution nodes are preferred. Therefore, convolution node 306 can be 1-24 convolution nodes, preferably 4 convolution nodes.

[0193] Reshape 307 takes the output of the final convolution node of convolution node 306 and changes the format with shape 307a. In this embodiment, shape 307a flattens each filter activation matrix from the output into a vector of scalar numbers. In this embodiment, flattening is necessary because the next step is a dense node that requires a vector. However, other embodiments may use different shapes, depending on what input data format is required for the next step.

[0194] The output vector of Reshape 307 is then input to two dense nodes, dense node A 308 and dense node B 311. Here, two dense nodes are used because the variational autoencoder may require both the mean and the standard deviation or log variance. However, other embodiments may use only one dense node or more than two dense nodes to directly output the latent variables.

[0195] The dense node A 308 may consist of a series of steps, such as a matrix multiplication 309, a bias add operation 310. The dense node A 308 may output the logarithm of the standard deviation or variance.

[0196] Matrix multiplication 309 multiplies its input data by kernel 309a. Kernel 309a is a weight matrix that can be randomly initialized. Kernel 309a is a learnable parameter that is changed through back propagation.

[0197] The output of the matrix multiplication 309 is then input to a bias addition operation 310. The bias addition operation 310 adds a bias 310a to each element of the input. The bias 310a is a set of scalar numbers and learnable parameters that are changed by back-propagation and can be randomly initialized or can be initialized at all zeros.

[0198] Dense node B 311 has a similar format as dense node A 308 and may output a mean.

[0199] The outputs of dense nodes 308 and 311 are sampled to generate latent variable 312. Although this embodiment of the present invention uses two dense nodes, a single dense node may also be used to directly generate latent variable 312.

[0200] These latent variables are then fed into a decoder, not shown, which can be a similar neural network but with deconvolutions instead of convolutions. The entire encoder-decoder structure is trained, back-propagating the different weights to the various learnable parameters throughout the structure.

[0201] Figure 4 An hourglass predictor for generating a crown according to an embodiment is shown. In this embodiment, the hourglass predictor is trained to predict the outer surface of a crown given a tooth surrounding environment. The outer surface can be connected to a mesh of a bottom surface suitable for further manufacturing and / or preparation. Figure 4 , for objects 402 , 408 , 409 , 419 , 420 , 429 , and 430 , the term “crown” may be used as shorthand for the outer surface of the crown.

[0202] The first step 400 trains the autoencoder to parameterize / output the crown. For the method of this embodiment, the autoencoder provides us with a set of latent variables to parameterize the crown, and a decoder to decode the latent variables back to the crown. In this embodiment, the autoencoder can be a variational autoencoder, which uses probability distributions to generate latent variables instead of generating them directly (Goodfellow, Ian et al., Deep learning, Vol. 1, Cambridge: MIT Press, 2016, Chapter 12). The embodiment can also be a normal autoencoder, which directly generates latent variables. Variational autoencoders using probability distributions can perform better inferences when data is sparse.

[0203] Tooth 401 is an example of an underlying object type represented by input training data 402, target training data 409, and output data 408. Figure 7 The method described in processes a plurality of teeth and / or crowns into a sampling matrix.

[0204] The sampling matrix of multiple crowns and / or teeth 402 is the first set of input training data. Since the hourglass predictor is only used to generate the outer surface of the crowns here, both the crowns and the actual teeth can be used as training data. The sampling matrix of multiple crowns 402 can be based on scans of physical crowns and / or teeth or digital objects representing crowns and / or teeth.

[0205] Convolutional neural network A (403) encodes the sampling matrix of multiple crowns 402 into a mean vector 404 and a standard deviation vector 405. Vector 405 can also be a vector of variance logarithms. The mean vector 404 and the standard deviation vector 405 are then sampled to estimate the latent variables, i.e., crown and / or tooth parameterization 406. Figure 3 This convolutional neural network example is further detailed in .

[0206] The latent variables, i.e. crown and / or tooth parameterization 406, can parameterize the tooth and can be decoded back into a sampling matrix with a corresponding 3D mesh of the tooth. The data format of these latent variables is a set of scalar numbers. There may be 3-50 scalar numbers, but the total number of group members should remain consistent for a given hourglass predictor.

[0207] The latent variables are a parameterization of the teeth and allow the shape of the teeth to be represented by a set of scalar numbers. These numbers are not easily interpreted by human perception, but do contain information about the tooth shape discovered by the machine learning method. Furthermore, these numbers can be converted back to the corresponding 3D mesh of the underlying teeth.

[0208] As a parameterization, latent variables can also be used to change the teeth. Changing the value of the latent variable will correspondingly change the shape of the corresponding 3D mesh of the tooth. This allows precise quantification of the changes.

[0209] The convolutional neural network B (407) decodes the latent variables, i.e., the crown and / or tooth parameterization 406, into an auto-encoding matrix 408 of the crown / tooth. The auto-encoding matrix 408 of the crown / tooth is Figure 2 An embodiment of the first set of output data 205.

[0210] Convolutional neural networks are trained using back propagation based on comparing output data with target data. An autoencoder is a special arrangement where the input data itself is the target data. Thus, the output data of convolutional neural network B (407), i.e., the autoencoder matrix of crown / tooth 408, is evaluated based on its similarity to the sampling matrix of crown / tooth 403 / 409.

[0211] The second step 410 trains the hourglass predictor to create a crown based on its surroundings. This step provides us with an encoder to encode a set of different but related input data into latent variables. The latent variables can then be decoded back into the underlying objects of the autoencoder. In this embodiment, the input data is a representation of the crown's surroundings, and the output data is a representation of the crown.

[0212] Matched crowns and surroundings 411 and 412 are examples of underlying objects of the training data set for the hourglass predictor. Surroundings 412 are a collection of objects surrounding a crown or crown preparation, including but not limited to: adjacent teeth, opposing teeth, gums, jaws, and / or preparations. Crown 411 is an existing crown designed, for example, by a dental technician for surroundings 412.

[0213] The surroundings 412 are the underlying objects of the input training data 413, and the crown 411 is the underlying object of the output data 419 and the target training data 420. Figure 7 The method described in processes multiple of these into a sampling matrix.

[0214] The sampling matrix of the matched tooth surrounding environment 413 is the second set of input training data. The sampling matrix of the matched tooth surrounding environment 413 is based on the tooth surrounding environment, such as 412 (for a method of obtaining a sampling matrix from a three-dimensional object, see Figure 7 ).

[0215] The convolutional neural network C (414) encodes the sampling matrix of the matched tooth surroundings 413 into a mean vector 415 and a standard deviation vector 416. The mean vector 415 and the standard deviation vector 416 are then sampled to estimate the latent variable, i.e., the matched crown / tooth parameterization 417. Figure 3 An example of a convolutional neural network is further explained in detail in .

[0216] The latent variables, ie matched crown parameterizations 417 , have the same data format as the latent variables from the first step 406 , and represent the same underlying objects.

[0217] Convolutional neural network B (418) decodes the latent variable, i.e., the matched crown parameterization 417, into a matrix of matched crowns 419. Convolutional neural network B (418) is the same as the convolutional neural network B in the first step (407). Using convolutional neural network B (407 / 418) as a decoder for both steps means that the latent variables are decoded into the same type of object, here the outer surface of the crown.

[0218] The matrix of the matching crown 419 is equivalent to Figure 2 The second set of output data 212 in is evaluated by comparing with the target training data set, namely the sampling matrix of matching crowns 420.

[0219] Each sampling matrix of the matching crown 420 corresponds to a sampling matrix of the matching tooth surroundings from 413. The decoder, i.e., the convolutional neural network B (407 / 418), is not changed, which means that a given set of latent variables always returns the same output. Therefore, only the encoder, i.e., the convolutional neural network C (414), is trained in step 410.

[0220] In summary, step 410 trains a convolutional neural network C (414) to return latent variables (417) that decode to a crown that matches a given surrounding environment.

[0221] The third step 421 predicts crowns for new environments. The hourglass predictor trained in the previous step is used for a new set of dental environments to generate crowns. The generated crowns are based on the predictions from the training data from the previous step and can be considered as the crowns that a dental professional would design given the environment.

[0222] New surroundings 422 is an example of an underlying object of the input data for the hourglass predictor. This is the same type of underlying object as matched surroundings 412. New surroundings 422 is an example of an underlying object of the input data for the hourglass predictor. Figure 7 The method in is processed into a sampling matrix 423. The sampling matrix 423 is equivalent to Figure 2 The third set of input data 215 in .

[0223] The convolutional neural network C (424) encodes the sampling matrix of the new tooth surroundings 423 into a mean vector 425 and a standard deviation vector 426. The mean vector 425 and the standard deviation vector 426 are then sampled to estimate the latent variable, i.e., the matched crown / tooth parameterization 427. The convolutional neural network C (424) is the same as the convolutional neural network C (414) in step 410.

[0224] The latent variables, ie, new crown parameterization 427, have the same data format as the latent variables from the first step (406) and the second step (416), and represent the same underlying objects.

[0225] Convolutional neural network B (428) decodes the latent variables, i.e., new crown parameterization 427, into a sampling matrix 429 for crown prediction. Convolutional neural network B (428) is the same as the convolutional neural network B from the first and second steps (407 / 418).

[0226] The matrix of crown prediction 429 is equivalent to Figure 2 The third set of output data 219 in . Note that there is no training in step 421, so there is no target data and no back propagation.

[0227] The matrix of the crown prediction 429 can now be used to generate a physical crown 430. The matrix of the crown prediction 429 is reconnected into a mesh, and the mesh is connected to a mesh of a bottom surface suitable for further manufacturing and / or preparation. If necessary, the mesh is converted into a data format configured to manufacture a physical object, which is then used to mill, 3D print, or otherwise manufacture the physical crown 430.

[0228] The physical crown 430 may be placed in the new surrounding environment 422 , creating a new environment with the crown 431 .

[0229] Figure 5 An hourglass predictor generating a proposed digital 2D image of a desired dental setting based on a pre-treatment digital 2D image according to an embodiment is shown. In this embodiment, the hourglass predictor is trained to predict a proposed digital 2D image given a pre-treatment digital 2D image.

[0230] The first step 500 trains an autoencoder to parameterize / output a proposed digital 2D image. For the method of this embodiment, the autoencoder provides us with a set of latent variables to parameterize the proposed digital 2D image and a decoder to decode the latent variables back to the proposed digital 2D image. In this embodiment, the autoencoder can be a variational autoencoder, as described above.

[0231] The desired tooth setting 501 is one example of the type of underlying object represented by the input training data 502, the target training data 509, and the output data 508. Potential desired tooth settings are discussed above.

[0232] The first set of input training data is a set of proposed digital 2D images of the desired tooth setting 502. Since the proposed digital 2D images are arrays of pixels, they can be represented by a matrix. Each pixel can be represented as a matrix element whose value corresponds to the RGB or grayscale value of the proposed digital 2D image. These values ​​can be normalized before further processing.

[0233] Convolutional neural network A (503) encodes the matrix of multiple proposed digital 2D images 502 into a mean vector 504 and a standard deviation vector 505, and then samples them to generate latent variables 506, similar to step 400 of Figure 4, as described above. Vector 505 can use logarithmic variance instead of standard deviation.

[0234] The latent variables, i.e. the proposed digital 2D image parameterization 506, can parameterize the proposed digital 2D image and can be decoded back into a matrix with the corresponding 2D digital image. These latent variables have the same Figure 4 The latent variables 406 in , are similar in properties in that they are scalar numbers with a range of values ​​and are parameterizations of the underlying objects, as described above. However, the underlying objects may be better represented by a different set of scalars, so the size and optimal values ​​of the groups may be different.

[0235] The convolutional neural network B (507) decodes the latent variable, i.e., the proposed digital 2D image parameterization 506, into the proposed digital 2D image auto-encoding matrix 508. The proposed digital 2D image auto-encoding matrix 508 is Figure 2 An embodiment of the first set of output data 205 in .

[0236] As mentioned above, the convolutional neural network requires back propagation for training. Therefore, the output data of the convolutional neural network B (507), i.e., the auto-encoding matrix of the proposed digital 2D image 508, is evaluated based on its similarity with the matrix of the proposed digital 2D image 503 / 509.

[0237] The second step 510 trains an hourglass predictor to create a proposed digital 2D image based on the corresponding pre-treatment digital 2D image.

[0238] The matched proposed digital 2D image 511 and the matched pre-treatment digital 2D image 512 are examples of underlying objects of the training data for the hourglass predictor. The pre-treatment digital 2D image is a digital image of the patient before treatment. It can be a 2D image showing the patient's smile, where a portion of the upper front teeth is visible.

[0239] The pre-treatment digital 2D image 512 is the underlying object of the input training data 513, and the proposed digital 2D image 511 is the underlying object of the output data 519 and the target training data 520. The matrix representations are performed in a similar manner to those discussed in step 500 to train the autoencoder.

[0240] The convolutional neural network C (514) encodes the matrix of the matched pre-treatment digital 2D image 513 into a mean vector 515 and a standard deviation vector 516. The mean vector 515 and the standard deviation vector 516 are then sampled to estimate the latent variable, i.e., the matched proposed digital 2D image parameterization 517. The vector 516 may use the logarithmic variance instead of the standard deviation.

[0241] The latent variables, ie the matched proposed digital 2D image parameterization 517 , have the same data format as the latent variables from the first step 506 , and represent the same underlying objects.

[0242] Convolutional neural network B (518) decodes the latent variable, i.e., the matching proposed digital 2D image parameterization 517, into a matrix of the matching proposed digital 2D image 519. Convolutional neural network B (518) is the same as the convolutional neural network B of the first step (507). Using convolutional neural network B (507 / 518) as a decoder for both steps means that the latent variables are decoded into the same type of object.

[0243] The matrix of the matched proposed digital 2D image 519 is equivalent to Figure 2 The second set of output data 212 in is evaluated by comparing with a target training data set, ie a matrix of matched proposed digital 2D images 520 .

[0244] Each matrix of the matched proposed digital 2D image 520 corresponds to a matrix of the matched pre-treatment digital 2D image 513. The decoder, i.e. the convolutional neural network B (507 / 518) is not changed, which means that only the convolutional neural network C (514) is trained in step 510 as described above.

[0245] In summary, step 510 trains the convolutional neural network C (514) to return latent variables (517) that decode to a proposed digital 2D image that matches a given pre-treatment digital 2D image.

[0246] The third step 521 predicts a proposed digital 2D image for the new pre-treatment digital 2D image, and can be considered as the content designed by the dental professional for the pre-treatment digital 2D image.

[0247] The new pre-treatment digital 2D image 522 is an example of an underlying object of the input data for the hourglass predictor, and is of the same type as the matched pre-treatment digital 2D image 512. The new pre-treatment digital 2D image 522 is processed into a matrix 523 according to the method discussed above.

[0248] The convolutional neural network C (524) encodes the matrix of the new pre-treatment digital 2D image 523 into a mean vector 525 and a standard deviation vector 526. As described above, these are sampled to estimate the latent variable, i.e. the matched proposed digital 2D image / tooth parameterization 527. The convolutional neural network C (524) is the same as the convolutional neural network C (514) from step 510.

[0249] The latent variables, namely the newly proposed digital 2D image parameterization 527, have the same data format as the latent variables from the first step (506) and the second step (516), and represent the same underlying objects.

[0250] Convolutional neural network B (528) decodes the latent variables, i.e. the newly proposed digital 2D image parameterization 527, into a matrix of proposed digital 2D image prediction 529. Convolutional neural network B (528) is the same as the convolutional neural network B from the first step and the second step (507 / 518).

[0251] The matrix of the proposed digital 2D image prediction 529 is equivalent to Figure 2 The third set of output data 219 in .

[0252] The proposed digital 2D image 530 may be incorporated into the new pre-treatment digital 2D image 522 , for example by superimposition, overlay and / or combination, thereby generating a new pre-treatment digital 2D image with a predicted proposed digital 2D image 531 .

[0253] Figures 6A-6E Possible representations of 3D objects in various embodiments are shown.

[0254] Fig. 6A A depth map of a molar in an embodiment is shown. A depth map can be derived from a picture of a 3D object, where the scale of each pixel is used to estimate its distance from the camera and therefore its representation in 3D space. Here, a molar is shown from multiple viewpoints.

[0255] Figure 6BA pseudo image of a molar in an embodiment is shown. A pseudo image is a 2D image where the grey value of each pixel represents the normalized distance at that point from the viewpoint of the 3D object. The distance can be derived from a 3D scan. Here, a molar is shown from multiple viewpoints.

[0256] Figure 6C A voxel representation of a molar in an embodiment is shown. A voxel representation uses a three-dimensional grid to represent a 3D object. Each voxel can be a cube in 3D space. If the object exists in that voxel, it is marked as present, and if it does not exist in that voxel, it is marked as not present. Here, the teeth are represented in voxels. Note that in this embodiment, the voxel size is particularly large to illustrate the voxels more clearly. Depending on the application, smaller voxels can also be used.

[0257] Fig.6D A point cloud representation of the molar surface in an embodiment is shown. A point cloud is a collection of points in 3D space, each with x, y, z coordinates. Here, the Figure 6C The molar surface is displayed as a point cloud, but a point cloud can represent a volume.

[0258] Fig. 6E A mesh representation of a molar surface in an embodiment is shown. A mesh is a collection of vertices, edges, and faces. A vertex is a single point representing a surface, an edge is a line connecting vertices, and a face is a continuous area enclosed by vertices and edges. Here, Figure 6C The molar surface in -D is represented as a triangle mesh.

[0259] Figure 7 A method of generating a corresponding 3D mesh from a scan or digital object according to an embodiment is shown. In 701, a physical object is scanned, here the surface of a tooth or crown. This step is optional, as some objects only exist as digital objects. The model may have been designed in a CAD / CAM program and was never a physical object. 702 is the original 3D mesh from the scan.

[0260] From 702 to 703, the original 3D grid is flattened into a planar grid through the above-mentioned grid flattening process. At this point, the original 3D grid and the planar grid are bijective. For some grid flattening processes, the flattening process can be completely reversible. Step 704 shows the sampled planar grid, although as mentioned above, there are several ways to do this. Step 705 shows the sampling matrix, where each sample is interpolated or taken from a sample on the planar grid.

[0261] Step 706 shows the corresponding 3D grid of the sampling matrix according to an embodiment. Here, the corresponding grid is a consistent grid, and the sampling matrix and the corresponding 3D grid can be converted back and forth. In addition, operations on either can be converted back to the other. For example, matrix operations on the sampling matrix can be converted to the corresponding 3D grid. Conversely, changes in the shape of the corresponding 3D grid in 705 will also change the sampling matrix.

[0262] Figures 8A-8C The correspondence between the 2D RGB matrix, the sampling matrix and the corresponding 3D grid according to an embodiment is shown.

[0263] Fig. 8A Grayscale images of red, green and blue matrices according to an embodiment are shown respectively. Image 800 (red matrix) represents the x value of a sample from a planar grid. Each pixel of the image is equivalent to a matrix cell of a matrix. The x value is standardized, and the scale value of each pixel is the standardized x value of its corresponding sample. In a similar manner, image 801 (green matrix) represents the y value of a sample of a planar grid, and image 802 (blue matrix) represents the z value of a sample of a planar grid.

[0264] Fig. 8A The red, green, and blue matrices in can be combined into a single color image, namely a 2D RGB matrix (not shown because it will be a color image). The 2D RGB matrix is ​​combined from Fig. 8A The red, green, and blue values ​​of each pixel are Fig. 8A The RGB value of each pixel in the matrix. Therefore, the RGB value of the pixel also corresponds to the three-dimensional coordinate of the sample corresponding to the pixel.

[0265] Figure 8B A subset of the sampling matrix according to an embodiment is shown. Specifically, they show rows 55-57, columns 1-3 of the sampling matrix. Due to limited available space, the entire sampling matrix is ​​not shown.

[0266] Samples from a planar grid have positions and three-dimensional coordinates. Positions are coordinates that include a numerical value representing the location of a sample relative to other samples. Three-dimensional coordinates include three numerical values, represent a point in three-dimensional space, and can be Euclidean coordinates.

[0267] In the sampling matrix, the location of the sample can be the location of its matrix element in the sampling matrix. The three-dimensional coordinates represented as x, y and z values ​​can be the values ​​of the matrix elements. This can be displayed in a variety of ways, as three two-dimensional arrays (803-805) with elements in one dimension, or as a two-dimensional array (806) with elements in three dimensions.

[0268] The sampling matrix represented by 803 is a two-dimensional array of elements with one dimension (i.e., the x dimension). Each matrix element represents a sample, where the position of the matrix element is the position of the sample, and the value of the matrix element is the absolute coordinate x value of the sample in millimeters. The sampling matrices represented by 804 and 805 are similar to 803, but are for y values ​​and z values ​​respectively.

[0269] The sampling matrix represented by 806 is a two-dimensional array with elements in three dimensions. Comparing the first element, it can be seen that the value of the matrix element of 806 is the value of the matrix element at the same position in 803-805. Therefore, 806 has all three coordinates x, y and z.

[0270] Since the sampling matrix represented by 806 and the sampling matrices represented by 803 - 805 are different data formats of the same object, both formats are referred to as “sampling matrices” herein.

[0271] The sampling matrix represented by 803 is similar to image 800, i.e., the red matrix. For a given sample, the x value can be represented by the value of the matrix element of 800 as normalized to a color scale, or as a Euclidean coordinate by the value of the matrix element of 803. The Euclidean coordinates can be absolute or normalized; Figure 8B The matrix shown shows absolute coordinates. Sample matrix subset 803 is for rows 55-57 and columns 1-3, which correspond to pixels at the same position in red matrix 800.

[0272] In a similar manner, the sampling matrices denoted by 804 and 805 are similar to 801 and 802, respectively.

[0273] The sampling matrix represented by 806 is similar to the 2D RGB sampling matrix. For a given sample, the x, y, z values ​​can be represented by the values ​​of the corresponding 2D RGB matrix elements as normalized to the RGB color scale, or represented as Euclidean coordinates by the values ​​of the corresponding 806 matrix elements.

[0274] Figure 8C The embodiment according to the present invention is shown Figure 8B The sampling matrix and Fig. 8A The 3D mesh corresponding to the image in . Note that unlike the mesh from the scan, Figure 8C The corresponding 3D meshes in are consistently connected. Figure 8C The vertices of the corresponding 3D mesh have Figure 8B The corresponding matrix elements of the sampling matrix represented by ; for example, Figure 8B The nine matrix elements shown in are the nine vertices along the left boundary of the tooth. Fig. 8A There are also corresponding pixels in the image, here are the pixels in rows 55-57 and columns 1-3.

[0275] Fig. 8A One or more RGB matrices in Figure 8B The sampling matrix and Figure 8C The corresponding 3D meshes in are different representations of the same underlying 3D object. Furthermore, these representations are bijective and can be converted back and forth between formats.

[0276] By changing the properties of any one representation, the other representations can be changed. For example, a matrix operation that changes the sampling matrix will change the 2D RGB matrix and the corresponding 3D mesh. This is particularly useful when applying machine learning methods that rely on matrix operations and / or uniform datasets. The sampling matrix allows machine learning methods to be performed on three-dimensional objects.

[0277] In one embodiment, a scalar transformation can be accomplished by multiplying each element of the matrix by a scalar number; the size of the corresponding 3D grid will change in proportion to the scalar number. In another embodiment, a principal component analysis can be performed on the sampled matrix. Any changes to the sampled matrix based on the principal components will also affect the corresponding 3D grid.

[0278] Fig. 9 An embodiment of a method of transforming an object using a transformation function is shown.

[0279] First, the autoencoder is trained in step 900. The details of training the autoencoder are similar to Figure 2 The autoencoder described in step 200, but to summarize: the training input data 901 is encoded by the encoder 902 into latent variables 903. The latent variables 903 are decoded by the decoder 904 into training output data 905. The training output data 905 is then compared with the training input data 901, and the weights of the decoder 904 and encoder 902 are changed accordingly. This cycle repeats until the training input data 901 and the training output data 905 are substantially the same, at which point the autoencoder is considered trained.

[0280] The encoder 902 and decoder 904 may be neural networks; Figure 3 An example architecture is described in . The training input data 901 may be a representation of a real-world object, for example, a flattened 3D mesh of a tooth.

[0281] Next, latent variables are found for the data pairs in step 906. As described above, each data pair 907 includes an initial object 908 and a transformed object 909. These objects are similar to the object types underlying the training input data 901, and thus can be read by the autoencoder from step 900. The initial object 908 and the transformed object 909 are each passed through an encoder 911 / 914, which is the encoder 902 trained in step 900. The encoding outputs two sets of latent variables - the initial state latent variables 912 and the transformed state latent variables 915.

[0282] Step 916 finds a transformation 919 function based on the initial state latent variables 917 and the transformed state latent variables 918. Specific embodiments are discussed above and below.

[0283] Step 920 applies a transformation function 925 to a new object 921. The new object 921 may be of the same type as the original object 908 and may be converted to new object input data readable by an encoder 923. The encoder 923 is the same as the encoders 911 / 914 from step 906 and the trained encoder 902 from step 900. Encoding the new object input data 922 generates new latent variables 924.

[0284] The new latent variable is then transformed by a transformation function 925 into a transformed new latent variable 926. The transformation function 925 is the transformation function 919 derived in step 916.

[0285] The transformed latent variables 926 are then decoded by a decoder 927 into transformed output data 928. The decoder 927 is the same as the trained decoder 904 from step 900. The transformed output data 928 can be converted into a transformed new object 929, which is a new object 921 that is transformed in the same way as the original object 908 to the transformed object 909.

[0286] Fig.10 A simple example of a vector-based transformation function is shown in Figure 1. It shows five data pairs, each with two latent variables.

[0287] Graph 1001 shows five groups of initial state latent variables in an XY plot, each group represented by a square, showing two latent variables for each initial object. Although there may be more latent variables, for ease of illustration only two are shown here, corresponding to the X and Y values ​​of the plot. Similarly, for illustration purposes, a given data pair is shown; training an autoencoder may involve more or fewer.

[0288] Diagram 1002 further illustrates five groups of transformed state latent variables in an XY plot, each group represented by a circle. Again, while there may be more latent variables, for ease of illustration only two are shown, with values ​​corresponding to the X and Y values ​​of the plot. In this example, the transformed state latent variables (circles) are clustered together away from the initial state latent variables (squares).

[0289] Diagram 1003 further shows vectors drawn as arrows between the initial state latent variable (square) and the transformed state latent variable (circle) for each data pair. Vectors typically have magnitude and direction; in this embodiment, the vectors act to add a constant to each latent variable. Because the transformed state is considered to be the completed state, the vector direction points to the transformed state latent variable.

[0290] Graph 1004 further illustrates an embodiment of a transformation function, here the mean vector (arrow) of the vectors of graph 1003. Note that it starts at the center of the cluster of initial state latent variables (squares) and ends at the center of the cluster of transformed state latent variables (circles). However, since vectors include magnitude and direction, the vectors need not start and end at the points shown here. Instead, the vectors are relative and therefore can be applied to a new group of latent variables, which can be represented by a new position on the XY graph.

[0291] Diagram 1005 further illustrates the latent variables for the new object, represented by asterisks. In this embodiment, the new object is similar to the original object, as can be seen by its position in the cluster of the original object latent variables (squares).

[0292] Diagram 1006 shows the transformation function, i.e., the vector (arrow) in diagram 1004, applied to the new object latent variable (asterisk at the beginning of the arrow). The asterisk at the end of the arrow is the transformed new latent variable. Note that the transformed new latent variable is located close to the transformed latent variables (circles), indicating that it is similar to those underlying objects.

[0293] Diagram 1007 shows a partial transformation using a vector (arrow). Here, the size of the vector has been halved, and the transformed new latent variable (asterisk at the end of the arrow) is now midway between the original object latent variable and the transformed object latent variable. This can be used to adjust the level of the transformation.

[0294] Figure 1008 shows an over-transformation using a vector (arrow). Here, the magnitude of the vector is increased by 20%, so the new latent variable of the transformation (asterisk at the end of the arrow) is extrapolated beyond the original transformation.

[0295] Fig.11 An embodiment of aging a tooth using a vector as a transformation function is shown. Figure 1101 is the new object, a molar. Figure 1105 is the transformed new object, an aged molar, at 100% of the vector. Figures 1102, 1103, and 1104 were transformed using 25%, 50%, and 75% of the vector, respectively.

[0296] FIG. 12 shows an embodiment in which the angular symmetry of the lingual side of the crown is increased as a transformation function. Fig. 12A The original crown is shown, and Fig. 12B A transformation crown is shown.

[0297] It may be desirable to increase angular symmetry when designing a crown because smoother crowns are more comfortable for the patient and easier to chew. However, perfectly round crowns do not look like real teeth and do not fit well with adjacent teeth. This can lead to problems adjusting the crown to look like a realistic tooth while minimizing certain undesirable features, such as overly prominent protrusions.

[0298] Transformation functions can be used to solve this problem in a consistent and quantifiable way. Fig. 12A The original crown is shown; Fig. 12B A dental crown transformed by a transformation function trained on a set of data pairs is shown, where the transformed object has increased angular symmetry on the lingual side.

[0299] Note that the original shape of the crown is generally maintained, but some features have been smoothed. For example, the large bump on the lingual side (shown on the right side of the images in Figures 1201 and 1204) is slightly flatter. Despite the slight difference in the figures, the patient may experience better chewing and a more comfortable fit due to this change. However, since the transformation function is limited to the lingual side, this change does not affect the adjacent teeth.

[0300] According to an embodiment, a computer program product embodied in a non-transitory computer readable medium is disclosed. The computer program product includes a computer readable program code executable by a hardware data processor to cause the hardware data processor to perform a method when the computer readable program code is executed by the hardware data processor. The method may include any one of the system components performing one or more functions of one or more steps of the method disclosed in one or more embodiments of the present disclosure.

Claims

1. A computer-implemented method for generating an object based on output data, comprising: training an autoencoder on a first set of training input data to identify a first set of latent variables and to generate a first set of output data, wherein the autoencoder comprises a first encoder and a first decoder, wherein the first encoder converts the first set of training input data into a first set of latent variables, wherein the first decoder converts the first set of latent variables into the first set of output data, wherein the first set of output data is at least substantially the same as the first set of training input data; training an hourglass predictor to return a second set of latent variables, wherein the hourglass predictor includes a second encoder and the first decoder, wherein the second encoder converts a second set of training input data into the second set of latent variables, wherein the second set of latent variables has a data format comparable to the first set of latent variables, and the first decoder converts the second set of latent variables into a second set of output data that is at least substantially the same as a set of training target data, the set of training target data has the same type of underlying objects as the first set of training input data, and the second set of training input data is different from the first set of training input data; and using the hourglass predictor on a third set of input data to generate a third set of output data, wherein the third set of input data and the second set of training input data have the same type of underlying objects and the same data format, and wherein the third set of output data is in a data format comparable to the first set of output data, and The object is generated based on the third set of output data, wherein the object is a dental restoration, an orthodontic appliance or an ear-related device or a proposed digital 2D image of a desired tooth setting based on the pre-treatment digital 2D image.

2. The method according to claim 1, wherein: The third set of input data contains information about objects surrounding the object to be generated.

3. The method according to claim 1, wherein: The matrix of samples of multiple crowns and / or teeth is the first set of training input data.

4. The method according to any one of claims 1 to 3, wherein: The first set of latent variables and / or the second set of latent variables are used as parameterization of input data and / or output data.

5. The method according to any one of claims 1 to 3, wherein: The first set of training input data, the second set of training input data, the set of training target data and / or the third set of input data are based on three-dimensional objects.

6. The method according to any one of claims 1 to 3, wherein: The first set of training input data, the second set of training input data, the third set of input data, the set of training target data, the first set of output data, the second set of output data and / or the third set of output data include one or more of the following: a 2D image, a depth map, a pseudo image, a point cloud, a 3D mesh and / or volume data of the object.

7. The method according to any one of claims 1 to 3, wherein: The first set of training input data, the second set of training input data, the third set of input data, the set of training target data, the first set of output data, the second set of output data and / or the third set of output data are corresponding 3D grids corresponding to a sampling matrix, wherein the sampling matrix is ​​generated by a method comprising: An initial three-dimensional mesh is transformed into a planar mesh, the initial three-dimensional mesh comprising a first set of vertices and edges, and the planar mesh comprising a second set of vertices and edges, wherein: - each vertex in the second set of vertices is a transformation of a vertex from the first set of vertices and includes the value from the vertex in the first set of vertices, and - each edge in the second set of edges is a transformation of an edge from the first set of edges and includes a value from the edge in the first set of edges; The planar grid is sampled to generate a plurality of samples, such that each sample in the plurality of samples includes: - three-dimensional coordinates comprising three values ​​representing a point in three-dimensional space, wherein said three values ​​are derived and / or obtained directly from said initial three-dimensional mesh, and - coordinates comprising numerical values ​​representing the position of the sample relative to other samples of the plurality of samples; and generating the sampling matrix based on the plurality of samples; The sampling matrix is ​​represented as a corresponding 3D grid.

8. The method according to any one of claims 1 to 3, wherein: The third set of output data is used to estimate at least one unscanned and / or erroneously scanned region.

9. The method according to any one of claims 1 to 3, wherein: The at least one un-scanned and / or erroneously scanned area, wherein the at least one un-scanned and / or erroneously scanned area is a tooth area or an ear area.

10. The method according to any one of claims 1-3, further comprising designing the dental restoration, orthodontic appliance or ear-related device using at least one of the third set of output data, any portion of the third set of output data and / or a combination thereof.

11. The method of any of claims 1-3, further comprising identifying non-movable objects in the three-dimensional scan, thereby allowing movable objects to be excluded from the three-dimensional scan.

12. The method of any one of claims 1-3, further comprising using the hourglass predictor to identify an object.

13. The method according to claim 5, wherein: The first set of training input data, the second set of training input data, the set of training target data and / or the third set of input data are based on a three-dimensional scan of a 3D object.

14. A computer-implemented method for transforming an object, comprising: - training an autoencoder on training input data to identify latent variables and generate training output data, wherein the autoencoder comprises an encoder and a decoder, wherein the encoder converts the training input data into the latent variables, wherein the decoder converts the latent variables into training output data, wherein the training output data is at least substantially the same as the training input data, and wherein the trained autoencoder encodes input data into the latent variables with the encoder and decodes the latent variables into output data with the decoder; - identifying a transformation function using at least one data pair, wherein each of the at least one data pair includes an initial object in an initial state and a transformed object in a transformed state, by: generating an initial state latent variable of each of the initial objects by using a representation of each of the initial objects as the input data of the encoder; generating a transformed state latent variable of each of the transformed objects by using a representation of each of the transformed objects as the input data of the encoder, and identifying the transformation function based on a difference between the initial state latent variable and the transformed state latent variable; and - transforming a new object with the transformation function by: generating new latent variables by using a representation of the new object as the input data of the encoder; generating transformed new latent variables by applying the transformation function to the new latent variables; generating transformed output data by decoding the transformed new latent variables with the decoder.

15. The method according to claim 14, wherein: The transformation function is a vector, and changing the vector also changes the degree of the transformation.

16. The method according to any one of claims 14-15, wherein: The initial object is the original tooth and the transformed new object is an aged version of the original tooth.

17. The method according to any one of claims 14-15, wherein: The initial object is the original tooth and the transformed new object is a tooth having increased angular symmetry compared to the original tooth.

18. The method of any of claims 14-15, further comprising outputting a data format suitable for manufacturing a physical object from at least one of the third set of output data, any portion of the third set of output data, and / or any combination of the foregoing.

19. The method of any one of claims 14-15, further comprising generating a physical object from at least one of the third set of output data, any portion of the third set of output data, and / or any combination of the foregoing by 3D printing or milling.

Images