Computer-implemented method and system for providing digital dental restoration, and related non-transitory computer readable medium

Abstract

A method and system receive a 3D digital tooth model representing at least a portion of a patient's dentition; automatically determine a virtual tooth preparation site in the 3D digital tooth model using a first neural network; automatically generate a 3D digital dental prosthesis model in the 3D digital tooth model using a second trained generative deep neural network; and automatically deliver the 3D digital tooth model containing the virtual 3D dental prosthesis model to a quality control ("QC") user.

Description

Technical Field

[0001] Cross-reference to related applications

[0002] This application claims priority to and the benefit of co-pending U.S. Provisional Patent Application No. 63 / 369,151, filed on July 22, 2022, entitled "Integrated Tooth Restoration Design Method and System," and U.S. Utility Application No. 18 / 353,550, filed on July 17, 2023, entitled "Integrated Tooth Restoration Design Method and System," which are hereby incorporated by reference in their entirety.

[0003] The present disclosure relates to an integrated tooth restoration design method and system. Prior Art

[0004] Background of the Invention

[0005] Recently, CAD / CAM dentistry (computer-aided design and computer-aided manufacturing in dentistry) has provided a wide range of dental restorations, including crowns, veneers, inlays and onlays, fixed bridges, dental implant restorations, and orthodontic appliances. In a typical CAD / CAM-based dental procedure, the treating dentist prepares the tooth to be restored as a crown, inlay, onlay, or veneer. The prepared tooth and its surroundings are then scanned using a three-dimensional (3D) imaging camera and uploaded to a computer for design. Alternatively, the dentist may obtain an impression of the tooth to be restored and either scan the impression directly or create a model to be scanned or uploaded to a computer for design.

[0006] Current dental CAD can often be cumbersome, time-consuming, and lead to inconsistent designs. Minimizing the time and errors involved in CAD dental restorations is of great interest. The ultimate goal is to provide an automated and integrated solution capable of consistently delivering quality designs. Building such a highly autonomous dental CAD system requires integrating a high level of expertise into the software. Summary of the Invention

[0007] Summary of the Invention

[0008] A computer-implemented method for providing a 3D digital tooth restoration is disclosed. The method may include receiving a 3D digital tooth model representing at least a portion of a patient's dentition; automatically determining a virtual tooth preparation site in the 3D digital tooth model using a first neural network; automatically generating a 3D digital denture model in the 3D digital tooth model using a second trained generative deep neural network; and automatically delivering the 3D digital tooth model including the virtual 3D denture model to a quality control ("QC") user.

[0009] Also disclosed is a non-transitory computer-readable medium storing executable computer program instructions for providing a 3D digital tooth restoration, the computer program instructions including instructions for: receiving a 3D digital tooth model representing at least a portion of a patient's dentition; automatically determining a virtual tooth preparation site in the 3D digital tooth model using a first neural network; automatically generating a 3D digital denture model in the 3D digital tooth model using a second trained generative deep neural network; and automatically delivering the 3D digital tooth model including the virtual 3D denture model to a quality control ("QC") user.

[0010] A system for providing a digital dental restoration is also disclosed. The system may include a processor and a non-transitory computer-readable storage medium including instructions executable by the processor to perform steps including: receiving a 3D digital dental model representing at least a portion of a patient's dentition; automatically determining a virtual tooth preparation site in the 3D digital dental model using a first neural network; automatically generating a 3D digital denture model in the 3D digital dental model using a second trained generative deep neural network; and automatically delivering the 3D digital dental model including the virtual 3D denture model to a quality control ("QC") user. Simple diagram description

[0011] FIG. 1 shows a diagram of an example of a cloud computing environment for an integrated recovery design service method and system in some embodiments.

[0012] FIG2 shows a diagram of an example of a computing environment in some embodiments.

[0013] FIG. 3 illustrates a diagram outlining an example of one or more features of an integrated resilience design service method and system in some embodiments.

[0014] FIG4 shows a flow chart illustrating an example of one or more features of some embodiments.

[0015] FIG5 shows a diagram of a convolutional neural network, such as in some embodiments.

[0016] FIG6 shows a top-down perspective view of an example of a 2D depth map of a digital model, such as in some embodiments.

[0017] 7( a ) and 7 ( b ) show diagrams of hierarchical neural networks, such as in some embodiments.

[0018] FIG8 shows a diagram of a deep neural network, such as in some embodiments.

[0019] FIG9 shows a diagram illustrating a computer-implemented method for automatic margin line proposal, such as in some embodiments.

[0020] 10 shows a perspective view of an example of a 3D digital model showing edge lines proposed from basic edge lines, for example in some embodiments.

[0021] 11( a ) and 11 ( b ) show perspective views of a 3D digital model with prepared teeth and proposed margin lines, such as in some embodiments.

[0022] FIG12 is a flowchart of a method for generating a 3D denture model using a deep neural network according to some embodiments of the present disclosure.

[0023] FIG13 is a graphical representation of the inputs and outputs of a deep neural network according to some embodiments of the present disclosure.

[0024] 14( a ) and 14 ( b ) are flowcharts of methods for training a deep neural network to generate 3D dentures according to some embodiments of the present disclosure.

[0025] FIG. 15( a ) illustrates an example of a GUI that may be used as part of a QC method in some embodiments.

[0026] FIG. 15( b ) illustrates an example of adjusting the proximal contact point in some embodiments.

[0027] FIG. 15( c ) illustrates an example of adjusting the telecentric contact point in some embodiments.

[0028] FIG15( d ) shows the generated 3D digital denture model as viewed from the occlusal direction.

[0029] 15( e ) and 15 ( f ) illustrate examples of a GUI in some embodiments that allows adjustment of the shape or contour of an automatically generated 3D digital denture.

[0030] FIG15( g ) shows an example of the margin line proposal determined by the QC method.

[0031] FIG. 16 illustrates an example of a flow chart in some embodiments.

[0032] FIG. 17 illustrates an example of a computer-implemented method in some embodiments. Implementation Method

[0033] Detailed description of preferred embodiments

[0034] For descriptive purposes, certain aspects, advantages, and novel features of embodiments of the present disclosure are described herein. The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Rather, the present disclosure relates to all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and subcombinations with each other. The methods, apparatus, and systems are not limited to any specific aspect or feature or combination thereof, nor do the disclosed embodiments require that any one or more specific advantages be present or problems be solved.

[0035] Although some operations in the disclosed embodiments are described in a particular sequential order for ease of presentation, it should be understood that this description encompasses reconfiguration unless specific language in the following text requires a specific ordering. For example, operations described sequentially may in some cases be reconfigured or performed simultaneously. Furthermore, for simplicity, the figures may not illustrate the various ways in which the disclosed methods can be used in conjunction with other methods. Furthermore, the description sometimes uses terms such as "provide" or "achieve" to describe the disclosed methods. The actual operations corresponding to these terms may vary depending on the specific implementation, and those of ordinary skill in the art can readily discern such operations.

[0036] As used in this application and the claims, the singular forms "a," "an," and "the" include the plural forms unless the context clearly dictates otherwise. Furthermore, the word "comprising" means "including." Furthermore, the words "coupled" and "associated" generally mean electrically, electromagnetically, and / or physically (e.g., mechanically or chemically) coupled or linked and, in the absence of specific language to the contrary, do not exclude the presence of intervening elements between the coupled or associated items.

[0037] In some instances, a value, procedure, or device may be referred to as "lowest," "best," "minimum," or the like. It should be understood that such descriptions are intended to indicate that many alternatives may be selected, and that such selections are not necessarily better, lesser, or otherwise advantageous than other selections.

[0038] In the following description, certain terms such as "upper," "lower," "upper," "lower," "horizontal," "vertical," "left," "right," and the like may be used. Where applicable, these terms are used to provide some clarity when dealing with relative relationships. However, these terms are not intended to imply absolute relationships, positions, and / or orientations. For example, with respect to an object, the "upper" surface can become the "lower" surface simply by turning the object over. However, it remains the same object.

[0039] As used herein, the term "denture" may refer to any dental restoration (prosthesis), including but not limited to crowns, bridges, dentures, partial dentures, implants, onlays, inlays, or veneers.

[0040] Some embodiments of the present disclosure may include providing an integrated workflow that automatically performs one or more steps in the integrated restoration design service method and system. As part of the integrated workflow, some embodiments of the present disclosure may include a computer-implemented method and / or system for designing a dental restoration associated with a dental model of a dentition. In some embodiments, the method and / or system may provide a streamlined, automated workflow for automatically digitally designing a dental restoration. In some embodiments, the integrated workflow, along with any corresponding computer-implemented method and / or system, may be implemented in a cloud computing environment. As is known in the art, a cloud computing environment may include, but is not limited to, one or more devices, such as one or more computing units, such as servers, networks, storage, and / or applications enabled via the internet and accessible by one or more permitted client devices. For example, one example of a cloud computing environment may include Amazon Web Services. Other types of cloud computing environments may be used, including, but not limited to, private cloud computing environments available to a limited set of clients (e.g., those within a company). In some embodiments, the system may be implemented in a cloud computing environment.

[0041] FIG1 illustrates an example of a cloud computing environment or system 102 for supporting an integrated digital workflow for providing dental restoration design and / or fabrication, according to some embodiments. In some embodiments, the cloud computing environment 102 may include a dental restoration cloud server 104, automated design features 103, quality control ("QC") features 106, storage 107, and other components commonly known in the art for cloud computing. In some embodiments, the components of the cloud computing environment 102 may be interconnected, for example, directly to each other via the dental restoration cloud server 104, or via one or more other servers or computers within the cloud computing environment or system 102. In some embodiments, one or more client devices 108 may be connected to the dental restoration cloud server 104 directly or via one or more networks 105. Each of the one or more client devices 108 may be connected to one or more scanners 109 known in the art to, for example, scan a patient's dentition or a dental impression and provide a 3D digital dental model of at least a portion of the patient's dentition. The dental restoration cloud server 104 can be connected to one or more manufacturing providers 110 directly and / or via one or more networks 105. FIG1 shows only one dental restoration cloud server 104, one automated design feature 103, one client device 108, one scanner 109, one third-party manufacturing provider 110, and one storage 107 to simplify and clarify the description. Embodiments of the cloud computing environment 102 may include multiple dental restoration cloud servers 104, automated design features 103, client devices 108, scanners 109, manufacturing providers 110, QC features 106, and storage 107. Similarly, the features and configuration / connections performed by the various entities in FIG1 may vary in different embodiments.

[0042] As part of the integrated workflow, in some embodiments, the dental restoration cloud server 104 can receive dental restoration cases from client devices 108 operated by clients, manage dental restoration cases between different clients, and provide completed dental restoration designs and / or milled dental restorations to the clients. In some embodiments, a dental restoration case may include a design-only case that only requests the dental restoration cloud server 104 to provide a digital design of a dental restoration. In some embodiments, a dental restoration case may request the dental restoration cloud server 104 to provide not only a design but also the fabrication of a dental restoration. In some embodiments, a dental restoration case may only request fabrication.

[0043] To manage dental restoration cases between different clients, the dental restoration cloud server 104 can optionally create accounts and manage profiles for each client. Furthermore, the dental restoration cloud server 104 can also create accounts for QC users to receive 3D digital tooth models and / or 3D digital tooth restorations, apply QC changes to the generated 3D tooth restorations using certain dental restoration QC software 106, manage QC user profiles, and, as needed, deliver dental restoration cases to and between QC users 106. Furthermore, the dental restoration cloud server 104 can determine the workflow of cases.

[0044] In some embodiments, the dental restoration cloud server 104 can create, read, update, and delete information related to case management, user management, materials, manufacturing, workflow, queuing, design automation, users, design, and user interface in storage, including, but not limited to, one or more databases. Other embodiments of the dental restoration cloud server 104 may include different and / or additional components. Furthermore, features may be distributed among the components in a manner different from that described herein. In some embodiments, the cloud computing environment 102 may include multiple dental restoration cloud servers 104 and / or other devices that perform tasks for multiple requesting clients.

[0045] In the example of FIG1 , the cloud computing environment 102 may include at least one processor in communication with at least one computer-readable storage medium that may include instructions. The instructions are executable by the at least one processor to perform steps such as receiving a virtual 3D dental model (also referred to as a "3D digital dental model") representing at least a portion of a patient's dentition from a client device 108, which may reside, for example, outside of the cloud computing environment 102. The cloud computing environment 102 may include an automated integrated restoration design service method and system 103 for automatically designing and generating a virtual 3D denture model for the virtual 3D dental model. In some embodiments, the cloud computing environment 102 may implement a quality control ("QC") feature 106 to evaluate the designed virtual 3D denture model and provide feedback to the automated integrated restoration design service method and system 103 to modify the current and / or future designs. In some embodiments, the QC feature may be implemented and / or used outside of the cloud computing environment 102. In some embodiments, the QC feature 106 and the automated design 103 may be implemented within the dental restoration cloud server 104. In some embodiments, the QC features 106 and the automated design 103 may be implemented in one or more separate computing environments within the cloud computing environment 102 .

[0046] In some embodiments, the cloud computing environment 102 may output a virtual 3D dental model with the designed virtual 3D dental prosthesis, or may simply output the designed virtual 3D dental prosthesis. The cloud computing environment 102 may provide the virtual 3D dental model with the designed virtual 3D dental prosthesis to a client device 108 or other device / system, and / or may optionally provide the virtual 3D dental model to a computer-aided manufacturing ("CAM") system / method 110 for fabrication. The CAM system / method 110 may fabricate a physical 3D model of the designed virtual 3D dental prosthesis.

[0047] One or more of the features depicted in FIG1 may be implemented by and / or in one or more computing environments. As shown in FIG2 , in some embodiments, computing environment 14042 may include one or more processing units 14030 and memory 14032. Processing units execute computer-executable instructions. Processing unit 14030 may be a central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In some embodiments, one or more processing units 14030 may, for example, execute multiple computer-executable instructions in parallel. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example, a representative computing environment may include a central processing unit and a graphics processing unit or a co-processing unit. Tangible memory 14032 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or a combination of both, accessible by the processing units. The memory stores software implementing one or more of the innovations described herein in the form of computer-executable instructions suitable for execution by the processing unit.

[0048] The computing system may have additional features. For example, in some embodiments, the computing environment includes storage 14034, one or more input devices 14036, one or more output devices 14038, and one or more communication connections 14037. An interconnection mechanism, such as a bus, controller, or network, interconnects the components of the computing environment. Typically, operating system software provides an operating environment for other software running in the computing environment and coordinates the activities of the components of the computing environment.

[0049] Tangible storage 14034 may be removable or non-removable and include magnetic or optical media such as disks, tapes or cartridges, CD-ROMs, DVDs, or any other media that can be used to store information in a non-transitory manner and accessed within a computing environment. Storage 14034 stores instructions for software implementing one or more of the innovations described herein.

[0050] An input device can be, for example, a touch input device such as a keyboard, mouse, stylus, or trackball; a voice input device; a scanning device; any of a variety of sensors; another device that provides input to the computing environment; or a combination thereof. For video encoding, the input device can be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or reads video samples to a CD-ROM or CD-RW in the computing environment. An output device can be a display, printer, speakers, CD writer, or another device that provides output from the computing environment.

[0051] A communication link enables communication with another computing entity via a communication medium. The communication medium carries information, such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, the communication medium may employ an electrical, optical, RF, or other carrier wave.

[0052] Any of the disclosed methods can be implemented as computer-executable instructions stored on one or more computer-readable storage media 14034 (e.g., one or more optical media discs, volatile memory devices (such as DRAM or SRAM), or non-volatile memory devices (such as flash memory or a hard drive)) and executed on a computer (e.g., any commercially available computer, including smartphones, other mobile devices that include computing hardware or programmable automated controllers) (e.g., the computer-executable instructions cause one or more processors of the computer system to perform the method). The term computer-readable storage medium does not include communication links, such as signals and carrier waves. Any of the computer-executable instructions used to implement the disclosed techniques, as well as any data created and used during the implementation of the disclosed embodiments, can be stored on one or more computer-readable storage media 14034. The computer-executable instructions can be, for example, part of a dedicated software application or a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or using one or more network computers in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network). In some embodiments, the computing environment may be part of a cloud computing environment and may include and / or access block storage, one or more databases, one or more queues, object storage, identification features, and one or more runtime environments that are also part of the cloud computing environment, either wired or wirelessly via the network or via a direct local connection. One or more features may be implemented in the cloud computing environment, which may be accessed via a device and / or interface capable of accessing the same.

[0053] FIG3 provides a more detailed overview of one or more features of the integrated design service workflow in some embodiments. Automated design 302 can receive a virtual 3D dental model generated by any method that scans a patient's dentition or a physical impression of the patient's dentition and generates a virtual 3D dental model of the patient's dentition. In some embodiments, the scanning method can generate, for example, an STL, PLY, or CTM file suitable for use with dental design software, such as FastDesign™ dental design software provided by Glidewell Laboratories of Newport Beach, California.

[0054] For example, in some embodiments, a virtual 3D dental model can be generated by a standard intraoral scanning device 308. In some cases, the intraoral scanning device can be handheld and can be used by a dentist, technician, or user to scan a patient's dentition. The standard intraoral scanning device and associated hardware and software can then generate the virtual 3D dental model as a standard STL file or other suitable standard format, which can be provided to a cloud computing environment or other network implementing the automated integrated restoration design service method and system 302.

[0055] In another example, the virtual 3D dental model can be generated by an X-ray computed tomography scanning system ("CT scanner") 306, known in the art, which can scan a physical dental impression of the patient's dentition. An example of a CT scan is described in U.S. Patent Application No. US20180132982A1 to Nikolskiy et al., which is hereby incorporated by reference in its entirety. In some embodiments, other types of CT scanners known in the art can be used.

[0056] In another example, the virtual 3D dental model may be generated by optically scanning a physical plaster cast 304 of the patient's dentition created by a dental laboratory, which may generate the virtual 3D dental model based on the scanned plaster cast.

[0057] In some embodiments, the virtual 3D tooth model can be provided to the automated integrated restoration design service method and system 302 via a graphical user interface ("GUI"), which can be displayed on a client device via a cloud computing environment. In some embodiments, for example, the GUI can provide an interface that allows a user to log into a dental restoration design server and upload a scanned virtual 3D tooth model.

[0058] In some embodiments, a cloud computing environment can receive a virtual 3D dental model from a client device, automatically identify virtual prepared teeth (also referred to as "digital prepared teeth") in the virtual 3D dental model, and automatically generate a virtual 3D denture model of the virtual prepared teeth (also referred to as a "3D digital denture"). In some embodiments, the cloud computing environment can evaluate the generated 3D dental model and optionally send the generated model along with the virtual 3D dental model to perform quality control ("QC") 310 on the generated virtual 3D denture model. In some embodiments, QC can optionally be performed by one or more QC users, including but not limited to laboratory technicians 312 and / or dentists / doctors 314, clients in some embodiments, and / or other users. In some embodiments, the final results of QC modifications can be determined as ground truth. In some embodiments, QC modifications can be automatically provided to the design feature 302 via a QC feedback loop 316. For example, the QC feedback loop 316 can provide QC modification data to improve future designs. In some embodiments, the current virtual 3D denture model undergoing QC can be regenerated based on the QC feedback loop 316. In some embodiments, once the generated virtual 3D denture model passes QC, it can be sent for physical generation by a CAM method 318. The CAM method 318 can include, for example, a chairside milling machine 320 or a milling center 322, or any other CAM manufacturing method known in the art.

[0059] FIG4 illustrates a flow chart showing the automated integrated restoration design service method and system from FIG3 in more detail. The automated integrated restoration design service method and system may include one or more of the steps illustrated in FIG4 . In some embodiments, the steps may be performed sequentially. In some embodiments, the steps may be performed in parallel. In some embodiments, the steps may be performed in any order. The automated integrated restoration design service method and system may receive a 3D digital dental model. The automated integrated restoration design service method and system may automatically identify a preparation site / teeth in the 3D digital dental model (402); automatically determine an occlusal direction in the 3D digital dental model (404); automatically determine a buccal direction in the 3D digital dental model (406); automatically detect margin lines in the 3D digital dental model (408); and / or automatically generate a 3D digital denture model design (410), for example, in some embodiments, for the 3D digital dental model in some embodiments. In some embodiments, the margin lines belong to the preparation site / teeth. One or more features may be determined as described in US Patent Application No. 17 / 245,944, which is hereby incorporated by reference in its entirety.

[0060] In some embodiments, a trained 3D deep neural network ("DNN") can be used to automatically determine one or more features on a volumetric (voxel) representation. In some embodiments, the DNN can be a convolutional neural network ("CNN"), which is a network that uses convolutions instead of conventional matrix multiplications in at least one of the hidden layers of the DNN. A convolutional layer can calculate its output value by applying a kernel function to a subset of the values ​​of the previous layer. A computer-implemented method can train the CNN by adjusting the weights of the kernel function based on training data. The same kernel function can be used to calculate each value in a particular convolutional layer.

[0061] FIG5 illustrates an example of a CNN in some embodiments. For illustration purposes, a 2D CNN is shown. A 3D CNN may have a similar architecture, but uses a three-dimensional kernel (xyz axes) to provide a three-dimensional output after each convolution. A CNN may include one or more convolutional layers, such as a first convolutional layer 502. First convolutional layer 502 may apply a kernel (also called a filter), such as kernel 504, to an input image, such as input image 503, and optionally apply an activation function to generate one or more convolution outputs, such as first kernel output 508. First convolutional layer 502 may include one or more feature channels. Applying a kernel, such as kernel 504, and optionally applying an activation function, may produce a first convolution output, such as convolution output 506. The kernel may then be advanced to a set of pixels in input image 503 based on a stride, and kernel 504 and, optionally, an activation function, may be applied to produce a second kernel output. The kernel may be advanced in this manner until it has been applied to all pixels in input image 503. In this manner, the CNN can generate a first convolution image 506 that may include one or more feature channels. In some embodiments, the first convolution image 506 may include one or more feature channels, such as 507. In some cases, the activation function may be, for example, a RELU activation function. Other types of activation functions may also be used.

[0062] The CNN may also include one or more pooling layers, such as a first pooling layer 512. The first pooling layer may apply a filter, such as a pooling filter 514, to the first convolved image 506. Any type of filter may be used. For example, the filter may be a maximum filter (which outputs the maximum value of the pixels to which the filter is applied) or an average filter (which outputs the average value of the pixels to which the filter is applied). The one or more pooling layers may downsample and reduce the size of the input matrix. For example, the first pooling layer 512 may downsample the first convolved image 506 by applying the first pooling filter 514 to provide a first pooled image 516. The first pooled image 516 may include one or more feature channels 517. The CNN may optionally apply one or more additional convolutional layers (and activation functions) and pooling layers. For example, the CNN may apply a second convolutional layer 518 and, optionally, an activation function, to output a second convolved image 520 that may include one or more feature channels 519. Second pooling layer 522 may apply a pooling filter to second convolved image 520 to generate second pooled image 524, which may include one or more feature channels. A CNN may include one or more convolutional layers (and activation functions) and one or more corresponding pooling layers. The output of the CNN may optionally be sent to a fully connected layer, which may be part of one or more fully connected layers 530. The one or more fully connected layers may provide output predictions, such as output prediction 524. For example, in some embodiments, output prediction 524 may include labels for the tooth and surrounding tissue. In some embodiments, output prediction 524 may include identification of one or more features in a 3D digital tooth model.

[0063] Some embodiments may include determining the occlusal orientation of a 3D digital dental model. In some embodiments, occlusal orientation can be automatically determined using a neural network trained on occlusal orientation. In some embodiments, the CNN trained on occlusal orientation may be a 3D CNN trained using one or more 3D voxel representations, each of which represents a patient's dentition, optionally utilizing augmented data such as the surface normal of each voxel. The 3D CNN may perform 3D convolutions, which use a 3D kernel instead of a 2D kernel and operate on 3D input. In some embodiments, the trained 3D CNN receives 3D voxel representations with voxel normals. In some embodiments, an N×N×N×3 floating-point tensor may be used. In some embodiments, N may be, for example, 100. Other suitable values ​​of N may be used. In some embodiments, the trained 3D CNN may include four levels of 3D convolutions and two linear layers. In some embodiments, the 3D CNN may operate using a regression mechanism that regresses voxels representing the patient's dentition and their corresponding normals into three numbers: the X, Y, and Z coordinates of a unit bite vector. In some embodiments, the training set for a 3D CNN may include one or more 3D voxel representations, each representing a patient's dentition. In some embodiments, each 3D voxel representation in the training set may include a bite direction labeled manually by a user or by other techniques known in the art. In some embodiments, the training set may include tens of thousands of 3D voxel representations, each with a labeled bite direction. In some embodiments, the training dataset may include 3D point cloud models, each with a labeled bite direction. Thus, each image / model (3D voxel representation) of a patient's dentition is labeled with a bite direction by a technician in the training dataset, and the training dataset may include tens of thousands of images / models (3D voxel representations) of the corresponding patient's dentition. In the training dataset, the coordinates of the unit bite vector may be such that, for example, in some embodiments, X² + Y² + Z² = 1.

[0064] Some embodiments may include determining a digital preparation site in a virtual 3D digital dental model of the patient's dentition. In some embodiments, the digital preparation site may be automatically determined using a neural network trained on the preparation site. In some embodiments, the neural network trained on the preparation site may determine the 3D center of the digital preparation site (also referred to as a digital preparation mold).

[0065] In some embodiments, the 3D center of the digital preparation can be automatically determined. For example, in some embodiments, a neural network can be used to determine the 3D center of the digital preparation on an occlusally aligned 3D point cloud. In some embodiments, the trained neural network can provide the 3D coordinates of the center of the digital preparation bounding box. In some embodiments, the neural network can be any neural network capable of segmenting a 3D point cloud. For example, in some embodiments, the neural network can be the PointNet++ neural network segmentation described in this disclosure. In some embodiments, the digital preparation can be defined by a fixed-radius sphere surrounding the 3D center of the digital preparation. In some embodiments, for example, for molars and premolars, the fixed radius can be 0.8 cm. In some embodiments, other suitable values ​​for the fixed radius can be determined and used. In some embodiments, training the neural network can include using a sampled point cloud of the digital jaw (without augmentation) centered at the center of mass of the jaw. In some embodiments, the digital jaw point cloud can be oriented such that the occlusal direction is vertically oriented. In some embodiments, a computer-implemented method may train a neural network to identify digital preparation sites / molds in a 3D digital dental model using a training dataset. The training dataset may include a 3D digital model of a patient's dentition, such as a digital jaw that may include a preparation site, where one or more points within the margins of the preparation site are marked by a user using an input device or any other technique known in the art. In some embodiments, the training dataset may be tens of thousands of points. In some embodiments, the neural network may utilize segmentation to return a bounding box containing the selected points. For example, in some embodiments, the segmentation used may be PointNet++ segmentation.

[0066] In some embodiments, the 3D center of a digitally prepared mold can be automatically determined based on a planar depth map image of the jaw. In a training dataset, the mold center can be determined as the geometric center of the edges marked by the technician. For example, in some embodiments, the final edge point from a completed case can be used. In some embodiments, the network can receive a depth map image of the jaw from an occlusal view and return the location (X, Y) of the mold center in pixel coordinates of the image. For training, a dataset containing depth map images and corresponding correct answers (floating-point X and Y values) can be used. In some embodiments, the training dataset can be tens of thousands.

[0067] In some embodiments, the 3D center of the digital preparation mold can be optionally set manually by the user. In some embodiments, the 3D center of the digital preparation mold can be set using any technology known in the art.

[0068] Some embodiments may include determining the buccal orientation of a digital 3D dental model. In some embodiments, the buccal orientation may be automatically determined using a buccal trained neural network. In some embodiments, the 3D digital model may include the buccal orientation. In some embodiments, the buccal orientation may be manually set by the user. In some embodiments, the buccal orientation may be determined using any technique known in the art.

[0069] In some embodiments, a trained 2D CNN operates on an image representation. In some embodiments, cheek orientation can be determined by providing a 2D depth map image of a 3D digital model mesh to the trained 2D CNN. In some embodiments, the method may optionally include generating a 2D image from the 3D digital model. In some embodiments, the 2D image may be a 2D depth map. The 2D depth map may include a 2D image that contains, for each pixel, the distance from an orthogonal camera to an object along a line passing through the pixel. For example, in some embodiments, the object may be a surface of a digital jaw model. In some embodiments, the input may include, for example, an object, such as a 3D digital model of a patient's dentition ("digital model"), such as a jaw, and a camera orientation. In some embodiments, the camera orientation may be determined based on an occlusal direction. The occlusal direction is orthogonal to the occlusal plane, and the occlusal plane of the digital model may be determined using any technique known in the art. Alternatively, in some embodiments, the occlusal direction can be specified by the user using an input device such as a mouse or touch screen to manipulate a digital model on a display, for example, as described herein. In some embodiments, the occlusal direction can be determined, for example, using the occlusal axis technique described in U.S. Patent Application No. 16 / 451,968 (U.S. Patent Publication No. US20200405464A1) to Nikolskiy et al., the entirety of which is incorporated herein by reference.

[0070] A 2D depth map can be generated using any technique known in the art, including, for example, a z-buffer or ray tracing. For example, in some embodiments, a computer-implemented method may initialize the depth of each pixel (j, k) to a maximum length and the pixel color to, for example, the background color. For each pixel in a polygonal projection onto a digital surface, such as a 3D digital model, the computer-implemented method may determine the depth z of the polygon at (x, y) corresponding to pixel (j, k). If z < the depth of pixel (j, k), the depth of the pixel is set to depth z. "z" may refer to the convention that the central axis of the camera's view is in the direction of the camera's z-axis and does not necessarily refer to the absolute z-axis of the scene. In some embodiments, the computer-implemented method may also set the pixel color to a color other than, for example, the background color. For example, in some embodiments, the polygon may be a digital triangle. In some embodiments, the depth in the figure may be the depth per pixel. Figure 6 illustrates an example of a 2D depth map for a digital model in some embodiments.

[0071] In some embodiments, the 2D depth map image may include the von Mises average of 16 rotated versions of the 2D depth map. In some embodiments, the cheek orientation may be determined after determining the occlusal orientation and the 3D center of the digital preparation mold. In some embodiments, the 2D depth map image may be an image of a portion of the digital jaw surrounding the digital preparation mold. In some embodiments, regression may be used to determine the cheek orientation. For example, in some embodiments, the 2D CNN may include GoogleNet Inception v3, known in the art. In some embodiments, the computer-implemented method may use a training dataset to train a neural network for cheek training. For example, in some embodiments, the training dataset may include cheek orientations labeled in a 3D point cloud model. In some embodiments, the training dataset may include tens to hundreds of thousands of images. For example, in some embodiments, the computer-implemented method preprocesses the training dataset by converting each training image into a 2D depth map as previously disclosed and training the 2D CNN using the 2D depth map.

[0072] In some embodiments, the computer-implemented method may include determining a margin line proposal. In some embodiments, the computer-implemented method may determine the margin line proposal by receiving a 3D digital model having a digitally prepared region, and determining an internal representation of the 3D digital model using a neural network trained using the internal representation.

[0073] Some embodiments of computer-implemented methods may include determining the internal representation of a 3D digital model using a neural network trained on an internal representation. In some embodiments, the neural network trained on the internal representation may include an encoder neural network. In some embodiments, the neural network trained on the internal representation may include a neural network used for 3D point cloud analysis. In some embodiments, the neural network trained on the internal representation may include a trained hierarchical neural network ("HNN"). In some embodiments, the HNN may include a PointNet++ neural network. In some embodiments, the HNN may be any message-passing neural network that operates on geometric structures. In some embodiments, the geometric structures may include shapes, meshes, and / or point clouds.

[0074] In some embodiments, computer-implemented methods may use an HNN, such as PointNet++, for encoding. PointNet++ is described in "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space" by Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas (Stanford University, June 2017), which is hereby incorporated by reference in its entirety. A hierarchical neural network can, for example, process a sampled point set in a metric space in a hierarchical manner. In some embodiments, an HNN, such as PointNet++ or another HNN, can be implemented by determining local structure induced by the metric. In some embodiments, an HNN, such as PointNet++ or another HNN, can be implemented by first partitioning the point set into two or more overlapping local regions based on a distance metric. The distance metric can be based on an underlying space. In some embodiments, local features can be extracted. For example, in some embodiments, granular geometric structure can be determined from a small local neighborhood. In some embodiments, small local neighborhood features can be grouped into larger units. In some embodiments, the larger units can be processed to provide higher-level features. In some embodiments, the method is repeated until all features for the entire point set are obtained. Unlike volumetric CNNs, which scan space with a fixed stride, the local receptive field in HNNs such as PointNet++ and other HNNs depends on both the input data and the metric. Furthermore, compared to CNNs that scan a vector space agnostic to the data distribution, the sampling strategy in HNNs such as PointNet++ and other HNNs generates the receptive field in a data-dependent manner.

[0075] In some embodiments, an HNN, such as PointNet++ or another HNN, may, for example, determine how to segment a point set and abstract a set of points or local features using a local feature learner. For example, in some embodiments, the local feature learner may be PointNet or any other suitable feature learner known in the art. For example, in some embodiments, the local feature learner may process an unordered point set to perform semantic feature extraction. The local feature learner may abstract one or more sets of local points / features into a higher-level representation. In some embodiments, the HNN may recursively apply the local feature learner. For example, in some embodiments, PointNet++ may recursively apply PointNet to nested portions of the input set.

[0076] In some embodiments, an HNN can define overlapping point set segments by defining each segment as a neighborhood sphere in Euclidean space, with parameters that may include, for example, the centroid location and scale. For example, the centroid can be selected from the input set using farthest point sampling, as is known in the art. One advantage of using an HNN can include, for example, efficiency and effectiveness, since the local receptive field can depend on the input data and the metric. In some embodiments, an HNN can utilize fields at multiple scales. This can, for example, allow for robustness and the capture of detail.

[0077] In some embodiments, the HNN may include hierarchical point set feature learning. For example, in some embodiments, the HNN may establish a hierarchical grouping of points and abstract increasingly larger local regions along the hierarchy. In some embodiments, the HNN may include several set abstraction levels. In some embodiments, at each level, the point set is processed and abstracted to produce a new set with fewer elements. In some embodiments, the set abstraction level may include three layers: a sampling layer, a grouping layer, and a local feature learner layer. For example, in some embodiments, the local feature learner layer may be a PointNet. For example, in some embodiments, the set abstraction level may take as input an N×(d+C) matrix derived from N points with d-dim coordinates and C-dim point features, and output an N'×(d+C') matrix of N' sub-sampled points with d-dim coordinates and new C'-dim feature vectors that summarize the local context.

[0078] In some embodiments, the sampling layer may select or sample a set of points from the input points. For example, in some embodiments, the HNN may define these selected / sampled points as the centroid of the local region. For example, in some embodiments, for the input points {x1, x2, …, xn} of the sampling layer, iterative farthest point sampling (FPS) may be used to select a subset of points { , , …, }, so that x ij is the distance from the rest of the points to the set { , , …, } furthest point. Compared to random sampling, given the same number of centroids, this can advantageously provide, for example, better coverage of the entire point set. Compared to convolutional neural networks (CNNs) that scan vector space independently of the data distribution, this can also advantageously generate, for example, receptive fields in a data-dependent manner.

[0079] For example, in some embodiments, the grouping layer may determine one or more sets of local regions by determining the neighboring points surrounding each centroid. In some embodiments, the input to this layer may be a point set of size N × (d + C) and the coordinates of the centroid of size N' × d. In some embodiments, the output of the grouping layer may include groups of point sets of size N' × K × (d + C). For example, in some embodiments, each group may correspond to a local region, and K may be the number of points in the neighborhood of the centroid point. In some embodiments, K may vary between groups. However, for example, the next layer (the PointNet layer) may convert a flexible number of points into a fixed-length local region feature vector. For example, in some embodiments, the neighborhood may be defined by a distance metric. For example, in some embodiments, a sphere query may determine all points within a radius of the query point. K may be capped. Alternatively, a k-nearest neighbor (kNN) search may be used. A kNN search may determine a fixed number of neighboring points. However, for example, in some embodiments, the local neighborhood of the ball query can guarantee a fixed region scale, thereby making one or more local region features more generalizable in space. For example, in some embodiments, this can be advantageous for semantic point labeling or other tasks requiring local pattern recognition.

[0080] In some embodiments, the local feature learner layer can encode the local region pattern into a feature vector. For example, given X = (M, d) is a discrete metric space whose metric system inherits from the Euclidean space Xn, where M⊆ [R] where n is a set of points and d is a distance metric, the local feature learner layer determines a function f that takes X as input and outputs semantically interesting information about X. The function f can be a classification function that assigns labels to X or a piecewise function that assigns a label to each member of M.

[0081] Some embodiments may use PointNet as a local feature learner layer, given an unordered set of points {​​ x 1, x 2, …, xn}, where xi ϵ [R] d, the learner layer can define the set function f: X à [R], which maps a set of points to vectors, such as:

[0082] f(x 1,x 2, …,xn) = γ( { h(xi)}).

[0083] In some embodiments, γ and h can be, for example, multilayer perceptron (MLP) networks or other suitable alternatives known in the art. For example, in some embodiments, the function f can be invariant to permutations of the input points and can approximate any continuous set function. In some embodiments, the response of h can be interpreted as a spatial encoding of the point. PointNet is described in R.Q. Charles, H. Su, M. Kaichun, and L.J. Guibas, "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 77-85, which is hereby incorporated by reference in its entirety.

[0084] In some embodiments, the local feature learner layer may receive N' local point regions. For example, the data size may be N' × K × (d + C). In some embodiments, each local region is abstracted by its centroid and, for example, encoding the local features of the centroid's neighborhood in the output. For example, in some embodiments, the output data size may be N' × (d + C). In some embodiments, the coordinates of a point in a local region may be translated into a local frame relative to the centroid point: , where i = 1, 2, ... K and j = 1, 2, ..., d, where is the centroid coordinate. For example, in some embodiments, using relative coordinates with point features can capture point-to-point relationships in a local area. In some embodiments, PointNet can be used for local pattern learning.

[0085] For example, in some embodiments, the local feature learner can address uneven density in the input point set via a density adaptive layer. As the input sampling density changes, the density adaptive layer can learn to combine features from regions of different scales. For example, in some embodiments, the density adaptive hierarchical network is a PointNet++ network. For example, in some embodiments, the density adaptive layer can include multi-scale grouping (MSG) or multi-resolution grouping (MRG).

[0086] In some embodiments, multi-scale patterns can be captured in MSG by applying a grouping layer with different scales followed by extracting features at each scale. For example, in some embodiments, feature extraction at each scale can be performed using PointNet. For example, in some embodiments, features at different scales can be concatenated to provide multi-scale features. In some embodiments, HNNs can learn optimal multi-scale feature combinations through training. For example, random input dropout can be used, where random input points are dropped with random probability. As an example, in some embodiments, a dropout ratio of Ɵ, uniformly sampled from [0, p], can be used, where p is less than or equal to 1. As an example, in some cases, p can be set to 0.95 to prevent the generation of empty point sets. For example, in some embodiments, other suitable values ​​can be used.

[0087] For example, in some embodiments, in an MRG, the features of a region at level Li may be the concatenation of two vectors, where the first vector is obtained by aggregating features from subregions at the lower level Li-1. This can be achieved by setting the abstraction level. In some embodiments, the second vector may be features obtained by directly processing the raw points of the local region using, for example, a single PointNet. In cases where the local region density is low, the second vector may be weighted more heavily in some embodiments because the first vector contains fewer points and includes sampling defects. In cases where the local region density is high, for example, the first vector may be weighted more heavily in some embodiments because it may provide finer details due to recursive detection at a higher resolution at a lower level.

[0088] In some embodiments, point features can be propagated for cluster segmentation. For example, in some embodiments, a hierarchical propagation strategy can be used. In some embodiments, feature propagation can include propagating point features from N l × ( d + C) points to N l-1 points, where N l-1 and N l (N l is less than or equal to N l-1) are the point set sizes for the input and output of a given abstraction level l. In some embodiments, feature propagation can be achieved by interpolating the feature values ​​f of N l points at the coordinates of N l-1 points. In some embodiments, an inverse distance weighted average based on the k nearest neighbors can be used (in the following equation, p = 2, k = 3; other suitable values ​​may be used). For example, in some embodiments, the interpolated features at N l-1 can be concatenated with skip-link point features from the given abstraction level. For example, in some embodiments, the concatenated features can be passed through a PointNet unit, which can be similar to a 1×1 convolution in a convolutional neural network. For example, in some embodiments, a shared fully connected and ReLU layer may be applied to update the feature vector of each point. In some embodiments, the method may be repeated until the features propagated to the initial set of points are determined.

[0089] f (j) ( x)= ,in , j=1 ,…… ,C

[0090] In some embodiments, computer-implemented methods may implement one or more neural networks disclosed or known in the art. Any specific structures and values ​​with respect to one or more neural networks, as well as any other features disclosed herein, are provided merely as examples, and any suitable variants or equivalents may be used. In some embodiments, as an example, one or more neural network models may be implemented based on the Pytorch geometry package.

[0091] Figures 7(a) and 7(b) illustrate examples of an HNN in some embodiments. The HNN may include a hierarchical point set feature learner 702, the output of which may be used to perform segmentation 704 and / or classification 706. The hierarchical point set feature learner 702 example uses points in 2D Euclidean space as an example, but can operate on an input 3D image in three dimensions. For example, as shown in the example of Figure 7(a), the HNN may receive an input image 708 having dimensions (N, d + C) and perform a first sampling and grouping operation 710 to generate a first sampled and grouped image 712 having dimensions (N1, K, d + C). The HNN may then provide the first sampled and grouped image 712 to a PointNet at 714 to provide a first abstract image 716 having dimensions (N1, d + C1). The first abstract image 716 may undergo sampling and grouping 718 to provide a second sampled and grouped image 720 having (N2, K, d + C1). The second sampled and grouped image 720 may be provided to a PointNet neural network 722 to output a second abstract image 724 having (N2, d + C2).

[0092] In some embodiments, the second abstract image 724 may be segmented by the HNN segmentation 704. In some embodiments, the HNN segmentation 704 may take the second abstract image 724 and perform a first interpolation 730, the output of which may be concatenated with the first abstract image 716 to provide a first interpolated image 732 having (N1, d + C2 + C1). The first interpolated image 732 may be provided to the PointNet unit at 734 to provide a first segment image 736 having (N1, d + C3). The first segment image 736 may be interpolated at 738, the output of which may be concatenated with the input image 708 to provide a second interpolated image 740 having (N1, d + C3 + C). The second interpolated image 740 may be provided to the PointNet unit 742 to provide a segmented image 744 having (N, k). For example, the segmented image 744 may provide a per-point score.

[0093] In some embodiments, as shown in the example of FIG7( b ), the second abstract image 724 may be classified by the HNN classifier 706. In some embodiments, the HNN classifier may take the second abstract image 724 and provide it to a PointNet network 760, whose output 762 may be provided to one or more fully connected layers, such as a connected layer 764, whose output may provide a class score 766.

[0094] Some embodiments of the computer-implemented method may include determining edge line proposals from basic edge lines and an interior representation of a 3D digital model using a neural network trained using shift values.

[0095] In some embodiments, a basic edge line may be pre-calculated once per network type. In some embodiments, the network types may include molars and premolars. For example, in some embodiments, other suitable network types may be used. In some embodiments, the network types may include other types. In some embodiments, the same basic edge line may be used as the initial edge line for each scan. In some embodiments, the basic edge line is three-dimensional. In some embodiments, the basic edge line may be determined based on edge lines from a training dataset used to train a neural network for internal representation training and a neural network for shift value training. In some embodiments, the basic edge line may be a pre-calculated mean or average of the edge lines in the training dataset. In some embodiments, any type of mean or average may be used.

[0096] In some embodiments, the edge line proposals may be free-form edge line proposals. In some embodiments, the shift value trained neural network may include a decoder neural network. In some embodiments, the decoder neural network may concatenate an internal representation with specific point coordinates to perform guided decoding. In some embodiments, guided decoding may generate closed surfaces, as described in T. Groueix, M. Fisher, VG Kim, BC Russell, and M. Aubry, "A Papier-Mache Approach to Learning 3D Surface Generation," 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 216-224, which is hereby incorporated by reference in its entirety.

[0097] In some embodiments, the decoder neural network may comprise a deep neural network ("DNN"). Referring now to FIG. 8 , a high-level block diagram illustrating the structure of a deep neural network (DNN) 800 according to some embodiments of the present disclosure is shown. DNN 800 comprises multiple layers N i, N h,1, N h,1-1, N h,1, N o, and so on. The first layer N i is the input layer, which may ingest one or more dental scan datasets. The last layer N o is the output layer. The deep neural networks used in the present disclosure may output probabilities and / or complete 3D margin proposals. For example, the output may be a probability vector comprising one or more probability values ​​for each feature or aspect of the dental model belonging to certain categories. Alternatively, the output may be margin proposals.

[0098] Each layer N may include multiple nodes connected to each node in the next layer N+1. For example, each computational node in layer Nh,1-1 is connected to each computational node in layer Nh,1. Layers Nh,1, Nh,1-1, and Nh,1 between input layer Ni and output layer No are hidden layers. The nodes in the hidden layers, denoted as "h" in Figure 8, may be hidden variables. In some embodiments, DNN 800 may include multiple hidden layers, such as 24, 30, or 50.

[0099] In some embodiments, DNN 800 may be a deep feedforward network. DNN 800 may also be a convolutional neural network, which uses convolutions in place of general matrix multiplications in at least one of the hidden layers of the deep neural network. DNN 800 may also be a generative neural network or a generative adversarial network. In some embodiments, training may use labeled training data sets to supervise the learning method of the deep neural network. Labels are used to map features to probability values ​​in probability vectors. Alternatively, training may use unstructured and unlabeled training data sets to train a generative deep neural network in an unsupervised manner, which does not necessarily require labeled training data sets.

[0100] In some embodiments, the DNN may be a multilayer perceptron (MLP). In some embodiments, the MLP may include four layers. In some embodiments, the MLP may include a fully connected MLP. In some embodiments, the MLP utilizes BatchNorm regularization.

[0101] FIG9 shows a diagram of a computer-implemented method for automatic margin line proposal as an example in some embodiments. For example, in some embodiments, before beginning margin line proposal for any 3D digital model, the computer-implemented method may pre-compute (901) basic margin lines 903 in three dimensions, where each point of the basic margin lines 903 has a 3D coordinate, such as coordinate 905. The computer-implemented method may receive a 3D digital model 902 of at least a portion of a jaw. In some embodiments, the 3D digital model may be in the form of a 3D point cloud. For example, the 3D digital model may include prepared teeth 904. The computer-implemented method may use an internal representation trained neural network 906 to determine an internal representation 908 of the 3D digital model. For example, in some embodiments, the internal representation trained neural network 906 may be a neural network that performs grouping and sampling 907, among other operations, on the 3D digital model (such as an HNN in some embodiments). In some embodiments, a computer-implemented method may use a shift-value trained neural network 910 to determine edge line proposals from the basic edge lines 903 and the internal representation 908 of the 3D digital model. In some embodiments, the shift-value trained neural network may provide, for example, one or more three-dimensional shift values ​​912 for digital surface points of the basic edge lines 903.

[0102] In some embodiments, a neural network trained with shift values ​​can determine three-dimensional edge line shift values ​​from basic edge lines. In some embodiments, the neural network trained with shift values ​​uses BilateralChamferDistance as a loss function. In some embodiments, a computer-implemented method can shift one or more points of a basic edge line by a shift value to provide an edge line proposal. FIG10 illustrates an example of some embodiments of adjusting basic edge lines 1002 of a 3D digital model 1000. In this example, one or more basic edge line points, such as basic edge line point 1004, can be shifted by a shift value and direction 1006. For example, other basic edge line points can be similarly adjusted according to their corresponding shift values ​​and directions to form edge line proposals 1008.

[0103] FIG11( a ) illustrates one example of a proposed digital margin line 1104 for a digitally prepared tooth 1102 in a 3D digital model 1105 . As can be seen, a margin line proposal can be generated even when the margin line is partially or completely covered by gum, blood, saliva, or other elements. FIG11( b ) illustrates another example of a proposed digital margin line 1106 for a digitally prepared tooth 1108 in a 3D digital dental model 1110 . In some embodiments, the proposed margin line is displayed on the 3D digital model and can be manipulated by a user, such as a dental technician or doctor, using an input device to adjust the margin line proposal.

[0104] In some embodiments, the same training dataset may be used to train the neural network for internal representation training and the neural network for shift value training. In some embodiments, the training dataset may include one or more training samples. In some embodiments, the training dataset may include 70,000 training samples. In some embodiments, each of the one or more training samples may include an occlusal direction, a preparation mold center, and a buccal direction as normalized positions and orientations for each sample. In some embodiments, the occlusal direction, the preparation mold center, and the buccal direction may be set manually. In some embodiments, the training dataset may include an untrimmed digit surface of the jaw and a target margin line on a surface corresponding to the trimmed digit surface. In some embodiments, the target margin line may be prepared by a technician. In some embodiments, training may use regression. In some embodiments, training may include using a loss function to compare margin line proposals with target margin lines. In some embodiments, the loss function may be a Chamfer loss function. In some embodiments, the Chamfer loss function may include:

[0105]

[0106] In some embodiments, training may be performed on a computing system that may include at least one graphics processing unit ("GPU"). For example, in some embodiments, the GPU may include two 2080-Ti Nvidia GPUs. Other suitable GPU types, numbers, and equivalents may be used.

[0107] In some embodiments, the computer-implemented method can be automatically performed. Some embodiments can further include displaying the free-form margin line on the 3D digital model. In some embodiments, the free-form margin line can be adjusted by a user using an input device.

[0108] Some embodiments may include using a trained deep neural network to generate a virtual 3D denture model based on a virtual 3D tooth model. Some embodiments may include using a trained generative deep neural network to automatically generate a 3D digital denture model (virtual 3D denture model) from a 3D digital tooth model. One example of using a deep neural network to generate a denture is described in U.S. patent application Ser. No. 15 / 925,078 (now U.S. Patent No. 11,007,040), which is hereby incorporated by reference in its entirety. Another example of using a deep neural network to generate a denture is described in U.S. patent application Ser. No. 15 / 660,073, which is hereby incorporated by reference in its entirety.

[0109] Exemplary embodiments of methods and computer-implemented systems for generating a 3D model of a dental prosthesis using a deep neural network are described herein. Certain embodiments of the method may include: training a deep neural network using a training dataset, using one or more computing devices, to generate a first 3D dental prosthesis model; receiving, by the one or more computing devices, patient scan data representing at least a portion of a patient's dentition; and generating the first 3D dental prosthesis model based on the received patient scan data using the trained deep neural network.

[0110] The training dataset may include a dentition scan dataset with preparation site data and a denture dataset. The preparation site above the gum line may be defined by a preparation margin or margin line above the gum line. The denture dataset may include scanned prosthesis data associated with each preparation site in the dentition scan dataset.

[0111] The scanned prosthesis can be a scan of a real patient's crown created based on a library of tooth templates, which can have 32 or more tooth templates. The dentition scan data set with preparation data can include scans of real preparations from the patient's scanned dentition.

[0112] In some embodiments, the training dataset may include a natural dentition scan dataset with digitally created preparation site data and a natural denture dataset. The natural denture dataset may include segmented tooth data associated with each digitally created preparation site in the dentition scan dataset. The natural dentition scan dataset may have two main components. The first component is a dataset comprising scanned dentition data of the patient's natural teeth. The data in the first component includes all of the patient's teeth in their natural, unmodified digital state. The second component of the natural dentition scan data is a missing tooth dataset, where one or more teeth have been removed from the scan data. In place of the missing teeth, the preparation sites created by the deep neural network can be placed at the locations of the removed teeth. This method generates two dentition datasets: complete, unmodified dentition scan data of the patient's natural teeth; and a missing tooth dataset (natural denture dataset), where one or more teeth have been digitally removed from the dentition scan data.

[0113] In some embodiments, the method further includes generating a digital model of the entire dental arch and segmenting each tooth in the entire dental arch to generate natural crown data for use as training data. The method may also include: using a natural dentition scan dataset with digitally created preparation data and a natural denture dataset to train another deep neural network to generate a second 3D denture model; using another deep neural network to generate a second 3D denture model based on received patient scan data; and blending features of the first and second 3D denture models to generate a blended 3D denture model.

[0114] FIG12 illustrates a method 1200 for denture generation using a deep neural network (DNN). Method 1200 begins at 1205, where a dentition scan dataset is received or ingested into a database. The dentition scan dataset may include one or more scans of a real patient's dentition, including tooth preparations and technician-generated (non-DNN-generated) dentures created for those preparations. A tooth preparation (also referred to as a tooth preparation or prepared tooth) is a tooth, multiple teeth, or an area on a tooth that has been prepared to receive a denture (e.g., a crown, bridge, inlay, etc.). A technician-generated or non-DNN-generated denture is a denture designed primarily by a technician. Alternatively, a technician-generated denture may be designed based on a tooth template library having multiple tooth restoration templates. Each tooth in an adult's mouth may have one or more tooth restoration templates in the tooth template library.

[0115] In some embodiments, the received dentition scan data set with tooth preparations may include scans of a real patient's dentition with one or more tooth preparations. The preparations may be defined by a preparation margin. The received dentition scan data set may also include scans of dentures after they have been installed in their corresponding tooth preparations. This data set may be referred to as a denture data set. In some embodiments, the denture data set may include scans of a denture created by a technician before the denture is installed.

[0116] In some embodiments, each received dental scan data set may optionally be pre-processed before using the data set as input to a deep neural network. Dental scan data is typically a 3D digital image or file representing one or more portions of a patient's dentition. A 3D digital image of the patient's dentition (3D scan data) can be acquired by intraorally scanning the patient's mouth. Alternatively, an impression or physical model of the patient's teeth can be scanned to generate 3D scan data of the patient's dentition. In some embodiments, the 3D scan data can be converted into a 2D data format using, for example, a 2D depth map and / or snapshot.

[0117] At 1210, a deep neural network can be trained (e.g., by a computer-implemented method or another method) using a dentition scan dataset comprising scans of real tooth preparations and their corresponding technician-generated dentures, either after and / or before installation. This combination of the dentition scan dataset of real tooth preparations and their corresponding technician-generated dentures can be referred to herein as a technician-generated dentition scan dataset. In some embodiments, the deep neural network can be trained using only the technician-generated dentition scan dataset. In other words, the training data only contains technician-generated dentures created based on one or more library templates of dental restorations.

[0118] The tooth templates in a dental restoration library can be considered optimal restoration models because they are designed with specific features for a particular tooth (e.g., the third tooth). Generally speaking, there are 32 teeth in a typical adult mouth. Therefore, a dental restoration library may contain at least 32 templates. In some embodiments, each tooth template may have one or more specific features that are specific to one of the 32 teeth (e.g., sidewall size and shape, buccal and lingual cusps, occlusal surfaces, buccal and lingual arches, etc.). For example, each tooth in the restoration library is designed to include features, landmarks, and orientations that will best fit the adjacent teeth, surrounding gums, and the tooth's location and position within the dental arch form. In this way, a deep neural network can be trained to recognize certain features that may be prominent for a particular tooth (e.g., sidewall size and shape, cusps, sulci, pits, etc.) and their relationships (e.g., the distance between cusps).

[0119] In some embodiments, a computer-implemented method or any other method may train a deep neural network to identify one or more dentition classes present or recognized in a training dataset based on an output probability vector. For example, assume the training dataset contains a large number of depth maps representing a patient's upper jaw and / or depth maps representing a patient's lower jaw. The computer-implemented method or another method may use the training dataset to train a deep neural network to identify individual teeth in a dental arch form. Similarly, the deep neural network may be trained to map a depth map of the lower jaw to a probability vector comprising probabilities of belonging to both the upper and lower jaw depth maps, where the probability of the depth map belonging to the lower jaw is the highest in the vector or is substantially higher than the probability of the depth map belonging to the upper jaw.

[0120] In some embodiments, a computer-implemented method or another method can use a dentition scan dataset with one or more scan datasets of real tooth preparations and corresponding technician-generated dentures to train a deep neural network to generate a complete 3D dental restoration model. In this way, the 3D dental restoration model generated by the DNN inherently incorporates one or more features of one or more tooth templates from a dental restoration library, which can be part of the database 150.

[0121] A computer-implemented method or another method may train a deep neural network, such as the networks discussed in Figures 8, 13, 14(a)-14(b), or another neural network, to generate a 3D model of a dental restoration using only a technician-designed dentition scan dataset. In this way, the 3D denture generated by the DNN will inherently include one or more features of the denture designed by the human technician using the library template. In some embodiments, the computer-implemented method or another method may train the deep neural network to output a probability vector comprising the probability that the occlusal surface of the technician-generated denture represents the occlusal surface of a missing tooth at the preparation site or margin. Alternatively, the computer-implemented method or another method may train the deep neural network to generate a complete 3D dental restoration model by mapping the occlusal surface and margin line data from the scanned dentition data with the highest probability to the preparation site. Additionally, a computer-implemented method or another method may train a deep neural network to generate sidewalls of a 3D dental restoration model by mapping sidewall data of a technician-generated denture to a probability vector comprising the probability of each of the sidewalls matching the occlusal surface and margin line data from the preparation site.

[0122] Referring again to FIG. 12 , to generate a new 3D model of the new patient's dentures, at 1215 , a new patient's dentition scan data (e.g., a scanned dental impression, a physical model, or an intraoral scan) is received and ingested. In some embodiments, the new patient's dentition scan data may be pre-processed to convert the 3D image data into 2D image data, making it more amenable to ingestion by certain neural network algorithms. At 1220 , one or more dental features in the new patient's dentition scan data are identified using a previously trained deep neural network. For example, the identified features may include the preparation site, corresponding margin lines, adjacent teeth and corresponding features, and surrounding gingiva.

[0123] At 1225, using the trained deep neural network, a complete 3D dental restoration model can be generated based on the features identified at 1220. In some embodiments, the trained deep neural network can be tasked with generating a complete 3D dental restoration model by generating an occlusal portion of the denture for the prepared site; obtaining margin line data from the generated margin proposals as described previously or from a patient's dentition scan; optionally optimizing the margin lines; and generating sidewalls between the generated occlusal portion and the margin lines. Generating the occlusal portion can include generating an occlusal surface having one or more of the following: a mesiobuccal cusp, a buccal groove, a distobuccal cusp, a distal cusp, a distobuccal groove, a distal pit, a lingual groove, a mesiobuccal cusp, etc.

[0124] In some embodiments, the trained deep neural network can obtain margin line data from edge proposals generated as previously described or from a patient's dentition scan data. In some embodiments, the trained deep neural network can optionally modify the contours of the obtained margin line by comparing and mapping the obtained margin line with thousands of other similar margin lines (e.g., margin lines of the same tooth preparation site) with similar adjacent teeth, surrounding gums, etc.

[0125] To generate a complete 3D model, a trained deep neural network can generate sidewalls that fit between the generated occlusal surface and the margin line. This can be achieved by mapping thousands of sidewalls of technician-generated dentures to the generated occlusal portion and margin line. In some embodiments, the sidewall with the highest probability value (in the probability vector) can be selected as the base model, from which the final sidewall between the occlusal surface and the margin line will be generated.

[0126] FIG13 illustrates exemplary inputs and outputs of a trained deep neural network 1300 (e.g., a GAN) according to some embodiments of the present disclosure. As shown, input dataset 1305 may be a scan of a new patient's dentition with a preparation site 1310. Using one or more trained deep neural networks 1300, a dental restoration server can generate a (DNN-generated) 3D model of a dental restoration 1315. The DNN-generated denture 1315 includes an occlusal portion 1320, a margin portion 1325, and a sidewall portion 1330. In some embodiments, the deep neural network can generate the sidewalls for the denture 1315 by analyzing thousands of technician-generated dentures generated based on one or more library templates and mapping them to the preparation site 1310. Ultimately, the sidewall with the highest probability score can be selected as the model for generating sidewall 1330.

[0127] Figure 14(a) is a high-level block diagram illustrating the architecture of a generative adversarial network (GAN) that can be used to identify and model tooth anatomical features and restorations according to some embodiments of the present disclosure. At a high level, a GAN uses two independent neural networks to generate an output model that is substantially indistinguishable from the true model. In other words, the GAN employs a minimax optimization problem to achieve convergence between the two competing neural networks. The GAN includes a generator neural network 1410 and a discriminator neural network 1420. In some embodiments, both neural network 1410 and discriminator neural network 1420 are deep neural networks structured to perform unstructured and unsupervised learning. In the GAN, both the generator network 1410 and the discriminator network (discriminator deep neural network) 1420 are trained simultaneously. The generator network 1410 is trained to generate samples 1415 from data input 1405. The discriminator network 1420 is trained to provide a probability that the sample 1415 belongs to the training data sample 1430 (which is from the real sample, real data 1425) rather than one of the data samples of the input 1405. The generator network 1410 is recursively trained to maximize the probability that the discriminator network 1420 cannot distinguish (at 1435) between the training data set and the output samples generated by the generator 1410.

[0128] At each iteration, the discriminator network 1420 may output a loss function 1440 that quantifies whether the generated sample 1415 is a real, natural image or an image generated by the generator 1410. The loss function 1440 may be used to provide the feedback needed by the generator 1410 to improve each subsequent sample generated in subsequent cycles. In some embodiments, in response to the loss function, the generator 1410 may change one or more of the weights and / or bias variables and generate another output.

[0129] In some embodiments, a computer-implemented method or another method may simultaneously train two adversarial networks: a generator 1410 and a discriminator 1420. The computer-implemented method or another method may use one or more of a patient's dental scan datasets to train the generator 1410 to generate sample models of one or more dental features and / or restorations. For example, the patient's dental scan data may be a 3D scan of the mandible including a prepared tooth / site and its adjacent teeth. Simultaneously, the computer-implemented method or another method may train the discriminator 1420 to distinguish between a generated 3D model of a crown of a prepared tooth (generated by the generator 1410) and samples of crowns from a real dataset (a collection of multiple scan datasets containing images of dental crowns). In some embodiments, the GAN network is designed for unsupervised learning, and therefore the input 1405 and the real data 1425 (e.g., the dental training dataset) may be unlabeled.

[0130] Figure 14(b) is a flow chart of method 1450 for generating a 3D model of a dental restoration, according to some embodiments of the present disclosure. Method 1450 can be performed by a computer-implemented method or another method on a dental restoration server or one or more other computers in a cloud computing environment. The instructions, methods, and algorithms of method 1450 can be stored in the memory of a computing device and, when executed by a processor, enable the computing device to perform training of one or more deep neural networks for generating a 3D dental prosthesis. Some or all of the methods and procedures described in method 1450 can be performed by one or more other entities or methods within the dental restoration server or another remote computing device. Furthermore, one or more blocks (methods) of method 1450 can be performed in parallel, in a different order, or even omitted.

[0131] At 1455, a computer-implemented method or another method may train a generative deep neural network (e.g., GAN generator 1410) using the unlabeled dentition dataset to generate a 3D model of a denture, such as a crown. In some embodiments, a labeled and classified dentition dataset may be used, but not necessarily. The generative deep neural network may be implemented by the computer-implemented method or another method, or in a separate and independent neural network, within or outside the tooth restoration server.

[0132] At 1460, and substantially simultaneously, the computer-implemented method or another method may also train a discriminative deep neural network (e.g., discriminator 1420) to identify a tooth restoration generated by the generative deep neural network as a model compared to a digital model of a true tooth restoration. In the identification method, the discriminative deep neural network may generate a loss function based on a comparison of the true tooth restoration with the generated model of the tooth restoration. The loss function provides a feedback mechanism for the generative deep neural network. Using information from the output of the loss function, the generative deep neural network can generate a better model that is more capable of fooling the discriminative neural network into believing the generated model is the true model.

[0133] The generative deep neural network and the discriminative deep neural network can be considered as mutually exclusive. In other words, the goal of the generative deep neural network is to generate a model (the generated model) that the discriminative deep neural network cannot distinguish as belonging to the distribution of real samples or fake samples. At 1465, if the generated model has a probability value indicating that it is likely fake, the training of the two deep neural networks repeats at 1455 and 1460 and continues again. This method continues and iterates until the discriminative deep neural network cannot distinguish the generated model from the real model. In other words, the probability that the generated model is fake is extremely low, and the probability that the generated model belongs to the distribution of real samples is extremely high.

[0134] Once the deep neural network is trained, method 1450 is ready to generate a model of the dental restoration based on the patient dentition data set received at 1470. At 1475, a model of the patient dentition data set is generated using the received patient dentition data set.

[0135] In some embodiments, a computer-implemented method may evaluate a generated 3D digital dental restoration model and determine whether to submit the generated 3D digital dental restoration model to quality control (QC). In some embodiments, the computer-implemented method may use the same DNN (e.g., GAN) used for generation to determine whether the generated dental restoration model should be submitted to quality control ("QC") based on a confidence score, which may be output by the same DNN (e.g., GAN) used to generate the 3D digital dental restoration model. Alternatively, in some embodiments, a separate neural network, such as a DNN (e.g., GAN), may be used to determine the confidence score, which is different from the neural network used to generate the 3D digital dental restoration model. In some embodiments, the confidence score may be based on the probability that the 3D digital dental restoration model belongs to a distribution of true samples. In some embodiments, the confidence score may be based on a loss function. In some embodiments, the confidence score may be determined by a discriminator neural network that compares the generated dental restoration model to the distribution of real samples and outputs a confidence score indicating the degree of match. A higher confidence score may indicate a closer match between the generated dental restoration model and the distribution of real samples. A lower confidence score may indicate a lower match between the generated dental restoration model and the distribution of real samples. In some embodiments, the computer-implemented method may determine whether the generated dental restoration model should be forwarded to a quality control (QC) based on the confidence score. For example, in some embodiments, if the confidence score output by the GAN equals or exceeds a confidence score threshold, the computer-implemented method may provide the generated 3D digital dental restoration model directly to an end user and / or may directly deliver the generated 3D digital dental restoration model to, for example, a milling machine or 3D printing device or facility, which may, for example, create a 3D physical dental restoration from the 3D digital dental restoration. On the other hand, if the confidence score output by the GAN is below a confidence score threshold, the computer-implemented method may determine that QC is required for the generated 3D digital dental restoration model. For example, in some embodiments, the computer-implemented method may use any other basis to determine whether the 3D digital dental restoration model requires QC based on the confidence score of a deep neural network, such as a GAN. In some embodiments, after determining that the generated 3D digital dental restoration model requires QC, the computer-implemented method may deliver the generated 3D digital dental restoration model to a QC. In some embodiments, delivering may include providing the generated 3D digital dental restoration to a QC user. In some embodiments, delivering may include placing the generated 3D digital dental restoration in a queue or other type of storage structure.

[0136] In some embodiments, the computer-implemented method may deliver the generated 3D digital denture model to QC without evaluation. Thus, in some embodiments, the computer-implemented method may automatically deliver the 3D digital dental model including the virtual 3D denture model to a quality control ("QC") user.

[0137] Some embodiments may include performing quality control ("QC") on the generated 3D digital denture model. In some embodiments, the computer-implemented method may assign the 3D digital denture model to a QC user / technician. In some embodiments, the computer-implemented method may assign the 3D digital denture model to a user queue or other type of storage structure associated with a QC user. In some embodiments, the computer-implemented method may notify one or more QC users that one or more 3D digital denture models are ready for QC. In some embodiments, the QC user or the computer-implemented method may retrieve the next generated 3D digital dental restoration model and perform QC on the model. In some embodiments, the computer-implemented method may display at least a portion of the 3D digital model of the patient's dentition including the generated 3D digital denture model in a graphical user interface ("GUI") on a display such as a computer screen. The GUI may include interactive controls that allow a dental technician, dentist, or other user to manipulate one or more features of the generated 3D digital denture model.

[0138] FIG15( a ) illustrates an example of a GUI 1500 that can be used as part of a QC method in some embodiments. In this example, a generated 3D digital restoration model is loaded for QC. The QC method may display at least a portion of a 3D digital tooth model 1502 of the patient's dentition and a generated 3D digital denture model 1504 in the GUI 1500, along with information regarding the 3D digital denture model 1504 and its position and orientation relative to the surrounding dentition. For example, in some embodiments, the computer-implemented method may display and indicate a digital tooth number 1506 along with its adjacent digital teeth in a representation of the maxilla or mandible. In some embodiments, the QC method may display information regarding the digital tooth 1504 and its adjacent digital teeth in a panel, such as panel 1508, or other suitable GUI display element known in the art. For example, panel 1508 may provide information regarding the occlusal, mesial, and distal relationships of the generated 3D digital denture model relative to the surrounding dentition. In some embodiments, the GUI 1500 may provide a control feature or features to adjust the 3D digital denture model 1504. For example, in some embodiments, the QC method may provide a control to adjust the contact points between the automatically generated 3D digital denture and adjacent digital teeth in the 3D digital tooth model.

[0139] For example, in some embodiments, the QC method may provide GUI controls for adjusting contact points, such as mesial, distal, and / or occlusal contact points. The mesial and distal contact points may be between the generated 3D digital denture model and adjacent digital teeth in the 3D digital dental model. The occlusal contact point may be between the occlusal surface of the generated 3D digital denture model and the corresponding digital teeth on the corresponding digital jaw. Figure 15(b) illustrates an example of adjusting a mesial contact point in some embodiments. As shown, the GUI 1520 may display at least a portion of the 3D digital dental model along with the mesial side of the generated 3D digital denture model 1522. The GUI 1520 may also display a mesial contact surface area 1524 between the 3D digital denture model 1522 and its mesial adjacent teeth (not shown) in the 3D digital dental model. For example, the GUI 1520 may provide an adjustment tool that allows a user to use an input device to select a mesial adjustment surface area 1526 that the computer-implemented method can reduce. In some embodiments, the adjustment may reduce the size of an adjustment surface area such as adjustment surface area 1526. In some embodiments, the adjustment may increase the size of adjustment surface area 1526. FIG15(c) illustrates an example of adjusting a distal contact point in some embodiments. As shown, the GUI 1530 may display at least a portion of a 3D digital dental model along with the distal side of a generated 3D digital denture model 1532. The GUI 1530 may display a distal contact surface area 1534 between the 3D digital denture model 1532 and its distally adjacent teeth (not shown) in the 3D digital dental model. For example, the GUI 1530 may provide an adjustment tool that allows a user to use an input device to select a distal adjustment surface area 1536 that the computer-implemented method can adjust. In some embodiments, the adjustment may reduce the size of an adjustment surface area such as adjustment surface area 1536. In some embodiments, the adjustment may increase the size of the adjustment surface area 1536.

[0140] In some embodiments, the QC method may provide a GUI control that allows the user to adjust occlusal contact points. For example, FIG15( d ) illustrates a generated 3D digital denture model 1540 viewed from the occlusal direction, showing occlusal surfaces such as occlusal surface 1542 . In some embodiments, the computer-implemented method may provide one or more digital tools for adjusting occlusal contact areas, such as occlusal contact area 1544 . In some embodiments, the adjustment may reduce the size of an adjusted surface area such as occlusal contact area 1544 . In some embodiments, the adjustment may increase the size of adjusted surface area 1544 .

[0141] In some embodiments, the QC method may provide a GUI control that allows the user to adjust the shape or contour of the automatically generated 3D digital denture, as illustrated in, for example, FIG15( e ) and FIG15( f ). In the example of FIG15( e ), the user may select a contour region, such as contour region 1556 , for adjustment. In the example of FIG15( f ), the user may select a contour region, such as contour region 1558 . In some embodiments, the contour control may include a visual indicator that defines the contour region. An example of such a visual indicator may include an "X" or a pound sign whose size can be adjusted by the user. An example of such a visual indicator is visual indicator 1559 . In some embodiments, when a pointer hovers over the generated 3D digital denture, the visual indicator may appear on at least a portion of the generated 3D digital denture model. In some embodiments, for example, the user may adjust the shape or contour of the contour region by pressing a button on an input device, such as a mouse, and dragging the contour region in the desired direction. In some embodiments, other GUI controls known in the art may be used to allow the user to adjust the shape or contour of the automatically generated 3D digital denture.

[0142] In some embodiments, the QC method may display at least a portion of a 3D digital dental model 1564 of the patient's dentition and automatically determine a margin line proposal, allowing the user to modify the determined margin line. FIG15( g) illustrates an example of the QC method displaying the determined margin line proposal 1560 in a GUI 1562. In some embodiments, the 3D digital denture is hidden or not displayed so that the determined margin line is visible. In some embodiments, the GUI 1562 may provide virtual handles 1566 to allow the user to adjust the automatically determined margin line. This may advantageously allow, for example, correction of the automatically determined margin line as part of the QC method.

[0143] In some embodiments, once the QC user completes any changes to the generated 3D digital denture model as part of the QC method, the computer-implemented method may apply the changes to the 3D digital denture model to provide a modified 3D digital denture model. In some embodiments where the changes to the model are major or fundamental, the computer-implemented method may regenerate the 3D digital denture using a generative neural network. In some embodiments, major and / or fundamental changes may include, but are not limited to, changes to margin line proposals. For example, in some embodiments, major and / or fundamental changes may be based on user-configurable values ​​of geometrically measured changes in the model. In some embodiments where the modified margin line proposal triggers regeneration of the 3D digital denture model, the computer-implemented method may provide the modified margin line as input to the 3D digital denture model generative neural network for regeneration. In some embodiments, the computer-implemented method may perform one or more of the features described herein on the regenerated 3D digital denture model and provide the regenerated 3D digital denture model for QC processing. In some embodiments where the QC changes are not major or fundamental, the computer-implemented method may apply the changes to the generated 3D digital denture model without regeneration.

[0144] In some embodiments, a computer-implemented method may assess whether the QC modifications made by the user to the 3D digital denture model during the QC method are significant enough to be used as part of a refined training dataset for generating a neural network. For example, in some embodiments, the computer-implemented method may geometrically assess the changes made, and if they exceed a user-configurable modification threshold, the computer-implemented method may determine that the changes made are significant and store the modified 3D digital denture model (or a regenerated 3D digital denture model) locally in a file system, database, or other storage device, or in a cloud computing environment. In some embodiments, the refined training dataset may include one or more 3D digital denture models modified or regenerated by the QC user (also referred to as refined 3D digital denture models). For example, the one or more refined 3D digital denture models may represent ground truth. In some embodiments, the computer-implemented method may add one or more modified and / or regenerated 3D digital denture models from the QC method to the refined training dataset. In some embodiments, the computer-implemented method may optionally always add the regenerated 3D digital denture model to the improved training dataset.

[0145] In some embodiments, based on a user-configurable retraining threshold, the computer-implemented method may update the training dataset of the 3D digital denture model generation neural network based on the improved training dataset and retrain the 3D digital denture model generation neural network. For example, in some embodiments, the updated training dataset is the training dataset used to train the discriminator neural network in a neural network such as a GAN. In some embodiments, the retraining threshold may be a user-configurable amount of time. For example, in some embodiments, the computer-implemented method may periodically retrain the 3D digital denture model generation neural network. In some embodiments, the retraining threshold may be the user-configurable number of improved 3D digital denture models in the improved training dataset. For example, the computer-implemented method may check and / or detect whether the number of improved 3D digital denture models in the improved training dataset meets and / or exceeds the retraining threshold, and if so, the computer-implemented method may retrain the 3D digital denture model generation neural network. In some embodiments, the computer-implemented method may incorporate an improved training dataset as part of a separate retraining method for generating 3D digital denture models. In some embodiments, the computer-implemented method may perform retraining after a manual trigger initiated by a user. In some embodiments, the computer-implemented method may retrain the 3D digital denture model generation neural network by adding the improved training dataset to the current training dataset of the 3D digital denture model generation neural network and then performing the training described in the present disclosure.

[0146] FIG16 illustrates an example of a flow chart in some embodiments. A computer-implemented method may generate a 3D digital denture model at 1602 using the trained neural network discussed previously. In some embodiments, the computer-implemented method may automatically provide the generated 3D digital denture model and / or 3D digital tooth model to a QC at 1608. After performing QC as previously described in this disclosure, the computer-implemented method may determine at 1610 whether any changes / modifications made during QC exceed a user-configurable modification threshold. If the QC changes / modifications do not exceed the user-configurable modification threshold, the 3D digital denture model may be output at 1606 and optionally sent to a CAM at 1607. In some embodiments, the CAM 1607 is known in the art and may physically generate the 3D digital denture model. In some embodiments, the computer-implemented method may output the 3D digital denture model to, for example, a 3D printer, or to a milling device and / or facility that is part of a cloud computing network or external to but in communication with the cloud computing network. If the QC changes / modifications exceed a user-configurable modification threshold, the computer-implemented method may add the modified / regenerated 3D digital denture model to a revised training dataset at 1612. In some embodiments, the revised training dataset may also include previously trained datasets used to train the 3D digital denture generation neural network. In some embodiments, the computer-implemented method may determine whether a retraining threshold is met at 1614. If the retraining threshold is met at 1614, the computer-implemented method may retrain the 3D digital denture model generation neural network at 1616. In some embodiments, retraining may be initiated manually.

[0147] FIG17 illustrates an example of a computer-implemented method in some embodiments. The method may include: receiving a 3D digital dental model representing at least a portion of a patient's dentition at 1702; automatically determining a virtual tooth preparation site in the 3D digital dental model using a first neural network at 1704; automatically generating a 3D digital denture model in the 3D digital dental model using a second trained generative deep neural network at 1706; and automatically delivering the 3D digital dental model including the virtual 3D denture model to a quality control ("QC") user at 1708.

[0148] In some embodiments, the method may include other features. In some embodiments, the neural network trained for 3D denture generation may include a generative adversarial network (GAN). In some embodiments, the first trained neural network may include a neural network trained for preparation sites. In some embodiments, the second trained neural network may include a neural network for 3D digital denture generation. Some embodiments may include automatically determining margin line proposals using a neural network trained for internal representation and a neural network trained for shift values. Some embodiments may include automatically determining buccal direction using a neural network trained for buccal direction. Some embodiments may include automatically determining occlusal direction using a neural network trained for occlusal direction. In some embodiments, in response to a QC user making a substantial change to the 3D digital denture model, the computer-implemented method may automatically regenerate the 3D denture model. In some embodiments, in response to a QC user making a substantial change to the 3D digital denture model, the computer-implemented method may add the modified 3D digital denture model to the improved training dataset. Some embodiments may include retraining a neural network trained for 3D denture generation using a training dataset comprising one or more 3D digital denture models from a modified training dataset. In some embodiments, the neural network trained for 3D denture generation may comprise a discriminator neural network in a GAN, and the retraining may comprise retraining the discriminator neural network.

[0149] In some embodiments, providing the 3D digital denture model includes generating the 3D digital denture model rather than deforming the library teeth. In some embodiments, one or more features are performed after the scan is completed and the 3D digital denture model is generated.

[0150] In some embodiments, the system and / or non-transitory computer-readable medium may implement / include one or more of the method features.

[0151] Some embodiments may further include generating a physical 3D denture using computer-aided manufacturing (CAM) as known in the art. In some embodiments, this may include additive manufacturing methods such as 3D printing or milling methods as known in the art.

[0152] For example, one or more features may include an integrated design workflow from automated design to generate physical 3D dentures, including quality control. For example, a 3D digital dental model with prepared teeth can be uploaded, and one or more features may automatically identify the preparation site, determine the occlusal direction, determine the buccal direction, detect margin lines, generate a 3D digital denture model design, and automatically assess the quality of the design. This can, for example, advantageously accelerate 3D digital denture design. For example, one or more features may also advantageously provide for empirically generated 3D digital denture designs. One or more features may also advantageously provide, for example, a consistent method for generating 3D digital denture designs from 3D digital dental models with prepared sites. One or more advantages may also include providing a reproducible method for generating 3D digital denture models starting from 3D digital dental models with prepared sites. For example, one or more advantages may include uniformity in the generation of 3D digital denture models starting from 3D digital dental models with prepared sites. For example, in addition to improving the current design, one or more advantages of one or more features may also provide for improvements to all future automated generation of 3D digital denture models based on the QC of the current design. For example, one or more advantages may include uniformly scaling improvements to future 3D digital denture models. One or more advantages may include allowing one or more neural networks to learn and improve based on feedback, such as from experts. For example, one or more advantages may include empirically determining the confidence level of generated 3D digital denture models and only sending those with defects to QC. This can advantageously improve the speed and efficiency of generating 3D digital denture models while maintaining design consistency and quality. For example, one or more advantages may include controlling the minimum confidence level required to avoid QC, thereby determining the minimum quality of generated 3D digital denture models. For example, one or more advantages may include reducing the number of future defects in generated 3D digital denture model designs by retraining the generative neural network to learn from errors and prevent or reduce the chance of their recurrence. For example, one or more advantages may include improved speed and efficiency in generating 3D digital denture models due to fewer defects. For example, one or more advantages may include minimizing the amount of time and errors involved in CAD of dental restorations by integrating features to generate 3D digital dentures, providing the ability to consistently deliver quality designs and allowing for the integration of high-level expertise into the software through, for example, QC feedback methods.

[0153] For example, in some embodiments, one or more features may be user-activated. For example, in some embodiments, some features, such as interacting with a GUI, may be activated by the user using an input device while viewing a digital model on a display. In some embodiments, the computer-implemented method may allow the input device to manipulate the digital model displayed on the display. For example, in some embodiments, the computer-implemented method may rotate, scale, move, and / or otherwise manipulate the digital model in any manner known in the art. In some embodiments, one or more features may be user-activated using an input device. In some embodiments, one or more automated methods may be initiated using techniques known in the art, such as user selection of a button. Other techniques known in the art may be used to select points or digital surfaces.

[0154] For example, in some embodiments, a computer-implemented method may display a digital model on a display and receive input from an input device, such as a mouse or touch screen, on the display. For example, the computer-implemented method may receive a 3D digital model of a patient's dentition. Upon receiving a start command or receiving the 3D digital model, the computer-implemented method may automatically generate a 3D digital denture using one or more of the features described herein. Upon receiving a manipulation command, the computer-implemented method may rotate, scale, translate, and / or otherwise manipulate the digital model in any manner known in the art.

[0155] One or more of the features disclosed herein may be automatically performed and / or achieved without manual or user intervention. One or more of the features disclosed herein may be performed by a computer-implemented method. The disclosed features, including but not limited to any method and system, may be implemented in a computing system. For example, the computing environment 14042 used to perform these functions may be any of a variety of computing devices (e.g., desktop computers, laptop computers, server computers, tablet computers, gaming systems, mobile devices, programmable automation controllers, video cards, etc.) that may be incorporated into a computing system comprising one or more computing devices. In some embodiments, the computing system may be a cloud-based computing system.

[0156] For the sake of clarity, only certain selected aspects of a software-based implementation are described. Other details well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented using software written in C++, Java, Perl, Python, JavaScript, Adobe Flash, or any other suitable programming language. Similarly, the disclosed technology is not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are well known and need not be elaborated in detail in this disclosure.

[0157] It should also be understood that any functionality described herein may be performed, at least in part, by one or more hardware logic components rather than software. By way of example and not limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0158] Furthermore, any of the software-based embodiments (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed via any suitable communication means, including, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communications means.

[0159] In view of the many possible embodiments to which the principles of the present disclosure may be applied, it should be recognized that the described embodiments are examples only and should not be taken as limiting the scope of the disclosure.

[0160] 102: Cloud computing environment or system 103: Automated Design Features / Automated Integrated Recovery Design Service Method and System / Automated Design 104: Tooth Restoration Cloud Server 105: Internet 106: Quality Control (QC) Features / Tooth Restoration QC Software / QC Users 107: Storage 108: User terminal device 109: Scanner 110: Third-Party Manufacturing Provider / Computer-Aided Manufacturing ("CAM") System / Method 302: Automated Design / Automated Integrated Resilience Design Service Methods and Systems / Design Features 304: Solid plaster 306: X-ray computed tomography scanning system ("CT scanner") 308: Standard intraoral scanning device 310: Quality Control ("QC") 312: Laboratory Technician 314: Dentist / Doctor 316:QC feedback loop 318:CAM Method 320: Chair side milling machine 322: Milling center 402,404,406,408,410,714,734,738,1205,1210,1215,1220,1225,1435,1455,1460,1465,1470,1475,1602,1606,1610,1612,1614,1616,1702,1704,1706,1708: Steps 502: First convolutional layer 503,708: Input image 504: Kernel 506: Convolution output / first convolution image 507,517,519: Feature channels 508: First kernel output 512: First pooling layer 514: First pooling filter 516: First pooled image 518: Second convolutional layer 520: Second convolution image 522: Second pooling layer 524: Second pooled image / output prediction 530:Fully connected layer 702: Hierarchical point set feature learner 704: HNN segmentation 706:HNN classification 710: First sampling and grouping operation 712: First sampled and grouped image 716: First Abstract Image 718: Sampling and Grouping 720: Second sampled and grouped image 722:PointNet Neural Network 724: Second Abstract Image 730: First interpolation 732: First interpolated image 736: First Segment Image 740: Second interpolated image 742: Unit PointNet 744: Segmented Image 760:PointNet Network 762: Output 764: Connection layer 766:Category score 800: Deep Neural Network (DNN) 901: Precalculation 902, 1000, 1105: 3D digital model 903,1002: Basic edge line 904: Teeth Preparation 905: coordinates 906: Internal Representation Training of Neural Networks 907: Grouping and Sampling 908: Internal representation 910: Shift Value Training Neural Network 912: 3D shift value 1004: Basic edge line point 1006: Direction 1008: Marginal Line Proposal 1102,1108: Digitally prepared teeth 1104,1106: Proposed digital edge lines 1110,1502,1564:3D digital tooth model 1200: Denture production method 1300: Trained deep neural network 1305: Input Dataset 1310: Preparation 1315: Tooth restoration / DNN generated dentures / prostheses 1320: Occlusal part 1325: Edge line 1330: Side wall part / side wall 1405: Data Input 1410: Generator Neural Network / GAN Generator 1415: Generated sample 1420: Discriminator Neural Network / Discriminator 1425: Real information 1430: Training data sample 1440: Loss Function 1450: Method 1500,1520,1530,1562:GUI 1504:Digital Teeth / 3D Digital Denture Model 1506: Digital tooth number 1508: Panel 1522,1532,1540:3D digital denture models 1524: Proximal contact surface area 1526: Near-center adjustment surface area 1534:Telecentric contact surface area 1536: Telecentric adjustment surface area 1542: Occlusal surface 1544: Occlusal contact area / adjustment surface area 1556,1558: Contour area 1559:Visual Indicators 1560: Determined margin line proposal 1566: Virtual Control Point 1607:CAM 1608:QC 14030:Processing unit 14032: Tangible Memory 14034: Tangible storage / computer-readable storage media 14036: Input device 14037: Communication connector 14038: Output device 14042: Computing Environment N h,1,N h,l-1,N h,1: layer Ni: first layer / input layer No: output layer

Claims

1. A computer-implemented method for providing a 3D digital dental prosthesis, the method comprising: receiving a 3D digital dental model representing at least a portion of a patient's dentition; automatically determining a virtual tooth preparation site in the 3D digital dental model using a first neural network; automatically generating a 3D digital dental prosthesis model in the 3D digital dental model using a second trained generative deep neural network; and automatically delivering the 3D digital dental model including the virtual 3D dental prosthesis model to a quality control ("QC") user, wherein in response to significant adjustments made by the QC user to the 3D digital dental prosthesis model, the modified 3D digital dental prosthesis model is added to an improved training dataset for the second trained generative deep neural network.

2. The method of claim 1, wherein the first neural network includes a pre-site training neural network and the second trained neural network includes a 3D digital dental prosthesis generation neural network.

3. The method of claim 1, further comprising displaying the generated 3D digital dental prosthesis model in a QC procedure to provide graphical user interface ("GUI") controls, thereby adjusting the 3D digital dental prosthesis model as part of a QC procedure.

4. The method of request item 3, wherein the QC procedure allows the QC user to adjust one or more contact points.

5. The method of claim 3, wherein, as part of the QC procedure, the QC procedure allows the QC user to adjust the shape or contour of the 3D digital dental prosthesis.

6. The method of claim 3, wherein the QC program displays at least a portion of the 3D digital tooth model and an automatically determined edge line proposal, and as part of the QC program, one or more control features allow the QC user to modify the determined edge line.

7. The method of request 1, wherein in response to the QC user making basic adjustments to the 3D digital dental prosthesis model, the 3D dental prosthesis model is automatically regenerated.

8. The method of claim 1, further comprising retraining the neural network for training 3D dental prostheses using a training dataset containing one or more 3D digital dental prosthesis models from the modified training dataset.

9. A non-transitory computer-readable medium storing executable computer program instructions for providing a 3D digital dental prosthesis, the computer program instructions including instructions for performing the following operations: receiving a 3D digital dental model representing at least a portion of a patient's dentition; automatically determining a virtual tooth preparation site in the 3D digital dental model using a first trained neural network; automatically generating a 3D digital dental prosthesis model in the 3D digital dental model using a second trained generative deep neural network; and automatically delivering the 3D digital dental model containing the virtual 3D dental prosthesis model to a quality control ("QC") user.

10. The media as claimed in claim 9, wherein the first neural network includes a pre-site training neural network and the second trained neural network includes a 3D digital dental prosthesis generation neural network.

11. The media as claimed in claim 9, further comprising displaying the generated 3D digital dental prosthesis model in a QC procedure to provide graphical user interface ("GUI") controls, thereby adjusting the 3D digital dental prosthesis model as part of a QC procedure.

12. The media as requested in item 11, wherein the QC procedure allows the QC user to adjust one or more contact points.

13. The media as requested in item 11, wherein, as part of the QC procedure, the QC procedure allows the QC user to adjust the shape or contour of the 3D digital dental prosthesis.

14. The media as requested in claim 11, wherein the QC program displays at least a portion of the 3D digital tooth model and an automatically determined edge line proposal, and as part of the QC program, one or more control features allow the QC user to modify the determined edge line.

15. As requested in item 9, the 3D digital dental prosthesis model is automatically regenerated in response to basic adjustments made by the QC user.

16. As requested in Item 10, in response to the QC user making significant adjustments to the 3D digital dental prosthesis model, the modified 3D digital dental prosthesis model is added to an improved training dataset for one of the second trained generative deep neural networks.

17. A system for providing a digital dental prosthesis, the system comprising: a processor; and a non-transitory computer-readable storage medium including instructions executable by the processor to perform steps including: receiving a 3D digital dental model representing at least a portion of a patient's dentition; automatically determining a virtual tooth preparation site in the 3D digital dental model using a first trained neural network; automatically generating a 3D digital dental prosthesis model in the 3D digital dental model using a second trained generative deep neural network; and automatically delivering the 3D digital dental model including the virtual 3D dental prosthesis model to a quality control ("QC") user.

18. The system of claim 17, wherein the first neural network includes a pre-site training neural network and the second trained neural network includes a 3D digital dental prosthesis generation neural network.

19. The system of request item 17, wherein in response to the QC user making basic adjustments to the 3D digital dental prosthesis model, the 3D dental prosthesis model is automatically regenerated.

20. The system of claim 18, wherein in response to a significant adjustment made by the QC user to the 3D digital dental prosthesis model, the modified 3D digital dental prosthesis model is added to an improved training dataset for one of the second trained generative deep neural networks.

21. The system of claim 17, wherein the first neural network includes a pre-site training neural network and the second trained neural network includes a 3D digital dental prosthesis generation neural network.

22. The system of claim 17 further includes displaying the generated 3D digital dental prosthesis model in a QC procedure to provide graphical user interface ("GUI") controls, thereby adjusting the 3D digital dental prosthesis model as part of a QC procedure.

23. The system of request item 22, wherein the QC procedure allows the QC user to adjust one or more contact points.

24. The system of claim 22, wherein, as part of the QC procedure, the QC procedure allows the QC user to adjust the shape or contour of the 3D digital dental prosthesis.

25. The system of claim 22, wherein the QC program displays at least a portion of the 3D digital tooth model and an automatically determined edge line proposal, and as part of the QC program, one or more control features allow the QC user to modify the determined edge line.

26. The system of claim 18 further includes retraining a neural network for training 3D dental prostheses using a training dataset containing one or more 3D digital dental prosthesis models from an improved training dataset.

Images