Dental panoramic view

Abstract

Provided herein are apparatuses and methods for generating a panoramic rendering of a subject's teeth. Methods and processes for imaging teeth of a subject using a dental scan are provided. Methods and processes for automatic 3D rendering of teeth of a subject using scanned images are also provided. Methods and apparatus for generating simulated panoramic views of a subject's dentition from various perspectives are also provided.

Description

[0001] This application is a divisional application of the invention patent application with Chinese application number 2020800637263 (corresponding to PCT international application number PCT / US2020 / 050256), application date September 10, 2020, and invention name “Dental Panoramic View”. Priority claim

[0002] This patent application claims priority to U.S. Provisional Patent Application No. 62 / 898,481, filed on September 10, 2019, entitled “3D SCREEN VIEW FROM MULTIPLE PANORAMIC VIEWS,” and U.S. Provisional Patent Application No. 62 / 991,532, filed on March 18, 2020, entitled “3D SCREEN VIEW FROM MULTIPLE PANORAMIC VIEWS,” each of which is incorporated herein by reference in its entirety. References

[0003] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. Background Art

[0004] Many dental and orthodontic procedures can benefit from an accurate three-dimensional (3D) description of the patient's tooth shape and intraoral cavity. Surface representations of 3D tooth surfaces have proven to be very useful in the design and manufacture of dental restorations (e.g., crowns or bridges) and in treatment planning.

[0005] Historically, ionizing radiation (e.g., X-rays) have been used for imaging of teeth. For example, X-ray bitewing radiograms are often used to provide non-quantitative images of teeth. However, in addition to the risks of ionizing radiation, such images are often limited in their ability to show features and may involve lengthy and expensive procedures. Other techniques, such as cone beam computed tomography (CBCT), can provide tomographic images, but still require ionizing radiation.

[0006] Specialized 3D scanning tools have also been used for dental imaging. The scan from the 3D scanning tool provides topographical data of the patient's tooth shape, which can be used to generate a 3D dental mesh model of the patient's teeth. The 3D dental mesh model may include a polyhedral object that depicts the teeth and / or other elements of the dental arch in a format that can be rendered on a display. However, generating a 3D dental mesh model may be very time-consuming and processor-intensive, and may result in low-resolution, low-detail 3D models that are unable to accurately and realistically visualize the surfaces of the patient's teeth.

[0007] Therefore, it would be beneficial to provide methods and apparatus (including devices and systems, such as intraoral scanning systems) that can be used to accurately and efficiently provide high-resolution, photo-realistic models of a patient's teeth. Improved methods and systems are needed for scanning a patient's oral cavity, and / or visualizing a patient's teeth. Summary of the invention

[0008] In general, methods and devices (e.g., apparatus and systems) for scanning and visualizing a patient's dental arch and teeth are described herein. These methods and devices can generate photorealistic color or grayscale renderings of a subject's teeth. Any of these devices can include a camera or intraoral scanner for scanning within or around the subject's oral cavity. The scanning device can also include one or more sensors for detecting the precise position of the scanning device during scanning. The generated model can be a panoramic image. Panoramic images may be referred to as 3D renderings herein, or can be used to generate 3D renderings. These panoramic views can be generated to replace or supplement three-dimensional volumetric models of teeth, which can include the internal structure of the teeth.

[0009] Using the panoramic images described herein to display a view of all or part of a patient's dentition (e.g., upper and / or lower dental arches) (where an input scan image, such as an intraoral scan image) can provide a number of advantages over other described methods in which the scan image is shown piecemeal or as a synthesized three-dimensional (e.g., digital) model. The piecemeal, or even collaged, images are not satisfactorily smooth and may show abrupt and jarring transitions and orientation changes, among other things, introduced artifacts. The synthesized (e.g., digital 3D) models may also be computationally and time consuming, and may have particular difficulty accurately representing color, transparency, light effects, and internal structure. Such techniques also require segmentation (and may introduce artifacts due to segmentation).

[0010] The methods and devices described herein can avoid these difficulties and can provide fast and realistic images that can be viewed at any scale and position. The panoramic images described herein can be shown in real time and can provide continuous, smooth transitions between different panoramic views, thereby allowing continuous, real-time viewing, rotation, translation, zooming, etc.

[0011] The methods described herein generally include methods for generating a 3D rendering of a subject's teeth using one or more panoramic images of teeth including surface features or internal features. Specifically, multiple images of the subject's teeth can be taken from multiple positions. The position data of the camera can be recorded during image acquisition, and the images can be blended together using the position data.

[0012] The methods and systems described herein can be used to generate 2D images and / or 3D renderings showing internal features of a subject's teeth, surface features of a subject's teeth, or both. Different internal features and surface features can be visualized using data collected using one or more scanning modes of an intraoral scanner. For example, 2D and 3D renderings can be generated using data that is collected using infrared (IR) light, visible light, or a combination thereof. In some cases, the data obtained from the intraoral scanner is combined with other image data.

[0013] The methods and systems described herein can be used to generate simulated views (e.g., bitewing views) based on selected images taken from one or more predetermined camera angles (positions and / or orientations) during one or more scanning operations performed on the teeth of a subject. Such methods can include determining a central jaw line to be determined, and identifying camera angles for those images in the scan data that correspond to the selected viewing angles used to generate the views.

[0014] The methods and systems described herein may enable a user to select a particular perspective to view a subject's teeth. For example, a user may select to view the dentition from above or below, to visualize an occlusal view of the dental arch from a lingual perspective and / or from a buccal perspective. In some cases, a user may rotate the image to update the various views.

[0015] For example, methods for displaying a panoramic view of a dental arch are described herein. These methods may include: receiving a plurality of two-dimensional (2D) infrared images of a dental arch, each two-dimensional (2D) infrared image being taken at an associated camera angle; identifying a viewing angle for viewing a panoramic view of the dental arch; identifying a central jaw line of the plurality of 2D images, and identifying a plurality of points along the central jaw line; generating a panoramic view from the central jaw line and the plurality of points at the viewing angle, and displaying the panoramic view. Generating the panoramic view from the central jaw line and the plurality of points at the viewing angle may include: selecting, for each of the plurality of points, an image based on one or more of the plurality of 2D images, the image including the point and having a camera angle corresponding to the viewing angle; and combining the selected images.

[0016] Identifying a central jaw line of the plurality of 2D images may include arranging the plurality of 2D images along the central jaw line based on one or more of: content of the 2D images, and position information collected for the 2D images when each 2D image was captured.

[0017] The combination may include: combining along a line corresponding to the central jaw line. In general, the central jaw line can be any line extending through the multiple 2D images. The central jaw line can approximately correspond to the path taken by the intraoral scanner to capture the image relative to the dental arch. The central jaw line (which for simplicity may be referred to as a trajectory line) may be curved, straight, or any other shape. The central jaw line may be wrapped or unwrapped. The central jaw line does not have to be absolutely centered (e.g., centered on the teeth), but may be approximately centered.

[0018] These points may be pixels (or groups of pixels) on the image. In some variations, these points may be virtual pixels on a virtual screen, for example, by center lines identified from the scanned dental arch.

[0019] In any of these devices and methods, combining the selected images, such as when arranged in a panoramic view, may include blending the selected images to match gradients at boundaries of adjacent selected images. Generating the panoramic view may include displaying teeth of a dental arch in a line according to a linear center jaw line.

[0020] In any of these methods and devices, a user can select and / or change the viewing angle of the panoramic view, including dynamically selecting and / or changing the viewing angle of the panoramic view. For example, the viewing angle can be changed to a second (or other) viewing angle based on user input, and any of these methods and devices can include generating a second panoramic view using a second viewing angle. The original (e.g., first) panoramic view can be selected or set by a user interface. The user interface can allow the user to rotate or otherwise move (translate, including rotation) a panoramic view or another representation of a dental arch. Thus, the viewing angle can be selected by the user via a user interface that allows the user to move the panoramic view. In some variations, the user interface can display a panoramic view and can include one or more tools to allow the user to interact and (e.g., in real time) manipulate one or more views. In some variations, the viewing angle (e.g., the initial viewing angle) can be perpendicular to a virtual screen passing through the center jaw line.

[0021] Any of these methods and apparatus may include rendering regions of the panoramic view having different densities or compositions of different shades or colors.

[0022] As will be described in greater detail, any of these methods and apparatus can be used to help plan / generate, modify, or track a treatment plan, such as an orthodontic treatment plan. For example, these methods can allow a comparison between a patient's actual dental arch (e.g., showing actual tooth positions) and a predicted (e.g., digitally modeled) dental arch (showing predicted tooth positions). A panoramic view of the actual dental arch and a panoramic view of the virtual (digitally modeled) dental arch can be compared. In some variations, these methods and apparatus can be used to generate a treatment plan based on the panoramic views, and / or can be used to implement a treatment plan on a patient.

[0023] Any of these methods and apparatus may include forming one or more dental appliances according to a treatment plan based on the panoramic view.

[0024] Furthermore, in some embodiments, the method may include the steps of scanning the patient's dental arch to collect multiple two-dimensional (2D) images (e.g., infrared images, such as near-infrared images) of the dental arch, which may be received from a separate (remote in time or location) intraoral scanner.

[0025] The step of selecting an image for each of the plurality of points (the image being based on one or more of the plurality of 2D images) comprises: selecting an image from the one or more of the plurality of 2D images, the image having a camera angle closest to the viewing angle. Alternatively or additionally, in some variants, selecting an image for each of the plurality of points (the image being based on one or more of the plurality of 2D images) may comprise: extrapolating an image from the one or more 2D images.

[0026] Also described herein is a device including a system configured to perform any of these methods. For example, the system described herein includes: one or more processors; and a memory connected to the one or more processors, the memory including a non-transitory computing device readable medium having instructions stored thereon, the instructions being executable by the one or more processors to perform a method, the method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch, each of the plurality of two-dimensional (2D) images being taken at an associated camera angle; identifying a viewing angle for viewing a panoramic view of the dental arch; identifying a central jaw line of the plurality of 2D images, and identifying a plurality of points along the central jaw line; generating a panoramic view at the viewing angle by (selecting, for each of the plurality of points, an image based on one or more of the plurality of 2D images, the image including the point and having a camera angle corresponding to the viewing angle; and combining the selected images); and, displaying the panoramic view. The instructions may also include: identifying the viewing angle by receiving the viewing angle from a user interface. The user interface may be configured to allow the user to dynamically change the viewing angle and display a corresponding panoramic view.

[0027] Also described herein is a method for displaying a dental arch, the method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch, each of the plurality of two-dimensional (2D) images being captured at an associated camera angle, wherein the plurality of 2D images include internal features of teeth of the dental arch; aggregating the plurality of 2D images to generate a panoramic view of the dental arch along a central jaw line; passing a virtual plane through the panoramic view at a specified height and angle relative to the central jaw line; and generating a 2D slice view based on the specified height and angle, the slice view including corresponding internal features of the teeth. Any of these methods may also include passing a second virtual plane through the panoramic view at a second specified height and angle relative to the central jaw line, and generating a second 2D slice view.

[0028] The 2D slice views may include different shading or colors for areas of the dental arch having different densities or compositions.

[0029] Any of these methods may include passing multiple virtual planes through the panoramic view at different specified heights and the same angle relative to a reference plane, and generating multiple 2D slice views based on the multiple virtual planes. These methods may also include displaying the multiple 2D slice views in an animation that shows the progression through the dental arch. The multiple 2D images may include images collected using a near infrared light source. The multiple 2D images may also include images collected using a visible light source.

[0030] Any of these methods may also include generating a treatment plan based at least in part on the 2D slice views, and in some variations implementing the treatment plan on the patient. Any of these methods may also or alternatively include: generating a treatment plan based at least in part on the 2D slice views; and manufacturing one or more orthodontic devices based on the treatment plan.

[0031] The present invention also describes a non-transitory computing device readable medium having instructions stored thereon, the instructions being executable by a processor to cause the computing device to perform a method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch, each of the plurality of two-dimensional (2D) images being captured at an associated camera angle, wherein the plurality of 2D images include internal features of the teeth of the dental arch; aggregating the plurality of 2D images to generate a panoramic view of the dental arch along a central jaw line; passing a virtual plane through the panoramic view at a position relative to the central jaw line; and generating a 2D slice view based on a specified position, the 2D slice view including corresponding internal features of the teeth.

[0032] A method of displaying a dental arch may include receiving a plurality of two-dimensional (2D) images of the dental arch, each of the plurality of two-dimensional (2D) images being captured at an associated camera angle, wherein the plurality of 2D images include internal features of teeth of the dental arch; aggregating the plurality of 2D images to generate a panoramic view of the dental arch at a viewing angle from a central jaw line passing through the dental arch; identifying a region of interest within the panoramic view, the region of interest corresponding to a volume at a specified location of the panoramic view; and rendering the panoramic view on a display such that at least a portion of the region of interest is partially transparent to show corresponding internal features within the region of interest. The plurality of 2D images may include images collected using a near infrared light source. The plurality of 2D images may also include images collected using a visible light source. In some variations, identifying the region of interest includes automatically identifying the region of interest based on density or composition. Identifying the region of interest may include receiving input from a user, the input identifying a specified location. Rendering the panoramic view may include rendering different internal features within the region of interest in different shades or colors.

[0033] Any of these methods may also or alternatively include generating a treatment plan based at least in part on the rendered panoramic view; and implementing the treatment plan on the patient. In some variations, these methods may include generating a treatment plan based at least in part on the rendered panoramic view; and manufacturing one or more orthodontic devices based on the treatment plan.

[0034] Also described herein is a non-transitory computing device readable medium having instructions stored thereon, the instructions being executable by a processor to cause the computing device to perform a method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch, each of the plurality of two-dimensional (2D) images being captured at an associated camera angle, wherein the plurality of 2D images include internal features of the teeth of the dental arch; aggregating the plurality of 2D images to generate a panoramic view of the dental arch at a viewing angle from a central jaw line passing through the dental arch; identifying a region of interest within the panoramic view, the region of interest corresponding to a volume at a specified location of the panoramic view; and rendering the panoramic view on a display such that at least a portion of the region of interest is partially transparent to show corresponding internal features within the region of interest.

[0035] Methods and apparatus for forming a new composite image at a selected viewpoint are also described. As described herein, these methods can be used to generate panoramic views.

[0036] For example, methods for generating new views from multiple intraoral scan views are described herein. These methods may include: receiving multiple two-dimensional (2D) images of a dental arch, each of the multiple two-dimensional (2D) images being taken at an associated camera angle and position; identifying a new viewpoint having a camera angle and position relative to the dental arch; identifying three or more contributing images from the multiple 2D images, wherein the contributing images have a minimum angle relative to the camera angle of the new viewpoint, and wherein the camera position of the new viewpoint is bounded by a triangle formed by the camera positions of the three or more contributing images; converting the contributing images into multi-planar layers; blending adjacent multi-planar layers of the contributing images to form a new view; and displaying all or part of the new view. Any of these methods may also include identifying a new viewpoint by camera position and orientation.

[0037] In some variations, exactly three contributing images may be used. The plurality of two-dimensional (2D) images of the dental arch may be received from data captured by an intraoral scanner. The data may be accessed, read or otherwise received.

[0038] Identifying the new viewpoint may include identifying a point (and camera angle / orientation) captured from a user interface (e.g., via user input), and / or identifying the viewpoint and camera angle as part of a panoramic view generation method. The three or more contributing images may be identified after confirming that the intersection over union of each of the three or more contributing images is greater than 0.5. If the intersection over union is not greater than 0.5, a different contributing image may be used.

[0039] The contributing image may have a minimum angle relative to the camera angle of the new viewpoint, which may be x degrees or less (e.g., 15 degrees or less, 12 degrees or less, 10 degrees or less, 8 degrees or less, 6 degrees or less, 5 degrees or less, 4 degrees or less, 3 degrees or less, 2 degrees or less, 1 degree or less, etc.). In some variations, the camera angle of the contributing image is approximately the same as the camera angle of the new viewpoint.

[0040] The method of claim 47, wherein blending comprises applying a trained machine learning agent to blend adjacent multi-planar layers of the contributing images to form the new view. The machine learning agent may be trained on a plurality of intraoral scan views.

[0041] Also described herein is a non-transitory computing device readable medium having instructions stored thereon, the instructions being executable by a processor to cause the computing device to perform any of these methods, including a method for generating a new view from a plurality of intraoral scan views, such as: receiving a plurality of two-dimensional (2D) images of a dental arch, each of the plurality of two-dimensional (2D) images being taken at an associated camera angle and position; identifying a new viewpoint having a camera angle and position relative to the dental arch; identifying three or more contributing images from the plurality of 2D images, wherein the contributing images have a minimum angle relative to the camera angle of the new viewpoint, and wherein the camera position of the new viewpoint is bounded by a triangle formed by the camera positions of the three or more contributing images; converting the contributing images into multi-planar layers; blending adjacent multi-planar layers of the contributing images to form a new view; and displaying all or part of the new view. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The novel features of the present invention are particularly described in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by referring to the following detailed description and accompanying drawings that illustrate exemplary embodiments in which the principles of the present invention are used, as shown below:

[0043] Figure 1A is a diagram illustrating an example of a computing environment configured to generate a panoramic model of a subject's teeth.

[0044] Figure 1B is a diagram illustrating an example of one or more angle selection engines.

[0045] Figure 1C is a diagram illustrating an example of one or more image projection engines.

[0046] Figure 1D is a diagram illustrating an example of one or more image combination engines.

[0047] Figure 2 is a flow chart describing an example of a process for generating a panoramic model of a subject's teeth from multiple images of the subject's teeth.

[0048] FIG. 3A to FIG. 3B A scan is shown being performed on a subject's teeth, including acquiring images from multiple camera positions.

[0049] Figure 3C Shown is the generation of a point cloud of all camera positions and images from a scan of a subject's teeth.

[0050] FIG. 4A to FIG. 4BThe generation and triangulation of a sphere to identify key camera angles required to construct a panoramic model of a subject's teeth is shown.

[0051] Figure 5 A technique for generating a two-dimensional grid of points required to generate a panoramic model of a subject's teeth is shown.

[0052] Fig. 6A is an example of an initial panoramic model before warping and blending the images.

[0053] Figure 6B is an example of the final panoramic model after warping and blending the images.

[0054] Figure 7 is a simplified block diagram illustrating an example of a data processing system for designing and manufacturing orthodontic appliances.

[0055] Figure 8 An example user interface is shown showing a 3D virtual model of a portion of a subject's teeth.

[0056] Fig. 9 An example 3D model of a subject's dental arch with partially transparent areas is shown.

[0057] Fig.10 Shown is an example dental arch rendered in various slice views.

[0058] Fig.11 Another example of a dental arch rendered in a slice view along a horizontal cut is shown.

[0059] Fig.12 An example is shown of how scan data of a curved dental arch can be used to generate a panoramic view in which the teeth are aligned in a straight line.

[0060] FIG. 13A to FIG. 13C Panoramic views of the dental arch before and after the blending procedure and at different viewing angles are illustrated.

[0061] FIG. 14A to FIG. 14C Examples of panoramic views of another dental arch taken using different scanning modes and at different viewing angles are shown.

[0062] Fig.15 is a flow chart describing an example of a process for generating a panoramic view (eg, a bitewing panoramic view) of a subject's dentition.

[0063] Fig.16 is a flow chart describing an example of a process for selecting parameters and updating a panoramic view of a subject's dentition.

[0064] FIG. 17A to FIG. 17CAn example of a cylindrical projection of a dental arch that may be generated and / or used in any of the methods and apparatus described herein is shown. Fig.17A An example of a dental arch displayed in a 3D projection and surrounded by a cylindrical "screen" on the lingual, occlusal and buccal sides of the dental arch is shown. Fig. 17B An occlusal view of the same dental arch shown in Figure 17 is shown, with a section through the arch showing the surrounding "screen". Fig. 17C Shown by Fig. 17B A cross-sectional view of the section shown, thereby providing a cross-section of the tooth and surrounding screen.

[0065] Fig.18A and Fig.18B Another example of a cylindrical projection of a dental arch is shown. Fig.18A A normal occlusal view of the dental arch is shown; Fig.18B Shown in Fig.18A Example of a cylindrical projection of the same dental arch shown in .

[0066] Fig.19 An example of selecting a new view camera direction (shown in bold) in comparison with the camera directions of three surrounding contributing images is shown, which new view camera direction may be used to generate a new view.

[0067] Fig. 20 The IoU of the three contributing images compared to the new view and the base state is shown, showing an IoU greater than 0.5.

[0068] Fig.21 An example of a method of determining a new composite image from a plurality of scanned images is shown.

[0069] FIG. 22A to FIG. 22E An example of generating a new composite image from a data set comprising intra-oral scan data is shown. FIG. 22A to FIG. 22C Three contributing images are shown that are identified as having minimum camera angles (e.g., approximately the same camera orientation) relative to the camera angle of a new viewpoint, wherein the new camera position is located within a triangle formed by camera positions of images from a dataset, which triangle has a minimum volume compared to other triangles formed by camera positions of images also having approximately the same camera angles. Fig.22D shows the predicted new synthetic image, and Fig.22E The ground truth image of the new viewpoint is shown.

[0070] FIG. 23A to FIG. 23E Another example of generating a new composite image from a data set including intra-oral scan data is shown. FIG. 23A to FIG. 23CThree contributing images are shown that are identified as having minimum camera angles (e.g., approximately the same camera orientation) relative to the camera angle of a new viewpoint, wherein the new camera position is located within a triangle formed by camera positions of images from a dataset, which triangle has a minimum volume compared to other triangles formed by camera positions of images also having approximately the same camera angles. Fig.23D shows the predicted new synthetic image, and Fig.23E The ground truth image of the new viewpoint is shown. DETAILED DESCRIPTION

[0071] The present disclosure relates to systems, methods, computing device readable media, and devices for generating models of regions within a subject's oral cavity (e.g., one or more teeth, gums, jaws, etc.). The systems, methods, and computing devices herein address technical problems associated with the design and display of models of a patient's dental arch, including quickly and efficiently displaying photorealistic images or models of a subject's dental surfaces.

[0072] The present invention describes a scanning system or camera system for generating a photo-realistic image of an intraoral region of a subject, wherein the intraoral region of the subject includes external features or surface features of teeth, or internal features of teeth, and a method for using such a scanning system or camera system is described herein. An intraoral scanner or camera system may include a wand, which may be held by an operator (e.g., a dentist, a dental hygienist, a technician, etc.) and may be moved on one or more teeth of a subject to scan the surface structure of the subject's teeth. The wand may include one or more detectors (e.g., cameras such as CMOS, CCD, etc.), one or more light sources (visible light, infrared light), and one or more sensors (e.g., accelerometers, GPS, etc.) for measuring and recording the position and / or orientation of an intraoral scanner or camera system during image acquisition. Specifically, an intraoral scanner or camera system may be configured to measure and record the precise position and orientation of a camera when an image is captured, and may be further configured to associate the position and orientation of the camera with each image captured at that position and orientation. The scanner may include one or more controllers (buttons, switches, dials, touch screens, etc.) to assist in control (e.g., turning the scanner on / off, etc.); alternatively or additionally, one or more controllers may be present on other parts of the intraoral scanner (such as a foot pedal, keyboard, console, touch screen, etc.).

[0073] As described herein, dental models generated using an intraoral scanner and camera system can be used for planning and manufacturing dental appliances, including elastic polymer positioning appliances as described in detail in U.S. Pat. No. 5,975,893 and published PCT application WO98 / 58596, which are incorporated herein by reference for all purposes. Systems of dental appliances using the technology described in U.S. Pat. No. 5,975,893 can be purchased from Align Technology, Inc. of San Jose, California, under the trade name "Invisalign System". It can also be used to assist in dental diagnosis, such as diagnosing caries and / or diagnosing gingival pockets.

[0074] Throughout the text describing the embodiments, the use of the terms "orthodontic appliance," "appliance," or "dental appliance" is synonymous with the use of the terms "appliance" and "dental appliance" in dental applications. For clarity, the embodiments are described below in the context of the use and application of an appliance, more specifically a "dental appliance."

[0075] As described herein, an intraoral scanner or camera system can image a patient's dental arch and generate a virtual model of the dental arch. In some examples, the model can be a three-dimensional (3D) model of the dental arch. During an intraoral scanning procedure (also referred to as a scanning session), a user of the intraoral scanner or camera system (e.g., a dentist) can generate multiple different images (also referred to as scans or medical images) of dental sites, models of dental sites, or other subjects. These images can be discrete images (e.g., point-and-shoot images) or frames from a video (e.g., continuous scanning). The images can be taken in visible light or infrared (IR) light (e.g., pure infrared light or near-infrared light) or other wavelengths. The intraoral scanner can automatically generate a model of the patient's teeth from the image, which can be used for treatment planning.

[0076] Figure 1A 1 is a diagram illustrating an example of a computing environment 100A configured to digitally scan a dental arch of a subject. The environment 100A includes a computer-readable medium 152, a scanning system 154, a dentition display system 156, and a panoramic modeling system 158. One or more modules in the computing environment 100A may be coupled to each other or to modules not explicitly shown.

[0077] The computer-readable medium 152 and other computer-readable media discussed in this disclosure are intended to represent various potential applicable technologies. For example, the computer-readable medium 152 can be used to form a network or a portion of a network. When two components are co-located and cooperatively located on a device, the computer-readable medium 152 may include a bus or other data pipeline or plane. In the case where a first component is co-located and cooperatively located on a device and a second component is located on a different device, the computer-readable medium 152 may include a wireless or wired back-end network or LAN. If applicable, the computer-readable medium 152 may also include a WAN or other relevant part of the network.

[0078] The scanning system 154 may include a computer system configured to capture static images, videos, and / or other media of a patient's dental arch. The scanning system 154 may include a memory, one or more processors, and a sensor for detecting contours on a patient's dental arch. The scanning system 154 may also include a sensor configured to measure and / or record the position and orientation of the scanning system during image acquisition. The scanning system 154 may be implemented as a camera, an intraoral scanner, an x-ray device, an infrared device, etc. The scanning system 154 may include a system configured to provide a virtual representation of a patient's dental arch model. When viewed from an occlusal perspective, the "dental arch" used herein may include at least a portion of the patient's dentition formed by the patient's maxillary teeth or mandibular teeth. The dental arch may include one or more maxillary teeth or mandibular teeth of the patient, such as all teeth located in the maxilla or mandible or on the patient. The scanning system 154 may be used as part of an orthodontic treatment plan. In some implementations, the scanning system 154 is configured to capture the patient's dental arch at the beginning, intermediate, etc. of an orthodontic treatment plan.

[0079] The dentition display system 156 may include a computer system configured to display at least a portion of the patient's dentition. The dentition display system 154 may include a memory, one or more processors, and a display device for displaying the patient's dentition. The dentition display system 156 may be implemented as a part of a computer system, a display of a dedicated intraoral scanner, and the like. In some implementations, the dentition display system 156 facilitates displaying the patient's dentition by using scans taken at an earlier date and / or at a remote location. It should be noted that the dentition display system 156 may also facilitate displaying scans taken simultaneously and / or partially. As described herein, the dentition display system 156 may be configured to display the expected or actual results of an orthodontic treatment plan that is applied to a dental arch scanned by the scanning system 154. These results may include a virtual representation or model of a dental arch, a 2D image or rendering of a dental arch, a 2D or 3D panoramic image or model of a dental arch, and the like.

[0080] The panoramic modeling system 158 may include a computer system configured to process scans or images of a patient's dentition taken by the scanning system 154. The panoramic modeling system 158 may include one or more angle selection engines 160, one or more image projection engines 162, and one or more image combination engines 164. One or more modules in the panoramic modeling system may be coupled to each other or to modules not shown.

[0081] As used herein, any "engine" may include one or more processors or a portion thereof. A portion of one or more processors may include some parts of hardware that are less than all of the hardware that includes any given one or more processors, such as a subset of registers, a portion of a processor dedicated to one or more threads in a multithreaded processor, a time slice (during which the processor is dedicated in whole or in part to executing part of the engine's functions), etc. Therefore, the first engine and the second engine may have one or more dedicated processors, or the first engine and the second engine may share one or more processors with each other or with other engines. Depending on the specific implementation or other considerations, the engine may be centralized, or its functions may be distributed. The engine may include hardware, firmware, or software contained in a computer-readable medium for execution by a processor. The processor converts data into new data using the implemented data structures and methods, as described with reference to the accompanying drawings herein.

[0082] The engine described herein, or the engine by which the systems and devices described herein can be implemented, can be a cloud-based engine. As used herein, a cloud-based engine is an engine that can run applications and / or functions by using a cloud-based computing system. All or part of the applications and / or functions can be distributed across multiple computing devices and need not be limited to only one computing device. In some embodiments, a cloud-based engine can execute functions and / or modules that an end user accesses through a web browser or container application without having to install these functions and / or modules locally on the end user's computing device.

[0083] As used herein, "datastores" may include repositories having any suitable data organization, including tables, comma-separated value (CSV) files, traditional databases (e.g., SQL), or other applicable known or convenient organizational formats. For example, a data store may be implemented as software contained in a physical computer-readable medium on a specific-purpose machine, in firmware, in hardware, a combination thereof, or in an applicable known or convenient device or system. Components associated with a data store (such as a database interface) may be considered "part" of the data store, part of some other system component, or a combination thereof, although the physical location and other characteristics of the components associated with the data store are not important to understanding the techniques described herein.

[0084] Data storage may include data structures. As used herein, a data structure is associated with a specific way of storing and organizing data in a computer so that the data structure can be used effectively in a given context. Data structures are generally based on the ability of a computer to obtain and store data at any location in its memory, which is specified by an address, a bit string that can itself be stored in the memory and manipulated by a program. Therefore, some data structures are based on using arithmetic operations to calculate the address of a data item; while other data structures are based on storing the address of a data item in its own structure. Many data structures use these two principles, and sometimes these two principles are combined in a non-trivial way. The implementation of a data structure generally requires writing a set of programs to generate and manipulate instances of the structure. The data storage described herein may be a cloud-based data storage. A cloud-based data storage is a data storage compatible with a cloud-based computing system and engine.

[0085] The one or more angle selection engines 160 may implement one or more automatic agents configured to identify key camera angles for which a panoramic model is to be built. In some implementations, the one or more angle selection engines 160 are configured to generate a sphere (or at least a portion of a sphere) representing the panoramic model. The one or more angle selection engines 160 may be further configured to triangulate the sphere (or at least a portion of a sphere) into a plurality of triangles, with the vertices of each triangle representing the key camera angles required to generate the panoramic model. The one or more angle selection engines 160 may provide key camera angles and / or other data to other modules of the panoramic modeling system 158.

[0086] One or more image projection engines 162 can implement one or more automatic agents that are configured to project images from the scan of the subject's teeth to form an initial panoramic model for each key camera angle. One or more image projection engines 162 can receive images and camera position and / or orientation data from the scanning system 154. In some implementations, one or more image projection engines 162 are configured to form a two-dimensional point grid that includes all pixel positions required to generate a panoramic model for a given key camera angle. In one implementation, the two-dimensional grid can be formed by dividing the central jaw line into equidistant segments, forming lines at each segment, and identifying equidistant points on each line. These lines can be perpendicular to the central jaw line and each key camera angle. The point cloud of all camera positions and orientations recorded during the scan can be compared with the points on each line, and one or more image projection engines 162 can be configured to select the most appropriate physical camera position, for example, for each point of each line, the camera has the orientation closest to the key camera angle. The most appropriate image for each point in the two-dimensional grid can be approximated with an orthographic camera to provide an image for each point of each line, thereby generating an initial panoramic model for each key camera angle. Alternatively, other image selection criteria can be used. One or more image projection engines 162 can provide the two-dimensional point grid, projected images, initial panoramic model, and / or other data to other modules of the panoramic modeling system 158.

[0087] One or more image combination engines 164 can implement one or more automatic agents that are configured to register, deform, and / or blend the images of the initial panoramic model to generate a final panoramic model for each key camera angle. In some implementations, one or more image combination engines 164 are configured to register and / or deform the images in the initial panoramic model so that the gradients at the boundaries of adjacent images match. One or more image combination engines 164 can be further configured to blend the resulting images to produce a final panoramic model for each key camera angle. In addition, the image combination engine 164 can be configured to render the final panoramic model for the selected key camera angle to the user.

[0088] Figure 1B is a diagram illustrating an example of one or more angle selection engines 160a. One or more angle selection engines 160a may include a triangulation engine 168 and a key camera angle data store 170. One or more modules in one or more angle selection engines 160a may be coupled to each other or to modules not shown.

[0089] The triangulation engine 168 may implement one or more automated agents configured to identify key camera angles for which a panoramic model is generated. In some implementations, the triangulation engine 168 triangulates a sphere or a portion of a sphere into a plurality of triangles and identifies a vector starting at coordinate 0 of the sphere or a portion of a sphere and terminating at each vertex of the plurality of triangles as the key camera angle required for generating the panoramic model.

[0090] The key camera angle data store 170 may be configured to store data related to key camera angles identified by the triangulation engine.The key camera angle data may include a matrix of camera angles and / or positions required to generate a panoramic model of the subject's teeth.

[0091] Figure 1C is a diagram showing an example of one or more image projection engines 162a. The one or more image projection engines 162a may include a grid generation engine 172, an orthographic camera engine 174, and an initial panoramic model data storage 176. One or more modules in the one or more image projection engines 162a may be coupled to each other or to modules not shown.

[0092] The grid generation engine 172 can implement one or more automated agents that are configured to form a two-dimensional grid of points that represent the images and camera positions required to generate the panoramic model. In one implementation, the two-dimensional grid can be formed by dividing the subject's central jaw line into equidistant segments and forming lines at each segment. These lines can be perpendicular to the central jaw line and each key camera angle. In addition, the grid generation engine can be configured to index points along each line to form a two-dimensional grid of points.

[0093] The orthogonal camera engine 174 may implement one or more automated agents configured to approximate images at points in the two-dimensional grid of points using images from the most appropriate scan, e.g., the camera having an orientation closest to the key camera angle for the desired point location. The combination of all approximated images at each point in the two-dimensional grid of points will generate an initial panoramic model for each key camera angle of the subject's teeth.

[0094] The initial panoramic model data store 176 may be configured to store data associated with the two-dimensional point grid generated by the grid generation engine and to store data associated with the initial panoramic model generated by the orthographic camera engine 174 .

[0095] Figure 1D1 is a diagram showing an example of one or more image combination engines 164a. The one or more image combination engines 164a may include a deformation engine 178, a blending engine 180, a panoramic rendering engine 181, and a final panoramic model data storage 182. One or more modules in the one or more image combination engines 164a may be connected to each other or to modules not shown.

[0096] The deformation engine 178 can implement one or more automatic agents that are configured to register and / or deform the images in the initial panoramic model so that gradients at the boundaries of adjacent images match. For example, certain areas of the initial panoramic model may not be properly registered due to different camera angles or perspectives used when building the model. In one implementation, the deformation engine 178 is configured to perform a global optimization method to identify the appropriate image deformation required to match the boundaries of adjacent images. Once the deformation is identified, the deformation engine 178 can be configured to apply the deformation to the image of the initial panoramic model to deform the image.

[0097] The blending engine 180 may implement one or more automated agents configured to blend images from the warping engine to produce a final panoramic model for each key camera angle. In one implementation, the blending engine 180 may use Poisson blending for each key camera angle to use target gradients from non-blended images to produce a blended image with gradients that best match those target gradients. The final panoramic model may be rendered to the user as an average image that includes key camera angles and images corresponding to triangle vertices (actual or approximate) with weights from the barycentric coordinates of the triangles of the triangulation sphere.

[0098] The final panoramic model data store 182 may be configured to store data related to the final panoramic model.

[0099] The panoramic rendering engine 181 may implement one or more automated agents configured to render the final panoramic model for the selected key camera angles to the user.

[0100] Figure 2A flowchart 200 is shown that describes an imaging process for collecting, processing, and displaying a panoramic model of a subject's teeth. Referring to operation 202 of flowchart 200, the process includes performing a scan of the subject's oral cavity to collect images of the subject's teeth. As described above, the scan can be performed, for example, using an intraoral scanner or camera system. In some aspects, the intraoral scanner or camera system can record multiple discrete images or a series of continuous images (e.g., a video) during the scan. Additionally, during the scan, the precise position and orientation of the intraoral scanner or camera system (for each image) can be tracked and recorded. The position and orientation of the intraoral scanner or camera system can also be associated with each corresponding discrete image or video frame. In another embodiment, a digital model of the patient's dental arch can be accessed or received (e.g., if a scan was previously performed).

[0101] FIG. 3A to FIG. 3C 200, the scan includes the maxillary ( Figure 3A ) and mandibular ( Figure 3B ). As described above, the scanning can be performed, for example, using an intraoral scanner or camera system. In some aspects, during the scan, the intraoral scanner or camera system can record multiple discrete images or a series of continuous images (e.g., a video). Point 300 represents the position and / or orientation of the intraoral scanner or camera system for each image acquired during the scan (i.e., the position and / or orientation of the approximate center of the aperture of the intraoral scanner or camera system). Field-of-view 302 represents the field of view of each image captured by the intraoral scanner or camera system. The intraoral scanner or camera system scans along all surfaces of the maxillary and mandibular surfaces to obtain images of all relevant tooth surface structures.

[0102] refer to Figure 3C, a point cloud 304 of all camera positions and / or orientations can be recorded, including an image associated with each camera position and / or orientation in the point cloud. Additionally, a central jaw line 306 of the subject's mouth can be determined. In one example, the central jaw line can be determined by applying principal component analysis (PCA) on the camera positions from the point cloud 304 to find the maximal variance axis. The other two axes of the central jaw line can be determined by applying quantile regression of a polynomial (e.g., a quartic polynomial) to each of the other two axes to the maximal variance axis. Thus, the central jaw line can be the centerline of the received scan data, which (for intraoral scanner data) typically corresponds to the central jaw line. Alternatively, the central jaw line can be solved or determined to correspond to the actual central jaw line based on analysis of the tooth image. Unless otherwise specified, the central jaw line can refer to the centerline of the received data, or the actual, estimated central jaw line.

[0103] Next, in Figure 2 In operation 204, the process includes identifying key camera angles for which a panoramic model will be constructed. These key camera angles provide viewpoints for which a panoramic model of the subject's teeth will be constructed. The camera angle refers to the camera view direction (i.e., the z direction of the camera aperture) used for the panoramic model.

[0104] In one example, the camera angle of a panoramic model can be represented by a sphere, such as Figure 4A 400. Within the sphere 400, all possible camera angles can be represented as unit vectors starting at coordinate 0 (eg, the center of the sphere). Figure 4A An example vector 1 is shown in . Figure 4B The unit sphere 400 may be triangulated into a plurality of triangles 402 using any known triangulation method, such as, for example, by recursively subdividing all triangles into octahedra. Vectors starting at coordinate 0 and ending at each vertex 404 of the plurality of triangles are identified as key camera angles required for constructing the panoramic model.

[0105] Return to reference Figure 2 In operation 206, the process may include projecting images from a scan of the subject's teeth to form an initial panoramic model for each key camera angle. Figure 5 , by dividing the central jaw line 556 into equidistant segments 551 (by Figure 5 ) and forming lines 552 at each segment, a two-dimensional grid can be formed, with lines 552 being perpendicular to the central jaw line 556 and perpendicular to each key camera angle or direction 554 selected in the previous step (e.g., Figure 4B, starting at coordinate 0 in the unit sphere 400 and ending at the key camera angle at each vertex 404 of the plurality of triangles). The endpoints of each line 552 may be indexed by the segment number and the line number to form a two-dimensional grid of points. Next, the point cloud of all camera positions during the scan is compared to the endpoints of each line 552, and the most appropriate physical camera position is selected, for example, the camera with the orientation closest to the key camera angle for each point is selected. It should be noted that as indexed in the two-dimensional grid of points Figure 5 Each line 552 in may not exactly match the received scan data (e.g., the line may be shorter than the distance between the physical camera position and the subject's tooth during the scan, or may otherwise not correspond to an exact location in space). In this case, selected images captured during the scan can be approximated using an orthogonal camera to provide an image from each endpoint. This results in a narrow field of view from the scanned image being used to approximate the view from each endpoint. In some variations, as described in more detail below, the approximated image can be a new view corresponding to a grid of points (e.g., endpoints of lines or segments), which can be solved using image-based rendering configured for use with intraoral scan data. All of the approximated images at each endpoint in the two-dimensional grid of points are combined to produce an initial panoramic model of the subject's teeth for each key camera angle. Fig. 6A An example of an initial panoramic model 600 generated by the projection of multiple images is shown.

[0106] like Fig. 6A It can be seen that some areas in the initial panoramic model 600 may not be properly registered due to the various camera angles or perspectives used when building the model. Figure 2 , in step 208 of the process, the images in the initial panoramic model may need to be registered and / or deformed so that the gradients at the boundaries of adjacent images match. The deformation may include several different steps, such as global optimization followed by local optimization only along the image boundaries. In one example, a global optimization method (such as projected image alignment by using enhanced correlation coefficients or ECC maximization) can be used to identify the appropriate image deformations required to match the boundaries of adjacent images. After applying the deformations identified in the global optimization, the image boundaries may still not match. Next, local optimization only along the image boundaries can be used to identify the appropriate deformations along the image boundaries that are required to match the boundaries of adjacent images. The identified boundary deformations can be analytically extended to the interior of each image for deforming the images in a smooth and realistic manner.

[0107] Next, in operation 210 of the process, the resulting images from the previous steps can be blended to produce a final panoramic model for each key camera angle. In one example, Poisson blending can be used to produce a final panoramic model for each key camera angle, which takes target gradients from the unblended images to produce an image with gradients that best match those target gradients. Each domain can be taken from a single image to maintain resolution.

[0108] Finally, in operation 212 of the process, the final panoramic model can be displayed to the user for the selected key camera angle. The final panoramic model can be used as an average image with the above triangles (e.g., Figure 4B The average image includes key camera angles and images corresponding to the triangle vertices (actually or approximately) and is rendered to the user based on the weights of the barycentric coordinates of the triangle 402 in FIG. Figure 6B An example of a blended final panoramic model 602 from key camera angles is shown in FIG.

[0109] The methods described herein may be performed by a device such as a data processing system, which may include hardware, software, and / or firmware for performing many of the steps described above. For example, Figure 7 5 is a simplified block diagram of a data processing system 500. The data processing system 500 typically includes at least one processor 502, which communicates with multiple peripheral devices through a bus subsystem 504. These peripheral devices typically include a storage subsystem 506 (a memory subsystem 508 and a file storage subsystem 514), a set of user interface input and output devices 518, and an interface 516 to an external network (including a public switched telephone network). The interface is schematically shown as a "modem and network interface" block 516 and is connected to corresponding interface devices in other data processing systems through a communication network interface 524. The data processing system 500 may include a terminal or a low-end personal computer or a high-end personal computer, a workstation or a mainframe.

[0110] User interface input devices typically include keyboards, and may also include pointing devices and scanners. The pointing device may be an indirect pointing device, such as a mouse, trackball, touchpad, or graphics tablet, or a direct pointing device, such as a touch screen incorporated into a display. Other types of user interface input devices may be used, such as a voice recognition system.

[0111] The user interface output device may include a printer and a display subsystem including a display controller and a display device coupled to the controller. The display device may be a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), or a projection device. The display subsystem may also provide non-visual displays, such as audio output.

[0112] The storage subsystem 506 maintains the basic programming and data structures that provide the functionality of the present invention. The software modules discussed above are typically stored in the storage subsystem 506. The storage subsystem 506 typically includes a memory subsystem 808 and a file storage subsystem 514.

[0113] The memory subsystem 508 typically includes a plurality of memories including a main random access memory (RAM) 510 for storing instructions and data during program execution and a read-only memory (ROM) 512 for storing fixed instructions. In the case of a Macintosh compatible personal computer, the ROM will include a portion of the operating system; in the case of an IBM compatible personal computer, this will include the BIOS (Basic Input / Output System). Distributed instructions, such as cloud-based instructions, may also or alternatively be used.

[0114] The file storage subsystem 514 provides persistent (non-volatile) storage for program and data files and may include, for example, at least one hard disk drive and at least one floppy disk drive (with associated removable media) and / or flash memory. Other devices may also be present, such as a CD-ROM drive and an optical disk drive (both with their associated removable media). In addition, the system may include a drive of the type with a removable media cartridge. One or more drives may be located at a remote location, such as in a server on a local area network, or at a site on the World Wide Web of the Internet.

[0115] In this article, the term "bus subsystem" is generally used to include any mechanism that enables various components and subsystems to communicate with each other as expected. Except for input devices and displays, other components do not have to be located in the same physical location. Therefore, for example, some parts of the file storage system can be connected through various local area network media or wide area network media (including telephone lines). Similarly, input devices and displays do not have to be located in the same location as the processor, although it is expected that the present invention will be most often implemented in the environment of PCS and workstations.

[0116] The bus subsystem 504 is schematically shown as a single bus, but may include multiple buses, such as a local bus and one or more expansion buses (e.g., ADB, SCSI, ISA, EISA, MCA, NuBus, or PCI), as well as serial and parallel ports. A network connection may be established through a device such as a network adapter located on one of these expansion buses or a modem located on a serial port. The client computer may be a desktop system or a portable system.

[0117] The scanner 520 (e.g., an intraoral scanner) can be responsible for directly scanning the patient's teeth, or scanning a cast of the patient's teeth obtained from the patient or an orthodontist, and providing the scanned digital data set information to the data processing system 500 for further processing. In a distributed environment, the scanner 520 can be located at a remote location and communicate the scanned digital data set information to the data processing system 500 via a network interface 524.

[0118] Manufacturing machine 522 manufactures dental appliances based on the intermediate data set information and the final data set information received from data processing system 500. In a distributed environment, manufacturing machine 522 may be located at a remote location and receive data set information from data processing system 500 through network interface 524.

[0119] The dental appliance manufactured by manufacturing machine 522 can be designed to implement at least a portion of a treatment plan, the dental appliance including a housing having a plurality of cavities therein, the cavities designed to receive teeth of a jaw.

[0120] System 500 may include software and / or firmware for executing instructions on one or more processors to perform any of the methods described herein. For example, Figure 7 The system 500 in may include a non-transitory computing device readable medium having instructions stored thereon that are executable by a processor to cause a computing device to receive scan data (e.g., intraoral scan data) via the computing device, and to generate and / or display a new view of teeth from the received data, to form one or more panoramic views or one or more sets of views (e.g., models) of teeth from the received (including new views), and / or to display one or more panoramic views or one or more sets of views. In some variations, the non-transitory computing device readable medium may include instructions for generating or modifying an orthodontic treatment plan, including a series of incremental tooth arrangements to define a proposed orthodontic treatment.

[0121] The methods and devices described herein can be used to allow a user to select and view (including interactive viewing) virtually any image of a patient's teeth from intraoral scan data. This can include visible light (including color light) or any other frequency of light (e.g., fluorescence, infrared light, such as near infrared light, etc.); the input scanned image can include scans of different wavelengths. The same data set can include multiple wavelengths (e.g., visible light / color light, near infrared light, fluorescence, etc.). For example, a data set can include images taken at multiple wavelengths from the same location, or images taken at multiple wavelengths at different locations. Therefore, the received data (scan data), such as data scanned by an intraoral scanner, can be marked or otherwise identified (in some cases, by a predetermined data file structure, or by a label on the scanned image), thereby indicating what the scanning wavelength is. This metadata can be included with other metadata (e.g., location, patient identifier data, date / time data, etc.).

[0122] These methods and devices can be used with a user interface that allows a user to select a camera angle (e.g., a key camera angle) to display from the patient's teeth. For example, the user interface can provide an image of the scanned teeth (e.g., a starting view) and a manipulation tool that allows the user to zoom in, zoom out, rotate, or simply select a position relative to the teeth for display. The user can also switch between different wavelengths (visible light, infrared light, fluorescence, etc.) or between superpositions of these wavelengths. The displayed view can be a panoramic view as described herein. In some variations, the displayed view can be a new view derived from a scanned view as described herein. Therefore, the method and device can allow a user to quickly and efficiently view the scanned image data directly as a panoramic image without building a 3D model of the patient's teeth. Although the scanned data is typically taken at a position close to the patient's teeth, gums and / or palate, so that each image is limited in its field of view, the method and device for generating a panoramic view described herein allows extremely fast, continuous and accurate viewing of a reduced panoramic image from almost any point around or outside the oral cavity.

[0123] Thus, the devices and systems described herein can be configured to provide an alternative for volumetric visualization of a patient's dentition. For example, the internal structure of a tooth can be seen using infrared light (e.g., near infrared light), and the internal structure of the tooth can be rendered in a 3D virtual model of the dentition. In some cases, the external features of the tooth can also be rendered in a 3D virtual model by combining data obtained by scanning using infrared light with scan data collected using a light source that emits one or more other wavelength ranges, such as visible light (e.g., fluorescence). In some cases, the scan data includes X-ray data. Figure 8An example user interface 800 is shown showing a 3D image of a mandible 802 combining near infrared and visible light data. Internal structures 804 (e.g., collected using near infrared light) and surface features 806 (e.g., collected using visible light) can be rendered in different shades or colors on the user interface. The resolution of the 3D image can vary and, in some cases, can be selected by the user. In some embodiments, the resolution of the internal structures is approximately 200 μm, allowing the user to view the morphology of the internal structures. Although in some variations as described herein, the 3D image can be based on a reconstructed 3D model (derived from data obtained from the scan), the 3D model can be generated from panoramic data.

[0124] For example, a panoramic view may be equivalent to a view generated from a 3D reconstructed model without using a reconstructed 3D model. Because the displayed image may be taken directly from the scan data (or from a new view derived from the scanned image, as described herein), the loss of spectral information provided, if any, is minimal, which loss may necessarily occur when generating a view from a synthesized 3D model. Thus, information such as reflection, transparency / translucency, hue, etc., including internal structures, which may be difficult or impossible to reliably segment when forming a digital 3D model, may be preserved in a panoramic view and / or panoramic model as described herein.

[0125] refer to Figure 8 , the image of the mandible 802 can alternatively be a panoramic view from the initial position (which is panoramic because it is formed from combined images from an intraoral scanner, as described herein), showing both the (combined and / or overlapping) visible light image 806 and the near infrared light image 804. By using one or more user interface input tools (e.g., buttons), the view of the dental arch (e.g., the lower dental arch) can be rotated, translated, zoomed, etc.; movement can be visually achieved by displaying different camera positions (angles) from key camera angles of each panoramic view. The transition can be done smoothly to show continuous / smooth motion. Additional panoramic (e.g., wide-angle) views can be pre-computed as part of the group (e.g., from a panoramic model), or can be computed on the fly.

[0126] The user interface may allow the user to rotate the 3D model and / or zoom in on certain areas in the model. In some variations, the user may zoom, pull / drag, and rotate the image of the tooth 802 using a mouse, a touch screen, or the like. Figure 8An example graphical user interface element 808 is shown, which may also or alternatively allow a user to manipulate the 3D image in various ways, such as selecting which part of the dentition (e.g., maxillary and / or mandibular) is shown in the panoramic 3D view, removing or adding layers (e.g., colored light, near infrared light, fluorescence, etc.), and / or playing / replaying animations. In one implementation, the user interface allows the user to display and / or replay the motion of the near infrared light image and the visible light image (and / or fluorescence image) side by side. In some embodiments, the user interface allows the user to select one or more regions of interest on the 3D image to zoom in. For example, a user can select an area by using a magnification ring or tool to show the selected region of interest at a higher magnification (e.g., again, generate a panoramic view of the sub-area, and / or identify a specific detailed view or generate a new view from a scanned image corresponding to the magnified camera angle). This can help with diagnosis by increasing the confidence of the diagnostic result, and can help reveal defects in the tooth, such as caries. Panoramic views of penetrating wavelengths (e.g., infrared light, near infrared light, etc.) can be used to visualize internal structures without requiring a lot of time and / or processor-intensive calculations compared to reconstruction techniques. The location of caries revealed by the panoramic view may be an indication of caries severity.

[0127] As described above, panoramic views of different wavelengths corresponding to the same viewing angle can be shown side by side or simultaneously. Fig. 9 Another example of a 3D panoramic view 900 of a dental arch is shown with a selected area 902. The selected area may correspond to a specified location of a tooth that may be displayed using an image of the tooth / teeth at a different wavelength than the rest of the view (e.g., infrared, fluorescent, and / or visible light). Fig. 9 , a portion of the visible light panoramic view 901 in the selected area 902 is replaced by a corresponding portion of the near infrared panoramic view. This can allow internal structures to be displayed, for example, because the partially transparent nature of infrared (e.g., near infrared) images can allow a user to visualize the depth and volume of internal structures. If both internal and surface structures are shown, the partially transparent areas can allow a user to view internal and external features related to each other in a single model. Various internal and / or external features can be displayed in different shades and / or colors based on certain aspects (such as density and / or composition).

[0128] In some embodiments, the user can select which portion of the panoramic view is displayed as which wavelength or wavelengths. For example, a user can identify a portion of the 3D panoramic view as an area of ​​interest for further analysis. The user can then use the user interface to indicate the area so that it is shown in more detail from the scan data and / or shown from the scan data as near infrared light when near infrared light data is included (e.g., in addition to visible light and / or fluorescent images). Alternatively or additionally, the system can determine suspicious areas of interest (e.g., based on density and / or composition) and display one or more panoramic views of the suspicious areas using near infrared light and / or fluorescent light data. The user can zoom / pan / rotate the perspective of the scan data (e.g., a virtual camera) to access different viewpoints of the selected area 902. In some cases, the user can choose to render the entire dentition (or entire dental arch) in near infrared light, only in visible light, and / or only in fluorescence.

[0129] In some embodiments, at least a portion of the dentition may be viewed in a slice representation. Fig.10An example virtual cross-sectional model 1000 of a dental arch is shown, as well as slice views 1004 and 1006 generated from a panoramic model. The slice view can be generated based on a virtual plane that passes through at least a portion of the panoramic view. A cross section through the view can be calculated from the scan data and one or more panoramic images. The resulting cross section can be shown in the panoramic view. The position (e.g., height and angle) of the virtual plane can be determined relative to a reference. In some embodiments, the reference can correspond to a reference plane or reference line (e.g., a central jaw line or a gum line). In one implementation, a user can position and / or draw a first line 1002 across a selected tooth so that the first virtual plane passes through the tooth, thereby generating a corresponding first slice view 1004 of the tooth. Similarly, a user can position and / or draw a second line 1006 across a selected dental arch area so that the second virtual plane passes through the selected dental arch area, thereby generating a corresponding second slice view 1008 of the area. The slice view can be generated using data obtained by using one or more modes (e.g., infrared light and / or visible light) of one or more scans. Therefore, the slice view can reveal the corresponding internal structure and / or surface structure of the tooth. In some embodiments, slice views can be generated using data that is collected using only one mode (e.g., infrared light or visible light), for example, to save computing time. Slice views can correspond to 2D images, or can correspond to sections of specified thickness generated from raw data and / or panoramic views. In slice views, various features can be represented in different shades and / or colors, for example, to represent different densities, compositions (e.g., teeth, caries, gums), and / or other aspects of the dentition. In some embodiments, density calculations and / or other data associated with the slice views can also be displayed to the user. Specifically, the slices described herein can be derived from panoramic views.

[0130] Fig.11Another example is shown, which shows a slice view of a series of cuts through a dental arch. In this example, a series of slice views represent the progression of horizontal cuts through a dental arch, from a first slice 1102 through the lower part of the dental arch closest to the gum line to a last slice 1104, which represents the upper part of the dental arch including the occlusal surface of the teeth. The slice view may include different shades and / or colors to indicate different densities, compositions, and / or other aspects of the dental arch. This representation may allow a user to analyze different aspects of a tooth associated with a root, gum line, and / or occlusal surface. For example, the depth and height of a tooth defect such as caries may be easily observed. The user interface may allow a user to select the number of slices through the dentition. The user interface may allow a user to manipulate each slice view, for example, by moving, superimposing, and / or rotating the slice view. In some embodiments, the user interface allows a user to animate the slice view. For example, the user interface may display slices 1102 to 1104 in a series of images, thereby showing a progression from the lower part of the dental arch to the upper part of the dental arch, and vice versa.

[0131] The panoramic views described herein can render the dentition of a subject in any of a variety of ways. In some embodiments, the panoramic view simulates a bitewing view of the dental arch. Traditionally, a bitewing is an X-ray film of the crowns of the upper and lower teeth that is taken simultaneously using a label or paper that is held in place between the teeth. The bitewing view typically shows at least a portion of the dental arch and may include the crowns of the teeth up to approximately the location of the supporting bone. In practice, bitewing views are often used to detect signs of any cavities between the teeth, diagnose abscesses or cysts, and to show any changes in the roots of the teeth and surrounding bone. Panoramic viewing technology can be used to generate a simulated bitewing view of the dental arch so that the dentist can view aspects of the patient's dentition similar to traditional X-ray bitewing images.

[0132] In some cases, simulated bitewing views depict the teeth within the dental arch along a straight line rather than along the curvature of the jaw centerline. Fig.12 1207 with the teeth aligned in a straight line. As shown, the jaw centerline 1206 of the curved dental arch can be straightened to effectively "unfold" the dental arch. Thus, in the example shown, the bitewing view depicts the teeth along a straight line. The bitewing view shows the teeth of the dental arch from a perspective of a particular viewing angle. Fig.12In the example of FIG. 1 , the teeth are viewed from a buccal viewing angle so that a region of interest 1210 in the interdental area can be viewed from a buccal perspective. The region of interest 1210 can be identified, marked, and / or analyzed for defects. Other viewing angles can include an occlusal viewing angle or a lingual viewing angle. A particular viewing angle can be selected by a user and / or automatically selected (e.g., a default setting). As will be described herein, in some variations, the panoramic view can simultaneously show the buccal, lingual, and occlusal surfaces (as will be described in reference to FIG. 1 ). FIG. 17A to FIG. 17C and FIG. 18A to FIG. 18B described above).

[0133] The central jaw line 1206 can be identified using the collected scan data, and the central jaw line 1206 can be determined using any of a variety of techniques. Because the central jaw line is used as a reference, in some cases, the central jaw line can be approximated and does not need to be exactly in the center of the jaw at every point along the jaw. In some embodiments, the central jaw line is approximated based on an image of the teeth and / or 3D scan data of the teeth. In some embodiments, the central jaw line can be determined by finding the axis of maximum variance (e.g., by applying principal component analysis (PCA) to the camera positions from the point cloud), as described. In some cases, the central jaw line is determined by using only the camera positions collected during the scan. For example, if the scan involves moving the scanner around the dental arch (e.g., from the buccal, occlusal, and lingual sides), the central jaw line can be approximated as an arched curve along the center of the scan path. In some cases, the user can manually enter and / or change the central jaw line or some portion of the central jaw line.

[0134] Once the central jaw line is identified, the virtual pixels used to construct the panoramic view can be arranged along the central jaw line (similar to the Figure 5In some embodiments, the virtual pixels are equidistantly distributed along the center jaw line. The virtual pixels can be spaced at a predetermined distance, for example, to generate an image with at least or at most a certain resolution. In one implementation, the resolution is approximately 200 μm. In order to select an appropriate image to generate a panoramic view at a specific viewing angle, the virtual camera can travel along a path according to the center jaw line so that the virtual camera direction is at a specific viewing angle relative to the center jaw line. For example, in one implementation for generating an occlusal panoramic view, the jaw centerline is along the xy plane, and the virtual camera direction is in the z direction relative to the center jaw line. For each pixel, one or more images whose associated viewing angle is most closely aligned with the specific viewing angle are identified and projected onto a virtual screen perpendicular to the virtual camera direction. For example, for an occlusal panoramic view, one or more images that are most closely aligned with the occlusal viewing angle (e.g., 90 degrees) are identified and projected onto the corresponding virtual screen. For each pixel, the process will continue as the virtual camera moves relative to the central jaw line until all pixels have an associated image projected onto the corresponding virtual screen to generate a panoramic view of the dental arch. In this way, each pixel can be associated with one or more images that can be used together to generate a panoramic view of the dental arch from a selected viewing angle. In other words, the selected images of each virtual pixel can be projected onto the corresponding virtual screen and combined together to construct a panoramic view. In some cases, only one image is selected for each pixel. In other cases, more than one image (e.g., 2, 3, 4, 6, etc.) is selected for each pixel. In some cases, all teeth in the dental arch are shown in the panoramic view (e.g., so that the dentist can view all teeth in one image). In other cases, only a portion of the teeth in the dental arch are shown in the panoramic view (e.g., if the dentist only wants to view a portion of the dental arch).

[0135] As described herein, the center jaw line can be "unfolded" so that the teeth are shown as a line (a straightened center jaw line). One of the advantages of depicting the dental arch along a single line is that the dental arch can be displayed as a narrow, long image, thereby taking up less display space and facilitating stacking and comparison with other panoramic views. However, the panoramic view can render the dental arch in any of a variety of ways. For each pixel, the identified image can be projected onto a virtual screen, regardless of the curvature of the center jaw line. Therefore, the collection of images identified in the panoramic view can be rendered along a line having any shape. In other embodiments, the dental arch is shown according to a curved center jaw line (such as the natural curve of an actual dental arch), or is shown with different curvatures that facilitate viewing and analyzing certain aspects of the teeth. In some embodiments, the view includes an occlusal view showing different sectors (quadrants) of the dental arch.

[0136] The images selected for generating the view (and any of the panoramic views described herein) may include those collected using any type of scanning mode (e.g., infrared light, visible light, fluorescence, X-ray, etc.) or combination of scanning modes. Fig.12 In an example, a panoramic view is generated from a near-infrared light image that shows the internal structure of a tooth. In other examples, a panoramic view can be generated from images collected using light of different scanning wavelengths. For example, a panoramic view can be generated using images collected using visible light (e.g., fluorescence) or X-rays. In some embodiments, images collected using two or more modes can be combined to generate an image that describes different aspects of the dentition (e.g., internal features and surface features of the teeth). In other embodiments, images collected using only one mode are used to generate one or more panoramic views, which can allow images to be processed and displayed more quickly and consume less computing power.

[0137] Fig.13A and Fig. 13B Examples of panoramic views of the dental arch at occlusal viewing angles (eg, bitewing views in these examples) are shown. Fig.13A An original panoramic view of the images stitched together before a blending operation is performed is shown. As shown, the various selected images may overlap and have different shades, colors, and / or deformations related to the conditions under which each image was taken. Fig. 13B shows the hybrid operation after Fig.13A In the embodiment of the present invention, a panoramic occlusion view in the image is used to align and / or deform the corresponding original panoramic view so that the gradients at the adjacent image boundaries match. As described herein, the deformation can include global optimization and / or local optimization along the image boundaries. In one example, a global optimization method can be used to identify the appropriate image deformation required to match the adjacent image boundaries. If the image boundaries still cannot be adequately matched after global optimization, local optimization along the image boundaries (e.g., only along the image boundaries) can be used to identify the appropriate deformation along the image edges required to match the adjacent image boundaries. The boundary deformation can be extended to the interior portion of each image, so that the image is deformed in a smooth and realistic manner.

[0138] Fig. 13C Shows Fig.13A and Fig. 13BA panoramic lingual view of a dental arch in and after a blending operation. Thus, various views of a dental arch can be generated based on a selected viewing angle. In some embodiments, the system is configured to allow a user to select a viewing angle, and the system will automatically update the panoramic image on a computer display based on the user's input. In some cases, the panoramic image can be updated in real time or near real time. In some cases, the panoramic image can be updated in approximately 15 seconds or less. The user interface for allowing a user to select a viewing angle may vary. In some cases, a user can use a cursor to select a panoramic image and rotate the image (e.g., up or down on a computer display) to select between various viewing angles. The panoramic image can be updated quickly to provide a smooth transition between various viewing angles. In some embodiments, the user interface includes one or more buttons, scroll bars, and / or other functional features for selecting a viewing angle.

[0139] In bitewing or non-bitewing (e.g., pseudo-realistic, realistic, such as FIG. 6A to FIG. 6BIn a panoramic view (shown in FIG. 1 ), a simulated panoramic view (and any of the panoramic views described herein) can be generated from images generated by one or more machine learning agents. Such a synthetic image can be generated based on a real image taken during one or more scans, or based on a synthetic new view generated from scan data, where the scan data does not include a scan taken at a specified location. In some variations, as described herein, a new view can be generated using an image-based rendering machine learning agent. For example, a machine learning agent (e.g., a machine learning algorithm) can be configured to recognize patterns in a real image and then use the recognized patterns to generate a new ("synthetic") image. Synthetic images may be useful, for example, if there are not enough images to choose from at a specific position and viewing angle to generate a panoramic image with sufficient detail or resolution. For example, if the camera is not at a precise angle to show a specific image of one or more teeth, the system can generate such an image and use the synthetic image to render a more detailed panoramic view. In one example, a first scan image and a subsequent second scan image can be collected from a specific viewing angle during a dental scan. However, not enough data may be collected to provide an image between the first image and the second image to generate a sufficiently detailed panoramic image. The machine learning algorithm can generate one or more new synthetic images at a specific viewing angle that correspond to one or more real images located between the first image and the second image. The one or more synthetic images can then be used with the first image and the second image (and any other selected or synthetic images) to generate a panoramic view. In other examples, the machine learning algorithm can use data collected from one or more viewing angles different from the specified viewing angle to generate a synthetic one or more images at the specified viewing angle. The panoramic view can be generated based solely on the synthetic images.

[0140] FIG. 14A to FIG. 14C Panoramic views of another dental arch at different viewing angles taken using different scanning modes are shown. Fig.14A Two lingual views of a dental arch are shown, including a panoramic view 1400 , which is an image captured using near-infrared light, and a panoramic view 1402 , which is an image captured using visible light. Fig. 14B Two buccal panoramic views of the dental arch are shown, including a near infrared light view 1404 and a visible light view 1406 . Fig. 14CTwo occlusal panoramic views of a dental arch are shown, including a near infrared light view 1408 and a visible light view 1410. Displaying the panoramic views using different modes can allow a user to visualize different aspects of a patient's dentition. For example, some dental features and defects (e.g., surface features) can be more easily identified using visible light, while other dental features and defects (e.g., internal structures) can be more easily identified using near infrared light. These examples also show how the various panoramic views can be stacked and aligned for ease of comparison (particularly compared to realistic / pseudo-realistic views).

[0141] It should be noted that Fig.12 , FIG. 13A to FIG. 13C as well as FIG. 14A to FIG. 14C The bite wing panoramic view shown in is shown as an example only, and other viewing angles and scanning modes may be alternatively or additionally shown on the computer interface. For example, a panoramic view of a dental arch at an angle between a lingual view and an occlusal view and / or between an occlusal view and a buccal view may be alternatively or additionally generated and displayed. Such views may be generated using real images collected during one or more scanning operations, and / or by using synthetic images generated via machine learning. Additionally or alternatively, images taken using other scanning modes (such as scanning modes using a specific range of electromagnetic radiation (e.g., ultraviolet, infrared, fluorescence, X-rays, etc.)) may be taken by a scanner to generate a corresponding panoramic image. As described above, panoramic views including bite wing panoramic views may be rotated, scaled, translated, etc. using a user interface. The rotation / scaling / translation of the bite wing panoramic view may be smoothly performed in a continuous manner, including allowing the user to manipulate the user interface to move (translate / zoom / rotate) the view. Bite wing panoramic views and non-bite wing panoramic views may be displayed simultaneously.

[0142] The user interface may arrange various panoramic views in any of a variety of ways, including but not limited to bitewing panoramic views. For example, any of the panoramic views may be vertically aligned (e.g., stacked) so that features in the image are vertically aligned. In some cases, various panoramic views are arranged horizontally (arranged side by side). Alternatively or additionally, the panoramic views may be capable of overlapping and / or displayed as partially transparent so that features may be aligned and shown on top of each other. The user interface may be configured to allow a user to move one or more panoramic images, for example, by using a drag-and-drop feature. In some cases, the user interface may allow a user to mark / mark an area of ​​interest in one or more panoramic views. In one implementation, a user may mark an area of ​​interest in one panoramic view, and the system generates a corresponding mark at a corresponding position in another panoramic view. The user interface may be configured to show a single panoramic view that is capable of switching to other panoramic views at different viewing angles and / or different scanning modes. In some embodiments, a user may scroll or rotate a panoramic view (or a user interface button) to display various viewing angles and / or scanning modes. For example, a user may view a dental arch in a lingual view, rotate the image to render an occlusal view, and further rotate to render a buccal view. In another example, a user can view a dental arch photographed in infrared light mode and rotate the image to render a dental arch photographed in visible light mode. The system can be configured to continuously update the view as the user rotates the image. In some cases, the image can be rotated in both directions.

[0143] Any of the panoramic views can be displayed in color and / or grayscale on the computer screen. In some cases, the user interface allows the user to choose whether to display one or more panoramic views in color or grayscale. The user interface can be configured to display features in two or more different colors (e.g., 2, 3, 4, 5, etc.). Different shades and / or colors can be used to visualize different aspects of the dentition. For example, gums can be displayed in one shade and / or color, while teeth can be displayed in another shade and / or color. In some embodiments, tooth regions with different densities, compositions or other attributes are rendered in different shades and / or colors. For example, a tooth region with a normal tooth composition can be displayed in one shade and / or color, while a tooth region with a defective tooth composition (e.g., caries) can be displayed in another shade and / or color.

[0144] The user interface can be configured to show a panoramic view based on the time when the scan was taken. In one case, a first scan is taken of the patient's teeth at a first time (e.g., before orthodontic treatment), and a second scan is taken of the patient's teeth at a second time (e.g., after orthodontic treatment, or after a portion of orthodontic treatment). The system can be configured to generate and display a first panoramic view based on the first scan, and to generate and display a second panoramic view based on the second scan. Similarly, the system can be configured to generate multiple (e.g., 2, 3, 4, 5, 6, etc.) panoramic views based on the scan time. In this way, the user can easily visualize the changes in the dentition over a period of time (e.g., before and after treatment). In some cases, the user interface is configured to display such time-based images in a time-lapse representation, wherein the progression of the dentition changes is sequentially displayed to the user. For example, a first panoramic image can be transformed into a second panoramic image, a second panoramic image can be transformed into a third panoramic image, and so on. In some cases, the transformation is presented as an animation. The user is able to change the display speed of the time-lapse image. Additionally or alternatively, the user can stop the time-lapse representation at any point, for example, to examine a particular image.

[0145] Fig.15 A flowchart 1500 is shown, which describes a process for generating a panoramic view of a subject's teeth. Referring to operation 1502, the process may include receiving images of a subject's dental arch, each image having an associated recorded camera angle. Images may be collected by performing one or more scans of the subject's oral cavity using, for example, an intraoral scanning device or camera system as described herein. In some aspects, the intraoral scanner or camera system may record multiple discrete images or a series of continuous images during the scan. Additionally, the position and orientation of the intraoral scanner or camera system may be tracked and recorded during the scan, and the position and orientation of the intraoral scanner or camera system may also be associated with each corresponding discrete image or video frame. In another embodiment, a digital model of the patient's dental arch may be accessed or received (e.g., if a scan was previously performed). These images may include images captured using any of a plurality of imaging modes or scanning modes. For example, images may be captured using one or more light sources, such as visible light and / or infrared light (e.g., near infrared light) light sources. The intraoral scanner or camera system scans along all surfaces of the maxillary and mandibular surfaces to obtain images of all relevant tooth surface structures. Thus, the received images may be part of image data, which may also include the camera position (e.g., camera angle) from which each image was taken, the mode (e.g., wavelength) used to take each image, and / or other metadata associated with each image. Thus, any of these methods may include accessing (e.g., receiving) image data including images.

[0146] Generally, as used herein, a panoramic view refers to a wide-angle view. The panoramic views described herein are not limited to specific angles (e.g., greater than 30 degrees, greater than 45 degrees, greater than 90 degrees, greater than 135 degrees, greater than 180 degrees, between 10 and 270 degrees, between 10 and 225 degrees, between 10 and 180 degrees, between 30 and 270 degrees, between 30 and 225 degrees, etc.).

[0147] Referring to operation 1504, the process may include identifying a specific viewing angle for constructing the panoramic view. The specific viewing angle may correspond to a viewpoint for which the panoramic view of the subject's dentition is generated and displayed on the computer screen. The user may select the specific viewing angle. For example, the user may select via the user interface whether the panoramic view is to be viewed from a buccal, occlusal, lingual, or other perspective (including cylindrical or partial cylindrical projections, such as FIG. 17A to FIG. 17C and FIG. 18A to FIG. 18B The user interface may include a graphical user interface, such as a switch, a button, a scroll bar, a drop-down menu, and / or other graphical user interface elements, which allow the user to select one or more specific viewing angles (viewing angles) to view the teeth of the subject. Alternatively or additionally, the computing device may be configured to automatically select a specific viewing angle, for example as a default setting. The devices and methods described herein may allow the user to dynamically (e.g., in real time) change the viewing angle, and thereby change the panoramic view.

[0148] Once the viewing angle is identified, the trajectory line of the image (e.g., the center jaw line) 1506 can be identified. The center jaw line can be estimated using images collected during one or more scans of the subject's teeth (e.g., 2D images, such as from received scan images), a virtual 3D model of the dentition, and / or position data of the scanner / camera. Typically (but not necessarily), multiple points (e.g., "virtual pixels") along the center jaw line can be automatically selected or determined. In some variations, these points (e.g., virtual pixels) can be distributed along the center jaw line, including being equally spaced. These virtual pixels can correspond to a point for which at least one of the images collected during one or more scanning operations can be selected to generate a panoramic view. Alternatively, these points (some or all of them) can be used as points between these scanned positions. In some embodiments, the virtual pixels are evenly distributed along the center jaw line. Typically, the denser the distribution of virtual pixels, the more images will be used to generate a panoramic view.

[0149] For each of these identified points (virtual pixels), one or more images from the scanned 2D image including the point can be identified at which the viewing angle matches or approximately matches the camera angle used to capture the image 1508. For example, the virtual camera can travel along a path adjacent to the central jaw line. This can be done to identify one or more images having an associated camera angle that most closely matches a specific viewing angle at each virtual pixel. For example, when the virtual camera passes through a first pixel, one or more images collected during one or more scanning operations can be identified as having a camera angle close to the specific viewing angle. When the virtual camera passes near the central jaw line, at a certain point, the virtual camera can be perpendicular (or approximately perpendicular) to the virtual pixel (e.g., in the z direction). When the virtual camera is perpendicular to the virtual pixel, one or more images can be selected that have one or more camera angles that are most similar to the specific viewing angle, thereby selecting one or more images captured at the most similar one or more positions, such as the perspective from the specific viewing angle. In some variations, the identified image may be a new composite image determined for a location corresponding to a viewing angle that does not have an exact match in the received image.

[0150] Upon (or after) selecting the images, the images may be collected 1510. For example, the collected images may be projected onto a virtual screen at each point. The virtual screen may be perpendicular to the viewing direction. The process may be repeated for all identified points on the central jaw line (e.g., as the virtual camera passes through each pixel along the central jaw line until a set of identified images is reached) to collect a resulting panoramic view. For the bitewing panoramic view, the central jaw line may be straightened.

[0151] In 1512, the set of identified images is used to generate a panoramic view. An approximate panoramic view can be generated using the collected images, which can be arranged along the central jaw line in the order in which the points are arranged along the jaw line. The panoramic view can optionally be blended to match the gradients at the boundaries of adjacent images and provide a more consistent panoramic view to the user 1514. As described herein, the blending operation can include using target gradients and position data associated with the images. The final panoramic view can be displayed on a computer display, stored in a memory, and / or sent as an output (e.g., to a network and / or a printer). In some embodiments, the final panoramic view is displayed on a user interface of the display.

[0152] The user interface may include user input features that allow the user to modify the panoramic view based on viewing angle, scan mode, scan time, and / or other image parameters. Fig.16A flowchart 1600 of a process for allowing a user to select image parameters for displaying a panoramic view of a subject's teeth is shown. Referring to 1602, for example, image parameters are received from a user (e.g., via a user interface) and / or are set to default parameters. The image parameters may include a specific viewing angle corresponding to a desired viewing angle for viewing a dentition (e.g., a dental arch). The specific viewing angle may (but typically does not) correspond to a camera angle from which the image is taken. The specific viewing angle may be specified as a viewing angle type, such as a buccal, occlusal, or lingual viewing angle. Additionally or alternatively, the specific viewing angle may be specified as an angle relative to a given reference (such as a jaw centerline) (e.g., 180°, 90°, 70°, 60°, 45°, 30°, 20°, 15°, 10°, 5°, etc.). The image parameters may additionally or alternatively include a scanning mode, including a light source wavelength range used during one or more scans. In some embodiments, visible light and / or infrared light (e.g., near infrared light) is used. Visible light images can include surface features of the dentition, and infrared light (e.g., near infrared light) images can provide internal structural features of the dentition. In some embodiments, only one scanning mode (e.g., visible light or infrared light) is used to generate a panoramic image to save computing time for generating the panoramic image. Image parameters may additionally or alternatively include the time at which the scan was performed. For example, a first scan may be performed on the dentition of a subject before treatment, a second scan may be performed at some time during the implementation of the treatment plan, and a third scan may be performed after the treatment plan is completed. The user may wish to view panoramic images from each of the first scan, the second scan, and the third scan to analyze the progress of the treatment and / or decide whether additional treatment is needed. As described herein, the user may choose to view the progress of the treatment sequentially on the display screen using a time-lapse or animation. Image parameters may additionally or alternatively include an identification of the subject so that panoramic images of different dentitions can be viewed and compared.

[0153] At 1604, one or more panoramic views are generated based on the image parameters (e.g., user-selected or default image parameters) and displayed on the user interface. In one embodiment, the one or more panoramic views show a dental arch with teeth arranged in a line (e.g., Fig.12 , FIG. 13A to FIG. 13C or FIG. 14A to FIG. 14C). The system can be configured to receive new image parameters 1606 for modifying one or more panoramic images. In some embodiments, the user interface allows the user to change one or more image parameters (e.g., viewing angle, scan mode, scan date / time, dentition / subject identification, bitewing / realism, etc.). This allows the user flexibility in how the panoramic image is displayed and in selecting the best panoramic image to analyze specific aspects of the dentition. Alternatively or additionally, the system can automatically generate a modified panoramic image based on, for example, providing a predetermined arrangement of panoramic images and / or a setting for a time-lapse animation. Once the new image parameters are received, the system can update the panoramic view (or generate one or more new panoramic views) 1608 based on the new image parameters. The system can be configured to receive new image parameters to update the panoramic view and / or display one or more new panoramic views.

[0154] Any of the images and / or virtual models described herein may be rendered using animation and / or time-lapse techniques. For example, images collected during one or more scans of the teeth may be replayed to the user. The replay may be in actual time (the same time as the scan was performed), at an increased speed (e.g., 1.25 times, 1.5 times, 1.75 times, 2 times, 2.25 times, 2.5 times the scan speed), or at a reduced speed (e.g., 0.75 times, 0.5 times, 0.25 times the scan speed, etc.). In some cases, a longitudinal animation may be displayed in which images of the dentition (or a portion of the dentition) in different scans taken at different times are correlated and displayed sequentially to the user. This can show the user what changes have occurred to the dentition over a period of time, such as before, during, and after orthodontic treatment.

[0155] Any image and / or virtual model described herein can allow a user to visualize a specific region of interest. Regions of interest can include those regions with defects (such as caries). In some cases, a system (e.g., one or more processors / one or more controllers) can be configured to detect suspicious regions of interest in the scan data. These suspicious regions of interest can be indicated in a panoramic view and / or at other locations in the user interface. Users can indicate, highlight or otherwise mark those regions determined to be regions of interest. In some embodiments, this detection can be automatically completed (e.g., as a default setting). In some cases, a user interface can allow a user to select whether to perform a detection operation and / or select detection parameters. The detection of such regions of interest can allow false positives with minimal false negatives. Cylindrical view

[0156] In general, what is described herein is a cylindrical view in which a panoramic view of a dental arch can be provided that simultaneously shows two or more of the buccal, lingual, and occlusal sides of the teeth arranged in a flat panoramic view. The cylindrical view (e.g., a cylindrical view that simultaneously shows the lingual, occlusal, and buccal sides of the dental arch) can be used to quickly and accurately analyze or compare the dental arches of a patient. It can also allow for a convenient 2D representation (panoramic view) of the entire dental arch. Therefore, these cylindrical projections can be used to analyze, store, and display dental arches.

[0157] Fig.17A An example of a 2D projection of a 3D surface of a patient's dental arch is shown, showing a tubular (enclosed on three sides) "screen" 1705 over the dental arch 1703. In this example, the virtual screen encloses the dental arch. Fig. 17B A top view (occlusal view) of the same dental arch and surrounding tube is shown, and section CC' is also shown. Fig. 17C A cross section through line CC' is shown. Fig. 17C The cross-section of teeth 1707 and shield 1705 in FIG. 1 shows that the tubular shield surrounds the dental arch on three sides (e.g., buccal, occlusal, and lingual) approximately 270 degrees. Other less enclosing "screens" may be used, such as 180 degrees, etc. Views taken at each point in the enclosing cylinder toward the centerline of the tooth may be identified and / or synthesized and stitched together, such as described herein (including blending), to form a cylindrical mapping of the patient's teeth. The density of the indicated points may be adjusted by the user or automatically (increased / decreased). For example, the density may be adjusted to prevent loss of detail while minimizing the density to increase speed.

[0158] FIG. 18A to FIG. 18B Another example of a dental arch is shown in an occlusal view as a 2D projection of a 3D model of the dental arch 1803 ( Fig.18A ), and for comparison the cylindrical projection 1821 ( Fig.18B ), the cylindrical projection 1821 shows Fig.18A 1821 (similar to the cylindrical panoramic views described above). In this example, the cylindrical projection 1821 shows the lingual, occlusal, and buccal sides of the teeth in the same image. Except for small occlusions, this projection shows all the visual data in the scan in one image.

[0159] As mentioned above, Fig.18B The panoramic views shown in can be generated from a dataset of 2D images taken from recorded or known camera positions, for example using an intraoral scanner. In some variations, a panoramic view can be identified by using trajectory lines that extend only beyond a single center jaw line (e.g., the center jaw line); multiple lines can be used.

[0160] The cylindrical projections described herein can be used in part for manual analysis, such as by a doctor, dentist, etc. In some variations, these projections can be used for machine learning, such as for machine learning techniques using images of dental arches and / or teeth. In this case, a machine learning engine can use the cylindrical projection images of the dental arches for processing by machine learning (e.g., for one or more of the following: segmentation, recognition, etc.). Including all visual data in a single image (e.g., a cylindrical projection) can result in better machine learning results.

[0161] For example, described herein are methods of generating cylindrical projections of a dental arch, including processing scan data (as described above) to form cylindrical projections of a dental arch including buccal, occlusal, and lingual views. These methods may include: determining one or more trajectory lines that encompass all three sides in an overlapping manner; identifying points on a tubular "screen" surrounding the dental arch, such as FIG. 17A to FIG. 17B and FIG. 18A to FIG. 18B as shown; and identifying one or more 2D images from, for example, intraoral scan data (including but not limited to near infrared / infrared intraoral scan data, and / or surface scan data); and then aggregating the images, for example by stitching them together, and blending, smoothing, or otherwise coordinating them to provide a final panoramic view Generate new views from scan data

[0162] In general, new view synthesis refers to the problem of generating a new camera perspective of a scene given a set of fixed images of the same scene (or overlapping parts of the same scene). Therefore, new view synthesis methods perform image and video synthesis based on camera positions. In classical computer vision, image-based rendering (IBR) methods typically rely on optimized multi-view stereoscopic methods to reconstruct scene geometry and warp the observations into the coordinate system of the new view. However, this may result in view-dependent effects and may cause afterimage-like defects and holes. The techniques described herein may be referred to as image-based neural rendering techniques, which are particularly suitable for use as described herein. For example, these techniques are suitable for use using intraoral scans (and similar medical / dental scans) as input, and may be particularly suitable for generating new views of biological structures, particularly new views of dental structures (e.g., teeth), which can have opacity in visible light and other penetrating (e.g., near-infrared) wavelengths. These techniques can be used with machine learning, which can be trained to reconstruct learned representations of scenes from observations by end-to-end learning using a differentiable renderer. This enables learning prior knowledge of geometry, appearance, and other scene properties in a learned feature space. Although the techniques described herein may be neural rendering techniques, classical methods may alternatively or additionally be used.

[0163] Neural image-based rendering is a combination between classical image-based rendering and deep neural networks, which can use learned components to replace manual heuristics. Classical IBR methods use a set of captured images and proxy geometry (e.g., from different viewpoints) to generate new images. The proxy geometry is used to reproject the image content from the captured images to a new target image domain. In this target image domain, the projections from the source images are blended to form the final image. This simplified process only provides accurate results for diffuse objects with precise geometry that are reconstructed using a sufficient number of captured views. However, artifacts such as afterimages, blurring, holes, or seams may occur due to view-dependent effects, imperfect proxy geometry, or too few source images. To address these issues, image-based neural rendering methods replace the heuristics commonly found in classical IBR methods with learned blending functions or corrections that take into account view-dependent effects.

[0164] As mentioned and described above, the methods and devices herein can generate new views from received scan data (e.g., received multiple 2D images and corresponding camera position information). Any of these methods and devices can include image-based rendering (IBR) to synthesize a new view (e.g., a new view of the tooth) from the sampled view by using contributing images (e.g., preferably 3 images) from the received images, wherein the contributing images are taken at a minimum angle relative to the new viewpoint. For example, three (or in some variations, more, such as five) contributing images can be selected by triangulation that encloses the desired (new) angle or point (for which a new novel image is to be generated), wherein the triangulation produces a minimum envelope that can be formed by the camera positions of the contributing images.

[0165] The method may allow a new image to be generated from a set of images taken at arbitrary positions (e.g., received multiple scanned 2D images, such as may be provided by an intraoral scanner). In some variations, it has been demonstrated that using three such images is computationally particularly fast, particularly compared to using multiple (e.g., 4 or more) received images, the three images being selected to have camera positions that provide the smallest triangle that can enclose the desired new camera position.

[0166] Each contributing image can then be decomposed into multi-planar layers (e.g., RGB alpha layers). These layers can be uniformly disparity sampled in the contributing image camera view's frustrum. For example, these layers can be formed by reprojecting each plane / layer to the sensor plane of the new view camera and alpha compositing these planes from back to front to encode the local light field. Adjacent layers in the three (or in some variants, for example, 5) contributing images can then be blended. Therefore, in some variants, three (or in some variants, more) contributing images can be projected into multiple layers (also referred to as planes) parallel to the new viewpoint they want to generate (e.g., the surface or screen on which they will be projected). They can then be blended as described above.

[0167] The methods described herein can be automated by using a machine learning agent (e.g., a new view generation machine learning agent) that can perform all or some of these steps, for example, including identifying contributing images, dividing each contributing image into multi-plane layers, and mixing adjacent layers in the multi-plane layers of the contributing images. Specifically, the mixing can be performed by a machine learning agent. The machine learning agent can be trained on a data set, for example, on a dental data set. In some variations, the machine learning agent can be trained on a data set (e.g., a scanned dental data set), wherein known points with known camera positions are removed from the data set and used as target points. An actual image at the known point (now the target point) can be used. Given a large number of images at known camera positions from dental scans, many such target training points (images) can be generated, allowing multiple iterations (e.g., thousands, tens of thousands, hundreds of thousands, etc.).

[0168] The methods and apparatus described herein for generating and displaying panoramic views may particularly benefit from techniques for generating new views from received scan data. New view generation may involve generating new views from a given set of images (where the camera position of each image is also known). One or more new views may be generated for new camera positions for which images were not captured. For example, new view generation may be used to generate new views for display from multiple (e.g., 3, 4, 5, 6, etc.) images, and / or for generating panoramic views. The multiple images, such as images captured using an intraoral scanner, may also record camera positions with high accuracy, for example by using a built-in accelerometer or one or more other position sensors. These images may be acquired during scanning (which may form an image "soup" comprising multiple images), and additional new views may be generated from the image soup after post-processing.

[0169] The generation of new images of teeth may be particularly desirable when multiple images are captured (e.g., in scan data) while the teeth are very close to an imaging camera (e.g., an intraoral scanner). For example, the pinhole of the camera (scanner) may be located, for example, about 7 mm above the teeth in some variations, even at a very wide angle (which may result in perspective distortion) of the scanner, compared to other scanners (where the teeth may be scanned at a distance of about 90 mm from the teeth), even at a location where the scanner has a very wide angle (which may result in perspective distortion). For example, a typical intraoral scanner may have a field of view that spans only a single tooth (where the camera scans, for example, between 1-20 mm from the tooth) or two teeth (where the camera scans, for example, between 25-100 mm from the teeth). Thus, even if the field of view is large enough to cover more than one tooth (e.g., 2-3 teeth), the field of view may be narrow enough to limit the viewing angle of one or more teeth within the patient's dentition (e.g., upper and / or lower arches), which may make it difficult to understand the relative position and orientation of the teeth in the jaw and to understand clinical findings in the image from the received image alone. However, despite the relatively small field of view, the scanned images acquired can have very high signal-to-noise ratio, clarity, resolution and other optical properties. In addition, many of these scanned image data sets can include hundreds or thousands of images.

[0170] Thus, the methods and apparatus described herein may allow for the formation of a panoramic image with a relatively large field of view from a collection of images acquired for post-processing (e.g., an image soup) or from multiple images obtained from a current (real-time) scan, wherein the scanner includes multiple offset imaging cameras (e.g., in some variations, 6 or more images may be imaged simultaneously), which may be used to generate a panoramic image and / or a new or composite image. For example, in some variations, the scanner is capable of simultaneously scanning multiple images from different offset positions on the scanner (e.g., a scanning gun of the scanner), and as described herein, the scanner may be configured to display a new composite image from a point located between multiple cameras on the scanner (including at a center region); this may be particularly useful in the case where the cameras are offset relative to the scanner (e.g., offset to the side of the scanner). Displaying the image from the tip and / or center of the scanning tool (e.g., a scanning gun) may be more intuitive than displaying one or more images from a camera on the side of the scanning tool.

[0171] Fig.21 An example of a method for generating a composite image at a new camera position is shown. As described above, the method may be included as part of a method or apparatus for generating a panoramic view. As an initial step, a new viewpoint may be identified 2101. For example, a new view direction may be selected from received scan data. Fig.19is an illustration showing the selection of a new view direction 1901 (shown in black) compared to three input images 1903, 1905, 1907 (shown in grey). Three (or in some variants, more, such as five) contributing images 2103 may be selected from the image dataset. For example, the three images may be selected based on camera positions such that the three images are viewing the same scene and their principal ray directions form a triangle that includes the principal ray directions of the desired composite image. This may alternatively be stated as selecting the contributing images as a set of images from a dataset with camera positions that form a triangle that encloses the new viewpoint with a minimal enclosing area (when three images are used). Alternatively or additionally, this may be understood as the set of images having a minimal angular difference relative to the viewpoint angle of the new viewpoint. In some variants, both of these may be true (e.g., having both a minimal area surrounding the camera positions of the new view and pointing in approximately the same direction (having minimal angular difference relative to the new viewpoint)).

[0172] In some variations, the method or apparatus for determining a new view may also require that the contributing images (e.g., contributing images forming a surrounding triangular structure) each have a field of view area of ​​about 10 mm from the camera (e.g., a pinhole camera), and the intersection will be greater than the union of about 0.5 of the field of view areas. For example, Fig. 20 The intersection-over-union (IoU) is shown, which shows an IoU greater than half (0.5). The IoU is an evaluation metric that can be used to measure the accuracy of an object detector on a specific dataset.

[0173] After determining the contribution image from the image dataset, the contribution image can be converted into a multi-plane layer 2105, for example by projecting onto a grid that includes multiple planes parallel to the screen of the new viewing camera. Each point in the screen can correspond to a pixel on the new view image. In some variations, after the contribution image from the input image is projected onto the grid, the contribution image can result in a tensor of size image width x image height x number of planes x 3. Adjacent layers in the converted multi-layer contribution image can then be blended 2107 to form the new view image.

[0174] As described above, in any of these methods and devices, a machine learning agent can be trained and used, for example, to mix layers of contributing images. For example, a machine learning agent can be trained using multiple scanned images received (e.g., from an intraoral scanner) using a "leave one out" approach, in which one image is removed and the camera position of the removed image is used as the true ("ground truth") image when using three neighboring images and trying to predict the removed image.

[0175] FIG. 22A to FIG. 22E and FIG. 23A to FIG. 23E An example of a prototype projection using a trained machine learning agent is shown. For example, FIG. 22A to FIG. 22C An example of three contributing images identified from a dataset including a plurality of scanned images, each having corresponding camera position information, is shown. Each of the three contributing images is shown to have approximately the same viewing angle relative to the new view to be determined (e.g., having a minimum difference relative to the new viewpoint angle) and to form a minimum triangle surrounding the new viewpoint. Each of these contributing images is then converted into multiple parallel planes (e.g., RGBα layers) and blended to form an image at the new viewpoint, as shown in FIG. Fig.22D As shown, the predicted image of the new viewpoint is shown. The predicted image compares favorably with the actual (ground truth) image at that viewpoint, as shown in Fig.22E As shown. FIG. 22A to FIG. 23E Similar results are seen in the example. FIG. 23A to FIG. 23C Three contributing images are shown in Fig.23D The predicted image is shown in . For comparison, Fig.23E The actual (ground truth) image is shown in .

[0176] In these examples, blending is performed using trained machine learning agents (networks), which is both simple and fast. As mentioned above, this technique can be particularly useful for determining images of panoramic views.

[0177] Various alternatives, modifications and equivalents may be used in place of the components described above.While computer-assisted techniques may be used to determine the final position of the teeth, the user may move the teeth to their final position by independently manipulating one or more teeth while satisfying prescription constraints.

[0178] Additionally, the techniques described herein may be implemented in hardware or software or a combination of the two. These techniques may be implemented in a computer program executed on a programmable computer, each programmable computer including a processor, a storage medium (including volatile and non-volatile memory and / or storage elements) that can be read by the processor, and suitable input and output devices. Program code is applied to data input by using an input device to perform the functions and generate output information. The output information is applied to one or more output devices.

[0179] Each program can be implemented in a high-level procedural or object-oriented programming language to operate in conjunction with a computer system. However, if desired, these programs can be implemented in assembly language or machine language. In any case, the language can be a compiled language or an interpreted language.

[0180] Each such computer program may be stored on a storage medium or device (e.g., a CD-ROM, hard disk, or magnetic disk) that can be read by a general or special programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the process. The system may also be implemented as a computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner.

[0181] Although preferred embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that these embodiments are provided as examples only. Without departing from the present invention, those skilled in the art will now appreciate many variations, changes and alternatives. It should be understood that in practicing the present invention, various alternatives to the embodiments of the present invention described herein may be adopted. A variety of different combinations of the embodiments described herein are possible, and these combinations are considered to be a part of the present disclosure. In addition, all features discussed in conjunction with any one of the embodiments herein may be easily applied to other embodiments herein. The appended claims are intended to define the scope of the present invention, and therefore will cover methods and structures and their equivalents within the scope of these claims.

[0182] When a feature or element is referred to as "on" another feature or element in this article, it can be directly located on another feature or element, or there can also be intermediate features and / or elements. On the contrary, when a feature or element is referred to as "directly on" another feature or element, there are no intermediate features or elements. It will also be understood that when a feature or element is referred to as "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to another feature or element, or there can be intermediate features or elements. On the contrary, when a feature or element is referred to as "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intermediate features or elements. Although described or shown with respect to one embodiment, the features and elements described or shown in this way can be applied to other embodiments. Those skilled in the art will also recognize that the reference to a structure or feature set to be "adjacent" to another feature can have a portion overlapping or located below the adjacent feature.

[0183] The terms used herein are used only to describe specific embodiments and are not intended to limit the present invention. For example, as used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any one and all combinations of one or more associated listed items, and may be abbreviated as " / ".

[0184] Spatially relative terms such as "under", "below", "lower", "over", "upper", etc. may be used herein to facilitate description of the relationship of one element or feature relative to another or multiple elements or features as shown in the accompanying drawings. It should be understood that, in addition to the orientations depicted in the accompanying drawings, spatially relative terms are intended to include different orientations of the device in use or operation. For example, if the device in the figure is inverted, the elements described as "under" or "beneath" other elements or features will be oriented to "over" other elements or features. Therefore, the exemplary term "under" can include orientations above and below. The device can be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used in this article can be interpreted accordingly. Similarly, unless otherwise specifically indicated, the terms "upwardly", "downwardly", "vertical", "horizontal", etc. used herein are for the purpose of explanation only.

[0185] Although the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context indicates otherwise. These terms may be used to distinguish one feature / element from another feature / element. Therefore, without departing from the teachings of the present invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0186] Unless the context requires otherwise, throughout the specification and the appended claims, the word "comprise" and variations such as "comprises" and "comprising" mean that various components can be used together in methods and articles (e.g., compositions and apparatus including devices and methods). For example, the term "comprising" will be understood to mean including any stated elements or steps but not excluding any other elements or steps.

[0187] In general, any apparatus and method described herein should be understood to be inclusive, but all or a subset of components and / or steps may alternatively be exclusive and may be expressed as "consisting of" or alternatively "consisting essentially of" the various components, steps, subcomponents or sub-steps.

[0188] As used in this specification and claims, including as used in the examples, unless otherwise expressly specified, all numbers can be interpreted as beginning with the word "about" or "approximately", even if the term does not appear explicitly. When describing size and / or position, the phrase "about" or "approximately" can be used to indicate that the value and / or position described is within the reasonable expected range of the value and / or position. For example, the numerical value can be + / -0.1% of the specified value (or range of values), + / -1% of the specified value (or range of values), + / -2% of the specified value (or range of values), + / -5% of the specified value (or range of values), + / -10% of the specified value (or range of values), etc. Unless the context indicates otherwise, any numerical value given herein should also be understood to include the approximate or approximate value of the value. For example, if the value "10" is disclosed, "approximately 10" is also disclosed. The range of any numerical value described herein is intended to include all sub-ranges contained therein. It should also be understood that when a value is disclosed, values ​​that are "less than or equal to" that value, values ​​that are "greater than or equal to that value," and possible ranges between values ​​are also disclosed, as appropriately understood by those skilled in the art. For example, if a value "X" is disclosed, then "less than or equal to X" and "greater than or equal to X" are also disclosed (e.g., where X is a numerical value). It should also be understood that throughout the application, data is provided in a variety of different formats, and that the data represents endpoints and starting points, as well as ranges for any combination of data points. For example, if a specific data point "10" and a specific data point "15" are disclosed, it can be understood that greater than, greater than or equal to, less than, less than or equal to, equal to 10 and 15, and between 10 and 15 are considered disclosed. It should also be understood that every single number between two specific single numbers is also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0189] Although various illustrative embodiments are described above, any of many changes may be made to the various embodiments without departing from the scope of the invention as described in the claims. For example, in alternative embodiments, the execution order of the various method steps described may be frequently changed, and in other alternative embodiments, one or more method steps may be skipped entirely. Optional features of the various device and system embodiments may be included in some embodiments and not included in other embodiments. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be construed as limiting the scope of the invention as described in the claims.

[0190] The examples and illustrations included herein illustrate specific embodiments by way of illustration and not limitation, in which the subject matter of the present invention can be practiced. As previously mentioned, other embodiments can be utilized and derived therefrom, so that structural and logical replacements and changes can be made without departing from the scope of the present disclosure. These embodiments of the subject matter of the present invention may be referred to herein individually or collectively by the term "invention", for the purpose of convenience only, and if more than one invention or inventive concept is in fact disclosed, it is not intended to voluntarily limit the scope of the present application to any single invention or inventive concept. Therefore, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose may replace the specific embodiments shown. The present disclosure is intended to cover any and all modifications or variations of various embodiments. The combination of the above-mentioned embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art after reading the above description.

Claims

1. A computer-implemented method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch; receiving a plurality of camera positions associated with the plurality of two-dimensional (2D) images; identifying one or more key camera angles for which a panoramic model of the dental arch is to be constructed; For each key camera angle of the one or more key camera angles: identifying a grid of points from which a plurality of approximate images of the dental arch are to be generated; projecting an approximate image for each point in the grid of points from the plurality of two-dimensional (2D) images, the approximate image having a camera position and orientation that best fits the position of each point in the grid of points; processing the approximation image to form a panoramic model; and Displays a panoramic model from a selected key camera angle.

2. The method according to claim 1, further comprising: The plurality of images and camera positions are received from a scan of the dental arch.

3. The method according to claim 1, wherein: Identify the key camera angles, including: representing the panoramic model as at least a portion of a sphere; triangulate the sphere into a plurality of triangles; and The key camera angle is identified as a vector originating from a center of the sphere and terminating at each vertex of the plurality of triangles.

4. The method according to claim 1, wherein: Identifying the point grid, comprising: identifying a central jaw line of the dental arch; dividing the central jaw line into a plurality of segments; and A line is formed at each of the plurality of segments, the line being perpendicular to the central jaw line and the key camera angle.

5. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the processor, cause the processor to perform a computer-implemented method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch; receiving a plurality of camera positions associated with the plurality of two-dimensional (2D) images; identifying key camera angles for which a panoramic model of the dental arch is to be constructed; For each key camera angle: identifying a grid of points from which a plurality of approximate images of the dental arch are to be generated; projecting an approximate image for each point in the grid of points from the plurality of two-dimensional (2D) images, the approximate image having a camera position and orientation that best fits the position of each point in the grid of points; processing the approximation image to form a panoramic model; and Displays a panoramic model from a selected key camera angle.

6. A non-transitory computing device readable medium having instructions stored thereon, the instructions being executable by a processor to cause a computing device to perform a method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch; receiving a plurality of camera positions associated with the plurality of two-dimensional (2D) images; identifying key camera angles for which a panoramic model of the dental arch is to be constructed; For each key camera angle: identifying a grid of points from which a plurality of approximate images of the dental arch are to be generated; projecting an approximate image for each point in the grid of points from the plurality of two-dimensional (2D) images, the approximate image having a camera position and orientation that best fits the position of each point in the grid of points; processing the approximation image to form a panoramic model; and Displays a panoramic model from a selected key camera angle.

7. A method for generating a new view from a plurality of intraoral scan views, the method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch, each two-dimensional (2D) image of the plurality of two-dimensional (2D) images being taken at an associated camera angle and position; identifying a new viewpoint having a camera angle and position relative to the dental arch; identifying three or more contributing images from the plurality of two-dimensional (2D) images, wherein the contributing images have a minimum angle relative to a camera angle of the new viewpoint, and wherein a camera position of the new viewpoint is defined by a triangle formed by the camera positions of the three or more contributing images; converting the contribution image into a multi-planar layer; blending adjacent multi-planar layers of the contributing images to form a new view; and All or part of the new view is displayed.

8. A non-transitory computing device readable medium having instructions stored thereon, the instructions being executable by a processor to cause a computing device to perform a method comprising: receiving a plurality of two-dimensional (2D) images of a dental arch, each two-dimensional (2D) image of the plurality of two-dimensional (2D) images being taken at an associated camera angle and position; identifying a new viewpoint having a camera angle and position relative to the dental arch; identifying three or more contributing images from the plurality of two-dimensional (2D) images, wherein the contributing images have a minimum angle relative to a camera angle of the new viewpoint, and wherein a camera position of the new viewpoint is defined by a triangle formed by the camera positions of the three or more contributing images; converting the contribution image into a multi-planar layer; blending adjacent multi-planar layers of the contributing images to form a new view; and All or part of the new view is displayed.

9. A method for displaying a panoramic view of a dental arch, the method comprising: receiving a plurality of two-dimensional (2D) images of the dental arch, each two-dimensional (2D) image of the plurality of two-dimensional (2D) images being taken at an associated camera angle and position; identifying a viewing angle for viewing a panoramic view of the dental arch; identifying a central jaw line of the plurality of two-dimensional (2D) images, and identifying a plurality of points along the central jaw line; The panoramic view is generated at the viewing angle by: generating, for each of the plurality of points, an image based on one or more of the plurality of two-dimensional (2D) images, the image including the point and having a camera angle corresponding to the viewing angle, wherein generating comprises selecting an image based on one or more of the plurality of two-dimensional (2D) images or synchronizing an image based on one or more of the plurality of two-dimensional (2D) images; as well as Combine selected images; as well as The panoramic view is displayed.

10. A system comprising: one or more processors; as well as a memory coupled to the one or more processors, the memory comprising a non-transitory computing device readable medium having instructions stored thereon, the instructions executable by the one or more processors to perform a method comprising: receiving a plurality of two-dimensional (2D) images of the dental arch, each two-dimensional (2D) image of the plurality of two-dimensional (2D) images being taken at an associated camera angle and position; identifying a viewing angle for viewing a panoramic view of the dental arch; identifying a central jaw line of the plurality of two-dimensional (2D) images, and identifying a plurality of points along the central jaw line; The panoramic view is generated at the viewing angle by: generating, for each of the plurality of points, an image based on one or more of the plurality of two-dimensional (2D) images, the image including the point and having a camera angle corresponding to the viewing angle, wherein generating comprises selecting an image based on one or more of the plurality of two-dimensional (2D) images or synchronizing an image based on one or more of the plurality of two-dimensional (2D) images; and combining selected images; and The panoramic view is displayed.

Images