3D dental scanning system and method with adaptive resolution

Abstract

The present disclosure relates to a computer-implemented method for generating a digital 3D representation of a dental object, the method comprising the steps of: acquiring one or more 2D images of the object; generating point cloud data based on the one or more 2D images, wherein the point cloud data includes a plurality of points representing surface points of the object; and generating mesh data including a plurality of polygons representing the surface of the object based on the point cloud data. The step of generating the mesh data includes: acquiring geometry information based on the one or more 2D images, wherein the geometry information indicates a rate of change of one or more surface characteristics in a plurality of surface sections of the object; determining the grid resolution of each surface section based on the geometric information; and generating a mesh for each surface cross-section using the resolution determined for the respective surface cross-section.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a computer-implemented method for generating a digital 3D representation of a dental object. The present disclosure also relates to a scanning system for generating a digital 3D representation of a dental object. BACKGROUND

[0002] The field of digital dentistry has made significant progress since its inception four decades ago. With the heavy push of CAD / CAM technology, digital dentistry has revolutionized the fabrication of dental restorations. Traditional methods require multiple visits and manual impression taking, which can be uncomfortable for the patient and are prone to inaccuracies. Digital impression taking using intraoral scanners, which do not require a physical mold, can provide a highly accurate 3D representation of a patient’s mouth. These digital models can be used to design and manufacture crowns, bridges, veneers, and other restorations with high accuracy and significantly reduced time consumption.

[0003] Despite these advances, there are still challenges in improving the workflow of the scanning process itself. For many dental procedures, current scanning systems are capable of providing 3D representations with far more resolution than is needed. Rendering high-resolution 3D representations requires additional processing power and lengthens the processing time. To address this issue, some current scanning systems manually select a lower resolution mode in which the 3D representation is generated at a lower resolution, thereby shortening the processing time.

[0004] However, for procedures such as the restoration of missing or damaged teeth, high resolution is essential to ensure that the outer surface of the denture created from the 3D representation can accurately match the patient’s other teeth, and the inner surface of the denture can match the prepared site or implant to which the denture will be attached. For these procedures, the dental practitioner must manually select a higher resolution mode in which the 3D representation is generated at a higher resolution, but this increases the processing time.

[0005] However, in the higher resolution mode, not only the target site is resolved at high resolution, but the entire scan volume is. While this can ensure that important features are resolved correctly, generating a high-resolution mesh of all scan features is slightly overkill. After all, for relatively flat surfaces such as the gum portion or the front of the incisors, a mesh consisting of a large number of polygons does not produce a more accurate representation than a mesh consisting of fewer polygons. Thus, part of the processing power used to generate the high-resolution mesh is wasted, as not all scan parts need to be at high resolution.

[0006] Therefore, there is a need to improve the scanning workflow and optimize the use of processing power in scanning systems. SUMMARY

[0007] The present invention aims to address these challenges by:

[0008] A first aspect of the present disclosure is a computer-implemented method for generating a digital 3D representation of a dental object. The method comprises a step of acquiring one or more sets of 2D images of the object. The method comprises a step of generating point cloud data based on the one or more sets of 2D images, wherein the point cloud data comprises a plurality of points representing a volume of the object. The method comprises a step of generating mesh data based on the point cloud data, the mesh data comprising a plurality of polygons representing a surface of the object. The step of generating the mesh data comprises acquiring, based on the one or more sets of 2D images, geometric information of a plurality of surface sections of the object, wherein the geometric information is indicative of a rate of change of one or more surface properties in the respective surface sections. The step of generating the mesh data comprises determining a mesh resolution for each surface section based on the geometric information of the respective surface section. The step of generating the mesh data comprises generating a mesh for each surface section, wherein the mesh of each surface section comprises a number of polygons corresponding to the mesh resolution determined for the respective surface section.

[0009] A second aspect of the present disclosure is a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of the first aspect.

[0010] A third aspect of the present disclosure is a non-transitory computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of the first aspect.

[0011] A fourth aspect of the present disclosure is a scanning system for generating a digital 3D representation of a dental object. The system comprises one or more projector units configured to project light onto the object. The system comprises one or more camera units configured to acquire one or more sets of 2D images of the object. The system comprises one or more processors. The processor(s) are configured to generate point cloud data based on the one or more sets of 2D images, wherein the point cloud data comprises a plurality of points representing a volume of the object. The processor(s) are configured to generate mesh data based on the point cloud data, the mesh data comprising a plurality of polygons representing a surface of the object. In generating the mesh data, the processor(s) are further configured to acquire, based on the one or more sets of 2D images, geometric information of a plurality of surface sections of the object, wherein the geometric information is indicative of a rate of change of one or more surface properties in the respective surface sections. In generating the mesh data, the processor(s) are further configured to determine a mesh resolution for each surface section based on the geometric information of the respective surface section. In generating the mesh data, the processor(s) are further configured to generate a mesh for each surface section, wherein the mesh of each surface section comprises a number of polygons corresponding to the mesh resolution determined for the respective surface section.

[0012] For all of these aspects, one advantage is that the resolution of the 3D representation is adaptive and selected based on the rate of change of one or more surface properties in the respective surface section. For example, if a certain surface section has many sharp features, a higher resolution will be selected for that section in order to resolve these features more accurately in the final 3D representation. On the other hand, if a section is relatively flat, a lower resolution will be selected so that processing power is not unnecessarily spent to generate a high resolution mesh for a section that can be done with a lower resolution.

[0013] Another advantage is that the resolution mode has to be manually selected by the previous user (i.e. the dentist) according to their needs for a particular dental procedure. Now, the resolution required to accurately represent the respective surface sections will be determined automatically, improving the scanning workflow. BRIEF DESCRIPTION OF DRAWINGS

[0014] The above-mentioned and other features and advantages of this application will be more clearly understood from the following detailed description of exemplary embodiments of the application, taken in conjunction with the accompanying drawings, in which:

[0015] Figure 1a An intraoral scanner for use in the scanning system or method of the present application is shown;

[0016] Figure 1b An intraoral scanner for use in the scanning system or method of the present application is shown;

[0017] Figure 2 A lab scanner for use in the scanning system or method of the present application is shown;

[0018] Figure 3 A block diagram of the scanning system of the present application is shown;

[0019] Figure 4 A block diagram of the scanning system of the present application is shown;

[0020] Figure 5 A scanning session using the scanning system of the present application is shown;

[0021] Figure 6 A scanning session using the scanning system of the present application is shown;

[0022] Figure 7 Point cloud data is shown;

[0023] Figure 8 A 3D representation including a 3D mesh is shown;

[0024] Figure 9a An example of a mesh is shown;

[0025] Figure 9b An example of a mesh is shown;

[0026] Figure 9c An example of a mesh is shown;

[0027] Figure 9d An example of a mesh is shown;

[0028] Figure 10a A 3D representation including mesh data is shown;

[0029] Figure 10b An enlarged portion of a 3D representation including a 3D mesh is shown;

[0030] Figure 10c An enlarged portion of a 3D representation including a 3D mesh is shown;

[0031] Figure 11 A 3D representation including a 3D mesh is shown;

[0032] Figure 12 An initial 3D mesh is shown;

[0033] Figure 13 A flowchart of the method of the invention is shown;

[0034] Figure 14a A 3D representation including mesh data is shown; and

[0035] Figure 14b A 3D representation including mesh data is shown. DETAILED DESCRIPTION

[0036] The detailed description set forth below, in connection with the appended drawings, is intended as a description of various configurations and is not intended to limit the scope of the concepts. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. Numerous aspects of devices, systems, media, programs, and methods are described by various blocks, functional units, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as “elements”). These elements can be implemented using electronic hardware, computer programs, or any combination thereof, depending on the particular application, design constraints, or other reasons.

[0037] A computer-implemented method for generating a digital 3D representation of a dental object is disclosed, the method comprising the steps of:

[0038] • acquiring one or more 2D images of the object;

[0039] • generating point cloud data based on the one or more 2D images, wherein the point cloud data comprises a plurality of points representing surface points of the object; and

[0040] • generating, based on the point cloud data, a 3D representation comprising mesh data, the mesh data having a plurality of polygons representing a surface of the object.

[0041] wherein the step of generating the mesh data comprises:

[0042] o acquiring geometry information based on the one or more 2D images, wherein the geometry information is indicative of a rate of change of one or more surface properties in the plurality of surface sections of the object.

[0043] o determining a mesh resolution for each surface section based on the geometry information.

[0044] o generating a mesh for each surface section at the resolution determined for the respective surface section.

[0045] By dividing the 3D representation into sub-sections representing individual surface sections of the dental object, a mesh can be generated at a mesh resolution that is determined to be suitable for representing the individual surface sections of the dental object. Thus, the mesh can accurately represent the surface of the object without requiring excessive processing power of the scanning system. As a result, the mesh can be generated more quickly, thereby saving time for the user and the patient and improving the user experience.

[0046] At least part of the geometry information related to a respective surface section can be based on a distribution of points of the point cloud data corresponding to the surface section. At least part of the geometry information related to a respective surface section can be based on ray tracing of light reflected by points of the point cloud data corresponding to the surface section. The geometry information can comprise information indicative of a rate of change of a profile in the surface section. Both the surface point distribution and / or the ray tracing can be used to acquire geometry information about the profile of the object, i.e. the shape of the surface of the object. The geometry information can then in turn be used to determine a mesh resolution required to generate a mesh that accurately represents the individual surface sections of the object. Thus, the generated mesh can have a higher mesh resolution in sub-sections representing surface sections with a large change in shape, and a lower mesh resolution in sub-sections representing surface sections that are relatively flat.

[0047] At least part of the geometry information related to a respective surface section can be based on color data corresponding to the surface section. The geometry information can comprise information indicative of a rate of change of a color in the surface section. By basing the geometry information on color data from the 2D images, it can be possible to provide geometry data that is indicative of a change in color of the surface of the object. This in turn can be used to determine a mesh resolution required to accurately represent the change in color in the individual surface sections of the object.

[0048] The step of acquiring one or more 2D images of the object comprises acquiring a plurality of 2D images of the object. The step of generating point cloud data can comprise solving a correspondence problem to determine which parts of a first image of the plurality of 2D images correspond to which parts of a second image of the plurality of 2D images. The step of generating point cloud data comprises registering a plurality of points acquired from the respective 2D images into a common coordinate system.

[0049] The step of generating mesh data comprises assigning a color value to each polygon based on the one or more 2D images. This has the benefit that the 3D representation can not only present the shape of the object, but also the color. This can allow the user to make more detailed diagnoses when inspecting the 3D representation, and it can also allow a prosthesis made based on the 3D representation to match the color of the adjacent teeth.

[0050] The step of acquiring one or more 2D images comprises projecting light onto the dental object. The light can be projected in a pattern. The projected pattern can be time static or time variable. By projecting light in a pattern, it can be easier to solve the correspondence problem when using multiple 2D images taken at different times and / or viewpoints, i.e. pattern features can be used to identify which parts of one 2D image correspond to which parts of another 2D image.

[0051] The step of acquiring one or more 2D images of the object can be performed using an intraoral scanner. The advantage of an intraoral scanner is that it allows a user, e.g. a dentist or a dental technician, to quickly and accurately create a digital 3D representation of one / more teeth of the user without the need to take physical dental impressions that are uncomfortable for the patient.

[0052] The step of acquiring one or more 2D images of the object can be performed using a lab scanner. While intraoral scanners are a faster and more comfortable way of acquiring 2D image(s) of a dental object, they are not available to all dental clinics. Therefore, for users that do not have access to an intraoral scanner, it can be an economically efficient method of digital 3D representation generation to acquire 2D images of a patient’s dental impression using a lab scanner.

[0053] The step of generating point cloud data can comprise performing a first transformation to generate a coordinate dataset of at least a portion of the object in a scanning device coordinate system. The step of generating point cloud data can comprise performing a second transformation on the coordinate dataset of at least the portion of the object in the scanning device coordinate system to generate at least a portion of the point cloud data in a real-world coordinate system.

[0054] The step of generating mesh data can further comprise generating an initial mesh based on the point cloud data, wherein the initial mesh comprises a plurality of initial polygons. The step of generating mesh data is based on the initial mesh.

[0055] The step of acquiring the geometric information of the surface section can comprise comparing normal vectors of adjacent initial polygons within the surface section to acquire information about a profile change rate of the surface section.

[0056] Dental object

[0057] Although only a singular dental object is referred to herein, the dental object can be a set of dental features in the mouth of a subject, i.e. a patient, such as a set of teeth and the surrounding gingiva. Examples of dental objects include one or more of: a tooth / tooths, gingiva, implant(s), dental restoration(s), dental prosthesis, edentulous ridge(s) and / or combinations thereof. Alternatively, the dental object can be a plaster model or a plastic model representing the teeth of the subject. As an example, the dental object can comprise the teeth and / or gingiva of the subject. The dental object can only be a part of the teeth and / or mouth of the subject, as not necessarily the entire set of teeth of the subject is scanned during a scanning session. A scanning session can be understood herein as a period of time during which data, such as 2D images, of the dental object is acquired.

[0058] Scanning device

[0059] The scanning device disclosed herein can be an intraoral scanning device for acquiring images of the mouth of a subject. The intraoral scanning device is a handheld scanning device, i.e. a device configured to be held by a human hand. Alternatively, the scanning device can be a fixed dental scanner, also referred to as a lab scanner, for acquiring images of a plaster model or a plastic model representing the teeth of the subject. The scanning device can employ any suitable scanning principle, such as triangulation-based scanning, stereo vision, motion structure reconstruction, confocal scanning or other scanning principles.

[0060] In some embodiments, the scanning device employs a triangulation-based scanning principle. For example, a projector unit and one or more camera units can be used to determine points in three-dimensional space based on triangulation. In other embodiments, the scanning device employs a focus-based scanning principle. A focus scanning device is further described in EP2442720B1, the entire content of which is incorporated herein.

[0061] Projector unit

[0062] The projector unit can be understood herein as a device configured for projecting light onto a surface, such as a surface of a dental object. In preferred embodiments, the projector unit is configured to project a light pattern onto a surface of a dental object. The projector unit can be configured to project a light pattern such that the light pattern is focused at a predefined focal distance measured along the optical axis of the projector.

[0063] Each projector unit can include one or more light sources. When the light source(s) are turned on / activated, the projector unit can be configured to project a light pattern defined by a plurality of projected light rays. The light source can be configured to produce light of a single wavelength, a distribution of wavelengths (e.g., a Poisson distribution), or a combination of wavelengths (monochromatic or polychromatic). The combination of wavelengths can be produced by a light source configured to produce light containing different wavelengths (e.g., white light).

[0064] In some embodiments, each projector unit includes a light source for producing white light. Alternatively, the projector unit can include a plurality of light sources, e.g., LEDs, that produce light of different wavelengths (e.g., red, green, and blue light) that can be combined to form light including different wavelengths. Thus, the light produced by the light source(s) can be defined by a wavelength defining a particular color or a range of different wavelengths defining a combination of colors (e.g., white light). In some embodiments, the light source is a diode, e.g., a white light diode or a laser diode.

[0065] In some embodiments, the scanning device includes a light source configured for exciting a fluorescent material to acquire fluorescence data from a dental object, e.g., from a tooth. Such a light source can be configured to produce a narrow range of wavelengths. In other embodiments, the scanning device includes an infrared light source configured to produce wavelengths in the infrared range, e.g., between 700 nm and 1.5 pm. In some embodiments, the scanning device includes one or more light sources selected from the group consisting of: an infrared (IR) light source, a near-infrared (NIR) light source, a blue light source, a violet light source, an ultraviolet (UV) light source, and / or combinations thereof. In some embodiments, the scanning device includes a first light source forming part of the projector unit, and one or more second light sources, e.g., infrared light emitting diodes (IR-LEDs) and / or ultraviolet light emitting diodes (UV-LEDs), located in a distal portion of the scanning device, e.g., in a tip of the scanning device.

[0066] The projector unit can include a digital light processing (DLP) projector using a micro-mirror array to generate a time-varying pattern, or a diffractive optical element (DOF), or a front reflective mask projector, or a micro-LED projector, or a liquid crystal on silicon (LCoS) projector, or a backlit mask projector, where the light source is placed behind a mask with a spatial pattern, thereby patterning the light projected onto the surface of the dental object. The pattern can be dynamic, i.e., such that the pattern varies over time, or static, i.e., such that the pattern remains constant over time. The projector unit can include one or more collimating lenses for collimating the light from the light source. The collimating lens(es) can be placed between the light source and the mask. The projector unit can also include one or more focusing lenses or lens elements configured for focusing the light at a predefined focal distance.

[0067] In some embodiments, the projector unit of the scanning device comprises at least one light source and a pattern generating element for defining a light pattern. The pattern generating element is preferably configured for generating a light pattern to be projected onto the surface of the dental object. As an example, the pattern generating element can be a mask having a spatial pattern. Accordingly, the projector unit can comprise a mask for defining a light pattern. The mask can be placed between the light source and one or more focusing lenses of the projector unit such that light transmitted through the mask is patterned into a light pattern. As an example, the mask can define a polygonal pattern comprising a plurality of polygons, e.g. a checkerboard pattern. The projector unit can further comprise one or more lenses, e.g. collimating lenses or projection lenses. In other embodiments, the pattern generating element generates a light pattern based on diffraction and / or refraction, e.g. a pattern comprising a discrete, unconnected array of dots.

[0068] The pattern generating element can be a mask having a spatial pattern, e.g. a transparent or transmissive mask. In other embodiments, the pattern generating element is configured to generate a light pattern with diffraction and / or refraction.

[0069] The spatial pattern can be a polygonal pattern comprising a plurality of polygons. The polygons can be selected from the group consisting of triangles, rectangles, squares, pentagons, hexagons, and / or combinations thereof. Other polygons can also be envisaged. Generally, a polygon is composed of sides and corners. In some embodiments, the polygons reoccur in the pattern in a predefined manner. As an example, the pattern can comprise a plurality of repeating units, wherein each repeating unit comprises a predefined number of polygons, and the repeating units reoccur throughout the pattern. Alternatively, the pattern can comprise a predefined arrangement comprising any of a stripe, a square, a dot, a triangle, a rectangle, and / or combinations thereof. In some embodiments, the pattern is non-coded such that no part of the pattern is unique.

[0070] In some embodiments, the generated light pattern is a polygonal pattern, e.g. a checkerboard pattern comprising a plurality of checkerboard squares. Similar to a common checkerboard, the checkerboard squares in the pattern can have alternating dark and light regions, corresponding to low (dark) and high (bright) light intensity regions, respectively. In some embodiments, the light pattern is a checkerboard pattern comprising alternating dark and light squares. In some embodiments, the light pattern comprises a discrete, unconnected distribution of light dots.

[0071] The pattern preferably comprises a plurality of pattern features. The pattern can be a high-density pattern, which can be understood as a pattern comprising more than 3000 pattern features. However, the system and method of the present disclosure is not limited to high-density patterns, as it is equally applicable to low-density patterns. In some embodiments, the pattern comprises at least 1000 pattern features, or at least 3000 pattern features, or at least 10000 pattern features. When projecting a pattern comprising pattern features onto the surface of a 3D object, the captured object image will likewise comprise a plurality of image features corresponding to the pattern features. A pattern / image feature can be understood as a well-defined single location in the pattern / image. Examples of image / pattern features include corners, edges, vertices, dots, transitions, spots, stripes, etc. In some embodiments, the image / pattern features comprise the corners of the squares in a checkerboard pattern. In other embodiments, the image / pattern features comprise the corners in a polygonal pattern, such as a triangular pattern.

[0072] The projector unit can be configured to generate a predefined static pattern, which can be projected onto the surface of the object. An advantage of using a static pattern is that it enables the simultaneous capture of all image data, thereby preventing distortions due to motion. Another advantage is that a static pattern makes it possible for each camera unit to capture only one image in a set of images, thereby reducing the power consumption, e.g. of the light source.

[0073] Alternatively, the projector unit can be configured to generate a dynamic pattern that varies over time. The projector unit can be associated with its own projector plane, which is determined by the projector optical system. For example, if the projector unit is a backlit mask projector, the projector plane can be understood as the plane in which the mask is contained. The projector plane comprises a plurality of pattern features of the projected pattern. Preferably, the camera unit and the projector unit are arranged such that the image sensor and the projector plane, e.g. defined by the mask, lie in the same plane.

[0074] The projector unit can define a projector optical axis. The optical axis can be understood as a line along which there is a certain degree of rotational symmetry in the optical system, e.g. the camera lens or the projector unit. In some embodiments, the projector optical axis of the projector unit is substantially parallel to the longitudinal axis of the scanning device. In other embodiments, the projector optical axis of the scanning unit forms an angle with the longitudinal axis of the scanning device, e.g. at least 45° or at least 75°. In other embodiments, the projector optical axis of the projector unit is substantially orthogonal to the longitudinal axis of the scanning device.

[0075] Camera unit

[0076] A camera unit can be understood herein as a device for capturing 2D images of a dental object. Each camera unit can comprise an image sensor for generating an image based on incident light, e.g. light received from an illuminated dental object. The image sensor can be an electronic image sensor, e.g. a charge-coupled device (CCD) or an active-pixel sensor (CMOS sensor), for example. In some embodiments, the image sensor is a global shutter sensor configured to expose the entire image area (all pixels) at the same time and generate an image at a single point in time. The image sensor can have an image frame rate of at least 30 frames per second, e.g. at least 60 frames per second, or even at least 90 frames per second.

[0077] The image sensor(s) can comprise a pixel array, wherein each pixel is associated with a respective camera ray. Similarly, each image in the image set can be composed of a pixel array, wherein each pixel contains a pixel color defined by one or more color channels. The pixel array can be a two-dimensional (2D) array. In some embodiments, the image sensor is a CMOS sensor comprising an analog-to-digital converter (ADC) for each column of pixels, thereby significantly shortening the conversion time and benefiting each camera unit with higher speed. Each image sensor can define an image plane, which can be understood as a plane containing a projected image of the object. Each image acquired by the image sensor(s) can comprise a plurality of image features, wherein each image feature originates from a pattern feature of the projection pattern. In some embodiments, one or more camera units comprise a light field camera. Preferably, each camera unit defines a camera optical axis. The camera unit can further comprise one or more focusing lenses for focusing light.

[0078] In some embodiments, the image sensor is a monochrome image sensor, wherein each pixel is associated with a single color channel, e.g. a grayscale color channel, wherein the value of each pixel only represents the amount of light. In other embodiments, the image sensor is a color image sensor or an image sensor comprising a color filter array on the pixel array. As an example, the color filter array can be a Bayer filter, which employs an arrangement of four color filters: red (R), green (G), green (G), and blue (B). The Bayer filter can also be referred to as an RGGB filter. When utilizing the image sensor data, the color pixels can be combined into monochrome pixels of 2x2 color pixels for 3D depth reconstruction. In this case, the resolution of the 3D depth reconstruction is only half of the resolution of the image sensor in each direction. When acquiring texture (color) images, the full original resolution is preferably used (using the color filtered pixels).

[0079] According to some embodiments, the projector optical axis and the camera optical axis of the at least one camera unit define a camera-projector included angle of about 5 to 15 degrees, preferably 5 to 10 degrees, more preferably 8 to 10 degrees. In some embodiments, the camera units are defocused at the probe opening of the scanning device and / or at the optical window surface in the probe. In some embodiments of the scanning device, the camera unit and the projector unit of a given scanning unit are focused at the same distance. In some embodiments, the field of view of each camera unit is 50-115 degrees, such as 65-100 degrees or 80-90 degrees.

[0080] Processor

[0081] According to some embodiments, the scanning device comprises one or more processors selected from the group of: a central processing unit (CPU), an accelerator (offload engine), a general purpose microprocessor, a graphics processing unit (GPU), a neural processing unit (NPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a special-purpose logic circuit, a special-purpose artificial intelligence processor unit, or a combination thereof.

[0082] The scanning device can further comprise a computer memory for storing instructions that, when executed, cause the processor(s) to perform the steps of determining image features in the set of images. The computer memory can further store instructions that, when executed, cause the processor(s) to perform the steps of generating a digital representation of a three-dimensional (3D) object. In general, the processor(s) can be configured to perform any of the computer-implemented methods disclosed herein in whole or in part, e.g. certain processors perform certain method steps while other processors perform other method steps when executed.

[0083] The processor(s) can all be located on the scanning device. Alternatively, the processor(s) or a subset thereof can be located on a computer system as described herein. As an example, a first processor can be configured to determine image features in the images and then provide data related to the determined image features to a second processor. The data can include image feature coordinates as well as other attributes, such as a camera index or other predefined attributes of the image features.

[0084] Computer system

[0085] A computer system can be understood as an electronic processing device for performing a series of arithmetic or logical operations. In the present context, a computer system refers to one or more devices comprising at least one processor (e.g. a central processing unit (CPU)) and some type of computer memory. Examples of computer systems that fit this definition include desktop computers, laptop computers, computer clusters, servers, cloud computing, quantum computers, mobile devices (e.g. smartphones and tablets), and / or combinations thereof.

[0086] A computer system may include hardware such as one or more central processing units (CPUs), graphics processing units (GPUs), and computer memory, such as random access memory (RAM) or read-only memory (ROM). The computer system may include a CPU configured to read and execute instructions stored in computer memory, such as random access memory. The computer memory is configured to store instructions for execution by the CPU and data used by those instructions. For example, the memory may store instructions that, when executed by the CPU, enable the computer system to perform, wholly or partially, any computer-implemented methods disclosed herein. The computer system may also include a graphics processing unit (GPU). The GPU may be configured to perform various tasks, such as video decoding and encoding, rendering of digital representations, and other image processing tasks.

[0087] The computer system may also include non-volatile storage in the form of hard disk drives. Preferably, the computer system also includes an I / O interface configured to connect peripheral devices used in association with the computer system. More specifically, a monitor can be connected and configured to display output from the computer system. For example, the monitor can display a 2D rendering of a generated digital 3D representation. Input devices can also be connected to the I / O interface. Examples of such input devices include a keyboard and a mouse, which allow a user to interact with the computer system. A network interface can also be included as part of the computer system to connect it to a suitable computer network, thereby receiving data (e.g., scan data, subscans, and / or images) from other computing devices and sending data (e.g., scan data, subscans, and / or images) to other computing devices. The CPU, volatile memory, hard disk drives, I / O interfaces, and network interfaces can be interconnected via a bus.

[0088] The computer system is preferably configured to receive data from the scanning device, either directly from the scanning device or via a computer network, such as a wireless network. The data can include images, processed images, sub-scans, point clouds, sets of data points, or other types of data. The data can be transmitted / received using a wireless connection, a wired connection, and / or a combination of both. In some embodiments, the computer system is configured to generate a digital representation of a dental object as described herein. In some embodiments, the computer system is configured to receive data, such as a sub-scan or a point cloud, from the scanning device, and then subsequently perform the steps of reconstructing and rendering a digital representation of the dental object. Rendering can be understood as the process of generating one or more images from three-dimensional data. The computer system can comprise a computer memory for storing a computer program comprising computer-executable instructions which, when executed, cause the computer system to perform a method of generating a digital representation of a dental object.

[0089] Acquiring 2D images

[0090] The methods disclosed herein comprise the step of acquiring or obtaining one or more 2D images. The 2D images can be digital images, such as digital color images. The 2D images can be acquired by the scanning devices disclosed herein, such as by an intraoral scanning device. Each camera unit can contain an image sensor with an array of pixels. Each pixel can be associated with a corresponding camera ray in three-dimensional (3D) space originating from that pixel. Typically, a point positioned along a given camera ray (corresponding to a given pixel) will be imaged on the image sensor of that pixel, which is then read out to generate an image pixel of the 2D image.

[0091] Each 2D image can consist of an array of image pixels, such as an array of two-dimensional image pixels, corresponding to the array of pixels on the image sensor(s). Each image pixel can contain a pixel color defined by one or more color channels. An example of a single color channel is a grayscale color channel, where the value of each pixel only represents the amount of light, thereby carrying intensity information. Thus, the images in the set of images can be grayscale images. Another example of a single color channel is red, green, or blue. Thus, the methods of the present disclosure are not limited to using pixel color information, but can also utilize pixel intensity information.

[0092] The scanning system can be configured to continuously acquire / obtain 2D images at a predetermined frame rate during a scanning session, in which one or more objects, such as the teeth of a patient, are scanned using an intraoral scanning device.

[0093] Generating point cloud data

[0094] A point cloud is a set of discrete data points in space. These points can represent a 3D shape or object, in this case a dental object. Each point position has its own set of Cartesian coordinates (X, Y, Z). Points can contain data other than position, such as RGB color, normal, timestamp, etc. As disclosed above, several principles are known to extract depth information from one or more 2D images, such as the triangulation or focus principle. When the 2D image(s) have been acquired, the depth information is extracted to generate a depth image, in which the image pixels are assigned a depth Z value, effectively converting the 2D image(s) or processed version of the 2D image(s) from 2D information to 3D.

[0095] When scanning a dental object using a scanning device, the captured point clouds generated from the plurality of 2D images each contain scene fragments taken at different times and / or from different positions, and thus need to be aligned to generate a complete map of the scanned dental object. This process is called point set registration (or simply registration or stitching), which aligns the point clouds taken from different 2D images to each other, bringing them into a common coordinate system, thus generating the final point cloud data. The Iterative Closest Point (ICP) algorithm can be used to align two point clouds that overlap each other and are separated by a rigid transformation.

[0096] Surface section

[0097] In general, the more polygons used to represent the surface of the scanned feature (i.e. the higher the resolution), the more detailed the scanned feature can be resolved. However, only when the scanned object has fine features to begin with, can more detailed resolution translate into higher accuracy. Thus, for dental objects with sparse or coarse surface features, e.g. flat surfaces and / or dental objects with generally uniform coloring, a less number of polygons can be sufficient to represent without sacrificing the accuracy of the final 3D representation. On the other hand, dental objects with fine surface features (e.g. sharp edges, textured / rough surfaces, grooves, etc.) and / or diverse coloring, can benefit from being represented by more polygons (i.e. at a higher resolution), as this can enable the mesh to display fine details.

[0098] However, dental objects are rarely composed entirely of either coarse or fine features, but mostly a combination of both. Therefore, according to the present application, the generation of the mesh can advantageously comprise dividing the scan data into a plurality of different surface sections, so that each individual surface section can be treated according to the requirements of the features present at that part of the dental object. As an example, the dental object can be the teeth of a patient's lower jaw, including a restoration site. This can be divided into a plurality of surface sections, wherein surface sections with rich details, such as the surface sections containing the restoration site or the occlusal surface, can be treated with a higher resolution, while surface sections with less details, such as the sides of the teeth and gums, can be treated with a lower resolution, so that the final mesh will be a variable resolution mesh, with higher resolution in the surface sections where it is needed, and lower resolution in the surface sections where less polygons are sufficient.

[0099] Geometric information / data

[0100] In order to determine the resolution required for a particular surface section, it is necessary to extract geometric information from the 2D image(s) and / or from the point cloud data. The geometric information is indicative of the rate of change of one or more surface properties, or in other words, it provides information about how much variation there is of the dental object in a given area, and thus also about how much amount of detail the surface section corresponding to that area has.

[0101] The one or more surface properties can comprise a profile, in which case the geometric information will be indicative of how much or how little curvature the surface in the different surface sections has. The profile can be estimated from the distribution of the points of the point cloud data of a given surface section. A larger spread of points can be indicative of a surface section with a higher curvature and / or a large amount of sharp features, i.e. a surface section with a higher rate of profile variation; while a lower spread of points can be indicative of a surface section with a lower curvature and / or less sharp features, i.e. a surface section with a lower rate of profile variation.

[0102] The one or more surface properties can comprise a color, in which case the geometric information will be indicative of the degree of color variation of the surface in the different surface sections. The color can be estimated from the 2D image and / or the point cloud data of a given surface section. A surface section with a relatively high color variation has a higher rate of color variation, while a surface section with a relatively low color variation has a lower rate of color variation.

[0103] Determining the mesh resolution

[0104] ​After acquiring the geometric information of each surface section, the mesh resolution of each surface section can be determined based on the rate of change of its surface properties (also known as surface features). As mentioned above, a high rate of change of the surface section will result in a higher mesh resolution, while a low rate of change will result in a lower mesh resolution. In this way, each surface section will be rendered at a mesh resolution suitable for the surface features in that surface section of the dental object, without unnecessarily consuming processing power to render the surface section at an excessively high mesh resolution. Besides unnecessary processing power, this also leads to excessive storage / data transfer requirements.

[0105] The mesh resolution can be selected from two or more fixed resolution levels, each associated with a range of rates of change for one or more surface properties. That is, if the rate of change for one or more surface properties falls within the range associated with that resolution level, then a resolution level is selected for the surface section. The two or more fixed resolution levels can include: a first resolution level, for example, approximately 45 polygons per square millimeter, and a second mesh resolution level, for example, approximately 180 polygons per square millimeter. When the rate of change for one or more surface properties falls within the first range, the first resolution level can be determined to be used for the surface section; when the rate of change for one or more surface properties falls within the second range, the second resolution level can be determined to be used for the surface section, where the midpoint of the first range is lower than the midpoint of the second range. Alternatively, the mesh resolution can be selected from a continuous range of resolutions based on geometric information.

[0106] Generate grid data

[0107] Points from point cloud data are used to form a mesh of multiple polygons representing the surfaces of a scanned dental object. These polygons are connected by their common edges (i.e., sides) and vertices (i.e., corners). In a preferred embodiment, the polygons are triangles. Each triangle, and other polygons used for this purpose, has a face, i.e., a surface defined by its edges. For a flat face, such as the face of a triangle, this face has a normal, also called a normal vector, which indicates the orientation of the face in space. Polygons with more than three edges may have non-flat faces, which do not have unique normal vectors, i.e., the normal vectors depend on the position on the face. Methods for generating meshes from point cloud data, such as Delaunay triangulation, are known and can be used for this purpose. Another approach could be to use Poisson reconstruction and construct a volumetric directed distance field from one or more 2D images and / or point clouds, then extract the initial mesh as an isosurface / profile of that directed distance field at zero values ​​(using the moving cube method).

[0108] The number of vertices used in the mesh data will vary and depend on the mesh resolution determined for the individual surface sections. Thus, in surface sections where the geometry information indicates a higher rate of change of one or more surface properties, more vertices are inserted into the mesh data so that the mesh can accurately represent the surface features in these surface sections. Similarly, in surface sections where the geometry information indicates a lower rate of change of one or more surface properties, fewer vertices are inserted into the mesh data so that processing power is not unnecessarily spent on rendering these surface sections at an unnecessarily high resolution.

[0109] Figure 1a An intraoral scanner 100 for the scanning system 700 or method of the present application is shown. The intraoral scanner 100 comprises a distal end 102 configured to be at least partially inserted into a patient's mouth. The distal end 102 comprises a scanning window 150 through which light can be projected onto a dental object 202, 402 to be scanned and an image of the illuminated object can be captured. The distal end 102 is further configured to have a sleeve 110 detachably attached thereto. The sleeve 110 is provided as a hygienic barrier, which can be exchanged between patients to avoid contamination between patients. The sleeve 110 comprises a sleeve window which at least partially overlaps the scanner window 150 when the sleeve 110 is attached to the distal end 102.

[0110] The intraoral scanner 100 further comprises a proximal end 104 opposite the distal end 102. The proximal end 104 can comprise an opening configured for receiving and housing a battery 120, preferably a rechargeable battery, for powering the intraoral scanner 100. Alternatively, the battery 120 can be arranged inside the intraoral scanner 100, in which case the battery can be charged through a power supply interface of the intraoral scanner 100.

[0111] The intraoral scanner 100 comprises a scanner body 106 between the distal end 102 and the proximal end 104, which is configured for being held by a user, e.g. a dentist or a dental technician. In order for the user to control the intraoral scanner 100 during a scanning session, the intraoral scanner 100 comprises a scanner interface 108, which can comprise one or more buttons and / or one or more touch interfaces.

[0112] Figure 1bA cross-sectional view of an intraoral scanner 100 for the scanning system 700 or method of the present application is shown. The shown intraoral scanner 100 comprises a plurality of camera units 140, e.g. four or eight camera units 140, each configured for obtaining a 2D image of the object 202, 402. By having a plurality of camera units 140, the intraoral scanner 100 can obtain a plurality of 2D images substantially simultaneously. The shown intraoral scanner 100 comprises a plurality of projector units 130 configured for projecting light, e.g. patterned light, onto the object 202, 402. The camera units 140 and the projector units 150 are each arranged with their optical axis parallel or slightly angled to the longitudinal axis of the intraoral scanner 100. In order for the camera units 140 and the projector units 150 to be able to view / illuminate the object 202, 402, a mirror 160 is arranged in their optical path to direct it towards the scanner window 150.

[0113] Figure 2 A lab scanner 200 for the scanning system or method of the present application is shown. The intraoral scanner 100 is typically used in a dental clinic, while the lab scanner is mainly used in a dental lab that manufactures or customizes various products for dental clinics. The lab scanner 200 is configured to scan a plastic or plaster model 202 that is molded from the subject’s teeth, the model being arranged within the scanning volume of the lab scanner 200.

[0114] Figure 3 A block diagram of the scanning system 700 of the present application is shown. The scanning system 700 comprises a scanning device, which can be an intraoral scanner 100 or a lab scanner 200, e.g. the devices shown in Figs. 1 and 2. Figure 2 The scanning device 100, 200 comprises one or more projector units 130 configured to illuminate the dental object 202, 402, and one or more camera units 140 configured to capture 2D images of the illuminated dental object 202, 402.

[0115] The scanning device 100, 200 can be operatively connected to a scanning station 300, e.g. a laptop or desktop computer, at least during a scanning session. The scanning station 300 comprises an input device 306, e.g. a mouse, touchpad and / or keyboard, which a user can use to control various parts of the scanning process and / or to post-process the 2D images to generate a 3D representation. The scanning station 300 comprises a display 302, which can be used to display the 3D representation and / or to provide a viewfinder to help the user navigate the intraoral scanner 100 and guide the user so that they can arrange the intraoral scanner 100 relative to the dental object 402.

[0116] To process one or more 2D images captured during a scanning session, the scanning system includes one or more processors 150, 350. The one or more processors 150, 350 may be located in scanning devices 100, 200 and / or scanning station 300. The scanning system 700 also includes computer-readable media 160, 360 containing instructions that, when executed by the scanning system, cause scanning devices 100, 200 and / or scanning station 300 to perform the methods of the present invention. Computer-readable media 160, 360 may be located in scanning devices 100, 200 and / or scanning station 300.

[0117] Figure 4 A block diagram of another scanning system 700 of the present invention is shown. Figure 4 The scanning system 700 and Figure 3 The difference lies in that it also includes a server 800 operatively connected to scanning station 300. Server 800 may be located locally, i.e., close to scanning station 300, and operatively connected to scanning station 300 via a local area network (LAN). Server 800 may be located remotely, i.e., far from scanning station 300, and operatively connected to scanning station 300 via a wide area network (WAN), such as the Internet. Server 800 may be configured to perform at least a portion of the method of the present invention, such as some or all of the steps related to data processing. For this purpose, server 800 includes at least some (if not all) of one or more processors 150, 350, 850, and optionally some or all of computer-readable media 160, 360, 860.

[0118] Typically, one or more processors 150, 350, 850 and / or computer-readable media 160, 360, 860 may reside in one of the scanning devices 100, 200, scanning station 300 or server 800, or they may be distributed in any combination of two or three of these devices.

[0119] Figure 5 and Figure 6 A scanning session using the scanning system of the present invention (including an intraoral scanner 100) is illustrated. During the scanning session, a dentist inserts the distal end 102 into the mouth of a subject 400, with the window facing a dental object 402. The dental object 402 can be a single tooth or a preparation site, or it can be an entire dental arch. The intraoral scanner 100 then moves around, capturing 2D images of the object from multiple perspectives.

[0120] During a scanning session, the intraoral scanner is operatively connected to a scanning station 300 comprising a display 302. During scanning, the display 302 can be used to display a viewfinder 304 showing the current field of view 404 of the intraoral scanner 100 to help the dentist navigate the intraoral scanner 100 within the mouth of the subject 400. After scanning, the display 302 can be used to display a 3D representation for inspection by the dentist.

[0121] Figure 7 A visualization of point cloud data 500 comprising a plurality of points 502 is shown, representing the volume of the dental object 202, 402. The point cloud data 500 is generated by extracting depth information from one or more 2D images acquired during a scanning session. The extraction of depth information is based on the working principle of the scanning device 100, 200. Based on a triangulation scanning system 700, the depth is determined from the projection of a point in 3D space on two or more camera pixels and / or projector pixels, while based on a focus scanning system 700, the depth is determined by linearly sweeping a lens and determining the focal length at which a given point in 3D space is in focus.

[0122] The point cloud data 500 collected from a plurality of vantage points, i.e. positions and / or orientations of the scanner 100, 200 relative to the dental object 202, 402, is registered into a common coordinate system. The registration can be performed using an Iterative Closest Point, ICP, algorithm, which keeps one point cloud (reference or target) fixed while transforming the other point cloud (source) to best match the reference. The transformation, i.e. translation and / or rotation, is estimated in an iterative fashion to minimize an error value, which is typically the sum of squared differences between the coordinates of matching pairs. In the shown figure, the final point cloud data represents a dental object 202, 402, which is a complete dental arch of the patient's lower jaw 400.

[0123] Figure 8 A 3D representation 600 of a dental object 202, 402 is shown, which is a complete dental arch of the patient's lower jaw 400. The 3D representation 600 comprises mesh data, which comprises a plurality of polygons 608, the mesh data representing the surface of the dental object 202, 402 being scanned. For 3D models, triangular polygons have advantages for a number of reasons.

[0124] Triangles are always planar, which means that they lie flat on their respective planes. This simplifies a number of mathematical calculations, such as rendering, shading and collision detection. Linear interpolation on a triangle is very simple, which is important for shading techniques such as Gouraud shading and texture mapping.

[0125] Any surface, no matter how complex or curved, can be approximated with a sufficient number of triangles. This makes triangles highly versatile in modeling various shapes. Triangle meshes can allow for varying levels of detail. Complex surfaces can be represented with a large number of small triangles in detail areas, while fewer but larger triangles are used in areas with less detail. Thus, triangle meshes provide a high degree of geometric flexibility.

[0126] Graphics hardware (GPU) is optimized for triangle processing. Operations such as transformation, lighting computation, and rasterization are very efficient for triangles. Algorithms for mesh operations, such as subdivision, simplification, and deformation, are well established and efficient for triangle meshes. Furthermore, many 3D graphics standards and file formats, such as STL, OBJ, and 3DS, natively support triangle meshes, ensuring wide compatibility across different software and platforms. Most 3D modeling and animation software is designed to handle triangle meshes and provides robust tools to edit and manipulate these meshes. Thus, triangle meshes are more compatible with various hardware / software platforms available to users for working with 3D representation 600.

[0127] Triangles ensure consistent behavior in various operations. For example, polygons 608 with more sides can be non-planar or concave, which complicates computations. Triangle meshes can be easily subdivided to create finer details. Subdivision algorithms such as Loop subdivision can work seamlessly with triangles. Reducing the number of triangles in a mesh while preserving the overall shape and appearance is a well-studied problem, and many efficient algorithms are available. In summary, triangle meshes strike a balance between mathematical simplicity, geometric flexibility, and computational efficiency, making them an ideal choice for representing 3D objects.

[0128] As will be described in greater detail below, the number of polygons 608 used for each surface section of the mesh data will depend on the level of detail of the area of the surface section indicated by the geometric information associated with the respective surface section.

[0129] Figures 9a-9d Examples of meshes according to various embodiments are shown. As described above, mesh data / meshes contain a plurality of polygons 608. A polygon 608 is defined by its corners (i.e., vertices 602) and its edges 604. An area enclosed by a certain number of edges is referred to as a face 606, the number of edges depending on the type of polygon 608 applied, e.g., three edges for a triangle. In various embodiments, there can be several differences in the mesh. Faces 606 can have a consistent number of vertices, or can have different numbers of vertices. The length of edges can be consistent, or edges can have different lengths. Non-boundary vertices can have different numbers of adjacent vertices, or can have a consistent number of adjacent vertices.

[0130] Figure 9a is a mesh according to an embodiment, whose faces have different numbers of vertices 602, different lengths of edges 604 and non-boundary vertices 602 with different numbers of adjacent vertices 602. Figure 9b is a mesh according to an embodiment, whose faces have a consistent number of vertices 602, different lengths of edges 604 and non-boundary vertices 602 with different numbers of adjacent vertices 602.

[0131] Figure 9c is a mesh according to an embodiment, whose faces 606 have a consistent number of vertices 602, a consistent length of edges 604 and non-boundary vertices 602 with a consistent number of adjacent vertices 602. Figure 9d is a mesh according to an embodiment, whose faces 606 have a consistent number of vertices 602, a consistent length of edges 602 and non-boundary vertices 602 with a consistent number of adjacent vertices 602. Different meshes have different functions, but in representing 3D objects, for the reasons mentioned above, meshes with triangular faces 606 (e.g. Figure 9b ) are typically used.

[0132] Each polygon 608 can be assigned a color value, so that the mesh data not only represents the shape of the dental object 202, 402 that was scanned, but also its color. This is very advantageous when deriving dental prostheses from the 3D representation 600, e.g. when deriving a dental bridge, a dental crown and / or a veneer, because the prosthesis(es) can be color-matched to other teeth of the patient, e.g. to the neighboring teeth of the restoration site where the prosthesis is to be placed.

[0133] Figure 10a - c shows an example of a 3D representation 600 generated according to the present application. The 3D representation 600 represents a dental object 202, 402, which is a set of teeth and gums. The mesh data used for this 3D representation comprises a plurality of triangles, wherein the resolution used for the individual surface sections of the mesh, i.e. the number of polygons 608, varies depending on the different degrees of detail.

[0134] Figure 10b A first region 610 of the mesh data is shown. The first region 610 contains the lingual surface of the incisors, which is a relatively flat surface, wherein the degree of detail / variation of the surface properties is relatively low. Accordingly, the resolution of the surface section 612 that is appropriate for this region 610 is also low. Accordingly, the mesh generated in the surface section 612 of this region 610 comprises fewer polygons 608 per unit area, because this is sufficient to resolve the relatively coarse features of the first region 610 without expending excessive processing power to generate superfluous polygons 608. Figure 10cA second region 620 is shown displaying the mesh data. The second region 620 contains the incisal surface of the tooth, which is a surface type having a greater curvature, and therefore a relatively higher degree of detail in the surface properties. Accordingly, the resolution of the surface sections determined to be appropriate for this region 620 is also higher. Therefore, the mesh generated in the surface sections 614 of this region 620 contains more polygons 608 per unit area, enabling the mesh to accurately represent the fine features. Figure 10c and Figure 10b Both have been magnified by the same factor to facilitate comparison. By comparing the two, it can be easily seen that the relatively flat lingual surface of the tooth located in the first region 610 is resolved at a lower resolution, resulting in a lower density of polygons 608 than the incisal surface located in the second region 620.

[0135] Figure 11 Another 3D representation 600 of a dental object 202, 402 is shown, which is a molar tooth of the patient 400. As can be seen from the magnified view on the right, the relatively flat side surfaces have been determined to have a lower rate of change in surface properties, in this case a lower profile rate of change, and therefore a lower mesh resolution has been determined to be appropriate for these surface sections. On the other hand, the occlusal surface has been determined to have a higher rate of change in surface properties, in this case a higher profile rate of change, and therefore a higher mesh resolution has been determined to be appropriate for these surface sections.

[0136] In order to determine the mesh resolution required for a surface section, the geometric information of that surface section must first be acquired. The geometric information indicates the rate of change of one or more surface properties, such as the profile and / or colour, and therefore provides the information required to determine the mesh resolution required for the mesh of the 3D representation 600 to accurately represent the surface of the dental object 202, 402 in that surface section.

[0137] The geometric information can be acquired from one or more 2D images and / or point cloud data 500, depending on the surface properties required. If colour information is required, this can be acquired from the colour information in the 2D images and / or from the colour information in the point cloud data extracted from the 2D image(s). If profile information is required, this can be acquired from the point cloud data 500. The profile information can be acquired by comparing the surface gradients of adjacent points, for example by performing ray tracing to determine the way in which light from one or more projector units 130 is reflected at a point in the point cloud data and comparing this to the way in which the light is reflected at its neighbouring points; or by determining how tightly or widely distributed the points are in the surface section, i.e. widely distributed points indicate a greater degree of change in the surface profile, whereas tightly distributed points indicate a lesser degree of change in the surface profile.

[0138] Figure 12An initial mesh containing a plurality of initial polygons generated based on the point cloud data 500 is shown. The initial mesh is generated prior to generating a mesh of a 3D representation at a predetermined initial resolution. The initial mesh can be used to obtain at least a portion of the geometric information. This can be achieved by deriving an error of fit measure for the initial mesh, which indicates how well the initial polygon fits the points in the point cloud data 500 used to generate it. If the points are highly distributed around the initial polygon, the geometric information will show that it has a poor error of fit; if the points are tightly distributed around the initial polygon, the geometric information will show that it has a good error of fit. A poor error of fit means that a higher mesh resolution should be used preferentially, while a good error of fit means that a lower mesh resolution is sufficient.

[0139] Figure 13 A flowchart of a computer-implemented method of the present application is shown. The method begins with a user initiating a scan session 900. After initiation 900, a step of obtaining / acquiring one or more 2D images 902 is performed. The step of obtaining / acquiring one or more 2D images 902 can be performed using an intraoral scanner 100 or a lab scanner 200 as described above.

[0140] After the step of obtaining / acquiring one or more 2D images 902, a step of generating point cloud data 904 is performed. The point cloud data 500 comprises a plurality of points representing surface points of the object 202, 402, which are extracted from the one or more 2D images or derived data therefrom. If a plurality of 2D images are acquired from different viewpoints, i.e. different positions and / or orientations of the scanning device 100, 200 relative to the dental object 202, 402, the step of generating point cloud data 904 can comprise registering points derived from the different 2D images into a common coordinate system. The method can subsequently comprise a step of generating an initial mesh 906 based on the point cloud data 500. The initial mesh is generated at a predetermined initial resolution and contains a plurality of initial polygons.

[0141] The method subsequently comprises a step of obtaining geometric information 908. The geometric information can be based on the one or more 2D images, the point cloud data 500 and / or the initial mesh. As described above, the geometric information indicates a rate of change of one or more surface properties in a plurality of surface sections of the object 202, 402, thereby indicating how much surface detail is present in each surface section and, in turn, indicating which mesh resolution is required to accurately represent each surface section of the dental object 202, 402.

[0142] The method then comprises a step of determining a mesh resolution for each surface section based on the geometry information 910. If the geometry information indicates that a surface section has rich surface details, e.g. large color variations with a complex profile, it will be determined that a higher mesh resolution is needed. On the other hand, if the geometry information indicates that a surface section lacks surface details, e.g. has a relatively flat profile with small color variations, it will be determined that a higher mesh resolution is sufficient. After determining the mesh resolution for each surface section, the method proceeds to a step of generating mesh data 912, which contains a plurality of polygons 608 representing the surface of the object 202, 402. How many polygons 608 are generated for each surface section depends on the mesh resolution needed for that surface section, determined in the previous step 910.

[0143] The method can then comprise a step of assigning a color value to each polygon 914. The color value can be based on one or more 2D images and can be determined based on the average color of the surface of the object 202, 402 in the area associated with the respective polygon 608. The mesh generated in step 912 and, optionally, step 914, will provide a 3D representation of the dental object 202, 402. Finally, the method can comprise a step of outputting the 3D representation 916. Outputting the 3D representation 916 can be performed using the display 302 of the scanning system 700, thereby allowing a user to inspect the dental object 202, 402 and determine the dental health condition of the patient 400.

[0144] Figure 14a and Figure 14b More examples of 3D representations generated by the method of the invention are shown. As in the case of Figures 10a-10c the surface sections at the occlusal surface and the edges connecting the occlusal surface and the side surface of the tooth have surface properties with high variation rates, e.g. profile variation rates, and thus require a higher mesh resolution to properly display these surface properties. On the other hand, the surface sections of the side surface and the gum have surface properties with smaller variations, e.g. profile, and thus require a lower mesh resolution to sufficiently display these surface properties. According to the invention, the mesh resolution has been adaptively determined so as to generate parts of the mesh at a higher resolution when needed, and at a lower resolution when a lower resolution is sufficient.

[0145] Further details

[0146] Embodiments of the invention are disclosed in the following list of items:

[0147] 1. A computer-implemented method for generating a digital 3D representation of a dental object, the method comprising the steps of:

[0148] • acquiring one or more 2D images of the object;

[0149] • generating point cloud data based on the one or more 2D images, wherein the point cloud data comprises a plurality of points representing surface points of the object; and

[0150] • generating mesh data based on the point cloud data, the mesh data comprising a plurality of polygons representing the surface of the object.

[0151] 2. The method of item 1, wherein the step of generating mesh data comprises:

[0152] • obtaining geometry information based on the one or more 2D images, wherein the geometry information is indicative of a rate of change of one or more surface properties in a plurality of surface sections of the object.

[0153] 3. The method of item 2, wherein the step of generating mesh data comprises:

[0154] • determining a mesh resolution for each surface section based on the geometry information.

[0155] 4. The method of item 3, wherein the step of generating mesh data comprises:

[0156] • generating a mesh for each surface section with the resolution determined for the respective surface section.

[0157] 5. The method of any one of items 2-4, wherein at least part of the geometry information related to a respective surface section is based on a distribution of points of the point cloud data corresponding to that surface section.

[0158] 6. The method of any one of items 2-5, wherein at least part of the geometry information related to a respective surface section is based on performing ray tracing of light reflected by points of the point cloud data corresponding to that surface section.

[0159] 7. The method of item 5 or 6, wherein the geometry information comprises information indicative of a rate of change of a profile of the surface section.

[0160] 8. The method of any one of items 2-7, wherein at least part of the geometry information related to a respective surface section is based on color data corresponding to that surface section.

[0161] 9. The method of item 8, wherein the geometry information comprises information indicative of a rate of change of a color of the surface section.

[0162] 10. The method of any one of the preceding items, wherein the step of obtaining one or more 2D images of the object comprises obtaining a plurality of 2D images of the object.

[0163] 11. The method of item 10, wherein the step of generating point cloud data comprises solving a correspondence problem to ascertain which portions of a first image of the plurality of 2D images correspond to which portions of a second image of the plurality of 2D images.

[0164] 12. The method of item 10 or 11, wherein the step of generating point cloud data comprises registering points from the plurality of points taken from each respective 2D image into a common coordinate system.

[0165] 13. The method of any of the preceding items, wherein the step of generating mesh data comprises assigning a color value to each polygon based on the one or more 2D images.

[0166] 14. The method of any of the preceding items, wherein the step of taking one or more 2D images comprises projecting light onto the dental object.

[0167] 15. The method of item 14, wherein the light is projected in a pattern.

[0168] 16. The method of item 15, wherein the projected pattern is time static.

[0169] 17. The method of item 15, wherein the projected pattern is time variable.

[0170] 18. The method of any of the preceding items, wherein the step of taking one or more 2D images of the object is performed using an intraoral scanner.

[0171] 19. The method of any of items 1-17, wherein the step of taking one or more 2D images of the object is performed using a lab scanner.

[0172] 20. The method of item 18 or 19, wherein the step of generating point cloud data comprises performing a first transformation to generate a dataset of coordinates of at least a portion of the object in a scanning device coordinate system.

[0173] 21. The method of item 20, wherein the step of generating point cloud data comprises performing a second transformation on the dataset of coordinates of at least a portion of the object in the scanning device coordinate system to generate at least a portion of the point cloud data in a real-world coordinate system.

[0174] 22. The method of any of the preceding items, wherein the step of generating mesh data further comprises:

[0175] o generating an initial mesh based on the point cloud data, wherein the initial mesh comprises a plurality of initial polygons.

[0176] 23. The method of item 22, wherein the step of generating mesh data is based on the initial mesh.

[0177] 24. The method of item 22 or 23, wherein the step of obtaining geometric information of the surface section comprises:

[0178] o comparing normal vectors of adjacent initial polygons within the surface section to obtain information about a rate of profile change of the surface section.

[0179] 25. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of the preceding items.

[0180] 26. A non-transitory computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any of items x-x.

[0181] 27. A scanning system for generating a digital 3D representation of a dental object, the system comprising:

[0182] • a scanning device comprising:

[0183] o one or more projector units configured to project light onto the object; and

[0184] o one or more camera units configured to obtain one or more 2D images of the object;

[0185] • one or more processors, wherein the processors are configured to:

[0186] o generate point cloud data based on one or more sets of 2D images, wherein the point cloud data comprises a plurality of points representing surface points of the object; and

[0187] o generate mesh data comprising a plurality of polygons representing the surface of the object based on the point cloud data.

[0188] 28. The scanning system of item 27, wherein the processor(s) are configured to:

[0189] o obtain geometric information based on the one or more 2D images, wherein the geometric information is indicative of a rate of change of one or more surface properties in a plurality of surface sections of the object.

[0190] 29. The scanning system of item 28, wherein the processors are configured to:

[0191] o determine a mesh resolution for each surface section based on the geometric information.

[0192] 30. The scanning system of item 29, wherein the processor is configured to:

[0193] o generate a mesh for each surface section with a resolution determined for the respective surface section.

[0194] 31. The scanning system of any one of items 28-30, wherein at least part of the geometric information related to a respective surface section is based on a distribution of the points of the point cloud data corresponding to that surface section.

[0195] 32. The scanning system of any one of items 28-31, wherein at least part of the geometric information related to a respective surface section is based on performing ray tracing of light reflected at points of the point cloud data corresponding to that surface section.

[0196] 33. The scanning system of item 31 or 32, wherein the geometric information comprises information indicative of a rate of profile variation of the surface section.

[0197] 34. The scanning system of any one of items 28-33, wherein at least part of the geometric information related to a respective surface section is based on color data corresponding to that surface section.

[0198] 35. The scanning system of item 34, wherein the geometric information comprises information indicative of a rate of color variation of the surface section.

[0199] 36. The scanning system of any one of items 28-35, wherein the one or more camera units are configured to acquire a plurality of 2D images of the object.

[0200] 37. The scanning system of item 36, wherein the processor(s) is configured to:

[0201] o solve a corresponding problem to ascertain which parts of a first image of the plurality of 2D images correspond to which parts of a second image of the plurality of 2D images.

[0202] 38. The scanning system of item 36 or 37, wherein the processor(s) is configured to:

[0203] o register points of the plurality of points acquired from each respective 2D image into a common coordinate system to generate the point cloud data.

[0204] 39. The scanning system of any one of items 28-38, wherein the processor(s) is configured to:

[0205] o assigning a color value to each polygon based on the one or more 2D images.

[0206] 40. The scanning system of any of items 28-39, wherein the light is projected in a pattern.

[0207] 41. The scanning system of item 40, wherein the projected pattern is time static.

[0208] 42. The scanning system of item 40, wherein the projected pattern is time variable.

[0209] 43. The scanning system of any of items 28-42, wherein the scanning device is an intraoral scanner.

[0210] 44. The scanning system of any of items 28-42, wherein the scanning device is a lab scanner.

[0211] 45. The scanning system of any of items 28-44, wherein the processor(s) are configured to:

[0212] o perform a first transformation to generate a dataset of coordinates of at least a portion of the object in a scanning device coordinate system when generating the point cloud data.

[0213] 46. The scanning system of item 45, wherein the processor(s) are configured to:

[0214] o perform a second transformation to the dataset of coordinates of at least a portion of the object in a scanning device coordinate system to generate at least a portion of the point cloud data in a real world coordinate system when generating the point cloud data.

[0215] 47. The scanning system of any of items 28-46, wherein the processor(s) are configured to:

[0216] o generate an initial mesh based on the point cloud data, wherein the initial mesh comprises a plurality of initial polygons.

[0217] 48. The scanning system of item 47, wherein the processor(s) are configured to:

[0218] o generate the mesh data based on the initial mesh.

[0219] 49. The scanning system of item 47 or 48, wherein the processor(s) are configured to:

[0220] o Compare normal vectors of adjacent initial polygons within a surface section to obtain information about a profile change rate of the surface section.

[0221] List of reference signs

[0222] 100 intraoral scanner 500 point cloud data / point cloud

[0223] 102 distal end 502 points

[0224] 104 proximal end 600 mesh data / mesh

[0225] 106 scanner body 602 vertex / vertices

[0226] 108 scanner interface 604 edge(s)

[0227] 110 sleeve 606 polygon(s)

[0228] 120 battery 608 polygon(s)

[0229] 130 projector unit 610 first region

[0230] 140 camera unit 612 surface section

[0231] 150 processor 614 surface section

[0232] 160 computer readable medium 620 second region

[0233] 170 scanner window 700 scanning system

[0234] 180 mirror 800 server

[0235] 200 lab scanner 850 processor

[0236] 202 dental object 860 computer readable medium

[0237] 300 scanning station 900 initiate a scanning session

[0238] 302 display 902 obtain / acquire one or more 2D images 304 viewfinder 904 generate point cloud data

[0239] 306 input device 906 generate initial mesh

[0240] 350 processor 908 obtain geometric information

[0241] 360 computer readable medium 910 determine mesh resolution

[0242] 400 subjects / patients 912 generate mesh data

[0243] 402 dental object 914 assign color values to each polygon

[0244] 404 current field of view 916 output 3D representation

[0245] The use of the terms “first,” “second,” “third,” “fourth,” “primary,” “secondary,” “tertiary,” etc. does not necessarily indicate any particular order or importance, but is used to distinguish one element from another. It is noted that the terms “first,” “second,” “third,” “fourth,” “primary,” “secondary,” “tertiary,” etc. are used herein and elsewhere merely to identify one element from another, and do not necessarily indicate any particular order or timing. Further, the use of the terms “first,” “second,” “third,” “fourth,” “primary,” “secondary,” “tertiary,” etc. does not mean that a second element must exist, or that a second element is dependent on the existence of a first element.

[0246] It is noted that the word “comprising” does not necessarily exclude the presence of additional elements or steps than those listed. It is further noted that the word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements.

[0247] It is further noted that any references to claim elements in the singular, for example, using the articles “a,” “an,” “the,” etc., are not to be construed as limiting of the number of elements to the single figure unless it is specifically called a “single” element or “only one” element. Further, it is to be understood that other number of elements than those specifically recited can be utilized without departing from the scope of the current embodiments.

[0248] It is to be understood that the use of “one embodiment,” “an embodiment,” “aspect,” or “the like” containing features, structures, or characteristics also contemplates the various features, structures, or characteristics of the described embodiments combining with one another in any manner. The foregoing description is provided for the purpose of explanation and is not intended to be exhaustive or to be limited to the precise embodiments disclosed. Modifications and variations are possible in light of the above teachings or can be acquired from practice of the various embodiments. The claims and representations, if any, are intended to cover all modifications and alterations.

[0249] While embodiments and features have been illustrated and described, it is understood that they are not intended to limit the application. It is understood that changes and modifications can be made by those skilled in the art, departing from the spirit of the application and without detracting from the scope of the application. The description and drawings are therefore to be regarded as illustrative in nature and not restrictive. The claimed application is intended to cover all alternatives, modifications and equivalents.

Claims

1. A computer-implemented method for generating a digital 3D representation of a dental object, the method comprising the steps of: • Obtain one or more 2D images of the object; • Generate point cloud data based on the one or more 2D images, wherein the point cloud data includes a plurality of points representing surface points of the object; and • Generate mesh data based on the point cloud data, the mesh data including multiple polygons representing the surface of the object. The characteristic feature is that the step of generating grid data includes: ○ Obtain geometric information based on the one or more 2D images, wherein the geometric information indicates the rate of change of one or more surface properties in a plurality of surface cross-sections of the object; ○ Determine the mesh resolution for each surface section based on the aforementioned geometric information; and ○ Generate a mesh for each surface section using the resolution determined for the corresponding surface section.

2. The method according to claim 1, characterized in that, At least a portion of the geometric information associated with the corresponding surface cross section is based on the distribution of points in the point cloud data corresponding to that surface cross section.

3. The method according to claim 1 or 2, characterized in that, At least a portion of the geometric information associated with the corresponding surface section is based on ray tracing of light reflected at points in the point cloud data corresponding to the surface section.

4. The method according to any one of the preceding claims, characterized in that, The geometric information includes information indicating the rate of change of the profile of the surface cross section.

5. The method according to any one of the preceding claims, characterized in that, At least a portion of the geometric information associated with the corresponding surface section is based on the color data corresponding to that surface section.

6. The method according to any one of the preceding claims, characterized in that, The geometric information includes information indicating the rate of color change of the surface cross-section.

7. The method according to any one of the foregoing items, characterized in that, The steps for generating mesh data include assigning color values ​​to each polygon based on the one or more 2D images.

8. The method according to any one of the foregoing items, characterized in that, The one or more 2D images of the object were acquired by an intraoral scanner.

9. The method according to any one of claims 1-7, characterized in that, The one or more 2D images of the object were acquired by a technician's scanner.

10. The method according to any one of the preceding claims, characterized in that, The steps for generating grid data also include: ○ Generate an initial mesh based on the point cloud data, wherein the initial mesh includes multiple initial polygons.

11. The method according to claim 10, characterized in that, The steps for generating grid data are based on the initial grid.

12. The method according to claim 10 or 11, characterized in that, The steps to obtain the geometric information of the surface cross-section include: ○ Determine the fitting error between each initial polygon in the initial polygon and its neighboring points in the point cloud data.

13. The method according to any one of claims 10-12, characterized in that, The steps to obtain the geometric information of the surface cross-section include: ○ Compare the normal vectors of adjacent initial polygons within the surface cross section to obtain information about the rate of change of the surface cross section profile.

14. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method described in any of the preceding claims.

15. A non-transitory computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-13.

Images