System and method for improving digital 3D surfaces

Abstract

The present disclosure relates to a system and method for improving or refining the accuracy and / or level of detail of a 3D surface based on pixel data from a set of images, such as color data of pixels. In particular, the present disclosure relates to a 3D scanner system comprising an intraoral 3D scanner, comprising: a projector unit comprising a light source and a pattern generating element for structuring light from the light source into a pattern to be projected onto a surface of an object; and two or more cameras configured to acquire a set of images, where the set of images includes images from each camera, where each image includes an array of pixels, each pixel having a pixel color; the scanner system further includes one or more processors operatively connected to the intraoral 3D scanner configured to: generate a three-dimensional (3D) surface of the object based on the set of images obtained from the camera, where the 3D surface includes a plurality of points and / or vertices; and generating a refined 3D surface by solving the optimization problem wherein points and / or vertices in the 3D surface are relocated such that for each point or vertex, a measure based on differences in pixel colors associated with that point or vertex will be minimized.

Description

Technical Field

[0001] This disclosure relates to a system and method for generating three-dimensional (3D) surfaces of objects. In particular, this disclosure relates to a system and method for improving or refining three-dimensional (3D) surfaces, for example, in terms of accuracy or level of detail. Background Technology

[0002] 3D scanning is the process of analyzing real-world objects or environments to collect three-dimensional data of their shape and possible appearance (such as color). The collected data can then be used to construct digital 3D models. 3D scanners can be based on many different technologies, such as depth of focus, defocus depth, triangulation, stereo vision, optical coherence tomography (OCT), motion structures, time-of-flight, and others.

[0003] Typically, a digital 3D model is reconstructed from multiple 3D representations or surfaces, sometimes called subscans. These subscans are brought into a common reference frame—a process commonly referred to as alignment or registration—and then merged to create a complete 3D model of the scanned object. Subscans are usually obtained from different perspectives of the object. The entire process from a single subscan to a complete 3D model is sometimes referred to as the 3D scan pipeline.

[0004] Sometimes, deviations can occur in individual 3D surfaces or sub-scans. An example of such deviation might be in depth; for instance, if the determined depth of an object is incorrect and deviates from baseline truth. This could mean that some areas of the 3D surface are incorrect in terms of depth, potentially leading to an inaccurate 3D model.

[0005] Therefore, there is interest in developing an improved system and method for refining 3D surfaces, for example, by removing the aforementioned biases, so that multiple such refined 3D surfaces can be merged to generate a more accurate digital 3D model. Furthermore, it is desirable to improve the level of detail of 3D surfaces. High accuracy and / or high level of detail are desired, for example, in many applications within digital dentistry, where digital impressions of human teeth are generated. Summary of the Invention

[0006] This disclosure addresses the aforementioned challenges by providing a system and method for improving or refining the accuracy and / or level of detail of 3D representations, such as 3D surfaces, based on pixel data, such as pixel color data, from images captured by two or more cameras. Specifically, this disclosure addresses and resolves the aforementioned challenges by providing a 3D scanner system comprising: - Intraoral 3D scanner, including: - A projector unit, the projector unit including a light source and a pattern generating element for structuring light from the light source into a pattern to be projected onto the surface of an object such as at least a portion of a human tooth; - Two or more cameras, operatively connected to the projector unit, the cameras being configured to acquire a set of images, wherein the set of images includes images from each camera, wherein each image includes a pixel array, and each pixel has a pixel color. c i ; - One or more processors, operatively connected to the intraoral 3D scanner, the processors being configured to: - Generate a three-dimensional (3D) surface of the object based on the set of images obtained from the camera, wherein the 3D surface includes a plurality of points and / or vertices; and - A refined 3D surface is generated by solving an optimization problem, wherein points and / or vertices in the 3D surface are repositioned such that, for each point or vertex, the pixel color associated with that point or vertex is used as the basis for the final resolution. c i The difference minimizes the metric.

[0007] According to some embodiments, the 3D scanner system includes: - Intraoral 3D scanner, including: - A projector unit, the projector unit including a light source and a pattern generating element, the pattern generating element being used to structure light from the light source into a pattern to be projected onto the surface of an object; - Two or more cameras, operatively connected to the projector unit, the cameras being configured to acquire a set of images, wherein the set of images includes images from each camera; - One or more processors, operatively connected to the intraoral 3D scanner, the processors being configured to: - Generate the three-dimensional (3D) surface of the object based on the set of images obtained from the camera; and - Generate camera rays emanating from each pixel in each image within the set of images and projecting them onto the 3D surface, thereby generating a plurality of camera rays incident on the 3D surface; and - A virtual image is generated by projecting each of the camera rays from a corresponding incident point on the 3D surface onto the projector image plane.

[0008] This disclosure further relates to a computer-implemented method, the computer-implemented method comprising the following steps: - Generate a three-dimensional (3D) surface of an object, wherein the 3D surface is generated based on a set of images obtained from two or more cameras, wherein the set of images includes at least one image from each of the cameras, wherein each image includes an array of pixels, and each pixel has a pixel color. c i ; - Generate camera rays emitted from each pixel to the 3D surface for each image, thereby generating multiple camera rays incident on the 3D surface; - A projector image is generated for each image by projecting each of the camera rays from a corresponding incident point on the 3D surface onto a predefined projector image plane, wherein the projector image comprises a plurality of projector image pixels, each projector image pixel having a color based on the pixel color. c i pixel color c p ; - Determine the color representing each pixel in the projector image. c p A measure of the difference across the projector images; and - A modified 3D surface is generated by translating one or more points and / or vertices of the 3D surface such that one or more cost functions associated with the projector image pixels are minimized.

[0009] According to some embodiments, the computer-implemented method includes the following steps: - Generate a three-dimensional (3D) surface of an object, wherein the 3D surface is generated based on a set of images obtained from two or more cameras, wherein the set of images includes at least one image from each of the cameras, wherein each image includes an array of pixels, and each pixel has a pixel color. c i ; - Define the projector image plane with a predefined pattern; - Generate camera rays emitted from each pixel in each image within the group of images to the 3D surface, thereby generating multiple camera rays incident on the 3D surface; - Generate a projector image that includes multiple projector image pixels, each with a pixel color. c p The projector image is generated by projecting each of the camera rays from a corresponding incident point on the 3D surface onto the projector image plane. - Determine the pixel color for each projector image pixel. c p Contributing pixel color c i The variance; and - Modify the 3D surface by changing the position of one or more points and / or vertices of the 3D surface such that one or more cost functions associated with the projector image pixels are minimized.

[0010] Computer-implemented methods can be implemented on the 3D scanner system disclosed herein. Accordingly, the 3D scanner system may include one or more processors configured to perform, wholly or partially, any of the computer-implemented methods embodied herein.

[0011] Given a 3D representation of a real-world 3D object, such as a 3D surface, the currently disclosed systems and methods provide a framework for modifying and refining the 3D representation based on image data from multiple images, for example, by comparing the pixel colors of pixels associated with similar 3D points in the representation. In a preferred embodiment, this is achieved by first obtaining the 3D surface of the object. The 3D surface can be provided as input to the currently disclosed computer-implemented methods and / or can be generated by a 3D scanner system as disclosed herein. As an example, an intraoral 3D scanner comprising two or more cameras can be used to generate a 3D representation based on a set of images acquired by the cameras. The 3D representation can be provided to a 3D scanner system comprising one or more processors configured to generate the 3D surface based on the 3D representation and / or based on the acquired set of images. In some embodiments, the 3D surface is represented as a grid or a signed distance field.

[0012] A 3D surface can be generated based on a set of images, including at least one image from each camera used to acquire the set. Points on the 3D surface can be determined by triangulation, i.e., by triangulating defined image features in the images in 3D space and determining the intersections of these image features with projector rays corresponding to pattern features projected or traced in 3D space. Therefore, an intraoral 3D scanner can include a projector unit configured to project a pattern onto the surface of the scanned 3D surface. Typically, the 3D resolution of the reconstructed 3D surface, i.e., the number of 3D points on the surface, is related to the density of the projected pattern, i.e., the number of pattern features. In some cases, high-density patterns are projected, such as patterns including at least 3000 pattern features. Patterns with a large number of pattern features typically increase the complexity of the correspondence problem, i.e., the problem of associating each image feature with a projector ray. This can lead to ambiguity in determining 3D points through triangulation. This ambiguity can further lead to the aforementioned deviations in depth.

[0013] Given a 3D surface, the disclosed method may include the steps of projecting image pixels from an acquired image into a 3D space and determining the intersection points of the image pixels with the 3D surface. This projection may also be referred to herein as ray tracing. When ray tracing is performed on a given image pixel, a ray can be emitted from a single point of the camera, also referred to as the camera's focal point or aperture, and pass through the given image pixel located in the image plane of the camera. This ray tracing can be performed on all image pixels in the acquired image, thereby generating multiple camera rays in 3D space that are incident on or intersect with the 3D surface, such as a 3D mesh. The intersection points can be determined and subsequently projected onto a virtual projector image placed in a predefined projector image plane. In some embodiments, multiple such virtual projector images are generated, for example, one projector image per camera image in a set of images.

[0014] As an alternative to ray tracing, rasterization of 3D surfaces can be performed or utilized. Rasterization is a common technique for rendering 3D models. Compared to other rendering techniques such as ray tracing, rasterization is very fast and is therefore commonly used in real-time 3D engines. Rasterization can be understood as the process of computing the mapping from surface geometry to pixels. A 3D surface can be represented as a polygonal mesh, such as a triangular mesh composed of multiple triangle vertices. Therefore, the disclosed methods may include the step of rasterizing one or more such vertices. Vertices can undergo various transformations, such as model transformations, view transformations, projection transformations, and / or combinations thereof. These transformations can position each vertex in the appropriate location and transform the vertex from 3D world coordinates to 2D screen coordinates. The entire mesh of the 3D surface can be rasterized and rendered as a series of pixels, thereby creating a 2D representation of the 3D object.

[0015] The disclosed method may include the following steps: modifying a 3D surface by translating one or more points and / or vertices in the 3D surface, thereby obtaining one or more modified surfaces, such as... N A modified 3D surface. Alternatively, the step of modifying the 3D surface can be performed before the aforementioned ray tracing of image pixels and the generation of the virtual projector image. Points or vertices of the 3D surface can be translated along a straight line, such as along the line of sight of the projector unit. In other words, the method may include the step of generating one or more modified 3D surfaces, such as variations of the original input 3D surface. In some cases, the modified surface can be understood as a contraction or expansion of the original 3D surface.

[0016] Following the generation of the N After modifying the 3D surface, the step of tracing image pixels from the camera image onto the surface and into one or more virtual projector images can be performed for all modifications to the 3D surface. Accordingly, multiple virtual projector images can be generated, such as one virtual projector image per camera for each modification. Then, for each projector image pixel, the mean and / or variance of the color can be determined across the projector images. This can be done for all modifications to the original input 3D surface. In particular, the method may include the step of determining a measure for each projector image pixel that expresses the difference in pixel color across the projector images. The measure may be selected from the group consisting of standard deviation, variance, coefficient of variation, or combinations thereof, or other similar measures suitable for expressing bias or difference within the dataset. Furthermore, an average projector image can be generated for each modification. All projector images can be used to generate the average projector image.

[0017] The method may include the step of generating one or more cost functions associated with pixels of a projector image. In some embodiments, at least one cost function is generated for each pattern feature in a pattern visible in one or more projector images. The pattern may be generated by a pattern generating element, such as a mask, that forms a portion of the projector unit. Thus, in this case, a corresponding or similar pattern will appear in the acquired image and also in the corresponding virtual projector image. The method may include the step of calculating a pixel-wise product of the pattern and the average projector image. Accordingly, in some cases, a cost function is defined or generated for each pattern feature in the projector image. The cost function may include the aforementioned metric and / or a weighted sum of pixel-wise products.

[0018] The method may include a step of solving an optimization problem, wherein a generated cost function is minimized for each projector image pixel to determine the optimal depth of the 3D surface. Therefore, a refined or optimized 3D surface can be output by the system and method. The refined 3D surface can be optimized, particularly in terms of the depth of points and / or vertices forming the portion of the 3D surface, thereby allowing for the reconstruction of a more enhanced and / or accurate 3D model from multiple such 3D surfaces. In particular, the disclosed method is efficient in minimizing or removing deviations in depth, as from... Figure 7 This is self-evident. Further details of the system and method can be found in the detailed description in this article.

[0019] This disclosure further relates to a data processing system including one or more processors, such as the 3D scanner system disclosed herein, wherein the one or more processors are configured to perform one or more steps of the computer-implemented methods disclosed herein.

[0020] This disclosure further relates to a computer program product including instructions that, when executed by a computer, cause the computer to perform the steps of the methods disclosed herein. This disclosure further relates to a computer-readable data carrier on which the computer program product is stored.

[0021] This disclosure further relates to a computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform the steps of the methods disclosed herein.

[0022] In summary, the currently disclosed 3D scanner systems and associated computer-implemented methods provide a novel and improved framework for generating and refining 3D surfaces; this framework is particularly well-suited for applications within digital dentistry, such as digital impressions of teeth or other dental objects. Attached Figure Description

[0023] Figure 1A 3D scanner system according to this disclosure is shown.

[0024] Figures 2 to 4 Flowcharts illustrating different embodiments of the computer-implemented method disclosed herein are shown.

[0025] Figure 5 Examples of embodiments thereof, or portions thereof, are shown that can be implemented as computer-readable code in a computer system.

[0026] Figure 6 An exemplary demonstration of a currently disclosed computer-implemented method is shown.

[0027] Figure 7 The diagram shows the 3D surface generated before and after surface modification / refinement, i.e., a single sub-scan. Detailed Implementation

[0028] The first step of the currently disclosed method may be to obtain or generate a three-dimensional (3D) surface of an object. The object may be a dental object, such as at least a portion of one or more teeth of a human subject. Other examples of dental objects include: teeth, gums, implants (one or more), dental restorations (one or more), dental prostheses, edentulous ridges (one or more), and / or combinations thereof. The 3D surface may also be referred to herein as a 3D representation or subscan. A 3D surface may be generated based on a set of images obtained from multiple cameras, such as forming part of an intraoral 3D scanner. The intraoral 3D scanner may be a handheld 3D scanner used to acquire images and / or subscans of the interior of the oral cavity of a subject.

[0029] 3D surfaces can be represented as three-dimensional meshes, such as polygonal meshes, signed distance fields, voxel gratings, implicit surface functions, B-spline surfaces, or other suitable data structures for representing 3D surfaces. A 3D mesh, such as a polygonal mesh, can include multiple vertices connected at its edges. A mesh can further include multiple points. In some embodiments, a 3D surface is represented as a triangular mesh, which includes a set of triangle vertices connected by their common edges. Various methods exist for mesh generation, including the traveling cube algorithm. Accordingly, currently disclosed systems and methods can employ one or more methods or algorithms for mesh generation, such as traveling cubes, Delaunay triangulation, forward propagation, quadtree / octree subdivision, or isosurface extraction.

[0030] 3D surfaces or sub-scans can be generated by an intraoral 3D scanner, which can form part of a 3D scanner system as disclosed herein. The 3D scanner system can be configured to continuously generate multiple 3D surfaces in real time during operation. In some embodiments, the intraoral 3D scanner is configured to generate a 3D representation of a scanned surface of an object, wherein the 3D representation is a point cloud. The 3D scanner system may include one or more processors operatively coupled to the intraoral 3D scanner, wherein the processor(s) are configured to generate, for example, a 3D surface in the form of a polygonal mesh based on the point cloud generated by the intraoral 3D scanner. In other embodiments, the 3D surface is generated entirely by the intraoral 3D scanner.

[0031] A 3D scanner system can be configured to register multiple such 3D surfaces or sub-scans with each other in a process known as registration, whereby the sub-scans are brought into a common reference frame. 3D surfaces can be stitched together to form a complete 3D model of an object. The 3D model can be a digital impression of a person's dental arch. Typically, each 3D surface corresponds to a single field of view of an intraoral 3D scanner. Therefore, by stitching together multiple such 3D surfaces, a 3D model with a surface area larger than that that can be captured in a single field of view can be reconstructed. The registration and / or stitching steps can be performed in real time by the 3D scanner system.

[0032] In a preferred embodiment, a 3D surface is generated based on a set of images obtained from two or more cameras. The cameras can be integrated into an intraoral 3D scanner as disclosed herein. The cameras can be arranged with at least one projector unit in a fixed, known relationship. Each camera can have a given predefined field of view, such as selected from about 50° to 115°, for example, about 65° to 100°, preferably about 65° to 85°. In some embodiments, the cameras of the 3D scanner have overlapping fields of view, such that the cameras observe or image substantially the same part or surface of an object. The intraoral 3D scanner can be based on the principle of triangulation scanning; therefore, each camera can define an angle relative to the projector unit. The same applicant further describes a triangulation-based intraoral 3D scanner in the following applications: PCT / EP2022 / 086763, filed December 19, 2022, "System and method for generating digital representations of 3D objects"; PCT / EP2023 / 058521, filed March 31, 2023, "Intraoral 3D scanning device for projecting high-density light patterns"; and PCT / EP2023 / 058980, filed April 5, 2023, "Intraoral scanning device with extended field of view," all of which are incorporated herein by reference in their entirety.

[0033] In some embodiments, the set of images includes at least one image from each of the cameras, such as exactly one image from each of the cameras. As an example, in the case of an intraoral 3D scanner including four cameras, the set of images may include four images, each acquired from a single camera, such that each camera contributes one image to the set. The cameras may be arranged symmetrically about an optical axis defined by the projector unit; and the cameras may be defined at similar angles to the optical axis.

[0034] The projector unit may include or constitute a digital light processing (DLP) projector that uses a micromirror array to generate time-varying patterns, or a diffractive optical element (DOE), or a front-illuminated reflective mask projector, or a micro-LED projector, or a liquid crystal on silicon (LCoS) projector, or a back-illuminated mask projector, wherein the light source is placed behind a mask having a spatial pattern. The projector unit may include a light source for emitting light and a pattern generating element for structuring the light from the light source into a pattern. In some embodiments, the projector unit further includes one or more collimating lenses for collimating the light from the light source before it passes through the mask having a spatial pattern. The light source of the projector unit may be configured to emit light in the visible wavelength range, for example, by utilizing a white light source. The advantage of using white light is that color (texture) and 3D information can be inferred from the same set of image frames, i.e., from a single set of images.

[0035] In some embodiments, the virtual image described herein is a color image generated based on visible light reflected from the surface of an object. In other embodiments, the virtual image is generated based on infrared light reflected from the object. The infrared light may be provided by one or more additional infrared (IR) or near-infrared (NIR) light sources configured to emit infrared light, such as light having a wavelength or wavelength range selected from about 700 nm to about 1.5 µm. The infrared light may penetrate into one or more teeth, such that one or more internal regions of one or more teeth are visualized in the virtual image. The virtual image may be a synthetic image that has a novel view (position and / or orientation) compared to the view of a camera used to generate the virtual image.

[0036] Images acquired by a camera can be two-dimensional (2D) images. Each image can include an array of pixels, such as arranged in rows and columns, where each pixel has a pixel color in the image. Pixel color can be given by one or more intensity values, such as three intensity values ​​(RGB values) corresponding to the intensities of red, green, and blue. Pixel color can be obtained from one or more color channels on an image sensor. Therefore, in some embodiments, the image sensor is a color image sensor including one or more color channels. In some embodiments, an array of color filters, such as Bayer filters, is arranged above the pixel array.

[0037] The currently disclosed methods may include the step of defining a projector image plane with a predefined pattern. The projector image plane can be understood as a virtual image plane. In some embodiments, an intraoral 3D scanner includes a back-illuminated mask projector unit, wherein the mask includes a spatial pattern that can be projected onto the surface of the scanned object. In some embodiments, the projector image plane coincides with the location of the mask. However, this is not necessary. The projector image plane can be in another location, where a projection pattern on the plane can be determined. In some embodiments, the location of the virtual image plane differs from the location of the image plane of an image belonging to the set of images. In some embodiments, the projection pattern is generated by a diffractive optical element (DOE). The projection pattern can be a static pattern or a dynamic pattern, i.e., such that the pattern changes over time. Teeth typically have large areas with minimal variations in color and geometry, which often complicates 3D reconstruction of the tooth surface. The advantage of projecting a pattern onto the surface of the tooth is that it creates greater contrast on the tooth surface, thus making reconstruction more feasible.

[0038] The currently disclosed method may include the following steps: preferably generating camera rays emitted from the pixel to a 3D surface for each pixel in each image within the set of images, thereby generating multiple camera rays incident on the 3D surface. Thus, each pixel can be associated with a corresponding camera ray originating from said pixel in three-dimensional (3D) space. Camera rays can be generated for multiple pixels in one or more images within the set of images. In some embodiments, camera rays are generated for each pixel in each of the images within the set of images. Each camera can be mathematically approximated by a pinhole camera model having a given camera aperture and focal length. This model can define an image plane where a 3D object or scene is projected through the camera aperture. The image plane can be located at a distance from the aperture of the pinhole camera. f At (focal length). Ray tracing of a given image pixel can be understood as projecting a straight line from the camera's focal point through that image pixel, where the line can extend infinitely in 3D space along that direction. A similar model can be used for projector units. In some cases, the mathematical model for one or more camera and / or projector units further considers the geometric distortion and / or blurring of unfocused objects caused by lenses and finite-sized apertures.

[0039] In other words, the method may include the steps of projecting or ray-tracing each pixel in an image onto a 3D surface and then onto a projector image plane, thereby generating one or more projector images in said plane, such as one projector image per camera image. This is in Figure 6 The example demonstrates images from a single camera and projector. The projector image can be understood as a virtual projector image. It is advantageous to refine the depth of a 3D surface using all pixels; however, in some embodiments, the 3D surface is corrected using only a subset of pixels on one or more sensors. The method may include the step of determining the intersection points of camera rays incident on the 3D surface. Specifically, each camera ray intersects the 3D surface at a given point that can be determined by the currently disclosed systems and methods. Alternatively or additionally, the method may include the step of determining the intersection points between camera rays; however, the camera rays may not intersect perfectly on the 3D surface, but rather within a certain tolerance. The intersection point or incident point may correspond to a point on the 3D surface.

[0040] The currently disclosed methods may include the following steps: generating one or more projector images by projecting each of the camera rays from a corresponding incident point or intersection point on a 3D surface onto a projector image plane. In some embodiments, a projector image is generated for each image in the set of images. The projector images (one or more) are preferably located in the projector image plane. The projector images (one or more) may each have pixel colors. c p Multiple projector image pixels. Pixel color can be defined by one or more intensity values, such as three intensity values ​​(RGB values) corresponding to the intensities of red, green, and blue. Typically, the pixel color in (one or more) projector images... c p Subject to the pixel color observed in the camera image c i The effects are similar; however, they do not necessarily correspond 1:1. For example, a given pixel in a projector image may have a color that is the result of some smoothing or interpolation of the colors of its neighboring pixels. c p As an example, if a given camera ray associated with a given camera image pixel is projected onto a given projector image pixel, the resulting color of that pixel is... c p It can have colors that are largely contributed by the camera's light, but it can also receive contributions from neighboring pixels.

[0041] The currently disclosed method may include the following steps: preferably determining the expression pixel color for each projector image pixel. c iA measure of difference. Examples of suitable measures for expressing difference are: standard deviation, variance, coefficient of variation, or combinations thereof. Typically, it refers to the pixel color of the pixels in the captured image. c i To select the pixel colors in one or more projector images c p Contribution. In some embodiments, a given image pixel in a virtual image has a given color based on the contribution of pixels from two or more images obtained from different cameras. (Regarding pixel color) c p The number of contributions depends on the number of cameras that captured the images in that set. Typically, if there is more than one camera, such as two or more cameras, then for a given pixel color... c p Contributing pixel color c i The colors can be different; this difference can be estimated and utilized to assess the effectiveness of evaluating the depth of a 3D surface. Therefore, the color of each pixel in (one or more) projector images... c p Different pixel colors from different images c i The result is due to the contribution of this method. As an example, in the case of four cameras, i.e., four images in this set of images, there may be different colors. c i For example{ c 1 , c 2 , c 3 , c 4 The four image pixels will then be color-coded for each pixel in the projector image(s). c p It makes a contribution. The method may include color... c i Calculate the average to generate pixels with color. c p The method involves generating one or more projector images. Preferably, all images within the set are used to generate an average image. The method may further include the step of calculating a pixel-by-pixel product of a predefined pattern and the average image. Thus, the average image can be multiplied pixel-by-pixel by the value of the predefined pattern.

[0042] The different colors can be determined for each pixel in one or more projector images. c iThe difference measures, such as mean and / or variance. Ideally, image pixels contributing to a given point on a 3D surface should have the same color. Therefore, if the generated or obtained 3D surface is perfect, i.e., corresponds to the baseline truth, then the color of a given pixel in one or more projector images should be consistent. c p All contributing pixel colors c i They should all have the same color. This corresponds to the case where the 3D points associated with the pixels in the projector image(s) are correctly located on the 3D surface. Conversely, if the pixel color... c i If the colors are not the same, this means that the 3D surface can be refined or corrected. Accordingly, the pixel color can be determined and utilized. c i The aforementioned measure of the difference is used to adjust the 3D surface until the measure reaches a minimum. In some embodiments, the measure is pixel color. c i The variance. Variance can be understood as the square of the standard deviation. A 3D surface can be adjusted by changing the position of one or more points and / or vertices of the 3D surface, for example, by translating one or more vertices of the 3D surface in the case that the 3D surface is represented as a polygonal mesh. In particular, a 3D surface can be adjusted to minimize one or more cost functions, as described further below.

[0043] In some embodiments, a projector image is generated for each image in the set of images; therefore, the number of projector images may correspond to the number of camera images or the number of cameras. In other words, the method may include the step of generating a pair of camera images with corresponding projector images. In this case, the method may include the step of: preferably determining an expression pixel color for each projector image pixel. c p A measure of difference across projector images. As an example, in the case of a 3D scanner with four cameras, the four cameras can be configured to preferably acquire a set of four images simultaneously. The disclosed computer-implemented method can then include the step of generating a projector image for each of the images; thus, in this example, four projector images are generated. The color difference of the four projector images can then be determined, for example, by variance. The method may further include the step of generating an average projector image from the projector images. In some embodiments, each of the projector images includes a pattern having multiple pattern features.

[0044] The currently disclosed method may further include the step of defining one or more cost functions associated with a projector image or specifically with pixels of a projector image. In some embodiments, at least one cost function is generated for each pattern feature in a pattern visible in the projector image(s). In mathematical optimization, the cost function is sometimes also referred to as a loss function or error function. Typically, the optimization problem seeks to minimize the cost function. Finite difference or similar techniques can be used to minimize the cost function(s). In some embodiments, minimization is performed iteratively to maximize the light consistency between the images. In this disclosure, the cost function(s) may take one or more variables as inputs. Examples of suitable input variables include the aforementioned metrics, such as the color of a given pixel in the projector image(s). c p Contributing pixel color c i The mean and / or variance of the predefined pattern. As another example, the method may include a step of determining the correlation between a predefined pattern and a corresponding observed or constructed pattern in one or more projector images. Ideally, the two patterns should be identical. In the case that the one or more projector images are located in a projector image plane, the pattern in that plane should be exactly the same as the predefined pattern projected by an intraoral 3D scanner. Any deviation can indicate that the 3D surface can be optimized, refined, or corrected. Therefore, some correlation measures can be defined to assess the similarity between the two patterns. This correlation can be used as an input variable for a cost function. In principle, there can be a cost function assigned to or associated with each pixel in one or more projector images. Alternatively, the method may include a step of defining a cost function for each feature in the predefined pattern. As an example, the predefined pattern may be a checkerboard pattern, where the features of the pattern correspond to the corners in the pattern, i.e., the corners of the checkerboard within the checkerboard pattern. The goal is then to optimize the 3D surface such that the cost function is minimized, thereby obtaining a corrected and optimized 3D surface. The cost function associated with each pattern feature may include pixel color. c i For example, the weighted sum of variances.

[0045] The currently disclosed methods may include steps of modifying a 3D surface by changing the position of one or more points and / or vertices, such as through translation. The position of each point or vertex on the 3D surface can be varied, and the position of each point or vertex can be selected to minimize the value of the cost function at that point / vertex. In some embodiments, the positions(one or more) may be constrained to move only along one or more directions of projector rays emanating from projector image pixels. In other words, the positions(one or more) may vary along the line of sight of the projector unit, corresponding to expanding or contracting the surface along said line.

[0046] Methods for finding the location of the minimum of a cost function may involve using finite difference methods or similar techniques to compute the numerical derivative of the cost function. In other words, gradient-based techniques such as Newton's method or gradient descent can be used to minimize one or more cost functions. The gradient can be computed through analysis or estimated using finite difference. The method may utilize iterative methods for determining the minimum of one or more cost functions, such as Newton's method, also known as the Newton-Raphson method. Newton's method is an iterative algorithm for finding the minimum of a cost function. The method approximates the cost function with a quadratic function and updates the input value based on the roots of the quadratic function. The disclosed method may employ one or more iterations or executions of Newton's method to provide more than one optimization step. Alternatively, other methods or algorithms may be used, such as conjugate gradient methods, gradient descent, steepest descent, or quasi-Newton methods, such as the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm.

[0047] The modified 3D surface associated with determining the minimum of one or more cost functions can correspond to a refined version of the 3D surface initially input to the algorithm. Therefore, the disclosed method may include the step of generating a refined 3D surface by solving an optimization problem, wherein points and / or vertices in the 3D surface are relocated such that the pixel colors corresponding to similar points are... c i Minimize differences across images. In some embodiments, the optimization problem is solved by relocating or translating points and / or vertices in the 3D surface, such that for each point or vertex, the difference is based on the pixel color associated with that point or vertex. c i The difference is minimized. In some embodiments, the positions of all points and / or vertices on the 3D surface are iteratively changed to maximize the light consistency between images. One measure of light consistency is pixel color. c iThe method may further include the step of outputting a refined 3D surface based on the original input 3D surface. The refined 3D surface may be stored in a memory device operatively coupled to one or more processors of the 3D scanner system. The refined 3D surface may be stitched onto an existing 3D model and / or onto multiple other 3D surfaces (also referred to as subscans), thereby generating a 3D model. Typically, the 3D model has a surface area larger than the surface areas of the individual 3D surfaces.

[0048] Accordingly, the currently disclosed methods may include steps of solving an optimization problem, wherein the 3D surface is modified by changing the positions of one or more vertices or points belonging to the 3D surface, wherein the 3D surface is modified such that one or more cost functions associated with the projector image(s) are minimized. Specifically, the cost functions(s) may be associated with the aforementioned metrics, such as the mean and / or variance of pixel colors, and / or with the correlation of the pattern. High variance of pixel colors can be associated with high costs in the optimization problem, and low correlation may similarly lead to high costs when the problem is solved, for example, using iterative methods. Accordingly, the currently disclosed methods provide a framework for optimizing a given 3D surface, particularly with respect to depth bias, such that the optimized or refined 3D surface has less bias. Therefore, more accurate and detailed 3D models can be generated based on refined 3D surfaces.

[0049] A 3D scanner system may include an intraoral 3D scanner as disclosed herein and one or more processors configured to perform one or more steps of the computer-implemented methods disclosed herein. The processors may be selected from or include one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), an accelerator, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processor unit, and / or combinations thereof.

[0050] The 3D scanner system may further include computer memory, such as random access memory (RAM) or read-only memory (ROM). One or more processors of the scanner system may be configured to read and execute instructions stored in computer memory, for example, in the form of random access memory. The computer memory may be configured to store instructions for execution by the processor(s) and data used by those instructions. As an example, the memory may store instructions that, when executed by the processor(s), cause the scanner system to perform any of the computer-implemented methods disclosed herein, either entirely or partially. The scanner system may further include a graphics processing unit (GPU). The GPU may be configured to perform various tasks, such as video decoding and encoding, rendering of digital 3D models, and other image processing tasks.

[0051] The 3D scanner system may further include a non-volatile storage area in the form of a hard disk drive. Preferably, the scanner system further includes an I / O interface configured to connect peripheral devices used in conjunction with the scanner system. More specifically, a display may be connected and configured to display output from the scanner system. The display may, for example, display a 2D rendering of the generated digital 3D model. In some embodiments, the display is configured to display virtual images as disclosed herein or video at a given frame rate based on a plurality of consecutively generated virtual images. The video may also be referred to as a “real-time view” of the 3D scanner, and it may be displayed on the display along with the rendered 3D model. The viewpoint of the “real-time view” may correspond to a view as seen along the optical axis of a projector unit. Input devices may also be connected to the I / O interface. Examples of such input devices include a keyboard and mouse that allow the user to interact with the scanner system. A network interface may further be part of the scanner system to allow the scanner system to connect to a suitable computer network to receive data (such as scan data and images) from other computing devices or systems and to transmit data to other computing devices or systems. One or more processors, volatile memory, hard disk drives, I / O interfaces, and network interfaces can be connected together via a bus.

[0052] The 3D scanner system is preferably configured to receive data from an intraoral 3D scanner, either directly from the intraoral 3D scanner or via a computer network, such as a wireless network. The data may include images, processed images, 3D data, point clouds, data point sets, or other types of data. Wireless connections, wired connections, and / or combinations thereof may be used to transmit / receive data. The scanner system may be configured to perform, in whole or in part, any of the computer-implemented methods disclosed herein. In some embodiments, the scanner system is configured to receive data, such as point clouds, from an intraoral 3D scanner and then subsequently perform the steps of reconstructing and rendering a digital 3D model of the scanned three-dimensional (3D) object. Rendering can be understood as the process of generating one or more images from the three-dimensional data. The scanner system may include a computer memory for storing a computer program including computer-executable instructions that, when executed, cause the scanner system to perform a method for refining a 3D surface.

[0053] Virtual images can be generated in various ways. In some embodiments, a virtual image is generated in the projector image plane by ray tracing of camera rays from pixels in a camera image to a 3D surface, and then ray tracing from the point of incidence on the 3D surface back to the projector image plane. About Figure 6 This method will be described further.

[0054] In other embodiments, a volumetric model can be generated. The volumetric model can refer to a three-dimensional representation of an object. The model can store spatial information about the object by dividing it into small volumetric units, commonly called voxels. Each voxel in the volumetric model can contain data about its properties, such as color, density, texture, or material composition. Therefore, the scanned object can be discretized into a regular voxel grid. The method may further include the step of rendering one or more virtual images from the volumetric model, for example, using volumetric rendering techniques. As an example, a virtual image can be generated by tracing the rays from image pixels of an image into a volume defined by the volumetric model.

[0055] In other embodiments, Neural Radiation Fields (NeRF) can be used to generate virtual images, a computer graphics and computer vision technique for modeling the 3D geometry and appearance of objects or scenes from 2D images. It provides a way to represent complex and realistic scenes by training a neural network to approximate a volumetric scene representation based on a set of 2D images captured from different viewpoints. Unlike traditional geometric or voxel-based representations, NeRF models the scene as a continuous function that maps 3D spatial coordinates to radiation values ​​(color and opacity) in volumetric space. The trained neural network can be used to generate light-realistic renderings of objects from any desired viewpoint. This allows for the synthesis of one or more virtual images with one or more novel views.

[0056] The camera and projector units of an intraoral 3D scanner can be configured relative to each other in a fixed, predefined relationship. As an example, the camera can be symmetrically arranged around the projector unit. The camera and projector units can be mounted in a fixed unit to ensure a fixed positional relationship between the units. The advantage of placing the camera and projector units in a fixed, known relationship is that it provides a good and accurate platform for generating volumetric models by the 3D scanner system. As an example, the NeRF technique explained above assumes a known camera position and that the illumination is the same in different images acquired by the camera. The advantage of a camera with at least partially overlapping fields of view is that the amount of light in the images is similar; particularly when the light is provided by a projector unit positioned at the center of the camera.

[0057] Detailed Explanation of the Diagram Figure 1A 3D scanner system 100 according to this disclosure is illustrated. The 3D scanner system is configured to generate a three-dimensional (3D) representation of an object 101, such as a dental object. As an example, object 101 may be at least a portion of the oral cavity, including any of the dentition, gingiva, retromolar trigone, hard palate, soft palate, and floor of the oral cavity. In this embodiment, the 3D scanner system includes an intraoral 3D scanner 102 for acquiring a set of images of the scanned object, for example, within the oral cavity of a person. The 3D scanner system further includes one or more processors for generating a three-dimensional (3D) representation of the scanned object based on the acquired images. Typically, the 3D representation may only represent a portion of the object surface, for example, captured by the field of view of the intraoral 3D scanner 102. Such a 3D representation may also be referred to herein as a subscan or a 3D surface. The processors (one or more) may be part of the 3D scanner 102, or they may be external to the intraoral 3D scanner, or a combination of both, i.e., causing some processing to be performed on the 3D scanner and further processing to be performed on the computer system 104. An intraoral 3D scanner can be configured to continuously, for example, in real time, acquire multiple sets of images and generate one or more 3D surfaces and / or subscans based on the images. It can be further configured to continuously transmit the subscans to a computer system 104 via wired or wireless means. The subscans can be registered and stitched together to form a digital 3D model of the scanned object. The 3D model can be displayed on a monitor, for example, connected to the computer system.

[0058] Figure 2 A flowchart 200 illustrates an embodiment of a computer-implemented method disclosed herein. In step 202, a three-dimensional (3D) surface of an object is generated based on a set of images. In step 204, a projector image plane with a predefined pattern is defined. In step 206, one or more projector images, each comprising a plurality of projector image pixels, are generated. In step 208, the 3D surface is modified by changing the positions of one or more points and / or vertices of the 3D surface such that one or more cost functions associated with the projector image(s) are minimized.

[0059] Figure 3 A flowchart 300 illustrates an embodiment of a computer-implemented method according to the method disclosed herein. In step 302, a three-dimensional (3D) surface of an object is generated, wherein each image comprises an array of pixels, and each pixel has a pixel color. c i In step 304, a projector image plane with a predefined pattern is defined. In step 306, camera rays emitted from each pixel to the 3D surface are generated, thereby generating multiple camera rays incident on the 3D surface. In step 308, one or more projector images are generated, each projector image including its own pixel color.c p Multiple projector image pixels. In step 310, the pixel color for each projector image pixel is determined. c p Contributing pixel color c i Differences, such as mean and / or variance. In step 312, the 3D surface is modified by changing the position of one or more points and / or vertices on the 3D surface.

[0060] Figure 4 A flowchart 400 is shown illustrating an embodiment of a computer-implemented method disclosed herein. In step 402, a three-dimensional (3D) surface of the object is generated, wherein the 3D surface is generated based on a set of images obtained from two or more cameras, wherein the set of images includes at least one image from each of the cameras, wherein each image includes an array of pixels, and each pixel has a pixel color. c i In step 404, a projector image plane with a predefined pattern is defined. In step 406, camera rays emanating from each pixel in each image within the set of images are generated and projected onto the 3D surface, thereby generating multiple camera rays incident on the 3D surface. In step 408, one or more projector images are generated, each projector image including its own pixel color. c p Multiple projector image pixels, wherein one or more projector images are generated by projecting each of the camera rays from a corresponding incident point on a 3D surface onto a projector image plane. In step 410, the pixel color for each projector image pixel is determined. c p Contributing pixel color c i The differences, such as mean and / or variance, are considered. In step 412, the 3D surface is modified by changing the positions of one or more points and / or vertices of the 3D surface, such that one or more cost functions associated with the projector image(s) are minimized. The method may further include the step of outputting a refined 3D surface based on the modified 3D surface. The refined 3D surface can be generated by the scanner system disclosed herein. Multiple refined 3D surfaces can be registered in a common coordinate system and stitched together to form a 3D model of the object. The 3D model can be output to a display forming part of the scanner system.

[0061] Figure 5A computer system 500, in which embodiments of this disclosure or portions thereof can be implemented as computer-readable code, is illustrated. The computer system may encompass a series of components for implementing data processing, storage, communication, and user interaction. The computer system described herein may include one or more processors 504, a communication interface 524, a hard disk drive 512, a removable storage drive 514, an interface 520, main memory 508, a display interface 502, and a display 530. One or more processors 504 may be configured to perform one or more steps of the computer-implemented methods disclosed herein. The communication interface 524 may be configured to allow the computer system to communicate with external devices and networks. The communication interface may support various communication protocols, such as Ethernet, Wi-Fi, or Bluetooth, thereby enabling data exchange and facilitating connectivity with other devices. The hard disk drive 512 may be configured to provide non-volatile storage for the computer system. The hard disk drive may store software programs, operating system files, user data, and other files on media such as magnetic media. The hard disk drive 512 may ensure persistent storage, thereby allowing data to be retained even when the system is powered off. For example, a removable storage drive 514, such as a CD / DVD drive or a USB port, can be configured to provide the computer system with the ability to read from and / or write to the removable storage unit 518. This allows the user to access external storage devices, such as optical discs or USB flash drives, and to transfer data to and from the computer system. Interface 520 can be configured to connect various external devices, such as a keyboard, mouse, printer, scanner, or audio equipment, to the computer system.

[0062] Figure 6An exemplary demonstration of an aspect of a currently disclosed computer-implemented method is shown. The figure illustrates camera rays emitted from pixels in a given image acquired by a camera, where the camera rays are incident on a 3D surface, illustrated here as a grid. The figure further illustrates the projection of a point given by its intersection with the surface, where this point is projected onto a projector image in a projector plane. The method may include the step of generating such a projector image for each of the images in the set of images. Thus, multiple pairs of images can be generated, each pair including a camera image and a projector image. The projector image may be a virtual image. Therefore, the disclosed method may include the step of generating a virtual image by projecting each of the camera rays from a corresponding incident point on the 3D surface onto a projector image plane. The 3D scanner can be configured to continuously generate virtual images during a scanning session, such that video at a given frame rate can be output, for example, to a display. A virtual image can be understood as an aggregated image generated based on one or more of the images in the set of images. In some embodiments, the virtual image represents a novel view relative to the view / or orientation of the 3D scanner's camera. Therefore, the virtual image may have a different position and / or orientation relative to the image acquired by the camera. In some embodiments, the virtual image corresponds to a view seen along the optical axis of the projector unit. The novel view may coincide with the projector image plane. Given a 3D surface, the method may include the steps of generating a total [data / data] by considering the changes in the surface caused by moving vertices in the mesh further away from or closer to the defined projector plane. N The method may further include the step of determining, for each of a plurality of image pixels, preferably each of each image pixel in an image, an intersection point between a surface and a camera ray associated with that image pixel. The method may further include the step of projecting the intersection point onto a projector plane to generate a per-camera image. N A projector planar image. The method further includes the following steps: for a 3D surface... N For each of the versions, calculate the mean and / or variance of the pixels that contribute to the projector planar image(s).

[0063] Figure 7The diagram shows the 3D surfaces generated before and after surface modification / refinement, i.e., a single sub-scan. The figure shows a black baseline truth corresponding to the object's true 3D surface, and a gray generated 3D surface. The left image corresponds to the two surfaces before any refinement, and the right image corresponds to the two surfaces after running the currently disclosed computer-implemented method. As can be seen from the left image, before refinement, there are larger "islands" of connected points where the depth deviates from the baseline truth; the points are either closer or farther away from the baseline truth. As can be seen from the right image, after refinement, the area of ​​these "islands" is much smaller, thus implying a smaller deviation in depth within the refined 3D surface. Specifically, the depth of the points appears to alternate between having too little depth and too much depth rather than favoring either of these two cases. In other words, points with too much or too little depth are connected as smaller groups of connected points after the refinement process.

[0064] Further details of the invention 1. A computer-implemented method, comprising the following steps: - Generate a three-dimensional (3D) surface of an object, wherein the 3D surface is generated based on a set of images obtained from two or more cameras, wherein the set of images includes at least one image from each of the cameras, wherein each image includes an array of pixels, and each pixel has a pixel color. c i ; - Generate camera rays emitted from each pixel to the 3D surface for each image, thereby generating multiple camera rays incident on the 3D surface; - A projector image is generated for each image by projecting each of the camera rays from a corresponding incident point on the 3D surface onto a predefined projector image plane, wherein the projector image comprises a plurality of projector image pixels, each projector image pixel having a color based on the pixel color. c i pixel color c p ; - Determine the color representing each pixel in the projector image. c p A measure of the difference across the projector images; and - A modified 3D surface is generated by translating one or more points and / or vertices of the 3D surface, thereby minimizing one or more cost functions associated with the projector image pixels.

[0065] 3. The method according to Clause 1, wherein the method further includes the step of generating an average projector image from the projector image.

[0066] 4. The method according to any one of the preceding clauses, wherein each of the projector images comprises a pattern having a plurality of pattern features.

[0067] 5. The method according to any one of Clauses 3 or 4, wherein the method further comprises the step of calculating the pixel-by-pixel product of the pattern and the average projector image.

[0068] 6. The method according to any one of the preceding clauses, wherein the method further comprises the step of generating one or more cost functions associated with the projector image pixels.

[0069] 7. The method according to Clause 6, wherein at least one cost function is generated for each pattern feature in the pattern.

[0070] 8. The method according to any one of Clauses 6 or 7, wherein the cost function comprises a weighted sum of the metric and the pixel-wise product.

[0071] 9. The method according to any one of the preceding clauses, wherein the measure is selected from the group consisting of: standard deviation, variance, coefficient of variation, or a combination thereof.

[0072] 10. The method according to any one of the preceding clauses, wherein the pixel color associated with a given projector image pixel c i They are weighted differently.

[0073] 11. The method according to any one of the preceding clauses, wherein the 3D surface is represented as any of the following: a polygonal mesh, a signed distance field, a voxel grid, an implicit surface function, or a B-spline surface.

[0074] 12. The method according to any one of the preceding clauses, wherein the 3D surface is a signed distance field defined relative to a voxel grid comprising a plurality of voxels.

[0075] 13. The method according to Clause 12, wherein the 3D surface is modified by changing the value of the signed distance field at a predefined location within the voxel grid.

[0076] 14. The method according to any one of the preceding clauses, wherein the 3D surface is a polygonal mesh comprising a plurality of points and / or vertices.

[0077] 15. The method according to any one of the preceding clauses, wherein the position of each point or vertex on the 3D surface is varied, and the position of each point or vertex is selected to minimize the value of the cost function at that point or vertex.

[0078] 16. The method according to Clause 15, wherein the position is constrained to move along a projector ray emanating from the projector image pixel.

[0079] 17. The method according to any one of the preceding clauses, wherein the points and / or vertices of the 3D surface are translated along projector rays emanating from the projector image pixels.

[0080] 18. The method according to any one of the preceding clauses, wherein an iterative method, such as Newton's method or gradient descent, is used to determine the minimum value of the cost function.

[0081] 19. The method according to any one of the preceding clauses, wherein the minimum value of the cost function is determined by iteratively changing the positions of the points and / or vertices until the minimum value of the metric is obtained.

[0082] 20. The method according to any one of the preceding clauses, wherein the projector image plane is similar for all projector images, such that all projector images lie in the same plane.

[0083] 21. The method according to any one of the preceding clauses, wherein the set of images comprises one image from each of the cameras, such that the number of images in the set of images corresponds to the number of cameras.

[0084] 22. A computer-implemented method comprising the following steps: - Generate a three-dimensional (3D) surface of an object, wherein the 3D surface is generated based on a set of images obtained from two or more cameras, wherein the set of images includes at least one image from each of the cameras, wherein each image includes an array of pixels, and each pixel has a pixel color. c i ; - Define the projector image plane with a predefined pattern; - Generate camera rays emitted from each pixel in each image within the group of images to the 3D surface, thereby generating multiple camera rays incident on the 3D surface; - Generate a projector image that includes multiple projector image pixels, each with a pixel color. c p The projector image is generated by projecting each of the camera rays from a corresponding incident point on the 3D surface onto the projector image plane. - Determine the pixel color for each projector image pixel. c p Contributing pixel colorc i The variance; and - Modify the 3D surface by changing the position of one or more points and / or vertices of the 3D surface, thereby minimizing one or more cost functions associated with the projector image pixels.

[0085] 23. A data processing system comprising one or more processors, said one or more processors being configured to perform the steps of the method according to any one of clauses 1 to 22.

[0086] 24. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of clauses 1 to 22.

[0087] 25. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of clauses 1 to 22.

[0088] 26. A computer-readable data carrier having a computer program product as described in Clause 24 stored thereon.

[0089] 27. A 3D scanner system, comprising: - Intraoral 3D scanner, including: - A projector unit, the projector unit including a light source and a pattern generating element for structuring light from the light source into a pattern to be projected onto the surface of an object; - Two or more cameras, operatively connected to the projector unit, wherein each camera includes an image sensor for acquiring one or more two-dimensional images, each image including a pixel array, and each pixel having a pixel color. c i ;as well as - One or more processors operatively connected to the intraoral 3D scanner, the processors being configured to perform the steps of the method according to any one of clauses 1 to 22.

[0090] 28. A 3D scanner system, comprising: - Intraoral 3D scanner, including: - A projector unit, the projector unit including a light source and a pattern generating element for structuring light from the light source into a pattern to be projected onto the surface of an object; - Two or more cameras, operatively connected to the projector unit, the cameras being configured to acquire a set of images, wherein the set of images includes images from each camera, wherein each image includes a pixel array, and each pixel has a pixel color. c i ; - One or more processors, operatively connected to the intraoral 3D scanner, the processors being configured to: - Generate a three-dimensional (3D) surface of the object based on the set of images obtained from the camera, wherein the 3D surface comprises multiple points and / or vertices; and - A refined 3D surface is generated by solving an optimization problem, wherein points and / or vertices in the 3D surface are repositioned such that the pixel color is used as the basis for the final result. c i Minimize the measure of difference across the images.

[0091] 29. A 3D scanner system, comprising: - Intraoral 3D scanner, including: - A projector unit, the projector unit including a light source and a pattern generating element for structuring light from the light source into a pattern to be projected onto the surface of an object; - Two or more cameras, operatively connected to the projector unit, the cameras being configured to acquire a set of images, wherein the set of images includes images from each camera; - One or more processors, operatively connected to the intraoral 3D scanner, the processors being configured to: - Generate the three-dimensional (3D) surface of the object based on the set of images obtained from the camera; and - Generate camera rays emanating from multiple pixels in one or more images within the image set and projecting onto the 3D surface, thereby generating multiple camera rays incident on the 3D surface; and - A virtual image is generated by projecting each of the camera rays from a corresponding incident point on the 3D surface onto the projector image plane.

[0092] 30. The 3D scanner system according to any one of clauses 27 to 29, wherein camera rays are generated for each pixel of each of the images in the set of images.

[0093] 31. A 3D scanner system according to any one of clauses 27 to 30, wherein a given image pixel in the virtual image has a given color based on contributions from pixels in two or more images obtained from different cameras.

[0094] 32. The 3D scanner system according to any one of clauses 27 to 31, wherein the position of the virtual image plane is different from the position of the image plane of the image belonging to the group of images.

[0095] 33. A 3D scanner system according to any one of clauses 27 to 32, wherein each camera has a given field of view, and the overlapping of the fields of view of the cameras causes the cameras to image approximately the same scene or object.

[0096] 34. The 3D scanner system according to any one of clauses 27 to 33, wherein the position and / or orientation of the virtual image represents a novel view relative to the view / orientation of the camera.

[0097] 35. The 3D scanner system according to any one of clauses 27 to 34, wherein the novel view corresponds to a view seen along the optical axis of the projector unit.

[0098] 36. The 3D scanner system according to Clause 35, wherein the novel view coincides with the image plane of the projector.

[0099] 37. The 3D scanner system according to any one of clauses 27 to 36, wherein the pattern generating element is a mask, and wherein the projector image plane coincides with the position of the mask.

[0100] 38. A 3D scanner system according to any one of clauses 27 to 37, wherein the virtual image is continuously generated or updated in real time during operation of the intraoral 3D scanner, such that video with a given frame rate is provided.

[0101] 39. The 3D scanner system according to any one of Clauses 27 to 38, wherein the 3D scanner system further includes a display configured to display the virtual image and / or the video.

[0102] 40. The 3D scanner system according to any one of clauses 27 to 39, wherein the light source of the projector unit is configured to emit light in the visible wavelength range.

[0103] 41. The 3D scanner system according to any one of clauses 27 to 40, wherein the virtual image is a color image generated based on visible light reflected from the surface of the object.

[0104] 42. The 3D scanner system according to any one of Clauses 27 to 41, wherein the 3D scanner system further includes one or more additional light sources.

[0105] 43. The 3D scanner system according to any one of Clauses 27 to 42, wherein the 3D scanner system further comprises an infrared light source configured to emit infrared (IR) light or near-infrared (NIR) light.

[0106] 44. The 3D scanner system according to Clause 43, wherein the IR light has a wavelength or wavelength range selected from about 700 nm to about 1.5 µm.

[0107] 45. The 3D scanner system according to any one of clauses 43 to 44, wherein the virtual image is generated based on infrared light reflected from the object.

[0108] 46. ​​A 3D scanner system, comprising: - Intraoral 3D scanner, including: - A projector unit, the projector unit including a light source and a pattern generating element for structuring light from the light source into a pattern to be projected onto the surface of an object; - Two or more cameras, operatively connected to the projector unit, wherein each camera includes an image sensor for acquiring one or more two-dimensional images, each image including a pixel array, and each pixel having a pixel color. c i ; - One or more processors, operatively connected to the intraoral 3D scanner, the processors being configured to: - Generate a three-dimensional (3D) surface of the object based on a set of images obtained from the cameras, wherein the set of images includes at least one image from each of the cameras, wherein each image includes an array of pixels, and each pixel has a pixel color. c i ; - Generate camera rays emitted from each pixel to the 3D surface for each image, thereby generating multiple camera rays incident on the 3D surface; - A projector image is generated for each image by projecting each of the camera rays from a corresponding incident point on the 3D surface onto a predefined projector image plane, wherein the projector image includes elements each having pixel color. c i Multiple projector image pixels; - Determine the color representing each pixel in the projector image.c i A measure of the difference across the projector images; and - A modified 3D surface is generated by translating one or more points and / or vertices of the 3D surface such that one or more cost functions associated with the projector image pixels are minimized.

[0109] 47. A 3D scanner system according to any one of clauses 27 to 46, wherein the points and / or vertices in the 3D surface are repositioned in an iterative method, wherein the value of the metric is determined for all iterations.

[0110] 48. The 3D scanner system according to any one of Clauses 27 to 47, wherein the intraoral 3D scanner comprises four or more cameras.

[0111] 49. A 3D scanner system according to any one of clauses 27 to 48, wherein the cameras are synchronized such that the cameras acquire images approximately simultaneously.

[0112] 50. The 3D scanner system according to any one of clauses 27 to 49, wherein the set of images comprises one image from each camera.

[0113] 51. A 3D scanner system according to any one of Clauses 27 to 50, wherein one or more processors are configured to use a neural network to determine image features in the images within the set of images.

[0114] 52. The 3D scanner system according to any one of clauses 27 to 51, wherein the neural network is implemented on an application-specific integrated circuit, such as a neural processing unit.

[0115] 53. The 3D scanner system according to any one of clauses 27 to 52, wherein the projector unit includes a light source for emitting white light.

[0116] 54. The 3D scanner system according to any one of clauses 27 to 53, wherein the projector unit includes a light source for emitting unpolarized light.

[0117] 55. The 3D scanner system according to any one of clauses 27 to 54, wherein the pattern is a polygonal pattern comprising a plurality of polygons in a repeating pattern, the polygons being selected from the group consisting of triangles, rectangles, squares, pentagons, hexagons and / or combinations thereof.

[0118] 56. The 3D scanner system according to any one of clauses 27 to 55, wherein the pattern is a checkerboard pattern or a distribution of discrete, unconnected light points.

[0119] 57. The 3D scanner system according to any one of Clauses 27 to 56, wherein the intraoral 3D scanner further comprises one or more additional light sources for emitting light in the infrared range, such as light having a wavelength between 700 nm and 1.5 µm.

[0120] 58. The 3D scanner system according to any one of Clauses 27 to 57, wherein the intraoral 3D scanner further comprises one or more additional light sources for emitting light in the ultraviolet range, such as light having a wavelength between 315 nm and 400 nm.

[0121] 59. The 3D scanner system according to any one of Clauses 27 to 58, wherein the image sensor is a color image sensor.

[0122] 60. A 3D scanner system according to any one of Clauses 27 to 59, wherein the image sensor comprises an array of color filters, such as Bayer filters.

[0123] Although some embodiments have been described in detail and illustrated, this disclosure is not limited to such details, but may be embodied in other ways within the scope of the subject matter defined in the following claims. In particular, it should be understood that other embodiments may be utilized, and structural and functional modifications may be made without departing from the scope of this disclosure. Furthermore, those skilled in the art will appreciate that, unless the embodiments are presented specifically as alternatives only, different disclosed embodiments may be combined to achieve a particular implementation within the scope of this disclosure.

Claims

1. A computer-implemented method comprising the steps of: - generating a three-dimensional (3D) surface of the object, wherein, generating the three-dimensional surface based on a set of images obtained from two or more cameras, wherein the set of images includes at least one image from each of the cameras, wherein each image includes an array of pixels, each pixel having a pixel color c i ; - generating, for each image, camera rays emanating from each pixel to the three- dimensional surface, thereby generating a plurality of camera rays incident on the three- dimensional surface; - generating a projector image for each image by projecting each of the camera rays from a respective point of incidence on the three-dimensional surface to a predefined projector image plane, wherein the projector image comprises a plurality of projector image pixels, each projector image pixel having a pixel color based on the color of the pixel c i of the camera ray that projected the camera ray onto the three-dimensional surface c p ; - determining for each projector image pixel an expression of the color of said pixel c p a measure of the difference across the projector image; and - generating a modified three-dimensional surface by translating one or more points and / or vertices of the three-dimensional surface such that one or more cost functions associated with the projector image pixels are minimized.

2. The method of claim 1, wherein, The method further comprises the step of generating an average projector image from the projector images.

3. The method according to any of the preceding claims, wherein, Each of the projector images comprises a pattern having a plurality of pattern features.

4. The method of any one of claims 2 or 3, wherein, The method further comprises the step of calculating a pixel-wise product of the pattern and the average projector image.

5. The method according to any of the preceding claims, wherein, The method further comprises the step of generating one or more cost functions associated with the projector image pixels.

6. The method of claim 5, wherein, At least one cost function is generated for each pattern feature in the pattern.

7. The method of any one of claims 5 or 6, wherein, The cost function comprises a weighted sum of the metric and the pixel-wise product.

8. The method of any of the preceding claims, wherein, The metric is selected from the group consisting of: standard deviation, variance, coefficient of variation, or a combination thereof.

9. The method of any of the preceding claims, wherein, the pixel color associated with a given projector image pixel c i are weighted differently.

10. The method of any of the preceding claims, wherein, The three-dimensional surface is represented as any one of: a polygonal mesh, a signed distance field, a voxel grid, an implicit surface function, or a B-spline surface.

11. The method of any of the preceding claims, wherein, The three-dimensional surface is a signed distance field defined with respect to a voxel grid comprising a plurality of voxels.

12. The method of claim 11, wherein, The three-dimensional surface is modified by changing values of the signed distance field in predefined locations within the voxel grid.

13. The method of any of the preceding claims, wherein, The three-dimensional surface is a polygonal mesh comprising a plurality of points and / or vertices.

14. The method of any of the preceding claims, wherein, The position of each point or vertex on the three-dimensional surface is varied and the position of each point or vertex is selected to minimize the value of the cost function in that point or vertex.

15. The method of claim 14, wherein, The position is constrained to move along a projector ray emanating from the projector image pixel.

16. The method of any of the preceding claims, wherein, The points and / or vertices of the three-dimensional surface are translated along projector rays emanating from the projector image pixels.

17. The method of any of the preceding claims, wherein, An iterative method, such as Newton's method or gradient descent, is used to determine the minimum of the cost function.

18. The method of any of the preceding claims, wherein, The minimum of the cost function is determined by iteratively changing the position of the points and / or vertices until a minimum of the metric is obtained.

19. The method of any of the preceding claims, wherein, The projector image plane is similar for all the projector images, such that all projector images lie in the same plane.

20. The method of any of the preceding claims, wherein, The set of images comprises one image from each of the cameras, such that the number of images in the set of images corresponds to the number of cameras.

21. A 3D scanner system comprising: - an intraoral 3D scanner comprising: - a projector unit, the projector unit comprising a light source and a pattern generating element for structuring light from the light source into a pattern to be projected onto a surface of an object; - two or more cameras operatively connected to the projector unit, the cameras configured for acquiring a set of images, wherein the set of images comprises an image from each camera, wherein each image comprises an array of pixels, each pixel having a pixel color c i ; - one or more processors, the one or more processors being operatively connected to the intraoral 3D scanner, the processor being configured for: - generating a three-dimensional (3D) surface of the object based on the set of images obtained from the cameras, wherein the three-dimensional surface comprises a plurality of points and / or vertices; and - projecting the pattern onto the three-dimensional surface of the object. - generating a refined three-dimensional surface by solving an optimization problem, wherein points and / or vertices in the three-dimensional surface are repositioned such that the color of the pixel is based on the color of the point and / or vertex c i a measure of the difference across the image is minimized.

22. The 3D scanner system of claim 21, wherein, A camera ray is generated for each pixel in each of the images within the set of images.

23. The 3D scanner system of any of claims 21-22, wherein, A given image pixel in the virtual image has a given color based on contributions from pixels in two or more images obtained from different cameras.

24. The 3D scanner system of any one of claims 21-23, wherein, The position of the virtual image plane is different from the position of the image planes belonging to the images in the set of images.

25. The 3D scanner system of any one of claims 21-24, wherein, Each camera has a given field of view, wherein the fields of view of the cameras overlap such that the cameras image approximately the same scene or object.

26. The 3D scanner system of any one of claims 21-25, wherein, The position and / or orientation of the virtual image represents a novel view compared to the views / orientations of the cameras.

27. The 3D scanner system of any one of claims 21-26, wherein, The novel view corresponds to a view seen along the optical axis of the projector unit.

28. The 3D scanner system of claim 27, wherein, The novel view coincides with the projector image plane.

29. The 3D scanner system of any one of claims 21-28, wherein, The pattern-generating element is a mask, wherein the projector image plane coincides with the position of the mask.

30. The 3D scanner system of any one of claims 21-29, wherein, The virtual image is generated or updated in real-time continuously during operation of the intraoral 3D scanner such that a video with a given frame rate is provided.

31. The 3D scanner system of any one of claims 21-30, wherein, The 3D scanner system further comprises a display configured for displaying the virtual image and / or the video.

32. The 3D scanner system of any one of claims 21-31, wherein, The light source of the projector unit is configured for emitting light in the visible wavelength range.

33. The 3D scanner system of any one of claims 21-32, wherein, The virtual image is a color image generated based on visible light reflected from the surface of the object.

34. The 3D scanner system of any one of claims 21-33, wherein, The 3D scanner system further comprises one or more additional light sources.

35. The 3D scanner system of any one of claims 21-33, wherein, The 3D scanner system further comprises an infrared light source configured for emitting infrared (IR) light or near-infrared (NIR) light.

36. The 3D scanner system of claim 35, wherein, The infrared light has a wavelength or wavelength range selected from the range of about 700 nm to about 1.5 µm.

37. The 3D scanner system of any one of claims 21-36, wherein, The virtual image is generated based on infrared light reflected from the object.

Images