Intraoral 3D scanning device for projecting high-density light patterns

Abstract

The present disclosure relates to systems and methods for generating a digital representation of a three-dimensional (3D) object. In particular, the present disclosure relates to a dental scanning system for acquiring an image of an object and generating a digital representation of the object. One embodiment relates to a dental scanning system for scanning a dental object, comprising an intraoral 3D scanning device comprising at least one projector unit configured to project a light pattern along a projector optical axis, the light pattern comprising a plurality of pattern features; one or more cameras having at least partially overlapping fields of view along different camera optical axes (and along a projector optical axis), each camera including an image sensor, where the system further includes one or more processors configured to generate a digital three-dimensional representation of the dental object based on triangulation.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for generating digital representations of three-dimensional (3D) objects. Specifically, the present disclosure relates to dental scanning systems for scanning a 3D object using a light pattern to acquire an image of the object and generate a digital representation of the object. The present disclosure also relates to dental scanning systems for acquiring an image of the object and generating a digital representation of the object. Background Art

[0002] Digital dentistry is growing in popularity and offers several advantages over non-digital technologies. Digital dental scanning systems typically utilize a scanning device (e.g., an intraoral 3D scanner) to generate a three-dimensional digital representation of a three-dimensional object or surface within a patient's mouth. Within scanning devices, there are various different technologies, such as triangulation-based scanning, confocal scanning, focused scanning, ultrasound scanning, X-ray scanning, stereopsis, and optical coherence tomography (OCT).

[0003] Optical scanning devices typically include a projector unit for projecting an illumination pattern onto the surface of a 3D object and an image sensor for capturing one or more images of the illuminated object. In focused scanning devices, the projector unit and camera are typically positioned along the same optical axis. However, in triangulation-based scanning devices, the projector unit and camera are offset so that they form a triangle with a given point on the illuminated object's surface. Computer stereo vision and 3D scanning devices based on optical triangulation typically use triangulation to determine the spatial dimensions and / or geometry of an object. The density of the structured light pattern projected onto the scanned object is important for image quality. A higher density of the light pattern provides increased sampling of the surface, higher resolution, and better ability to stitch surfaces from multiple images. However, a higher density of light patterns often leads to more complex image generation, particularly due to the so-called correspondence problem. A central task in scanning systems employing triangulation is solving the correspondence problem. Given two or more images of the same 3D object acquired from different viewpoints, the correspondence problem refers to the task of finding a set of points in one image that can be identified as the same point in the other image. To achieve this, points or features in one image are matched with corresponding points or features in the other image. As the density of the projected light pattern increases, the number of spots to be addressed increases and becomes much more complex.

[0004] In order to generate a three-dimensional digital representation of a scanned object (e.g. a person's teeth), a corresponding problem generally needs to be solved, at least when an image of the object is acquired using a triangulation-based scanning device. Generally, it is desirable to generate the 3D digital representation in real time, or at least in a manner that is perceived by the user (e.g. a dentist) as real time. Therefore, the 3D representation is generally generated simultaneously with the acquisition of the image / scan data, so that the user can immediately view the generated 3D representation while the patient is being scanned. The 3D representation can be generated in real time. This is also considered to be important feedback for the user, as it is very obvious / clear when new scan data is added to the digital 3D representation.

[0005] The optical system of a 3D scanning device is typically modeled internally using a mathematical geometric model that models all of the optical components of the optical system. This geometric model typically includes information about the relative position and orientation of the camera and projector relative to each other. The arrangement of optical components (e.g., camera and projector) is typically assumed to be fixed in time. However, in reality, the arrangement of these optical components may undergo some changes over time, for example due to thermal expansion. It is therefore important to develop a scanning device that can mathematically account for changes in the physical configuration of the optical components in the scanning device. Summary of the Invention

[0006] Therefore, it is of interest to develop improved systems and methods for generating digital 3D representations of 3D objects. In particular, it would be desirable to develop scanning systems that can utilize the improved solutions to the presently disclosed correspondence problem. Furthermore, it would be of interest to develop scanning devices in which the geometric model of the optical system of the scanning device can be dynamically updated, thereby enabling dynamic calibration.

[0007] The present inventors have solved the correspondence problem in a fast and reliable manner, enabling rapid generation of 3D representations by adding sets of image features for which the correspondence problem has been solved. Thus, rather than solving the correspondence problem feature by feature, it can be solved for sets of features, such that a consistent and reliable solution can be found for all features in the set, as also described in detail herein. Furthermore, the present inventors have recognized that by including a less dense pattern within the projected pattern, for example by having multiple fiducial markers within the pattern, the correspondence problem can be initially solved for the less dense pattern. The solution to this problem can then be used as input to update the mathematical geometric model of the scanning device, thereby enabling dynamic calibration of the scanning device.

[0008] Therefore, the present invention relates to a 3D scanning system for scanning an object, such as a dental object, comprising:

[0009] -Intraoral 3D scanning device, the intraoral 3D scanning device comprising:

[0010] - at least one projector unit configured to project a light pattern of predetermined density onto the dental object along a projector optical axis;

[0011] one or more cameras, for example at least two cameras, having at least partially overlapping fields of view along different camera optical axes (and along the projector optical axis), each of said camera optical axes defining an angle with the projector optical axis, each of said camera optical axes comprising an image sensor having an array of pixels, and

[0012] - one or more processors, wherein at least one processor is configured to generate a digital three-dimensional representation of the dental object based on triangulation between the camera and the light pattern.

[0013] Preferably, each of said camera optical axes defines an angle of at least 3 degrees with the projector optical axis, such as about 5 to 15 degrees, preferably 5 to 10 degrees, even more preferably 8 to 10 degrees.

[0014] A preferred embodiment of the presently disclosed scanning system employs one or more projector units configured to project a static light pattern having a predetermined density onto the dental object along the projector optical axis. An advantage of using a static light pattern is that no moving parts are required in the projector for projecting the static light pattern. The associated electronics for generating the static light pattern can be kept quite simple. The static light pattern enables the use of one or more rolling shutter cameras. The static light pattern enables high contrast in the imaging system. Furthermore, the projector for generating the static light pattern can be made very small so that, for example, it can be integrated into the head of an intraoral scanning device, thereby reducing the overall size of the head and / or the intraoral scanning device. Another advantage of projecting a static pattern is that it allows all image data to be captured simultaneously, thereby preventing distortions caused by movement between the scanning device and the object.

[0015] In particular, it has been found that it is advantageous if the density of the projected light pattern (i.e. the density of features) is chosen such that a density measure of the projected light pattern is above a predetermined threshold, since a high-density pattern improves the resolution of the generated digital representation and captures an increased number of features of the scanned object. A high-density pattern is to be understood as a pattern comprising more than 3000 pattern features. In general, dense light patterns lead to a more complex correspondence problem, since there is a large number of features for which the correspondence problem needs to be solved. Furthermore, as will be described below, high-density patterns are more difficult to resolve due to the small features, which therefore places high demands on the optics of the scanning device. The inventors have found that a pattern comprising more than 3000 pattern features provides a very good resolution of the corresponding 3D representation of the scanned object, since the large number of features can provide a large number of 3D points.

[0016] Projecting high-density patterns with a large depth of focus (i.e., within a large focus range) is challenging for optical systems (e.g., projector units) with short working distances. Typically, projected features will not be imaged as perfect points by the scanning device, but rather will have some spread (blurring) when imaged by the scanning device. The degree of feature spread can be described by a point spread function (PSF). The resolution of the 3D representation is limited by the spread of features described by the PSF, as features must be sharp in the image to accurately determine the 3D points used to generate the 3D representation. In some cases, features are described by a PSF with an Airy disk radius equal to or less than 100 μm (e.g., equal to or less than 50 μm). The minimum feature size in the pattern is limited by the imaging resolution capability of the scanning device's optics. Imaging resolution is primarily limited by three effects: defocus, lens aberrations, and diffraction. A small aperture is beneficial for minimizing the negative effects of defocus and lens aberrations, as cameras or projectors with small apertures are highly tolerant of defocus. However, a small aperture results in increased diffraction, which negatively impacts imaging resolution. Therefore, it is difficult to optimize imaging resolution by changing the aperture size because the three effects are affected differently by the aperture size.

[0017] Through experiments performed by the inventors, it has been determined that an optical system (e.g., including a projector unit and / or camera(s) disclosed herein) having a numerical aperture between 0.0035 and 0.015 can achieve the ability to resolve very fine details between 50-200 μm in size within a focus range between 10 mm and 50 mm (e.g., between 12 mm and 36 mm). In some applications, the optical system is configured to have a working distance between 15 mm and 50 mm, such as the working distance of the projector unit and / or camera(s). In some cases, the working distance can be longer than 50 mm, for example, if the scanning device includes a mirror disposed at the distal end of the scanning device. Conversely, if a scanning unit is provided in the scanning device without a mirror, the working distance can be shorter than 15 mm in some cases. A numerical aperture between 0.0035 and 0.015 can correspond to an aperture that provides a pupil diameter between 0.2 mm and 0.7 mm. Thus, a technical effect of selecting the numerical aperture is that this provides the ability to project a high density pattern, wherein the pattern is focused within a relatively wide focus range very close to the scanning device, and wherein the blurring of the pattern features is below a given tolerance, such as the tolerance given by the Airy disk mentioned above.

[0018] Thus, a more accurate 3D representation can be generated because the positions of the 3D points can be determined more accurately, i.e. with less uncertainty, and also because smaller features allow more features to be present in the pattern, resulting in a 3D representation comprising more 3D points.

[0019] In a preferred embodiment, the light pattern comprises a predetermined number of similar pattern features and / or polygons, as known from a checkerboard pattern of alternating black and white squares. In the context of a checkerboard herein, the terms "square" and "checker" are used interchangeably. A checkerboard pattern is an example of a pattern that can carry a particularly high number of pattern features, since each black square is surrounded by a white square and each white square is surrounded by a black square. The pattern may comprise between 1000 and 50000 pattern features, for example between 3000 and 25000 pattern features. In a preferred embodiment of the presently disclosed scanning device, the total number of pattern features is at least 3000, preferably at least 10000 and even more preferably at least 19000. For example, this would be achieved by using a checkerboard light pattern having 140x140 checkerboards, which corresponds to a total number of 19600 pattern features. Such a high density light pattern has never been used in the dental field. As an example, a light pattern projected onto a surface measuring approximately 20x20 mm is used to generate a light pattern. 2 A 140x140 checkerboard pattern in area means that each square in the checkerboard pattern is approximately or less than 0.15x0.15 mm. The high density of features in such a pattern results in an unprecedented level of detail in digital representations.

[0020] The present disclosure also relates to a computer program comprising instructions that, when executed by one or more processors, cause the processor(s) to perform any of the computer-implemented methods disclosed herein. The processor(s) may be part of a scanning device or a computer system. The present disclosure also relates to a computer-readable medium having the computer program stored thereon.

[0021] Preferably, the processor(s) are configured to continuously execute the steps of the computer-implemented method(s) such that the digital representation is continuously updated during image acquisition in a scanning session. More preferably, the processor(s) are capable of executing the steps of the method in real time or near real time such that the digital representation of the 3D object can be generated in real time concurrently with a user operating the scanning device to acquire images of the object. Ideally, during execution of the method, the image group (e.g., a group of four images in the case of four cameras) is processed so rapidly that processing is complete by the time a new group of images is acquired, wherein the images are acquired at a predetermined frame rate, e.g., 25 frames per second (FPS) or higher. Such a scenario is an example of real-time processing.

[0022] In one embodiment, an intraoral 3D scanning system comprises:

[0023] - at least one projector unit configured to project a light pattern onto the dental object along a projector optical axis, the light pattern comprising a plurality of pattern features, wherein the pattern comprises a predetermined number of fiducial markers;

[0024] one or more cameras, for example at least two cameras, having at least partially overlapping fields of view along different camera optical axes and along the projector optical axis, each of said camera optical axes defining an angle of at least 3 degrees with the projector optical axis, each of the cameras comprising an image sensor for acquiring one or more images, and

[0025] - one or more processors, wherein at least one processor is configured to generate a digital three-dimensional representation of the dental object based on triangulation between the camera and the light pattern.

[0026] The present disclosure also relates to a method for generating a digital three-dimensional representation of a dental object, the method comprising the following steps:

[0027] -projecting a pattern onto the surface of the dental object using an intraoral scanning device;

[0028] - acquiring one or more sets of images, wherein each set of images comprises at least two images, wherein each image comprises at least a portion of the projected pattern;

[0029] - determining one or more image features within each set of images;

[0030] - solving the correspondence problem within each set of images using triangulation such that a point in 3D space is determined based on the image features, wherein the point forms a solution to the correspondence problem, wherein the correspondence problem is solved for the set of image features; and

[0031] - generating a digital three-dimensional representation of the dental object, wherein the solution of the corresponding problem is used to generate the three-dimensional representation.

[0032] The present disclosure also relates to a 3D scanning system, comprising:

[0033] - an intraoral scanning device for scanning a dental object, the scanning device comprising:

[0034] - one or more projector units configured to project a pattern onto a surface of the dental object, wherein the pattern comprises a predetermined number of fiducial markers;

[0035] - two or more cameras for each projector unit, wherein the cameras are configured to acquire one or more sets of images, wherein each set of images comprises at least one image from each camera, wherein each image comprises at least a portion of the projected pattern, said portion comprising at least one of said fiducial markers;

[0036] - One or more processors configured to:

[0037] - identifying one or more fiducial markers within at least one of the acquired set of images;

[0038] - solving a correspondence problem associated with the identified fiducial markers, wherein solving the correspondence problem results in determining a point in 3D space based on the identified fiducial markers, wherein said point forms a solution to the correspondence problem; and

[0039] - calibrating the scanning device by adjusting one or more parameters of a mathematical geometric model associated with the scanning device, wherein the adjustment is based on a solution to the corresponding problem.

[0040] The present disclosure also relates to a method for calibrating an intraoral scanning device, the method comprising the following steps:

[0041] - projecting a pattern onto a surface of the dental object using an intraoral scanning device, wherein the pattern comprises a predetermined number of fiducial markers;

[0042] - acquiring one or more sets of images, wherein each set of images comprises at least two images, wherein each image comprises at least a portion of the projected pattern, said portion comprising at least one of said fiducial markers;

[0043] - identifying one or more fiducial markers within the first set of acquired images;

[0044] - solving a correspondence problem associated with the fiducial markers, wherein solving the correspondence problem results in determining a point in 3D space based on the identified fiducial markers, wherein said point forms a solution to the correspondence problem; and

[0045] - calibrating the scanning device by adjusting one or more parameters of a mathematical geometric model associated with the scanning device, wherein the adjustment is based on a solution to the corresponding problem.

[0046] In some embodiments, the step of calibrating the scanning device includes the following steps: mathematically projecting one or more camera rays and projector rays together in 3D space, wherein the rays are associated with a reference marker; and minimizing the distance between the camera ray and the given associated projector ray by dynamically adjusting one or more parameters of the mathematical geometric model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A perspective view of a scanning unit comprising a projector unit and four cameras is shown, wherein the projector unit is configured to project a light pattern.

[0048] Figure 2 An embodiment of a scanning device including a scanning unit is shown. The scanning unit includes a reflective element, here taking a reflective mirror as an example.

[0049] Figure 3 An embodiment of a scanning unit according to the present disclosure is shown. In this embodiment, the scanning unit comprises a projector unit and four cameras.

[0050] Figure 4 A cross section through a scanning device according to the present disclosure is shown.In this embodiment, the scanning device comprises two scanning units arranged in series.

[0051] Figure 5 A flow chart of the disclosed method according to an embodiment is shown.

[0052] Figure 6 A flow chart of a disclosed method according to an embodiment is shown, wherein the method is more Figure 5 The method is described in more detail.

[0053] Figure 7 A flow chart of a disclosed method according to an embodiment is shown, wherein the method is more Figure 5 The method described is even more detailed.

[0054] Figure 8 A flow chart illustrating an example of how to determine the depth of an image feature according to an embodiment of the disclosed method.

[0055] Figure 9 A flowchart showing an example of how to generate an image feature group according to an embodiment of the disclosed method.

[0056] Figure 10 A flow chart showing an example of how groups of image features may be evaluated according to predetermined criteria and one or more may be added to a queue.

[0057] Figure 11 A flowchart illustrating another example of how to generate seed proposals according to an embodiment of the disclosed method.

[0058] Figure 12 A flow chart showing an example of how a digital representation may be generated according to an embodiment of the disclosed method.

[0059] Figure 13 A schematic diagram of a computer system is shown.

[0060] Figure 14 An embodiment of a dental scanning system including a scanning device and a computer system is shown.

[0061] Figure 15 shows examples of input data for the neural networks described herein.

[0062] Figure 16 An example of a first type of output from a neural network is shown.

[0063] Figure 17 An example of a second type of output from a neural network is shown.

[0064] Figure 18 An example of a third type of output from a neural network is shown.

[0065] Figure 19 An example of a fourth type of output from a neural network is shown.

[0066] Figure 20 An example of a checkerboard pattern is shown.

[0067] Figure 21 Shown Figure 20 Measured intensity signal of the checkerboard pattern shown in .

[0068] Figure 22 An example of a random pattern of black and white squares is shown.

[0069] Figure 23 Shown Figure 22 Measured intensity signal of the random pattern shown in .

[0070] Figure 24 An example of a polygonal pattern composed of tessellation cells is shown, forming a checkerboard pattern, with each tessellation cell / square comprising four sides and four corners.

[0071] Figure 25 Shown Figure 24 Examples of repeating units of the pattern shown in .

[0072] Figure 26 An example of a checkerboard pattern is shown, where pattern features are indicated as circles, the features corresponding to interior corners in the pattern.

[0073] Figure 27 An example of a checkerboard pattern is shown, where pattern features are indicated as circles, the features corresponding to corners in the pattern, including corners located on the outer boundary of the pattern.

[0074] Figure 28 An example of a polygonal pattern composed of triangles, each of which includes three sides and three corners, is shown.

[0075] Figure 29 Shown Figure 28 Examples of repeating units of the pattern shown in .

[0076] Figure 30 An example of a polygonal pattern composed of triangles is shown, where pattern features are indicated as circles, corresponding to corners in the pattern.

[0077] Figure 31 Shown Figure 30 An example of a repeating unit of the pattern shown in , with one of the edges highlighted.

[0078] Figure 32 An example of a polygonal pattern composed of hexagons, each of which includes six sides and six corners, is shown.

[0079] Figure 33 Shown Figure 32 A pattern in which a repeating unit 1 is indicated, as well as some of the pattern features 2 and edges 3 of the pattern.

[0080] Figure 34 An example of a polygonal pattern composed of triangles, each of which includes three sides and three corners, is shown.

[0081] Figure 35 An example of a pattern with a fiducial mark (here a cross is exemplified) is shown.

[0082] Figure 36 An example of a pattern with a fiducial mark (here a cross is exemplified) is shown.

[0083] Figure 37 An example of a pattern with a fiducial mark (here a cross is exemplified) is shown. DETAILED DESCRIPTION

[0084] 3D objects

[0085] A three-dimensional (3D) object can be a dental object. Examples of dental objects include one or more of the following: one / more teeth, gums, (one or more) implants, (one or more) dental restorations, (one or more) dentures, (one or more) toothless ridges and / or combinations thereof. Alternatively, the dental object can be a plaster cast or plastic model representing the teeth of a subject. For example, the three-dimensional (3D) object can include the teeth and / or gums of a subject. The dental object can be just a portion of the teeth and / or oral cavity of a subject, as the entire set of teeth of a subject is not necessarily scanned during a scanning session. A scanning session can be understood in this article as a period of time during which data (e.g., images) of a 3D object are obtained.

[0086] Scanning device

[0087] Therefore, the present invention relates to a scanning system, such as an intraoral 3D scanning device, for scanning an object, in particular a dental object, comprising:

[0088] - at least one projector unit configured to project a light pattern of predetermined density onto the dental object along a projector optical axis;

[0089] one or more cameras, for example at least two cameras, having at least partially overlapping fields of view along different camera optical axes (and along the projector optical axis), each of said camera optical axes defining an angle with the projector optical axis, each of the cameras comprising an image sensor having an array of pixels, and

[0090] - one or more processors, wherein at least one processor is configured to generate a digital three-dimensional representation of the dental object based on triangulation between the camera and the light pattern.

[0091] Generally speaking, the present disclosure also relates to a dental scanning system for generating a digital representation of a three-dimensional (3D) object, the scanning system comprising a scanning device, such as an intraoral 3D scanning device, the scanning system comprising:

[0092] - one or more projector units / projector units configured to project a predetermined pattern (preferably in the form of a light pattern of predetermined density such that the pattern comprises a plurality of pattern features) onto the 3D object along a projector optical axis, i.e. each projector unit is associated with a projector plane;

[0093] one or more cameras, for example at least two or more cameras, (for each projector unit) having at least partially overlapping fields of view along different camera optical axes (and along the projector optical axis), each of said camera optical axes defining an angle with the projector optical axis such that each of the cameras has a predetermined fixed position relative to the one or more projector units, wherein each camera is configured to acquire an image of at least a portion of the object, wherein each of the cameras comprises an image sensor having an array of pixels,

[0094] - at least one processor / processing unit, preferably comprising means for performing any computer-implemented method disclosed herein, e.g. configured to generate a digital three-dimensional representation of a 3D object based on triangulation between a camera and a light pattern.

[0095] The scanning device may be a handheld intraoral 3D scanning device. The scanning device may include an elongated probe having a tip. The intraoral 3D scanning device may be configured to capture images of the interior of a subject's oral cavity.

[0096] The scanning device may be an intraoral scanning device for acquiring images of the subject's oral cavity. In a preferred embodiment, the scanning device adopts a scanning principle based on triangulation. The scanning device includes at least one projector unit and at least one camera. Preferably, the scanning device includes one or more scanning units, for example as part of a handheld elongated probe having an end, wherein each scanning unit includes a projector unit and one or more cameras. As an example, the scanning device may include a scanning unit including a projector unit and one or more cameras, for example at least two cameras. As another example, the scanning device may include a scanning unit including a projector unit and four cameras. In yet another example, the scanning device may include at least two scanning units, wherein each scanning unit includes a projector unit and two or more cameras.

[0097] Specifically, the camera can be configured to capture a set of images, wherein the correspondence problem within the set of images is solved based on triangulation. The images within the set of images can be captured by separate cameras of the scanning device. The images in the set of images are preferably captured simultaneously, i.e., at the same time, with each camera contributing one image to the set of images. One advantage of this is that the light budget is improved; therefore, the light source and projector unit consume less power. The images in the set of images preferably capture substantially the same area of ​​the dental object. The images can include multiple image features that correspond to pattern features in a structured pattern projected onto the surface of the dental object. The correspondence problem can generally refer to the problem of determining which parts or image features of one image correspond to which parts of another image in the set of images. Specifically, in this context, the correspondence problem can refer to the task of associating each image feature with a projector ray emitted by the projector unit. In other words, the problem can also be expressed as the task of associating points in an image with points in the projector plane of the projector unit.

[0098] The projector unit may be configured to project a plurality of projector rays, which are projected onto the surface of the dental object. Solving the correspondence problem may comprise the steps of determining image features in an image of a set of images and further associating said image features with specific projector rays. In a preferred embodiment, the correspondence problem is solved jointly for groups of projector rays, rather than, for example, solving the correspondence problem one projector ray at a time. The group of projector rays considered may form a connected subset within the pattern. The inventors have found that by jointly solving the correspondence problem for groups or sets of projector rays, a particularly reliable and robust solution may be obtained, resulting in a more accurate 3D representation. Subsequently, the depth of each projector ray may be calculated, so that a 3D representation of the scanned object may be generated.

[0099] A projector unit is understood herein to be a device for generating an illumination pattern to be projected onto a surface (e.g., the surface of a three-dimensional object). The projector unit(s) may be selected from the following group: a digital light processing (DLP) projector, a diffractive optical element (DOF) projector, a front-illuminated reflective mask projector, a micro-LED projector, a liquid crystal on silicon (LCoS) projector, or a back-illuminated mask projector. In the case of a back-illuminated mask projector, a light source is placed behind a mask having a spatial pattern, so that the light projected onto the surface of the dental object is structured or patterned. A back-illuminated mask projector may include one or more collimating lenses for collimating the light from the light source, the collimating lens(es) being placed between the light source and the mask. In a preferred embodiment, the projector unit(s) of the scanning device include at least one light source and a mask having a spatial pattern. The mask may be a chrome-coated glass mask. In some embodiments, the projector unit(s) include a diffractive optical element as an alternative to a structured mask.

[0100] Examples of spatial patterns are Figure 24-34 As shown. The spatial pattern can be a polygonal pattern comprising a plurality of polygons. The polygons can be selected from the group consisting of triangles, rectangles, squares, pentagons, hexagons, and / or combinations thereof. Other polygons are also contemplated. Generally, polygons are composed of sides and corners. In a preferred embodiment, the polygons are repeated in a predetermined manner within the pattern. As an example, the pattern can include a plurality of repeating units, wherein each repeating unit includes a predetermined number of polygons, and wherein the repeating unit is repeated throughout the pattern.

[0101] The projector unit(s) may further comprise one or more lenses, such as collimating lenses or projection lenses. The collimating lens(es) may be placed between the light source and the mask. In some embodiments, the collimating lens comprises a first lens and a second lens, wherein there is a predetermined ratio between the curvatures of the lenses, wherein the ratio is between 0.40 and 1.15, for example, between 0.60 and 0.95. In some embodiments, the collimating lens(es) are Fresnel lenses. The projector unit(s) may further comprise one or more focusing lenses or lens elements configured to focus light at a predetermined working distance. Preferably, the projector unit(s) are configured to generate a predetermined pattern that can be projected onto a surface. Each of the projector units is associated with its own projector plane, which is determined by the projector optics. As an example, if the projector unit is a back-illuminated mask projector, the projector plane may be understood as a plane containing the mask. The projector plane includes a plurality of pattern features of the projected pattern.

[0102] The projector unit may include an aperture of a predetermined size such that it provides a pupil diameter between 0.2 mm and 0.7 mm (e.g., between 0.3 mm and 0.6 mm). In experiments performed by the inventors, it was found that a pupil diameter between 0.2 mm and 0.7 mm was particularly useful because it provided a particularly well-focused projected pattern over a large focus range (e.g., a focus range between 16 mm and 22 mm). In particular, it was found that a pupil diameter between about 0.3 mm and about 0.5 mm provided a good compromise between imaging resolution (e.g., resolution of the pattern) and depth of focus (i.e., focus range). In some cases, depth of focus can be understood as the maximum range over which an object appears to be in acceptably focus (e.g., within a given predetermined tolerance). In some embodiments, the aperture and / or pupil diameter are substantially the same for the projector unit and the camera.

[0103] For certain applications, such as dental scanning applications, the scanning device preferably has a working distance of between 10 mm and 100 mm. In experiments performed by the inventors, it was found that a working distance of the projector unit between 10 mm and 70 mm (e.g., between 15 mm and 50 mm) is particularly useful because the optical system (e.g., the scanning unit(s)) takes up less space within the scanning device, and because it is desirable to be able to scan objects that are very close to the scanning device. Since the optical system takes up less space within the scanning device, it also allows multiple scanning units to be placed in succession within the scanning device. In a preferred embodiment, the scanning device is capable of projecting a focused pattern at the exit of the scanning device head (e.g., at the optical window of the scanning device or at an opening in the surface of the scanning device). The working distance can be understood as the distance from the object to the lens when the image is in sharpest focus. The working distance can also or alternatively be understood as the distance from the object to the front lens (e.g., the front lens of the projector unit). The front lens can be one or more focusing lenses of the projector unit.

[0104] In some embodiments, the choice of aperture and working distance results in a numerical aperture of the projector unit between 0.0035 and 0.015, which was found to provide good imaging resolution, i.e. a pattern with pattern features that are well focused within a given focus range, and further provides a good compromise in terms of defocus, lens aberrations and diffraction. In experiments performed by the inventors, it was found that a numerical aperture of the projector unit between 0.005 and 0.009 provides an ideal compromise between imaging resolution and depth of focus. Therefore, a numerical aperture within this range was found to be the best balance to mitigate the negative effects of defocus, lens aberrations and diffraction on resolution. The numerical aperture of the projector unit and the camera(s) may be the same. The numerical aperture may be understood as the object-space numerical aperture.

[0105] The (one or more) light sources can be configured to generate light of a single wavelength or a combination or range of wavelengths (monochromatic or polychromatic). The wavelength combination can be generated by a light source configured to generate light within a wavelength range (e.g., white light). In a preferred embodiment, each projector unit includes a light source for generating white light. The white light source can be used in combination with a mask to form a back-illuminated mask projector unit. Alternatively, the (one or more) projector units can include multiple light sources (e.g., LEDs) that individually generate light of different wavelengths (e.g., red, green, and blue), which can be combined to form light comprising different wavelengths. Thus, the light generated by the (one or more) light sources can be defined by wavelengths that define a specific color or by different wavelength ranges that define a color combination (e.g., white light). In some embodiments, the projector unit includes a laser, such as a blue or green laser diode for generating blue or green light, respectively. This has the advantage of enabling a more efficient projector unit, which can achieve faster exposures compared to using, for example, a white light diode.

[0106] In some embodiments, the scanning device includes a light source configured to excite a fluorescent material to obtain fluorescence data from a dental object (e.g., from a tooth). Such a light source can be configured to produce a narrow range of wavelengths, such as within a range between 380 nm and 495 nm, such as a wavelength between 395 nm and 425 nm. In some embodiments, the scanning device includes one or more infrared light sources configured to emit infrared light that can penetrate dental tissue. Infrared light can be understood as light having a wavelength in the range of about 700 nm to about 1.5 μm. The scanning device may also include one or more NIR light sources, such as a plurality of NIR light sources, for emitting near-infrared light having a wavelength in the range of about 750 nm to about 1.4 μm. Light in the infrared range can be used for diagnostic purposes, such as for detecting caries areas in the oral cavity.

[0107] The projector unit can be configured to sequentially turn the light source on and off at a predetermined frequency, wherein the light source is on for a predetermined time period. For example, the time period can be between 3 milliseconds (ms) and 10 milliseconds (ms), for example, between 4ms and 8ms. The predetermined frequency for turning the light source on and off can be between 25Hz and 35Hz, for example, approximately 30Hz. In some embodiments, the exposure time of the image sensor is less than 15 milliseconds (ms), for example, less than 10ms, for example, between 4 and 8ms. These indicated exposure times preferably correspond to the time period during which the light source of the projector unit flashes as described above. Therefore, the advantage of configuring the light source to flash during the time period as described above is that blurring caused by relative motion between the scanner and the scanned object is minimized. This blurring is also referred to as motion blur. The light source of the projector unit can be configured to generate unpolarized light, for example, unpolarized white light.

[0108] In some embodiments, the projector unit includes a light source for generating white light. This has the advantage that white light enables the scanning device to simultaneously capture data or information related to both surface geometry and surface color. Thus, the same set of images can be used to provide both the geometry of the object (e.g., in terms of 3D data / 3D representation) and the color of the object. Consequently, a digital 3D representation of the object expressing both the color and geometry of the object can be generated without aligning the recorded data related to the surface geometry and the recorded data related to the surface color.

[0109] The pattern is preferably defined by a mask having a spatial pattern. The pattern may include a predetermined arrangement comprising any of stripes, squares, dots, triangles, rectangles, pentagons, hexagons, and / or combinations thereof. In a preferred embodiment, the generated illumination pattern is a checkerboard pattern comprising a plurality of checkerboard cells. Figure 1 Such a pattern is exemplified in . Similar to a regular checkerboard, the checkerboard cells in the pattern have alternating dark and light colors, corresponding to areas of low light intensity (black) and areas of high (relatively high) light intensity (white). Therefore, for ease of reference, the checkerboard cells of the checkerboard pattern will be referred to herein as white and black checkerboard cells.

[0110] The pattern includes multiple pattern features. When a pattern including such pattern features is projected onto the surface of a 3D object, the collected object image will similarly include multiple image features corresponding to the pattern features. Pattern / image features can be understood as separate, clearly defined positions in a pattern / image. Examples of image / pattern features include angles, edges, vertices, points, transitions, dots, stripes, etc. In a preferred embodiment, the image / pattern features include the corners of the checkerboard in a checkerboard pattern. In other embodiments, the image / pattern features include the corners in a polygonal pattern (e.g., a triangular pattern).

[0111] A camera, as used herein, may be understood as a device for capturing an image of an object. Each camera includes an image sensor for generating an image based on incoming light (e.g., light received from an illuminated 3D object). Each camera may also include one or more lenses for focusing the light. For example, the image sensor may be an electronic image sensor, such as a charge-coupled device (CCD) or an active pixel sensor (CMOS sensor). The image sensor may be a global shutter sensor or a rolling shutter sensor. An advantage of using a rolling shutter sensor is that, for a given pixel array size, the sensor can be made smaller compared to, for example, a global shutter sensor. Global shutter sensors typically include more electronics per pixel, resulting in a larger sensor area or volume footprint, thus occupying more space. Therefore, rolling shutter sensors are advantageous for space-constrained applications, such as intraoral scanning devices, particularly for compact intraoral scanning devices. Each image sensor may define an image plane, which is a plane containing the projected image of the object. Typically, each image captured by the image sensor(s) may include multiple image features, where each image feature is derived from a pattern feature of the projected pattern. In some embodiments, a color filter array, such as a Bayer color filter, is disposed on the image sensor(s). A color filter array may include a mosaic of tiny color filters placed over the pixel sensors of each image sensor to capture color information.

[0112] According to some embodiments, the scanning device includes one or more processors configured to perform one or more steps of the method. The scanning device may include a first processor configured to determine image features in the acquired image. The first processor may be configured to determine the image features using a neural network. The first processor may be selected from the following group: a central processing unit (CPU), an accelerator (offload engine), a general-purpose microprocessor, a graphics processing unit (GPU), a neural processing unit (NPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a dedicated logic circuit, a dedicated artificial intelligence processor unit, or a combination thereof. The scanning device may also include a computer memory for storing instructions that, when executed, cause the first processor to perform the step of determining image features in the acquired image.

[0113] The scanning device may further include a second processor configured to execute the steps of a computer-implemented method for generating a digital representation of a three-dimensional (3D) object. For example, the second processor may be configured to run a tracking algorithm comprising the following steps:

[0114] - assigning one or more depths to each image feature, wherein the depth(s) are determined using triangulation;

[0115] - generating image feature groups, wherein each image feature group includes one or more image features;

[0116] - generating one or more seed proposals, wherein each seed proposal includes at least a predetermined number of image feature groups, wherein generating each seed proposal comprises the following steps:

[0117] - adding one or more image feature groups to the seed proposal based on one or more predetermined first criteria;

[0118] - accepting a seed proposal(s) comprising at least a predetermined set of image features; and

[0119] - generating a digital representation of the 3D object by adding the seed proposal(s) to the digital representation based on one or more predetermined second criteria.

[0120] The computer memory may also store instructions that, when executed, cause the second processor to perform a method for generating a digital representation of a three-dimensional (3D) object. For example, the second processor may be a central processing unit (CPU), a processor utilizing a reduced instruction set computer (RISC) instruction set architecture (e.g., an ARM processor), or other suitable microprocessor. The second processor may include computer memory. The first and second processors may both be located on the scanning device, and they may be operatively connected such that the first processor provides input to the second processor. Alternatively, the first processor may be located on the scanning device, and the second processor may be located on the computer system described herein. For example, the first processor may be configured to determine image features in an image and then provide data related to the determined image features to the second processor. The data may include image feature coordinates and other attributes, such as a camera index or predetermined characteristics of (one or more) image features, such as a phase. The second processor may then be configured to generate a digital representation of the 3D object, such as in the form of a point cloud. The scanning device may further be configured to provide the digital representation to the computer system for rendering the representation. The computer system may further process the digital representation, such as by stitching the point cloud received from the scanning device and / or by fitting one or more surfaces to the stitched point cloud. This further processing by the computer system may also be referred to herein as reconstruction. The output of the reconstruction is a digital 3D model of the scanned object. The digital 3D model can be rendered and displayed on a display, such as a display connected to the computer system.

[0121] The scanning device is preferably configured to acquire an image map, wherein a set of images includes an image from each camera of the scanning device. As an example, if the scanning device includes four cameras, the scanning device can successively acquire four sets of images, wherein the correspondence problem is solved successively within each set of images. The scanning device can be configured to solve the correspondence problem by the following two main steps: a first step, wherein image features in a set of images are determined, and a second step, wherein 3D points are determined, for example based on triangulation. Finally, a 3D representation of the scanned object can be generated based on the 3D points. In some embodiments, the scanning device includes a first processor, wherein the first processor is configured to perform the step of determining image features in the images.

[0122] The scanning device preferably includes a module for transmitting data (e.g., an image or a point cloud) to one or more external devices (e.g., a computer system). The module can be a wireless module configured to wirelessly transmit data from the scanning device to the computer system. The wireless module can be configured to perform various functions required for the scanning device to wirelessly communicate with a computer network. The wireless module can utilize one or more of the IEEE 802.11 Wi-Fi protocol / integrated TCP / IP protocol stack that allows the scanning device to access a computer network. The wireless module can include a system-on-chip with different types of built-in network connectivity technology. These can include commonly used wireless protocols such as Bluetooth, ZigBee, Wi-Fi, WiGig (60GHz Wi-Fi), etc. The scanning device can further (or alternatively) be configured to transmit data using a wired connection (e.g., an Ethernet cable).

[0123] Overview of a solution to the corresponding problem

[0124] As previously mentioned, the present inventors have solved the correspondence problem in a fast and reliable manner, enabling the rapid generation of 3D representations by adding sets of image features for which they have solved the correspondence problem. Thus, rather than solving the correspondence problem feature by feature, it can be solved for sets of features, resulting in a consistent and reliable solution for all features in the set, as described in detail herein. Systems and methods for solving the correspondence problem are further described in PCT / EP2022 / 086763 and PA202370115 by the same applicants, which are incorporated herein by reference in their entirety.

[0125] The images may be acquired from different viewpoints, at different times, or with general motion of objects in the scene relative to the camera(s). The correspondence problem may arise in the stereo case when two images of the same object are acquired, or it may be generalized to the N-view correspondence problem. In the latter case, the images may come from N different cameras taking the images simultaneously, or from a single camera in motion relative to the object / scene. Similarly, the correspondence problem arises with one camera and one projector for scanning devices based on triangulation. The problem becomes even more difficult when the objects in the scene are in motion relative to the camera(s).

[0126] According to a preferred embodiment of the presently disclosed method, the method comprises the following steps:

[0127] - acquiring two or more images of a three-dimensional (3D) object, wherein each image includes a plurality of image features;

[0128] - determining image characteristics for each of the images;

[0129] -determining one or more depths associated with each image feature using triangulation;

[0130] - generating image feature groups, wherein each image feature group includes one or more image features;

[0131] - generating one or more seed proposals, wherein each seed proposal includes at least a predetermined number of image feature groups, wherein generating each seed proposal comprises the following steps:

[0132] - adding one or more image feature groups to the seed proposal based on one or more predetermined first criteria;

[0133] - accepting a seed proposal(s) comprising at least a predetermined set of image features; and

[0134] - generating a digital representation of the 3D object by adding the seed proposal(s) to the digital representation based on one or more predetermined second criteria.

[0135] Thus, when generating the digital representation, the entire set of image features can be added to the representation.Thus, the presently disclosed method provides a more robust and reliable solution to the corresponding problem in dental scanning systems.

[0136] A scanning device (e.g., the intraoral 3D scanning device disclosed herein) can be used to acquire two or more images of a three-dimensional (3D) object. The scanning device can be configured to provide / transmit the images to a processor and / or computer system that includes means for performing at least some of the steps of the computer-implemented methods disclosed herein. Thus, the steps of the above-described methods can be performed by one or more processors or computer systems. In a preferred embodiment, the scanning device includes one or more processors configured to perform, at least in part, any of the computer-implemented methods disclosed herein.

[0137] The following description is intended to provide an overview of the disclosed method at a general level. A more detailed description of the system and method is provided elsewhere herein. The first step of the method is to acquire two or more images of a three-dimensional (3D) object, wherein each image includes a plurality of image features. Typically, the two or more images will be acquired by a scanning device (such as the scanning device currently disclosed), which includes one or more cameras for acquiring images. The scanning device also includes one or more projector units for projecting a predetermined pattern onto the surface of the 3D object. The projected pattern includes a plurality of pattern features, so that the acquired image similarly includes a plurality of image features. As an example, the projected pattern can be a light pattern, such as a checkerboard pattern, wherein the image / pattern features are defined as the corners of the checkerboard pattern. Further examples are provided in the detailed description.

[0138] The next step of the method is to determine image features, preferably in each acquired image. This can be accomplished using a neural network trained to determine image features. The next step of the method can be to associate, for each pattern feature / projector ray, multiple image features from the determined image features with the pattern feature / projector ray. Each pattern feature can be associated with zero or more image features (e.g., one or more image features, e.g., multiple image features). The next step of the method can be to determine, for each image feature, one or more possible depths for that image feature. Thus, multiple depths can be assigned to each image feature, as it is not known which depth is the true solution to the corresponding problem for that particular image feature. The depth(s) are preferably determined by triangulation. Depth can be understood as the distance from the projector position along a given projector ray to a 3D midpoint where the projector ray intersects the camera ray within a given tolerance / distance. In particular, triangulation can be utilized when each camera phase of the scanning device has a predetermined fixed position relative to one or more projector units, particularly when the fixed position corresponds to an angle formed between the camera's viewing axis and the corresponding projector unit axis.

[0139] The next step in the method may be to sort the image features associated with a given pattern feature / projector ray by depth. This sorting is preferably performed / repeated for all pattern features. The sorted image features may be stored in a list. The method may then include a step of grouping the image features into groups of predetermined sizes, thereby generating image feature groups. This step may be performed iteratively, wherein a sliding window is used to navigate through the sorted image feature list to determine groups of image features each having a unique camera index, i.e., such that image features within the same group originate from images acquired using different cameras. In this context, a sliding window should be understood as a mathematical technique in which sublists (or subarrays) are operated on an underlying set / list / array. Sliding window techniques are well-known in mathematics. As an example, in the case of a scanning device comprising a projector unit and four cameras, the sliding window may have an initial size of four elements, thereby initially generating groups of four image features. In the next iteration, the sliding window size may be reduced to generate groups of three image features, and so on. This method may be performed for all pattern features in the pattern.

[0140] The next step of the method may be to generate a set of image feature groups, also referred to herein as a seed proposal. The seed proposal preferably includes at least a predetermined number of image feature groups. Thus, the seed proposal is generated such that it includes equal to or more than the predetermined number of image feature groups. The image feature groups are preferably added to the seed proposal based on one or more first criteria, examples of which are given in the detailed description.

[0141] The next step of the method is to generate a digital representation of the 3D object based on the seed proposal(s). The digital representation may be in the form of one or more point clouds comprising a plurality of 3D data points. The digital representation may be provided to a computer system for reconstructing a digital 3D model (e.g., a 3D surface model) based on the representation. The reconstruction may include the step of stitching the point clouds received from the scanning device, and it may also include the step of fitting one or more surfaces to the stitched point clouds, thereby generating a 3D surface model (digital 3D model). The 3D model may be rendered by the computer system and displayed on a display.

[0142] Thus, the method provides a framework for generating digital representations of 3D objects, wherein the correspondence problem is solved for a set of image features such that features within such a set are considered to form a consistent and reliable solution to the correspondence problem.

[0143] Light patterns and their characterization

[0144] As previously mentioned, the density of the projected light pattern is advantageously selected such that a density measure of the projected light pattern is above a predetermined threshold to increase detail in the digital representation of the scanned object.

[0145] A density measure is an indicator of the amount of information carried in a pattern, and there are different ways to define a density measure so that it can be objectively measured. In many cases, a light pattern can be characterized by similar features (e.g., only two different features) that are repeated only in two dimensions within the pattern. A checkerboard pattern is an example of such a pattern, such as a pattern of alternating black and white squares. The density measure used herein can then simply be the total number of squares in the checkerboard pattern. However, there are many other types of patterns with similar pattern features that are suitable for this purpose. For example, similar pattern features can be selected from the group consisting of checkerboard, triangles, hexagons, dots, lines, and striped lines. Thus, typically, a light pattern can include a predetermined number of similar pattern features. The density measure can then be defined as the total number of similar pattern features in the pattern. In a preferred embodiment, the light pattern is non-coded, so that no part of the pattern is unique. The light pattern can be a static light pattern, i.e., a pattern that does not change over time. The light pattern can be a structured light pattern having a discrete distribution of unconnected light spots, where the light spots correspond to pattern features.

[0146] In a preferred embodiment, the light pattern is a polygonal pattern comprising a plurality of polygons. The polygons may be selected from the group consisting of triangles, rectangles, squares, pentagons, hexagons, and / or combinations thereof. The polygons may be composed of sides and corners, wherein the pattern features correspond to the corners in the pattern. Preferably, the polygons are repeated in a predetermined manner within the pattern. As an example, the pattern may include multiple repeating units, wherein each repeating unit includes a predetermined number of polygons, and wherein the repeating unit is repeated throughout the pattern. Preferably, the light pattern includes a large number of polygons and / or pattern features, for example, at least 3,000, or at least 10,000, preferably at least 15,000, and even more preferably at least 19,000. The number of polygons and pattern features need not be the same. As an example, a polygonal pattern comprising 100x100 squares (each square forming a polygon) would have a total of 10,000 squares / polygons. If a pattern feature is understood as the intersection of four squares, such a pattern would have 99x99 pattern features, or 9,801.

[0147] In some embodiments, the light pattern constitutes a polygonal pattern comprising a repeating pattern of squares, such as a checkerboard, wherein pattern features are understood to be the corners of any given square. Thus, for a pattern of 100x100 squares, such embodiments and definitions would mean that, if all corners (including corners at the outer boundaries of the pattern) are included, the pattern comprises 10201 pattern features. In general, the most preferred embodiments of the light pattern comprise a large number of polygons and pattern features. A checkerboard pattern is a pattern that can carry a particularly large number of pattern features because each black square is surrounded by white squares, and each white square is surrounded by black squares. In some embodiments, the length of each checkerboard in the checkerboard pattern is between 100 μm and 200 μm. When the pattern is imaged on (one or more) image sensors, this can in some cases correspond to a pixel size of between 4-16 pixels (e.g., 8-12 pixels). Through experiments, the inventors have realized that at least 4 resolvable pixels per checkerboard period are sufficient to obtain reasonable contrast and clearly defined edges of the projected checkerboard pattern. In some embodiments, the pattern includes at least 100x100 checkerboard cells arranged in a checkerboard pattern, such as those of the aforementioned sizes. Such a pattern has a large number of pattern features, for example, where the corners of the checkerboard cells constitute features. Therefore, such a pattern places high demands on the optical system of the scanning device. Thus, the light pattern can resemble a checkerboard pattern with alternating checkerboard cells of varying light intensities. In other embodiments, the light pattern comprises a distribution of discrete, unconnected light spots.

[0148] Another way to objectively define a density metric for the projected light pattern is to analyze / measure the projected pattern in an area A on a planar surface. For example, the density metric can be defined as in, and where I = I(x, y), 0 ≤ I(x, y) ≤ 1, is the intensity signal measured in area A. This definition of the density metric can be used for any type of light pattern, as it is simply a parameter that provides the number of measured intensity variations integrated over area A. If the light pattern includes a predetermined number of similar pattern features, such as the checkerboard pattern exemplified below, then this more generalized density metric is not necessarily equal to the number of similar pattern features, but is at least proportional to the number of similar pattern features, such as the number of similar pattern features measured in area A on a planar surface.

[0149] In some cases, in order to objectively characterize the pattern, it may be advantageous to select a distance from the tip of the probe to the plane such that the projected light pattern is at least substantially focused, thereby simulating the situation during scanning, in which the pattern is clearly outlined on the scanned object. It may be desirable to specify a distance from the tip of the probe to a planar surface having an area A, which may be, for example, between 5 and 50 mm, preferably 15 mm, corresponding to the typical distance between the tooth being scanned and the tip of a handheld probe of an intraoral scanning device during intraoral scanning.

[0150] Many different types of standard light sensors can be used to perform such measurements. It may be necessary to define the noise of the sensor. For example, the intensity signal I may be measured using a sensor with a predetermined sensor noise level of 1% at its peak value. That is, the pattern is characterized by measuring the signal in area A using a sensor such that the noise level at its peak value is 1%. However, a fairly objective approach is to measure the intensity signal using a beam profiler. For example, a scanning device may be arranged such that the light pattern is projected onto a planar surface above the beam profiler.

[0151] The area A can be, for example, between 100 and 5000 mm 2 Between, preferably between 100 and 1000 mm 2 Between, for example 400mm 2 , for example an area of ​​20 mm x 20 mm, which would correspond to a typical viewing area of ​​the intraoral scanning device during an intraoral scan.

[0152] Defined as The generalized density measure is preferably at least 10,000, more preferably at least 20,000, even more preferably at least 30,000, most preferably 35,000, and may even exceed 40,000, wherein A is preferably between 100 and 1000 mm 2 Between, for example 400mm 2 .

[0153] The light pattern may also be a combination of a pattern having similar pattern characteristics (as exemplified above) and another type of light pattern (e.g., a coded pattern or a random or pseudo-random pattern). In this regard, it is preferred that at least 50%, more preferably at least 75%, most preferably at least 90%, or even 100% of the light patterns are patterns having similar repeating pattern characteristics.

[0154] Example

[0155] Consider a checkerboard pattern projected by a projector with a focal length of 0.7 mm and a checkerboard size of 7 μm. At a distance of 15 mm from the projector, a window of size 20 mm x 20 mm will contain a grid of 140 x 140 = 19,600 such checkerboard cells. This checkerboard pattern is Figure 20 Assume that the radius of the aperture in the projector is 0.1 mm - this determines the degree of diffraction of the pattern. Assume that the noise of the sensor used to measure the pattern in this scene is purely shot noise and has a magnitude of 1% at its peak. The resulting sensor signal is Figure 21 The integral can be calculated by adding the values ​​of the above graph and multiplying by the area of ​​each pixel, in this case giving Figure 21 The density metric of the medium signal is approximately 36993.

[0156] For comparison, Figure 22 A random pattern is shown. As mentioned above, the radius of the aperture in the projector is assumed to be 0.1 mm - this determines the degree of diffraction of the pattern. Assume that the noise of the sensor used to measure the pattern in this scene is purely shot noise and has a magnitude of 1% at its peak. The resulting sensor signal is Figure 23 The integral can be calculated by adding the values ​​of the above graph and multiplying by the area of ​​each pixel, in this case giving Figure 22 The density metric of the medium signal is approximately 24342, which is significantly lower than the density metric of the checkerboard pattern. This also proves that patterns with similar repetitive features (such as checkerboard patterns) are very efficient when high-density patterns are required. That is, there are indeed a large number of features in the checkerboard pattern. If the corresponding problem is solved accordingly, these features will directly translate into an increased number of features and details in the digital representation of the scanned object.

[0157] These examples demonstrate that light patterns with repeating pattern features are preferred because the information density in such patterns is indeed higher. Determining an objective density metric for light patterns with repeating pattern features is also much easier because it only requires counting the number of similar pattern features (e.g., counting the total number of checkerboards in a checkerboard pattern in a predetermined area). However, as demonstrated in the examples, for light patterns with non-trivial / non-repeating patterns, such as random or pseudo-random light patterns, a general method for objectively determining a density metric is necessary.

[0158] Pattern with fiducial marks

[0159] In some embodiments, the pattern includes a predetermined number of fiducial marks. As an example, the pattern may include at least one fiducial mark for every 20 pattern features, every 25 pattern features, every 50 pattern features, or every 100 pattern features. Figures 35-37Some examples are shown in FIG. A fiducial marker can be understood as a unique local portion of a pattern that can be identified from the rest of the pattern. A fiducial marker can have a unique geometric shape that makes it stand out from the rest of the pattern and thus can be easily identified. A fiducial marker can be dark (black), light (white) or a combination thereof, such as Figure 37 shown.

[0160] As an example, the geometric shape of the fiducial markers can be selected from the following group: a small dot, a triangle, a rectangle, a square, a pentagon, a hexagon, a polygon and / or a combination thereof. In some embodiments, each fiducial marker resembles a cross. As an example, the light pattern can constitute a checkerboard pattern comprising alternating black and white squares, and the pattern can include at least one fiducial marker for a block of 10x10 squares in the checkerboard pattern. The fiducial markers can constitute a sub-pattern within a larger (global) pattern. The sub-pattern can be described by a predetermined spatial frequency between the fiducial markers. As another example, there can be one or more predetermined distances between the fiducial markers. Therefore, the sub-pattern can be characterized by one or more parameters, such as the distance between the fiducial markers, the geometric shape of the fiducial markers, and the brightness (dark / bright) of the fiducial markers.

[0161] In some embodiments, the processor of the scanning device is configured to identify fiducial markers in the projected pattern. As an example, the processor can be configured to identify fiducial markers using a neural network. In other words, the neural network can be trained to recognize fiducial markers in the pattern. The advantage of using a pattern with fiducial markers (wherein the processor of the scanning device is configured to recognize fiducial markers in the pattern) is that it enables a more robust and / or more reliable solution to the corresponding problem, which solution can be used as input to update the geometric model associated with the scanning device. A more reliable solution can be understood as a solution in which fewer candidate points are considered for each projector ray, thereby reducing the risk of incorrectly assigning a point to a projector ray. A more robust approach can be understood as an approach that is more tolerant to any possible errors in the geometric model.

[0162] In general, the greater the density of the pattern, the more computationally intensive the solution of the correspondence problem becomes. Therefore, by providing a pattern with fiducial markers, the global pattern actually has two different densities: a first density associated with a first set of pattern features, and a second density associated with a second set of pattern features. As an example, if the pattern is a polygonal pattern, the first set of pattern features may be the corners of a polygon in the pattern. The fiducial markers may constitute a second set of pattern features, wherein the pattern formed by the fiducial markers has a lower density than the first set of pattern features. The correspondence problem may then be solved first for the pattern constituted by the fiducial markers by identifying the fiducial markers in the pattern. This solution may be used to correct / adjust the mathematical geometric model so that the scanning device may be calibrated dynamically, i.e., on-the-fly, preferably in real time. Preferably, the pattern formed by the fiducial markers has a significantly lower density than the rest of the pattern, so that the correspondence problem associated with the fiducial markers may be solved more quickly. The correspondence problem may be solved in a manner similar to that described elsewhere herein. Alternatively, any known technique for solving the correspondence problem may be used to solve the correspondence problem associated with the fiducial markers.

[0163] Dynamic calibration

[0164] An advantage of initially solving the correspondence problem for the fiducial markers is that the solution can be used to dynamically calibrate the scanning device. As previously mentioned, the optical system of the scanning device is typically mathematically modeled by a geometric model. The geometric model may include the relative position and orientation of the camera and projector with respect to each other. The geometric model may also include internal parameters related to the camera and projector, such as the focal length of these components. In addition, the geometric model may include one or more lens models of the projector and model the distortion in the image / pattern caused by the lens. Therefore, in some embodiments, one or more parameters of the mathematical geometric model are selected from the following group: the position of (one or more) cameras, the orientation of (one or more) cameras, the intrinsic parameters of (one or more) cameras and / or a combination thereof. Some examples of intrinsic parameters include the focal length of (one or more) cameras, the offset of the lenses in (one or more) cameras, the distortion of one or more lenses in the scanning device and / or a combination thereof.

[0165] The geometric model is typically updated during calibration, for example from the factory. If the physical configuration of the scanning device has changed since calibration, for example due to thermal expansion, wear or general degradation of the material, the geometric model will no longer be accurate (for example, the relative positions of the camera and projector may have changed). As a result, depth determination (for example, determination of 3D points by triangulation) becomes less accurate. In some cases, the accuracy may be so low that the correspondence problem can no longer be solved reliably. Therefore, it is desirable to calibrate the scanning device dynamically (i.e., over time) even during scanning. This can be achieved by solving the correspondence problem for fiducial markers and then using the solution as input / constraint for updating the geometric model associated with the scanning device. As a result, the generation of the digital 3D representation becomes more accurate.

[0166] The following outlines how the geometric model can be updated on the fly, for example, during a scanning session. Assume that the scanning device projects a pattern with multiple fiducial markers that define sub-patterns within the projected pattern. The density of the sub-patterns is less than that of the rest of the pattern, i.e., the number of fiducial markers is less than the number of image features in the entire pattern. Therefore, the correspondence problem is computationally easier and faster to solve for the sub-patterns defined by the fiducial markers. Once the knowledge is determined, this means that in each image, there are multiple image features for which the correspondence problem is solved, i.e., it is known which projector ray the feature originates from. If image features in multiple different images (obtained from different cameras) correspond to the same projector ray, then the 3D camera rays corresponding to these image features should intersect at exactly the same point in 3D. Therefore, in the event that they do not intersect exactly, the parameters of the geometric model can be changed by minimizing the distance between camera rays corresponding to the same projector ray. Thus, the mathematical geometric model can be updated based on the solution found for the pattern defined by the fiducial markers. Thus, in some embodiments, the step of calibrating the scanning device includes the steps of: mathematically projecting one or more camera rays and projector rays together in 3D space, the rays being associated with fiducial markers; and minimizing the distance between the camera rays and the given associated projector rays by dynamically adjusting one or more parameters of the mathematical geometric model. In some embodiments, calibration is performed in real time during scanning.

[0167] Alternatively, instead of using a solution to the correspondence problem based on identifying fiducial markers, another set of image features may be used instead. It is important to use a set of points for which the correspondence problem can be reliably solved. In the present disclosure, it is outlined how the correspondence problem can be solved for a set of image feature groups, wherein consistent and reliable solutions can be found in the set. It is further described how the reliability of an image feature group can be assessed, i.e., to what extent a potential solution can be trusted as a true solution. Thus, in some cases, a set of image feature groups may be found to be particularly reliable (e.g., if the pattern features are observed / imaged by many cameras at the same time, and / or if the camera rays and projector rays pass particularly close to each other, or if the set is of a certain size). This solution can then form the basis for updating the geometric model, as previously described.

[0168] Thus, in some embodiments, one or more processors of the scanning device are configured to identify one or more image features within the first set of acquired images and solve a corresponding problem associated with the image features, wherein solving the corresponding problem results in determining a point in 3D space based on the identified image features, wherein the point forms a solution to the corresponding problem; and wherein the (one or more) processors are configured to calibrate the scanning device by adjusting one or more parameters of a mathematical geometric model associated with the scanning device, wherein the adjustment is based on the solution to the corresponding problem.

[0169] Computer system

[0170] A computer system can be understood as an electronic processing device used to perform a series of arithmetic or logical operations. In this context, a computer system refers to one or more devices that include at least one processor (such as a central processing unit (CPU)) and some type of computer memory. Examples of computer systems that fall under this definition include desktop computers, laptop computers, computer clusters, servers, cloud computers, quantum computers, mobile devices (such as smartphones and tablet computers), and / or combinations thereof.

[0171] The computer system may include hardware, such as one or more central processing units (CPUs), graphics processing units (GPUs), and computer memory, such as random access memory (RAM) or read-only memory (ROM). The computer system may include a CPU configured to read and execute instructions stored in the computer memory, such as in the form of random access memory. The computer memory is configured to store instructions executed by the CPU and data used by these instructions. As an example, the memory may store instructions that, when executed by the CPU, cause the computer system to perform, in whole or in part, any computer-implemented method disclosed herein. The computer system may also include a graphics processing unit (GPU). The GPU may be configured to perform various tasks, such as video decoding and encoding, rendering of digital representations, and other image processing tasks. In some embodiments, the NPU or GPU is configured to perform steps of determining image features using a neural network.

[0172] The computer system may also include a non-volatile storage device in the form of a hard drive. The computer system preferably also includes an I / O interface, which is configured to connect peripheral devices used in conjunction with the computer system. More specifically, a display may be connected and configured to display output from the computer system. The display may, for example, display a 2D rendering of a generated digital 3D representation. An input device may also be connected to the I / O interface. Examples of such input devices include a keyboard and a mouse, which allow a user to interact with the computer system. A network interface may also be part of the computer system to allow it to be connected to an appropriate computer network, thereby receiving data (e.g., scanned data and images) from other computing devices and transmitting data to other computing devices. The CPU, volatile memory, hard drive, I / O interface, and network interface may be connected together via a bus.

[0173] The computer system is preferably configured to receive data from the scanning device, either directly from the scanning device or via a computer network (e.g., a wireless network). The data may include images, processed images, point clouds, data point sets, or other types of data. Data may be transmitted / received using a wireless connection, a wired connection, and / or a combination thereof. The computer system may be configured to perform, in whole or in part, any of the computer-implemented methods disclosed herein. In some embodiments, the computer system is configured to perform the computer-implemented method described herein for generating a digital representation of a three-dimensional (3D) object. In some embodiments, the computer system is configured to receive data (e.g., a point cloud) from a scanning device and then subsequently perform the steps of reconstructing and rendering a digital representation of the three-dimensional (3D) object. Rendering may be understood as the process of generating one or more images from three-dimensional data. The computer system may include computer memory for storing a computer program comprising computer-executable instructions that, when executed, cause the computer system to perform the method for generating a digital representation of a three-dimensional (3D) object.

[0174] Acquire images

[0175] According to a preferred embodiment of the present disclosure, the method comprises the step of acquiring an image of a three-dimensional (3D) object. Preferably, the image is acquired using a scanning device comprising one or more scanning units, wherein each scanning unit comprises a projector unit and one or more cameras. Figure 3 An embodiment of a scanning unit according to the present disclosure is shown. The center-to-center distance between the projector unit and a given camera of the scanning unit may be between 2 and 5 mm, preferably between 3 and 4 mm. The scanning device may be an intraoral scanning device for acquiring images of the interior of the oral cavity of a subject. The (one or more) projector units of the scanning device are configured to project a predetermined illumination pattern (e.g., a static light pattern) onto a surface, such as onto the surface of a three-dimensional object. Once projected onto the surface, some of the light will be reflected from the surface and may then enter the (one or more) cameras of the scanning device, so that an image of the 3D object may be acquired.

[0176] Preferably, one or more cameras per projector unit are used to capture images, for example, at least two cameras or at least four cameras are used per projector unit. In a preferred embodiment, each scanning unit of the scanning device comprises a projector unit and four cameras. The fields of view of the cameras preferably overlap. The advantage of having the camera fields of view overlap is that the accuracy is improved due to the reduced amount of image stitching errors. Another advantage of using multiple cameras (for example, two or more cameras) is that the reliability of determining 3D points is improved, thereby improving the accuracy of the generated 3D representation. The image can be processed by a processor located on the scanning device and then subsequently transmitted to the computer system. The image can also be transmitted to the computer system without any processing. In some embodiments, both the original image and the processed image are transmitted to the computer system by the scanning device. In some embodiments, the processor located on the scanning device receives the image as input and provides one or more point clouds as output by executing one or more steps of the disclosed computer-implemented method.

[0177] Determine image features

[0178] In a preferred embodiment of the method disclosed herein, the method includes the step of determining image features in the acquired images. This step is preferably performed by a processor, which may be located on the scanning device or computer system. Preferably, this step includes determining all image features in each image acquired by each camera. According to a preferred embodiment, the image features are determined using a neural network that is trained to recognize image features based on input data including the image. Examples of how to train the neural network are provided elsewhere in the specification. The neural network may form part of a computer program configured to run on a processor (which is, for example, located on the scanning device).

[0179] The neural network can be configured to receive as input a plurality of captured two-dimensional images and then output, for each image, image features determined in the image and the phase of the features. Specifically, the plurality of 2D images can be captured by a plurality of cameras, thereby obtaining a set of 2D images, wherein a set of images includes at least one image from each camera. Then, as described herein, a correspondence problem can be solved for the set of 2D images. In a preferred embodiment, the image features are corners in a checkerboard pattern. As previously described, the image is obtained by projecting an illumination pattern onto the surface of an object and then capturing an image of the light reflected from the object. Therefore, the captured image will include a pattern similar to the projected pattern, but the pattern in the image may be distorted due to the contours of the object surface compared to the mask pattern.

[0180] Since the projected pattern includes multiple pattern features, such as corners in a chessboard pattern, the captured image will similarly include multiple image features. Each image feature is derived from a pattern feature. Then, one problem is to figure out how to match the features in the image with the corresponding features in the projector plane. This problem is also commonly referred to as the correspondence problem in computer vision and stereo vision. If multiple images are captured by multiple cameras, that is, each camera captures at least one image, the correspondence problem can be expressed as how to identify corresponding image features in each captured image, that is, which image features in one image correspond to which image features in another image, and further which image features correspond to which pattern features. The projector plane can also be mathematically regarded as an image plane, even if the pattern of the projector plane is known a priori. Therefore, the pattern features are known in advance. Therefore, the purpose of the present disclosure is to describe a method for solving the correspondence problem in a new and efficient way.

[0181] Training of neural networks

[0182] Neural networks can be trained using supervised learning, where a set of example inputs and the expected output for each input is provided. This set of example inputs and outputs can constitute the neural network's training dataset. The difference between the network's output and the expected output can be quantified according to some cost function (e.g., cross-entropy). Furthermore, one or more parameters of the neural network can be adjusted during training to minimize the cost function using standard techniques (e.g., gradient descent, backpropagation, etc.). If the set of example inputs and outputs (the training dataset) is large enough, the neural network can deliver outputs close to the expected output, even for inputs never encountered during training.

[0183] The desired output for a given input depends on what the neural network is asked to do. In some embodiments, the desired output includes pixel-level annotations of the locations of structured light pattern features, i.e., the locations of pattern features, a pixel-level depth map of the corresponding 3D surface, or labels for classifying pixels as light or dark checkerboard.

[0184] Training datasets can include digital renderings of dental objects based on existing 3D representations. This enables the desired output to be calculated, eliminating the need for manual annotation. Furthermore, the positions of the intraoral scanning device used to generate the original 3D representation can be recorded at different times and used as the poses from which training images are rendered. This can improve the neural network, making it perform better in the real-world poses used during scanning.

[0185] The parameters of the rendering can vary between different images used for training, such as exposure, color balance, and camera parameters. This can create more data without having to perform more renderings, and can make the network more robust to changes in lighting, geometry, etc.

[0186] For example, by suppressing specular reflections and subsurface scattering, parts of the training dataset can be made to simulate diffuse surfaces, such as plaster. This ensures that the network performs well on materials other than enamel and gum. The training dataset can also be chosen to over-represent challenging geometries, such as scan landmarks, tooth preparations, and margin lines. This makes it more likely that the network will deliver the expected output for such situations in the field.

[0187] Generate neighborhood graph

[0188] A neighborhood graph may be understood herein as a two-dimensional (2D) graph (or discrete graph) that includes the location(s) of a plurality of image features associated with an image. The neighborhood graph preferably also includes predetermined properties of these features, such as their phase within the pattern and neighborhood information about each feature, which will be described in more detail below. The location(s) of the image feature(s) may be defined by image coordinates. Image coordinates may be understood as numerical coordinates, such as defined in a pixel coordinate system or a Cartesian coordinate system. As an example, an image coordinate may include an x-coordinate and a y-coordinate, such that an (x, y) pair uniquely defines the location of the image feature in the image. Each pair may also be associated with a pixel intensity or grayscale intensity corresponding to the point defined by these coordinates. The neighborhood graph may include additional information, such as one or more predetermined properties of the image features and a camera index indicating which camera acquired the image associated with the neighborhood graph. The camera index may also be a property of an image feature in the neighborhood graph, such that image features within the same neighborhood graph have the same camera index. As an example, the projected pattern may have a checkerboard pattern of alternating black and white checkerboards. Each checkerboard can be described as having four corners, four sides, and a color (black / white). In this example, image features and pattern features can constitute the corners of the checkerboard. Therefore, each of the image / pattern features (here, the corners) is associated with a phase, which can only take two values ​​(binary). Therefore, an example of a predetermined attribute of an image feature is phase. Phase can be black-white (BW) or white-black (WB). A corner in a checkerboard pattern will be surrounded by four checkerboards, two of which are black and the other two are white. The phase of BW corresponds to the corner, where the upper left checkerboard is black and the upper right checkerboard is white. The phase of WB is exactly the opposite of the phase of BW. Therefore, the phase of WB corresponds to the corner, where the upper left checkerboard is white and the upper right checkerboard is black. Alternatively, the phase can also be well defined by the bottom checkerboard (i.e., the checkerboard below the corner). The phase can also be defined by all four checkerboards surrounding the corner. However, for simplicity, the phase is only defined by two of the four checkerboards. As mentioned earlier, "white" can be understood as referring to areas with higher light intensity than "black." Ideally, the transition between the white and black checkerboards is completely sharp, but in reality, due to blurring, the transition can appear somewhat smeared.

[0189] According to a preferred embodiment, the method comprises the step of determining image features (e.g. corners) in the acquired image using, for example, a neural network. The neural network is preferably further configured to determine a predetermined property of the image feature (e.g. phase). This can be performed simultaneously with the determination of the image feature. The neural network can be further configured to determine the color of the checkerboard, i.e. to classify each pixel in the image as black or white. The advantage of using a projected pattern (where each pattern feature is associated with a phase, which can only take two values) is that the correspondence problem is simplified to two smaller correspondence problems, which are easier to solve. Therefore, the simplified problem places less computational and physical demands on the system. As a result, the correspondence problem becomes computationally faster to solve. It is generally desirable to solve the correspondence problem as quickly as possible in order to continuously and preferably in real time generate a digital representation of a three-dimensional (3D) object.

[0190] In a projector pattern, each pattern feature will typically be surrounded by multiple adjacent / neighboring pattern features, e.g., eight neighbors in the case of a checkerboard pattern (except near the border). For each feature, the neighborhood graph can store references to other features in the graph that correspond to these neighbors. In the case of a checkerboard pattern, these neighbors can be found by removing a portion of the image around each feature and flood-filling the area adjacent to the feature in question with either black or white pixels, depending on the pixel classification. Features that lie on the border of the flood-filled area can then be found as neighbors. Furthermore, if the neighbors around a given feature in the projector pattern are ordered in a certain way, e.g., starting from the top left corner and proceeding clockwise, the neighbors in the image can be ordered in the same way so that for each neighbor index in the projector pattern, the neighborhood graph stores a reference to the corresponding neighbor feature in the image (provided it exists).

[0191] Thus, the method may comprise the step of generating a neighborhood graph for each image, wherein generating each neighborhood graph comprises the step of determining a plurality of image features for each acquired image, wherein each image feature corresponds to a pattern feature, and wherein each image feature is associated with a set of 2D image coordinates, a camera index indicating which camera acquired the image, and a predetermined property (e.g., the phase of the feature). Finally, generating the neighborhood graph preferably comprises the step of storing, in the graph, references of each feature to neighboring features as described above.

[0192] Reject image features

[0193] In some embodiments, the method includes the step of determining whether one or more of the determined image features should be rejected based on one or more predetermined criteria. Examples of predetermined criteria include: Is the phase of the image feature similar to the phase of the pattern feature in the projector plane? Are the phases of adjacent image features correct in a predetermined area surrounding each image feature? In the case of a checkerboard pattern, is the length of the checkerboard edge surrounding the image feature being evaluated correct, i.e., does it correspond to the length of the edge in the pattern in the projector plane? In the event that one or more of the criteria are not met, i.e., if the answer to one or more of the above questions is no, the image feature being evaluated may be rejected. In some embodiments, an image feature is only rejected if two or more of the predetermined criteria are not met. If any image feature is rejected, the neighborhood graph(s) may be updated so that each neighborhood graph only includes image features that were not rejected.

[0194] Associating image features with pattern features

[0195] At this point, a neighborhood graph including image feature information (position, neighborhood information, and other properties, such as phase) has preferably been generated for each acquired image. As a next step, the method may include the step of associating the image features with pattern features. Each pattern feature may be associated with a certain number of image features, such as zero or more image features. To associate image features with pattern features, any suitable technique may be used, such as techniques employing projective geometry or epipolar geometry.

[0196] This paragraph is used to illustrate an example of how epipolar geometry can be used to associate image features with pattern features. Typically, epipolar geometry describes the relationship between two resulting views (i.e., when two cameras obtain images of the same 3D object from two different viewpoints). The relationship between a camera and a projector unit can similarly be described by epipolar geometry. In the following example, images are obtained using two or more cameras, each of which has a predetermined fixed position relative to the other cameras and one or more projector units of the scanning device. This example can be generalized to more cameras and more projector units. Each camera is configured to capture a 2D image of a 3D object. This conversion from 3D to 2D is called perspective projection. The projection operation can be modeled by a ray (also referred to herein as a camera ray) emitted from the camera and passing through the center of its focal point. Each emitted ray corresponds to a single point in the image of the camera.

[0197] As an example, to associate a given image feature with a pattern feature, consider a camera ray emitted from a first camera that passes through the center of its focal point and the image feature in the image plane of that camera. This camera ray will appear as a point in the image plane of the first camera. However, the projector unit (or second camera) will view the ray emitted from the first camera as a line / curve in the projector plane (in a real-world scene with lens distortion, this line will not be a straight line) because the projector unit and the camera observe the object from two different perspectives. The projected line / curve can also be called an epipolar line. Therefore, when a camera ray associated with an image feature is projected onto the image plane or projector plane of another camera, the ray forms a curve in these planes, and these curves are referred to as epipolar lines in this article. It should be noted that epipolar lines are straight lines only in an ideal situation (i.e., in the absence of lens distortion). Typically, when a 3D line (e.g., a camera ray or a projector ray) is projected onto a plane, the line forms a curve in the plane. The act of (virtually) projecting camera rays (which pass through image features and the center of the focus of the camera that captured the image) onto the projector plane to obtain epipolar lines may be referred to herein as simply projecting the image features onto the projector plane or simply as projecting (one or more) image features.

[0198] When each image feature in each image is projected onto the projector plane, it generates an epipolar line in the projector plane. According to a preferred embodiment, the method comprises the steps of: for each pattern feature in the projector plane, associating a plurality of image features from the determined image features, wherein associating the image feature with the pattern feature comprises the steps of:

[0199] - for each image, projecting each of the determined image features onto the projector plane, thereby obtaining an epipolar line for each projected image feature; and

[0200] - For each pattern feature, associate any image features whose corresponding epipolar lines pass within a predetermined distance from the pattern feature.

[0201] The above method is exemplified for a scanning device comprising at least one projector unit and one or more cameras. However, the method can be equally well extended to a scanning device comprising two or more scanning units, wherein each scanning unit comprises a projector unit and one or more cameras, for example, two or more cameras. Therefore, the method may include the following steps: for each pattern feature in each projector plane, associating multiple image features from the determined image features. The distance between a given pattern feature and a given epipolar line can be defined as the shortest distance from the pattern feature to any point on the epipolar line. This distance can be calculated using known methods. The method may specify a predetermined distance, such as a 2D distance, and set a threshold for when an epipolar line is close enough to be associated with a given pattern feature. Any epipolar line where the shortest distance from the line to the pattern feature exceeds the predetermined distance is not associated with that particular pattern feature. For some pattern features, it may not be possible to associate any image features, i.e., in this case, zero image features are associated. This may be the case, for example, if the projector ray corresponding to the given pattern feature does not hit the 3D object at all.

[0202] The above-mentioned method of associating image features with pattern features is based on projective geometry. Such a method ideally assumes that there is no lens distortion. In fact, lens distortion in an optical system, especially lens distortion of large values, means that the 3D lines in space do not form straight lines when projected onto a plane. Traditionally, the epipolar line in epipolar geometry refers to a straight line in the image plane or the projector plane. However, in this context, the epipolar line only refers to a line in 3D projected onto a 2D plane, thereby forming a curve. If there is lens distortion in an optical system (such as a scanning device), such a curve will not be a straight line. There are many known mathematical models that can correct radial distortion and tangential distortion caused by imperfect alignment of physical elements in the lens. An example of such a mathematical model is the Brown-Conrady model, which can be used to correct distortion or model distortion.

[0203] In this case, a neighborhood graph is preferably generated for each acquired image, which includes information about the image features (position and other attributes, such as phase). Thus, each neighborhood graph stores information about the image features, such as the 2D position of each determined image feature and predetermined attributes. In addition, each pattern feature is associated with multiple image features. Preferably, the next step of the disclosed method is to determine (and assign) possible depths of the determined image features in the acquired image.

[0204] Determining the depth of image features

[0205] Below is an example of how depth can be determined using triangulation. Each pattern feature in the projector plane can be projected in 3D by emitting a projector ray from the projector through the pattern feature. This is similar to emitting a camera ray from the camera through the image feature as described above. Similarly, a camera ray can be projected in 3D for each image feature, where the camera ray is emitted from the camera for that particular image feature. It should be noted that these projections are performed virtually by the processor implementing or executing the method. In other words, for each pattern feature, the projector ray is considered in 3D together with the camera ray of the associated image feature. In theory, the camera rays should perfectly intersect each other at a unique 3D point. However, in the real world, due to geometric distortion caused by lenses and finite apertures or blurring of unfocused objects, for example, camera rays do not intersect precisely at a given 3D point. Therefore, it is more important to determine the 3D point where the camera ray and the projector ray intersect each other within a predetermined distance or tolerance. If multiple camera rays intersect a given projector ray within a predetermined distance and at similar depths, this is equivalent to the cameras "agreeing" on a particular image feature, i.e., that image feature appears in both / all images, and the intersection point can be used to assign a depth to the image feature. The method steps are repeated for all pattern features, i.e., all projector rays and camera rays associated with the image feature are considered, so that one or more depths can be assigned to each image feature. Therefore, multiple depths can be assigned to each image feature, for example, one depth for each projector ray that the camera ray corresponding to the image feature approaches. Here, the term "approaching" can be understood as intersecting within a predetermined distance as described above. It should be noted that the depth is measured along each projector ray, i.e., from the projector position to the intersection point described previously.

[0206] As an example, consider a scanning device comprising a projector unit and four cameras. In this example, each camera captures at least one image, thereby capturing at least four images. A neighborhood graph can be generated for each image, and image features can be associated with pattern features, as described above. Then, for each pattern feature in the projector plane, a projector ray is projected in 3D together with up to four camera rays (one from each camera), wherein each camera ray passes through the image feature associated with the projector ray in question. All intersection points (3D points) where a camera ray intersects a given projector ray within a predetermined distance are then determined, so that one or more depths can be assigned to the image feature for the projector ray. In some cases, a depth cannot be assigned to the image feature for the projector ray in question. The method may further comprise a step of sorting the determined image features according to depth. The image features may be stored in a list, wherein the list is sorted according to image feature depth. Such a list may be generated for each pattern feature / projector ray.

[0207] According to some embodiments of the method, the step of determining one or more depths associated with each image feature comprises the following steps:

[0208] - for each camera, project a camera ray for each of the image features in the acquired image, wherein each camera ray is projected in 3D;

[0209] - casting a projector ray for each pattern feature, wherein each projector ray is cast in 3D;

[0210] - determining points in 3D where camera rays associated with image features of different camera indices intersect projector rays within a predetermined distance, wherein each of the determined points corresponds to a depth;

[0211] - assigning depth(s) to each image feature; and

[0212] - Optionally sorting image features associated with each pattern feature according to depth.

[0213] Generate image feature groups

[0214] As a next step, the image features are preferably grouped into image feature groups. Thus, the method may include the step of generating image feature groups, wherein each image feature group includes one or more image features. Preferably, each image feature within an image feature group has a unique camera index.

[0215] Below, an example of how to generate an image feature group is provided. In this example, the method further comprises the step of sorting the determined image features (for each pattern feature) according to depth before generating the image feature group. The image feature group can then be generated using a sliding window approach, wherein the image features are grouped using a sliding window of a predetermined size, wherein the image features within an image feature group each have a unique camera index. Thus, a sliding window can be applied to a list of image features sorted according to depth, wherein each instance of a group of image features is stored, the group having an amount of image features corresponding to the predetermined size of the window, wherein each image feature within the group has a unique camera index. Each time an image feature group is generated / stored, the image features of that particular group are preferably marked to keep track of used features, i.e., image features that have formed part of the group. Image features within the same group are corresponding image features obtained from different images. Thus, image features within an image feature group are expected to have approximately the same depth.

[0216] Typically, the number of cameras in the scanning device can be represented as N. In some embodiments of the disclosed method, a sliding window of size N is applied to the image feature list. As an example, the image can be acquired by a scanning device with four cameras, which corresponds to N=4. In this case, the sliding window will similarly have a size of 4, so that image feature groups with four image features in each group are generated. In some embodiments, the image feature groups are generated using an iterative process, in which the size of the sliding window is iteratively changed from an initial size N to a final size less than N. In each iteration, the size of the sliding window can be reduced by 1 until a predetermined size (e.g., 1) is reached. In the example above where N=4, the initial size of the sliding window will be 4. Then, in subsequent iterations of generating image feature groups, the sliding window will be 3, and so on. In the last iteration, the sliding window can be a predetermined size less than N. In some cases, this may even be 1, meaning that each image feature group includes only one image feature. The above method can be useful for grouping image features having approximately the same depth. The sliding window method can be performed for each pattern feature (i.e., for each projector ray).

[0217] Generating image feature groups may include determining a standard deviation of the depths of image features within each image feature group. A standard deviation may be assigned to each image feature group. Furthermore, each image feature group may be assigned a depth to which a potential 3D point is generated by projecting a projector ray. This depth may be the average of the depths of the image features within the image feature group. Alternatively, the assigned depths may be selected so that the sum of the squared distances between the image features and the projections of the 3D points onto the image plane where the image features reside is as small as possible. Furthermore, each group may be assigned a number corresponding to the size of the sliding window used to generate the group. Finally, each image feature group may be assigned a reliability score, where the reliability score indicates how reliable the image feature group is considered. The reliability score preferably indicates how close the projector and camera rays pass to each other, with a high reliability score indicating that the camera ray passes very close to a given projector ray. The standard deviation and / or the size of the sliding window may be used to indicate how reliable a particular image feature group is considered. Alternatively, the sum of the squared distances may be used to assign a reliability score to an image feature group. By way of example, an image feature group with four image features is generally considered more reliable than a group with only two image features. Thus, the standard deviation and / or the size of the sliding window can be used to determine the reliability score for each image feature group. Other parameters that form part of the reliability score are contemplated. The image feature groups can be stored in a list. In some embodiments, the list of image feature groups is sorted based on the standard deviation and / or reliability score of the group.

[0218] The determined image feature group should not be considered a definitive solution to the corresponding problem. Instead, the group (or list of groups) can be considered candidates for solutions to the problem, with some candidates considered more promising (i.e., more likely to be the true ideal solution) than others, as reflected in their reliability scores. The most promising image feature group, i.e., the group with the highest reliability score, is also denoted herein as a pre-seed, provided that the group includes N image features, where N represents the number of cameras in the scanning device used to acquire the image. A high reliability score corresponds to a low standard deviation of the depths within the group, i.e., the image features within the group have approximately the same depth. A high reliability score also corresponds to a sliding window size of N. Therefore, the reliability score can be a summary score formed by considering any of the following metrics: standard deviation, sum of squared distances, sliding window size, and / or a combination thereof, or other suitable metrics. The image feature groups can be sorted lexicographically, i.e., such that all N-cam groups (image feature groups comprising N image features) are sorted according to their reliability scores, the (N-1)-camera group is similarly sorted, and so on. There can only be one pre-seed per pattern feature / projector ray, however, there may be cases where no pre-seed is determined for a given pattern feature. This may be the case if some image features are only verified by n cameras (where n is less than the total number of cameras N).

[0219] Generate seed proposals

[0220] According to a preferred embodiment, the method includes the step of generating one or more seed proposals. Each seed proposal preferably includes at least a predetermined number of image feature groups, such as at least 10 image feature groups, or at least 15 image feature groups, or at least 20 image feature groups. In some embodiments, the predetermined number of image feature groups in the seed proposal is 15, such that each seed proposal includes at least 15 image feature groups.

[0221] A seed proposal may be understood as a collection of image feature groups, where each image feature group comprises at least one image feature. At least one attribute of an image feature group may be a point in 3D space, for example described by a set of Cartesian coordinates (x, y, z), where z represents the aforementioned depth assigned to the image feature group. An image feature group may preferably comprise further attributes, such as: a projector ray index and / or an index of individual image features within the group. A projector ray index may be understood as an index indicating which projector ray is associated with an image feature group. More generally, a seed proposal may be understood as a set of data points in space, such as a point cloud. The presently disclosed method provides a framework for generating a digital representation of a 3D object by continuously generating seed proposals and adding them to the digital representation of the 3D object (initially, i.e. before any (one or more) seed proposals are added, the representation may be empty).

[0222] To generate a seed proposal, one or more image feature groups are incrementally added to the proposal, preferably starting from a pre-seed.In a preferred embodiment of the method, only image feature groups that meet one or more predetermined first criteria are added to the proposal.

[0223] The step of adding the feature group to the seed proposal may include checking that the neighborhood information at the image feature level is correlated with the neighborhood information at the pattern feature level. More specifically, if a feature group corresponding to a given pattern feature has already been added to the seed proposal, then when considering whether to add a feature group corresponding to an adjacent pattern feature, the image features of the candidate image feature group should be the corresponding neighbors of the given image feature group in the neighborhood graph. In other words, if the candidate image feature group is a neighbor of the original image feature group with a given index, the image features in the original image feature group should have the features of the candidate image feature group as their neighbors with that index.

[0224] As an example, the one or more predetermined first criteria may include a criterion that the standard deviation of a given image feature group must be below a predetermined threshold before the image feature group is added to the seed proposal. As another example, the predetermined first criterion may include a check similar to that described in the previous paragraph. As yet another example, the reliability score of a given image feature group must be above a predetermined value before the group is added. Preferably, if the one or more predetermined first criteria are met, the image feature groups are added to the seed proposal(s) in a prioritized order based on their reliability scores.

[0225] The following paragraphs provide examples of how seed proposals are generated. First, a pre-seed associated with a given pattern feature in the projector plane is added to a queue. The pre-seed may constitute the first image feature group (3D point) in the seed proposal. The pre-seed is typically surrounded by a plurality (typically eight in the case of a checkerboard pattern) of adjacent image feature groups corresponding to neighbors in the projector pattern. Adjacent image feature groups are also referred to as "neighbors" hereinafter. The number of neighbors may be low near the edges of the image(s) or in areas where the surface of the 3D object is not hit by the projected illumination pattern. As a next step, the feature group may be taken from the queue and added to the seed proposal. Once a feature group is removed from the queue, all of its neighbors are preferably evaluated to determine whether any of them should be added to the queue. The process of taking one image feature group from the queue and possibly adding one or more new image feature groups to the queue may be performed iteratively until the queue is empty.

[0226] The purpose of the evaluation is to evaluate whether a given group of image features (e.g., pre-seeded neighbors) should be added to a queue. For each neighbor, it can be determined whether the reliability score exceeds a certain predetermined threshold. If so, it is preferably added to the queue. If not, the next neighbor is preferably considered. It can also be determined whether the image feature group being evaluated has the correct properties, such as the correct phase. In addition, it can be checked whether all image features of a given image feature group have the correct neighbors in each of their associated neighborhood graphs. The image feature group is preferably added to the queue as long as one or more predetermined first criteria are met. Preferably, the image feature groups are taken out of the queue in priority order according to their reliability scores, so that the most reliable image feature groups are evaluated before image feature groups with lower reliability.

[0227] Therefore, the queue is preferably configured as a priority queue, where the reliability score is the priority. In the case of a checkerboard pattern, up to eight image feature groups (the number of neighbors of a given image feature) can be added to the queue for each considered pre-seed / projector ray. However, preferably, only one image feature group can be taken from the queue in a single step. Therefore, image feature groups are removed from the queue and added to a given seed proposal in priority order.

[0228] Once the queue is empty, the seed proposal is evaluated to see if it includes at least a predetermined number of image feature groups. A seed proposal may include more groups than the predetermined number, but in a preferred embodiment, it is required to include at least the predetermined number. If the seed proposal includes fewer than this number of image feature groups, the seed proposal is preferably rejected, and the image feature groups can then form part of another seed proposal. If the seed proposal includes at least the predetermined number, the seed proposal is accepted. Preferably, the image feature groups that form part of the stored seed proposal are marked as used. The outlined method can then be repeated for all projector rays, thereby potentially generating multiple seed proposals. This multiple seed proposal can also be referred to as a set of seed proposals. Previously processed / evaluated projector rays are preferably marked so that such marked projector rays do not need to be re-evaluated in subsequent step(s). It should be noted that the above-described method for generating seed proposals is preferably repeated in one or more subsequent passes using a different set of criteria. The different set of criteria is preferably less stringent than the first set of criteria. As an example, in such a second pass of seed proposal generation, the size n (i.e., the number of image features within the group) of each image feature group can be less than N, where N represents the number of cameras. Thus, the method of generating seed proposals can be repeated iteratively, where n is iteratively reduced by 1 between each iteration. In subsequent passes, accepted seed proposals can be allowed to merge with each other and / or (one or more) new seed proposals can be allowed to merge with (one or more) accepted seed proposals. It is also possible to allow the accepted seed proposals to be grown under more relaxed criteria. The term "grow" can be understood as increasing the number of image feature groups in a given seed proposal.

[0229] Thus, the method may comprise the step of generating one or more seed proposals, wherein the generation of each seed proposal comprises the following steps:

[0230] i. adding one or more image feature groups to a seed proposal based on one or more predetermined first criteria; and

[0231] ii. Accepting seed proposals that include at least a predetermined number of image feature groups.

[0232] Generate digital representation

[0233] The final step of the disclosed method is to generate a digital representation of the 3D object by adding one or more seed proposals to the digital representation of the 3D object. In a preferred embodiment, the seed proposal(s) are added based on one or more predetermined second criteria. In some embodiments, the second criteria include the following: the seed proposal should include at least a predetermined amount of unlabeled image features in order for the seed proposal to be added to the digital representation.

[0234] The digital representation may initially be empty, i.e., contain no information. The digital representation may then be constructed by adding one or more seed proposals from a set of previously generated seed proposals to the (initially empty) representation. Preferably, seed proposals comprising the highest number of image feature groups are considered / added before seed proposals comprising fewer image feature groups. Thus, seed proposal(s) may be added to the digital representation in a prioritized order based on the number of image feature groups in each seed proposal. Preferably, once a seed proposal is added to the representation, its image feature groups are marked as being in use. Preferably, subsequently considered seed proposals do not include the marked image feature groups. However, in some embodiments, there may be overlap in image feature groups, such that some image feature groups belong to more than one seed proposal. Therefore, in some embodiments, each seed proposal should include at least a predetermined number of unlabeled image features in order for the seed proposal to be added to the digital representation. Once at least one seed proposal has been added, e.g., to an initial empty representation or pre-seed, a digital representation is generated. Subsequently, further seed proposals may be added to the representation, thereby increasing the amount of data (image feature groups) forming part of the representation. The digital representation may be provided in the form of one or more point clouds.

[0235] Detailed description with accompanying drawings

[0236] Figure 1 A perspective view of a scanning unit comprising a projector unit and four cameras is shown, wherein the projector unit is configured to project a light pattern (here, a checkerboard pattern) onto a surface of an object. The light pattern can be a static pattern over time. The light pattern can be a high-density pattern as described herein.

[0237] Figure 2 An embodiment of a scanning device including a scanning unit including a reflective element, here exemplified by a mirror, is shown. In this embodiment, the scanning device is configured to transmit scan data using a wired connection (e.g., a USB cable or an Ethernet cable). Furthermore, the scanning device can be powered via a wired connection to a power source. The scanning device can transmit the data to a computer system including a display for displaying a 3D representation of the dental object.

[0238] Figure 3 An embodiment of a scanning unit according to the present disclosure is shown. In this embodiment, the scanning unit includes a projector unit and four cameras, wherein the projector unit is arranged at the center of the scanning unit and the four cameras are symmetrically arranged around the projector unit. The scanning unit may also include a fixing unit configured to mount the projector unit and the camera(s) in the scanning unit.

[0239] Figure 4A cross section through a scanning device according to the present disclosure is shown. In this embodiment, the scanning device includes two scanning units that are arranged in series along the longitudinal axis of the scanning device to increase the field of view of the scanning device. Having a larger field of view enables large smooth features (such as the overall curve of a given tooth) to appear in each image, which improves the accuracy of subsequent stitching of corresponding 3D scan data (such as 3D surfaces) obtained from different sets of images. Each scanning unit is arranged in combination with a reflective element (such as a mirror), wherein the reflective element is configured to change the direction of light projected by a given scanning unit. The scanning device may also include a housing for accommodating the scanning unit and the frame. The housing may include an optical window arranged at the distal end of the scanning device. The optical window may be made of a polymer (such as polymethyl methacrylate (PMMA)), or may be made of glass (such as sapphire glass).

[0240] Figure 5A flow chart of an embodiment of the method of the present invention is shown. The first step 101 of the method is to receive or capture an image of a three-dimensional (3D) object. The image can be captured / provided by a scanning device (e.g., an intraoral 3D scanning device). The image can be captured by projecting a predetermined pattern onto the surface of the 3D object, wherein the pattern includes multiple pattern features, such as corners in a checkerboard pattern. The captured image will similarly include multiple image features. In step 102, the image features are determined. The image features can be automatically determined by a neural network trained to determine image features. The neural network can be further configured to output other properties of the pattern, such as a phase associated with each image feature. In step 103, the depth (or depths) of each image feature are determined using triangulation. For example, if an image feature is associated with five pattern features, that is, if the camera ray corresponding to the image feature intersects with five different projector rays within a predetermined tolerance, then five possible depths are assigned to the image feature. Each depth is measured along a projector ray from the projector plane to the intersection point. In step 104, image feature groups (3D points) are generated by grouping image features into groups of predetermined size, where each image feature within a group has a unique camera index. In other words, the image features within such a group originate from different images acquired by different cameras. The image feature groups are preferably generated using an iterative method, where a sliding window is applied to an ordered list of image features. The size of the sliding window can be reduced between each iteration, so that the size of the generated image feature groups similarly decreases between each iteration. Furthermore, each image feature group can be assigned a reliability score. The reliability score is a measure of how closely the camera rays of the image features within the group pass through the projector rays associated with the image feature group. In step 105, one or more seed proposals are generated. A seed proposal is a collection of image feature groups. When generating seed proposals, whether a given image feature group should be added to a queue is evaluated based on predetermined criteria. The queue is a priority queue, where the priority is the reliability score. As a starting point, a preseed is added to the queue, and then neighboring image features pointing to the preseed are evaluated. Any neighbors that meet the predetermined criteria are then added to the queue. Generally, many image feature groups can be added to the queue, but preferably, only one is removed from the queue at a given instance. As soon as an image feature group is added to the queue, the image feature group with the highest priority (i.e., highest reliability score) is removed from the queue and added to the seed proposal. The neighbors of the image feature group that was recently removed from the queue are then evaluated and possibly added to the queue. This can then continue until the queue is empty. Typically, image feature groups are taken out of the queue one by one based on their priority / reliability scores. However, each time an image feature group is removed from the queue and added to the seed proposal, its neighbors are evaluated based on the criteria and possibly added to the queue. Once the queue is empty, it is evaluated whether the number of image feature groups in the seed proposal exceeds a predetermined lower limit.If the seed proposal includes at least this minimum number of image feature groups, the seed proposal is stored. More seed proposals can be generated according to the outlined process, and some predetermined criteria can be relaxed in subsequent seed proposal generation. Thus, a set of seed proposals can be generated and stored. In step 106, a digital representation is generated based on the (one or more) seed proposals. The digital representation can be generated by adding (one or more) seed proposals to an initially empty representation. Preferably, the seed proposals are added to the representation in a priority order according to the size of the seed proposal (i.e. according to the number of image feature groups within the seed proposal). Each seed proposal includes a set of 3D points in space. Therefore, the digital representation similarly includes a set of 3D points, for example in the form of a point cloud. In subsequent method steps, a 3D surface model can be constructed based on the point cloud.

[0241] Figure 6 Flowchart showing an embodiment of the method disclosed in the present invention. Steps 201-203 are similar to those of Figure 5 In this embodiment, the method further comprises step 204, wherein the image features within a given projector ray, i.e., all image features associated with a given pattern feature, are sorted according to depth. This can be done for all pattern features, thereby generating an ordered list of image features for each pattern feature. Steps 205-207 are similar to those described with respect to Figure 5 Steps 104-106 are described.

[0242] Figure 7 Flowchart showing an embodiment of the method of the present disclosure. Steps 301-302 are similar to those of Figure 5Steps 101-102 described above. In this embodiment, the method further includes step 303, wherein a neighborhood graph is generated for each image based on the image features and the phase. The neighborhood graph can be provided as a 2D graph, showing information such as the position (one or more) of the image features, the connectivity of the image features (i.e., the relationship between the image features), the phase of the image features, or a combination thereof. The method further includes step 304, wherein the image features are associated with the pattern features or vice versa. This can be achieved by projecting a line in 3D (projector ray, camera ray) onto a 2D plane (image plane, projector plane), thereby obtaining a curve. As an example, for a given projector ray (line in 3D) projected onto the image plane, at each instance where the projector ray passes close to an image feature in the image plane (e.g., within a predetermined distance), the pattern feature corresponding to the projector ray is associated with the image feature. Conversely, for a given camera ray (line in 3D) projected onto the projector plane, at each instance where the projected camera ray passes close to a pattern feature, the image feature corresponding to the camera ray is associated with the pattern feature. Thus, each image feature can be associated with multiple different pattern features / projector rays. Thus, multiple depths can be assigned to the same image feature. Step 305 is similar to Figure 5 Step 306 is similar to step 103 described above. Figure 6 Steps 307-308 are similar to those described in step 204. Figure 5 Steps 104-106 of the relevant description.

[0243] Figure 8 A flowchart is shown that illustrates how the possible depths of a given image feature are determined according to the disclosed method. In step 401, a camera ray corresponding to the image feature is projected in 3D, i.e., emanating from the image feature. In step 402, a projector ray is projected in 3D for each pattern feature, such that each projector ray emanates from the pattern feature. In step 403, points in 3D are determined where the camera ray and one or more projector rays intersect within a predetermined distance. In step 404, the depth of each intersection point is determined by measuring the distance from the projector plane along each projector ray that intersects the camera ray. In step 405, the depth(s) are assigned to the image feature. Thus, multiple depths (one for each intersection point) can be assigned to an image feature. It should be noted that at this point / step in the method, the correspondence problem has not yet been solved. Therefore, it is unknown which of the assigned depths represents the true solution to the correspondence problem. Steps 401-405 can be repeated for all image features in all images.

[0244] Figure 9A flowchart is shown illustrating how an image feature group is generated according to the disclosed method. In step 501, image features associated with a given pattern feature are sorted / ordered by depth. It should be noted that within a given projector ray corresponding to a pattern feature, each image feature can only have a single depth. Image features can be assigned multiple depths because they can be assigned to several different pattern features. Therefore, there is no ambiguity when sorting image features associated with a given pattern feature by depth. Then, in step 502, a sliding window technique is employed to scan the ordered list of image features within the given projector ray / pattern feature. The sliding window has a predetermined first size n. In step 503, an evaluation is performed to determine whether n image features from n different cameras exist within the sliding window. If so, an image feature group is generated that includes the n image features. In step 505, the standard deviation of the depths within the image feature group is determined. In step 506, a reliability score is assigned to the image feature group. The standard deviation and / or the size of the sliding window can be used to determine the reliability score. However, other parameters for determining the score are contemplated. In general, the reliability score should reflect the likelihood that a given image feature set (3D point) forms part of the correct solution to the corresponding problem. In step 507, the size of the sliding window is reduced by 1, so that its size is n-1. Then, steps 502-507 can be repeated until the sliding window reaches a predetermined second size, such as 1. Therefore, an iterative method can be used to generate image feature sets, where the size of the sliding window is reduced by 1 in each iteration.

[0245] Figure 10 A flow chart is shown which illustrates how to evaluate whether a given image feature group should be added to a priority queue. In step 601, the image feature group is evaluated for correctness of a predetermined property (e.g., phase). If it has the wrong phase, the image feature group is ignored and one of the adjacent image feature groups is considered in step 602. If the predetermined property is correct, then in step 603 it is evaluated whether the image features within the group have the correct neighbors. This can be determined by consulting a neighborhood graph for each image feature. If this criterion is met, then in step 604 it can be evaluated whether the standard deviation of the image feature group is below a predetermined threshold. If all criteria (601, 603, 604) are met, then the group can be added to the priority queue in step 605. After adding the group, the method can continue by evaluating the next neighbor until all neighbors of the given image feature group that it is considered to start with have been evaluated.

[0246] Figure 11A flow chart is shown illustrating how image feature groups may be removed from a queue and added to seed proposals. The priority queue may include a number of image feature groups, each with a reliability score. Groups with the highest scores take precedence over groups with lower scores. In step 701, the image feature group with the highest reliability score is removed from the queue and added to the seed proposals. In step 702, the neighbors of the image feature group that was recently removed from the queue are evaluated. Preferably, only neighbors that have not been evaluated before are evaluated in step 702. Step 702 may then include the following steps: Figure 10 The steps 601-604 listed above are relevant. In step 703, any image feature groups that meet a predetermined first criterion are added to a queue. In some embodiments, the method may further include steps 704-707. In step 704, it is evaluated whether the queue is empty. If this is not the case, steps 701-704 may be repeated until the queue is empty. Once empty, it may be evaluated in step 705 whether the size of the seed proposal is sufficient, for example, whether the number of image feature groups within the seed proposal is equal to or exceeds a predetermined number. In step 706, if the size is less than the predetermined number, the seed proposal may be rejected. In step 707, if the seed proposal includes at least the predetermined number of image feature groups, the seed proposal may be accepted.

[0247] Figure 12 A flowchart is shown that illustrates how a digital representation can be generated based on seed proposals. In step 801, an initial digital representation is defined, which can be empty. In step 802, the seed proposals are considered in order based on their size (i.e., the number of image feature groups within the seed proposal). Thus, seed proposals with many image feature groups take precedence over seed proposals with fewer image feature groups. In step 803, an evaluation is performed to determine whether any image feature groups of the seed proposal are currently part of the digital representation. If not, the seed proposal is added to the digital representation in step 804. In step 805, the image feature groups belonging to the most recently added seed proposal(s) can be marked. It should be noted that steps 801-805 merely form an embodiment / example of how a digital representation can be generated based on seed proposal(s). As another example, steps 803 and 805 are not necessarily performed, such that a representation is generated by adding one or more seed proposals in order based on size. In yet another example, only steps 801 and 804 are performed, such that a representation is simply generated by adding previously generated seed proposal(s).

[0248] Figure 13A schematic diagram of a computer system is shown. The computer system may include a central processing unit (CPU) 901, which is configured to read and execute instructions stored in a computer memory, which may take the form of volatile memory, such as random access memory (RAM) 902. The computer memory stores instructions executed by the CPU and data used by these instructions. For example, the instructions may be related to the methods disclosed herein. The computer may also include a non-volatile storage device, such as a hard disk 906. The computer also includes an I / O interface 904, to which other devices can be connected. Such devices may include (one or more) displays 907, (one or more) keyboards 908, and (one or more) pointing devices 909 (e.g., a mouse). The display 907 can be configured to display a 2D reproduction of a digital representation of a dental object. A user can interact with the computer system via the keyboard 908 and / or mouse 909. A network interface 905 allows the computer system to be connected to an appropriate computer network to receive data from and transmit data to other computing devices. The CPU, computer memory (RAM / ROM), hard disk, I / O interface, and network interface can be connected via a bus.

[0249] Figure 14 1 shows an embodiment of a dental scanning system according to the present disclosure. The dental scanning system may include a scanning device 1001, such as an intraoral 3D scanning device. The system may also include Figure 13 The computer system 1003 described in . The scanning device 1001 can provide scan data 1002, such as an image or a point cloud, to the computer system 1003, and the computer system 1003 can be configured to generate a digital 3D surface model based on the scan data. The scanning device can include one or more projector units configured to project a predetermined pattern onto a 3D object, wherein each projector unit is associated with a projector plane; and one or more cameras for each projector unit, wherein each camera is configured to capture an image of at least a portion of the 3D object. The scanning device can also include one or more processors configured to execute any of the methods disclosed herein. In some embodiments, the dental scanning system includes only the scanning device 1001.

[0250] Figure 15 A rendered image of a 3D object, here some teeth of a subject, is shown. An illumination pattern (here a checkerboard pattern) is visible in the image. Such an image is an example of an image suitable for use as training input for the neural network described herein.

[0251] Figure 16 An example of a neural network output is shown. The output here is a checkerboard pattern, where multiple checkerboard squares (black / white) have been recognized by the neural network and classified as black or white.

[0252] Figure 17-18 Two graphs showing image features represented by points, where the image features correspond to Figure 16 The corners in the checkerboard pattern shown in . Figure 17 In , the image features belong to the BW (black-white-white-black) type, while in Figure 18 In the example, they are of the WB (white-black-black-white) type. These two graphs can be provided as the output of the neural network.

[0253] Figure 19 Another example of a neural network output is shown. Here, the neural network outputs an image showing some of the boundaries identified by the network.

[0254] Figure 20 An example of a static light pattern is shown, where the pattern is a checkerboard pattern, comprising a number of alternating black and white checkerboards. In this example, the checkerboard pattern is projected by a projector with a focal length of 0.7 mm and a checkerboard size of 7 μm. At a distance of 15 mm from the projector, a window of size 20 mm x 20 mm will contain a grid of 140 x 140 such checkerboards. Assume that the aperture in the projector has a radius of 0.1 mm - this determines the degree of diffraction of the pattern. Assume further that the noise of the sensor used to measure the pattern in this case is purely shot noise and its magnitude is 1% at the peak. The resulting sensor signal is as follows Figure 20 shown.

[0255] Figure 21 Shown Figure 20 The total density metric can be calculated by summing the intensity values ​​in the image and multiplying by the area of ​​each pixel, which in this case gives a value of approximately 36993. The mathematical expression for the total density metric is given elsewhere in the specification. Therefore, if the projected pattern is similar to Figure 20 The pattern in , then the integral The value given is 36993.

[0256] Figure 22 An example of a random pattern of black and white squares is shown. Assume that the radius of the aperture in the projector is 0.1 mm - this determines the degree of diffraction of the pattern. Further assume that the noise of the sensor used to measure the pattern in this case is purely shot noise and has a magnitude of 1% at its peak. The resulting sensor signal is Figure 23 shown.

[0257] Figure 23 Shown Figure 22 The density metric, the integral above, can be calculated by summing the intensity values ​​of the above image and multiplying by the area of ​​each pixel, in this case giving Figure 22 The density measure of the medium signal is approximately 24342, ie, a significantly lower density measure than the checkerboard pattern.

[0258] Figure 24 An example of a polygonal pattern composed of checkerboard cells, forming a checkerboard pattern, is shown, with each cell / square comprising four sides and four corners. This example checkerboard pattern comprises 6 x 6 cells, for a total of 36 cells. However, in a preferred embodiment where the light pattern is a checkerboard pattern, the checkerboard pattern may comprise a plurality of cells, for example, at least 100 x 100 cells.

[0259] Figure 25 Shown Figure 24 Examples of repeating units for the pattern shown. Other examples of repeating units can be given. For this example, the repeating unit shown is repeated three times horizontally and three times vertically to form Figure 24 pattern.

[0260] Figure 26 An example of a checkerboard pattern is shown, where pattern features are indicated as circles in the figure, which correspond to interior corners in the pattern. The pattern features may correspond to all corners in the pattern, or they may correspond to interior corners in the pattern as shown in this figure.

[0261] Figure 27 An example of a checkerboard pattern is shown, with pattern features indicated as circles corresponding to corners in the pattern, including corners on the pattern's outer boundary. Generally, as the checkerboard pattern expands with more squares, the difference between the number of interior corners and the total number of corners, including those at the boundary, decreases.

[0262] Figure 28 An example of a polygonal pattern composed of triangles, each of which includes three sides and three corners, is shown.

[0263] Figure 29 Shown Figure 28 Examples of repeating units of the pattern shown in .

[0264] Figure 30 An example of a polygonal pattern composed of triangles is shown, with pattern features indicated as circles, corresponding to corners in the pattern.

[0265] Figure 31 Shown Figure 30 An example of a repeating unit of the pattern shown in , with one of the edges highlighted.

[0266] Figure 32 An example of a polygonal pattern composed of hexagons, each of which includes six sides and six corners, is shown.

[0267] Figure 33 Shown Figure 32 A pattern in which a repeating unit 1 is indicated, as well as some pattern features 2 and edges 3 of the pattern.

[0268] Figure 34 An example of a polygonal pattern composed of triangles, each of which includes three sides and three corners, is shown.

[0269] Figure 35 An example of a pattern with a fiducial marker (here a cross is used as an example) is shown. In this example, the pattern is a checkerboard pattern, where some originally white squares have been replaced by dark squares, thereby creating multiple fiducial markers.

[0270] Figure 36 An example of a pattern with a reference mark (here a cross is used as an example) is shown. Figure 35 The example in is very similar, except that the spacing between the fiducial markers is larger.

[0271] Figure 37 An example of a pattern with a reference mark (here a cross is used as an example) is shown. Figure 35 The example in is very similar, except that some of the originally dark squares have been replaced by white squares, creating multiple fiducial markers.

[0272] Example

[0273] The following provides an example of how a neural network can be trained to determine image features. As input, the network can be provided with an image of a scene illuminated with a given pattern having three color channels, i.e., a WxHx3 tensor, where W is the width of the image and H is the height of the image. As training input, the neural network can be provided with a rendering of a 3D object (e.g., a dental object). Figure 14 An image suitable for training purposes is shown. The goal of a neural network is to deliver one or more types of data as output. Here are some examples:

[0274] - a channel containing the probability that a pixel is part of the pattern at phase 0 (corresponding to a black checkerboard);

[0275] - a channel containing the probability that a pixel is part of the pattern in phase 1 (corresponding to white pixels);

[0276] - a channel containing the probability that the pixel is in the background so that no pattern is visible in the pixel (marked in grey in the output image);

[0277] - a channel containing the probability that a pixel is part of a BW type (black-white-white-black) image feature;

[0278] - a channel containing the probability that a pixel is part of a WB type (white-black-black-white) image feature;

[0279] - a channel containing the probability that a pixel is not part of a feature; and

[0280] - A channel containing the probability that a pixel corresponds to a depth discontinuity (i.e., adjacent pixels have significantly different depths).

[0281] The neural network can be trained using supervised learning with a large number of input-output pairs, where an input-output pair is understood to be an image (a WxHx3 tensor) and at least one desired type of output data. Preferably, all seven types of output data are used in each input-output pair during supervised learning. As an example, the training set can include 15,000 such input-output pairs, but a larger set may improve the performance and reliability of the network.

[0282] As training data, rendered images of 3D objects (e.g. teeth) can be used. Figure 14 Examples of images suitable for training purposes are shown. The following data can be used to generate training data:

[0283] - a reference 3D model of the jaw or jaw part;

[0284] - Color images of 3D models;

[0285] - the trajectory of the scanning device as it moves while capturing the reference scan;

[0286] - one or more intrinsic parameters of the camera used to render the image;

[0287] - one or more intrinsic parameters of the projector used to generate the pattern; and

[0288] -Relative positions of camera and projector.

[0289] By using the above training data as input, we can generate true values ​​(desired output data) and input data for each pose along the trajectory through ray tracing. Figure 15-18 Different types of output data are shown. Figure 15 Shown is a checkerboard pattern projected onto teeth, where the neural network has identified multiple checkerboard squares (black / white). Figure 16-17 Two graphs showing image features represented as dots, where the image features correspond to corners in a checkerboard pattern. Figure 16 In , the image features are of the black-white-white-black type, while in Figure 17 In the , they are of the white-black-black-white type. Figure 18 The boundaries identified by the neural network are shown.

[0290] An example of a suitable network architecture for a neural network is the Lite-HRNet architecture, as described in the article “Lite-HRNet: A Lightweight High-Resolution Network” by Changqian Yu et al. (available at https: / / arxiv.org / abs / 2104.06403 ).

[0291] Further details of the invention

[0292] 1. A 3D scanning system for scanning a dental object, comprising:

[0293] -Intraoral 3D scanning device, the intraoral 3D scanning device comprising:

[0294] - at least one projector unit configured to project a light pattern onto the dental object along a projector optical axis, the light pattern comprising a plurality of pattern features;

[0295] - one or more cameras having at least partially overlapping fields of view along different camera optical axes and along the projector optical axis, each camera comprising an image sensor for acquiring one or more images, and

[0296] - one or more processors, wherein at least one processor is configured to generate a digital three-dimensional representation of the dental object based on triangulation between the camera and the light pattern.

[0297] 2. The 3D scanning system of item 1, wherein each of the camera optical axes defines an angle of at least 3 degrees with the projector optical axis.

[0298] 3. The 3D scanning system according to any of the preceding items, wherein the light pattern is a polygonal pattern comprising a plurality of polygons.

[0299] 4. The 3D scanning system of any preceding item, wherein the pattern comprises at least 10,000 pattern features and / or polygons.

[0300] 5. The 3D scanning system according to any one of items 2-4, wherein the polygon is selected from the following group: a triangle, a rectangle, a square, a pentagon, a hexagon and / or a combination thereof.

[0301] 6. A 3D scanning system according to any one of items 2-5, wherein a polygon is composed of edges and corners, and wherein the pattern features correspond to the corners in the pattern.

[0302] 7. A 3D scanning system according to any one of items 2-6, wherein the polygon is repeated in a pattern in a predetermined manner.

[0303] 8. The 3D scanning system of any one of items 2-7, wherein the pattern comprises a plurality of repeating units, wherein each repeating unit comprises a predetermined number of polygons, wherein the repeating unit is repeated throughout the pattern.

[0304] 9. A 3D scanning system according to any preceding item, wherein the pattern is non-encoded such that no part of the pattern is unique.

[0305] 10. A 3D scanning system according to any preceding item, wherein the pattern is encoded such that at least some parts of the pattern are unique.

[0306] 11. The 3D scanning system according to any of items 3 to 9, wherein the light pattern comprises at least 12,000 polygons, preferably at least 15,000 polygons, even more preferably at least 19,000 polygons.

[0307] 12. The 3D scanning system according to any of the preceding items, wherein the light pattern comprises at least 12,000 pattern features, preferably at least 15,000 pattern features, even more preferably at least 19,000 pattern features.

[0308] 13. A 3D scanning system according to any of the preceding items, wherein the light pattern is a checkerboard pattern comprising alternating black and white squares.

[0309] 14. A 3D scanning system according to any of the preceding items, wherein the light pattern is a checkerboard pattern comprising at least 100 x 100 alternating black and white squares such that the pattern comprises at least 9800 pattern features.

[0310] 15. A 3D scanning system according to any of the preceding items, wherein the light pattern is a checkerboard pattern comprising at least 140 x 140 alternating black and white squares such that the pattern comprises at least 19,000 pattern features.

[0311] 16. A 3D scanning system according to any preceding item, wherein the pattern comprises a predetermined number of fiducial marks.

[0312] 17. The 3D scanning system of item 16, wherein the pattern comprises at least one fiducial mark for every 20 pattern features, or every 25 pattern features, or every 50 pattern features, or every 100 pattern features.

[0313] 18. The 3D scanning system according to any one of items 16-17, wherein the geometric shape of the fiducial marker is selected from the group consisting of a dot, a triangle, a rectangle, a square, a pentagon, a hexagon and / or a combination thereof.

[0314] 19. A 3D scanning system according to any one of items 16-18, wherein the fiducial marker resembles a cross.

[0315] 20. The 3D scanning system of any one of items 16-19, wherein the light pattern is a checkerboard pattern comprising alternating black and white squares, and wherein the pattern comprises at least one fiducial marker for every 25 squares in the checkerboard pattern.

[0316] 21. A 3D scanning system according to any of items 16-20, wherein one of the processors of the scanning device is configured to identify a fiducial marker in the projected pattern.

[0317] 22. The 3D scanning system of item 21, wherein the processor is configured to identify the fiducial marker using a neural network.

[0318] 23. The 3D scanning system of item 23, wherein the processor is a neural processing unit (NPU).

[0319] 24. A 3D scanning system according to any one of items 16-23, wherein the (one or more) processors of the scanning system are configured to first solve the corresponding problems related to the reference mark before solving the corresponding problems related to the pattern features / image features.

[0320] 25. The 3D scanning system of any one of items 16-24, wherein the one or more processors are further configured to:

[0321] - identifying one or more fiducial markers within at least one of the acquired set of images;

[0322] - solving a correspondence problem associated with the identified fiducial markers, wherein solving the correspondence problem results in determining a point in 3D space based on the identified fiducial markers, wherein the point forms a solution to the correspondence problem; and

[0323] - calibrating the scanning device by adjusting one or more parameters of a mathematical geometric model associated with the scanning device, wherein the adjustment is based on a solution to the corresponding problem.

[0324] 26. A 3D scanning system according to item 25, wherein one or more parameters of the mathematical geometric model are selected from the following group: the position of (one or more) cameras, the orientation of (one or more) cameras, the intrinsic parameters of (one or more) cameras and / or a combination thereof.

[0325] 27. A 3D scanning system according to item 26, wherein the intrinsic parameters of (one or more) cameras are selected from the following group: focal length of (one or more) cameras, offset of lenses in (one or more) cameras, distortion of one or more lenses inside the scanning device and / or a combination thereof.

[0326] 28. The 3D scanning system of any one of items 25-27, wherein the one or more processors are configured to perform calibration in real time during scanning of the dental object.

[0327] 29. The 3D scanning system according to any one of items 25 to 28, wherein the step of calibrating the scanning device comprises the following steps:

[0328] - mathematically casting one or more camera rays and projector rays together in 3D space, said rays being associated with the fiducial markers; and

[0329] - Minimizing the distance between a camera ray and a given associated projector ray by dynamically adjusting one or more parameters of a mathematical geometric model.

[0330] 30. A 3D scanning system for scanning a dental object, comprising:

[0331] - at least one projector unit configured to project a light pattern of predetermined density onto the dental object along a projector optical axis;

[0332] one or more cameras, for example at least two cameras, having at least partially overlapping fields of view along different camera optical axes (and along the projector optical axis), each of said camera optical axes forming an angle of at least 3 degrees with the projector optical axis, each camera comprising an image sensor having an array of pixels, and

[0333] - one or more processors, wherein at least one processor is configured to generate a digital three-dimensional representation of the dental object based on triangulation between the camera and the light pattern,

[0334] Wherein the density of the projected light pattern is selected such that a density measure of the projected light pattern is above a predetermined threshold.

[0335] 31. The 3D scanning system of item 2, wherein the density measure of the projected light pattern in area A on the planar surface is defined as in, And where I=I(x,y), 0≤I(x,y)≤1 is the intensity signal measured in area A.

[0336] 32. The 3D scanning system of item 31, wherein the intensity signal is measured with a sensor having a predetermined sensor noise having a magnitude of 1% at a peak value.

[0337] 33. A 3D scanning system according to any of the preceding items 31-32, wherein the intensity signal is measured using a beam profiler.

[0338] 34. The 3D scanning system according to any one of items 31 to 33, wherein the area A is between 100 and 5000 mm 2 Between, preferably between 100 and 1000 mm 2 The most preferred size is 20mmx20mm.

[0339] 35. The 3D scanning system according to any of the preceding items 31-34, wherein the distance from the probe tip to the flat surface is between 5 and 50 mm, preferably 15 mm.

[0340] 36. A 3D scanning system according to any of the preceding items 31-35, wherein the distance from the probe tip to the plane is selected such that the projected light pattern is at least substantially focused.

[0341] 37. The 3D scanning system according to any of the preceding items 31-36, wherein the density measure is at least 25,000, preferably at least 30,000, even more preferably at least 35,000.

[0342] 38. A 3D scanning system according to any of the preceding items 31-37, wherein the light pattern comprises a predetermined number of similar pattern features such that the density measure is proportional to the number of similar pattern features in an area A on the planar surface.

[0343] 39. The 3D scanning system of any preceding item, wherein the light pattern comprises a predetermined number of similar pattern features.

[0344] 40. The 3D scanning system of item 39, wherein the density metric is defined as the total number of similar pattern features.

[0345] 41. A 3D scanning system according to any of items 39-40, wherein the total number of pattern features is at least 3,000, preferably at least 5,000, more preferably at least 10,000, even more preferably at least 15,000, and most preferably at least 19,000.

[0346] 42. The 3D scanning system according to any one of the preceding items 39-41, wherein the similar pattern features are selected from the group consisting of checkerboards, triangles, hexagons, dots, lines, and striped lines.

[0347] 43. A 3D scanning system according to any of the preceding items, wherein each of the camera optical axes defines an angle of at least 4, 5, 6, 7, preferably at least 8 degrees, even more preferably at least 9 degrees, most preferably at least 10 degrees with the projector optical axis.

[0348] 44. A 3D scanning system according to any of the preceding items, wherein the light pattern is non-coded such that no part of the pattern is unique.

[0349] 45. The 3D scanning system according to any of the preceding items, wherein at least 50%, more preferably at least 75%, most preferably at least 90% or even 100% of the light pattern is a checkerboard pattern.

[0350] 46. ​​The 3D scanning system according to any of the preceding items, wherein each camera is configured to capture a plurality of images depicting at least a portion of the light pattern projected on the dental object.

[0351] 47. A 3D scanning system according to any of the preceding items, wherein the processor is configured to determine intersection points between adjacent features of the projected pattern, and wherein these intersection points are associated with three-dimensional points in space.

[0352] 48. A 3D scanning system according to any of the preceding items, wherein the processor is configured to identify the three-dimensional location of the projected pattern of light based on the camera agreeing that there are light patterns cast by projector rays at certain intersection points.

[0353] 49. A 3D scanning system according to the preceding item, wherein the scanning device is a handheld intraoral scanning device.

[0354] 50. The 3D scanning system according to the preceding clause, wherein the scanning device is configured to transmit data and / or images wirelessly to an external device.

[0355] 51. The 3D scanning system according to the preceding clause, wherein the scanning device comprises an elongated probe having a tip.

[0356] 52. A 3D scanning system according to item 51, wherein the projector unit(s) and the camera are located at the distal end of the elongated probe.

[0357] 53. A computer-implemented method for generating a digital representation of a three-dimensional (3D) object, the method comprising the steps of:

[0358] - acquiring and / or receiving one or more sets of images, wherein a set of images comprises two or more images of a 3D object, wherein each image comprises a plurality of image features;

[0359] - determining one or more image features in each image;

[0360] - determining one or more possible depths associated with each image feature;

[0361] - generating image feature groups, wherein each image feature group includes one or more image features;

[0362] - generating one or more seed proposals, wherein each seed proposal includes at least a predetermined number of image feature groups, wherein generating each seed proposal comprises the following steps:

[0363] - adding one or more image feature groups to the seed proposal based on one or more predetermined first criteria;

[0364] - accepting seed proposals that include at least a predetermined number of sets of image features; and

[0365] - generating a digital representation of the 3D object by adding seed proposals to the digital representation(s) based on one or more predetermined second criteria.

[0366] 43. The method of item 48, wherein the image features are determined using a neural network.

[0367] 44. A method according to any preceding item, wherein the one or more possible depths are determined using triangulation.

[0368] 45. A method according to any of the preceding items, wherein the image is provided by a scanning device comprising a projector unit and two or more cameras.

[0369] 46. ​​A method according to any preceding item, wherein the image is provided by a scanning device comprising a projector unit and four or more cameras.

[0370] 47. A method according to any of items 44-46, wherein at least a first number of images are obtained using a first camera and at least a second number of images are obtained using a second camera.

[0371] 48. A method according to any preceding clause, wherein each image feature is associated with a set of image coordinates and a camera index indicating which camera acquired the image associated with the image feature.

[0372] 49. The method of item 48, wherein the step of determining image features comprises determining image coordinates and a camera index for each image feature.

[0373] 50. A method according to any preceding clause, wherein the method further comprises the step of evaluating whether one or more determined image features should be rejected.

[0374] 51. A method according to any preceding item, wherein each image feature within an image feature group has a unique camera index.

[0375] 52. A method according to any preceding clause, wherein the method further comprises the step of associating zero or more image features (eg one or more image features) with each pattern feature.

[0376] 53. A method according to any preceding clause, wherein the method further comprises the step of sorting the determined image features associated with a given pattern feature according to depth before generating the set of image features.

[0377] 54. A method according to any of the preceding items, wherein the image feature groups are generated using a sliding window method, wherein image features having approximately the same depth are grouped using a sliding window of a predetermined size.

[0378] 55. The method of item 54, wherein the number of cameras is denoted as N, and wherein the set of image features is generated using an iterative process in which the size of the sliding window is iteratively changed from an initial size N to a final size less than N.

[0379] 56. A method according to any preceding clause, wherein the generation of the image feature groups comprises the step of assigning a depth to each image feature group.

[0380] 57. The method of item 56, wherein the assigned depth is an average of the depths of the image features within the image feature group.

[0381] 58. A method according to any of the preceding items, wherein each image feature group is assigned a reliability score, wherein the reliability score indicates how reliable the image feature group is considered to be.

[0382] 59. The method of item 58, wherein the reliability score indicates how closely projector rays associated with a group of image features and camera rays associated with image features within the group pass each other.

[0383] 60. A method according to any preceding clause, wherein a standard deviation of the depths of the image features within each image feature group is calculated and assigned to each image feature group.

[0384] 61. A method according to any of items 54-59, wherein the standard deviation and / or the size of the sliding window are used to determine the reliability score for each image feature group.

[0385] 62. A method according to any of items 54-61, wherein the image feature groups are stored in a list, which is sorted according to standard deviation or reliability score.

[0386] 63. A method according to any of items 54-61, wherein a group of image features having a reliability score above a predetermined threshold is determined.

[0387] 64. A method according to item 63, wherein the method further comprises the step of calibrating the scanning device by adjusting one or more parameters of a mathematical geometric model associated with the scanning device, wherein the adjustment is based on the determined set of image features.

[0388] 65. The method of any preceding clause, wherein the predetermined number of image feature groups in a seed proposal is 5, such that each seed proposal includes at least 5 image feature groups.

[0389] 66. The method of any preceding clause, wherein the predetermined number of image feature groups in a seed proposal is 10, such that each seed proposal includes at least 10 image feature groups.

[0390] 67. The method of any preceding clause, wherein the predetermined number of image feature groups in a seed proposal is 15, such that each seed proposal includes at least 15 image feature groups.

[0391] 68. The method according to any of the preceding items, wherein the predetermined number of image feature groups in the seed proposal is 20, such that each seed proposal includes at least 20 image feature groups.

[0392] 69. A method according to any of items 58-67, wherein the image feature groups are added to the seed proposal(s) in a prioritized order according to their reliability scores if one or more predetermined first criteria are met.

[0393] 70. A method according to any of items 58-68, wherein the method further comprises the step of adding one or more image feature groups to a queue before adding them to the seed proposal.

[0394] 71. A method according to item 70, wherein the queue is a priority queue and wherein the reliability score is the priority.

[0395] 72. A method according to any of items 70-71, wherein image feature groups are removed from the queue one by one according to priority / reliability scores.

[0396] 73. A method according to any of items 70-72, wherein each time an image feature group is removed from the queue, adjacent image feature groups belonging to the most recently removed group are evaluated according to one or more predetermined criteria.

[0397] 74. A method according to any of items 70-73, wherein any one or more groups of adjacent image features that meet predetermined criteria are added to the queue.

[0398] 75. A method according to any of items 70-74, wherein each time an image feature group is removed from the queue, it is added to the seed proposal.

[0399] 76. A method according to any of items 59-68, wherein the one or more predetermined first criteria include a criterion that a standard deviation of a given image feature group must be below a predetermined threshold in order for the image feature group to be added to the seed proposal.

[0400] 77. A method according to any of the preceding items, wherein the step of generating (one or more) seed proposals is repeated iteratively, wherein the predetermined criteria changes at each iteration.

[0401] 78. A method according to item 77, wherein a previously generated seed proposal is allowed to grow in one or more subsequent iterations, wherein the (one or more) iterations are performed under a set of criteria that is different from the initial set of criteria.

[0402] 79. The method of claim 78, wherein the different set of standards is less stringent than the initial set of standards.

[0403] 80. A method according to any of the preceding items, wherein previously generated seed proposals may be allowed to be merged with each other and / or new seed proposal(s) may be allowed to be merged with previously generated seed proposal(s).

[0404] 81. A method according to any preceding clause, wherein the seed proposal(s) are added to the digital representation in a prioritized order based on the number of image feature groups in each seed proposal.

[0405] 82. The method according to any of the preceding items, wherein the method further comprises the step of marking image features that have been added to the digitally represented seed proposal(s).

[0406] 83. The method according to any of the preceding items, wherein the one or more predetermined second criteria include the criterion that the seed proposal should include at least a predetermined number of unlabeled image features in order for the seed proposal to be added to the digital representation.

[0407] 84. A method according to any of the preceding items, wherein the method further comprises the step of projecting a predetermined pattern onto the 3D object, wherein the pattern comprises a plurality of pattern features, wherein the pattern is projected by one or more projector units, each projector unit having a projector plane.

[0408] 85. The method according to any of the preceding items, wherein the method further comprises the step of acquiring two or more images of the 3D object, wherein the images are acquired by two or more cameras.

[0409] 86. The method of item 84, wherein each pattern feature is associated with a predetermined attribute.

[0410] 87. The method according to any one of items 84-86, wherein the predetermined pattern comprises any one of stripes, squares, dots, triangles, rectangles and / or combinations thereof.

[0411] 88. A method according to any of the preceding claims, wherein the image features are selected from the group consisting of corners, edges, vertices, points, transitions, dots, stripes and / or combinations thereof.

[0412] 89. The method of any one of items 84-88, wherein the image features are selected from the group consisting of corners, edges, vertices, points, transitions, dots, stripes and / or combinations thereof.

[0413] 90. The method according to any one of items 84-89, wherein the predetermined pattern is a checkerboard pattern comprising a plurality of checkerboard cells.

[0414] 91. A method according to any of items 84-90, wherein the image features are corners in a checkerboard pattern.

[0415] 92. A method according to any one of items 88-91, wherein the pattern features are corners in a checkerboard pattern.

[0416] 93. A method according to any preceding clause, wherein each image feature is associated with a predetermined attribute.

[0417] 94. A method according to any of the preceding items, wherein the predetermined property of the image / pattern feature is the phase of the pattern, wherein the phase can be white-black or black-white.

[0418] 95. A method according to any preceding clause, wherein each camera has a predetermined field of view, wherein the cameras are configured such that their fields of view at least partially overlap.

[0419] 96. A method according to any of the preceding items, wherein the field of view of each camera is between 50-115 degrees, such as 65-100 degrees or 80-90 degrees.

[0420] 97. A method according to any of items 84-95, wherein each camera has a predetermined fixed position relative to one or more projector units.

[0421] 98. A method according to any of the preceding claims, wherein the step of determining image features comprises generating a neighborhood graph for each image, wherein each neighborhood graph stores the image feature position of each determined image feature.

[0422] 99. A method according to item 98, wherein each neighborhood graph also stores information related to the connectivity of image features, i.e., which image features are neighbors of a given image feature.

[0423] 100. A method according to any of items 98-99, wherein each neighborhood graph also stores predetermined attributes of each image feature, such as phase.

[0424] 101. A method according to any of items 98-100, wherein (one or more) neighborhood graphs are generated using a neural network.

[0425] 102. The method of item 101, wherein the projected pattern is a checkerboard pattern, and wherein the neural network is trained to determine corners in the checkerboard in the captured image.

[0426] 103. The method of any one of items 101-102, wherein the neural network is trained to classify squares of a chessboard into black and white squares.

[0427] 104. A method according to any one of items 84-103, wherein the method further comprises the step of associating, for each pattern feature, a certain number of image features from the determined image features, such as zero or more image features.

[0428] 105. The method of item 104, wherein associating the image feature with the pattern feature comprises the following steps:

[0429] - for each image, projecting each image feature onto the projector plane, thereby obtaining an epipolar line for each projected image feature; and

[0430] - For each pattern feature, associate any image features whose corresponding epipolar lines pass within a predetermined distance from the pattern feature.

[0431] 106. The method of any one of items 84-104, wherein the step of determining the depth(s) associated with each image feature comprises the following steps:

[0432] - for each camera, casting a camera ray for each image feature in the acquired image, wherein each camera ray is cast in 3D;

[0433] - casting a projector ray for each pattern feature, wherein each projector ray is cast in 3D;

[0434] - determining points in 3D where camera rays associated with image features of different camera indices intersect projector rays within a predetermined distance, wherein each determined point corresponds to a depth; and

[0435] - Assign depth(s) to each image feature.

[0436] 107. A processor or computer system comprising means for performing the method according to any one of items 48-106.

[0437] 108. A computer program comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of items 48-106.

[0438] 109. A computer-readable medium having stored thereon the computer program of item 108.

[0439] 110. A method of generating a digital representation of a three-dimensional (3D) object, the method comprising the steps of:

[0440] - projecting a predetermined pattern onto the 3D object, wherein the pattern comprises a plurality of pattern features, each pattern feature being associated with a predetermined attribute, wherein the pattern is projected by one or more projector units, each projector unit having a projector plane;

[0441] - acquiring two or more images of a 3D object, wherein the images are acquired by a scanning device comprising two or more cameras, wherein each image comprises a plurality of image features, wherein each image feature is associated with a set of image coordinates and a camera index indicating which camera acquired the image associated with the image feature, and wherein each image feature is associated with a predetermined attribute;

[0442] - determining image features in each image, wherein the determining comprises determining image coordinates and a camera index for each image feature;

[0443] -determining one or more depths associated with each image feature using triangulation;

[0444] - generating image feature groups, wherein each image feature group includes one or more image features, each image feature having a unique camera index, wherein a depth is assigned to each image feature group;

[0445] - generating one or more seed proposals, wherein each seed proposal includes at least a predetermined number of image feature groups, wherein generating each seed proposal comprises the following steps:

[0446] - adding one or more image feature groups to the seed proposal based on one or more predetermined first criteria; and

[0447] - accepting a seed proposal(s) comprising at least a predetermined set of image features; and

[0448] - generating a digital representation of the 3D object by adding the seed proposal(s) to the digital representation of the 3D object based on one or more predetermined second criteria.

[0449] 111. A method according to item 110, wherein each camera has a predetermined field of view, wherein the cameras are configured so that their fields of view at least partially overlap, and wherein each camera has a predetermined fixed position relative to one or more projector units.

[0450] 112. A dental scanning system for generating a digital representation of a three-dimensional (3D) object, the scanning system comprising:

[0451] - a scanning device comprising:

[0452] - one or more projector units configured to project a predetermined pattern onto the 3D object, wherein the pattern comprises a plurality of pattern features, wherein each projector unit is associated with a projector plane;

[0453] - two or more cameras for each projector unit, wherein each camera is configured to capture an image of at least a portion of the object, wherein each camera has a predetermined fixed position relative to the one or more projector units; and

[0454] - One or more processors configured to perform a method according to any of items 48-106.

[0455] 113. The dental scanning system of item 112, wherein the predetermined pattern is a static pattern.

[0456] 114. A dental scanning system according to any of items 112-113, wherein the scanning device comprises at least four cameras for each projector unit.

[0457] 115. A dental scanning system according to any of items 112-114, wherein the one or more processors are located on the scanning device.

[0458] 116. A dental scanning system according to any of items 112-115, wherein the one or more processors include a field programmable gate array and / or a central processing unit (CPU).

[0459] 117. A dental scanning system for generating a digital representation of a three-dimensional (3D) object, the scanning system comprising:

[0460] - a scanning device, the scanning device comprising:

[0461] - one or more projector units configured to project a predetermined static pattern onto the 3D object, wherein the pattern comprises a plurality of pattern features, wherein each projector unit is associated with a projector plane;

[0462] - two or more cameras for each projector unit, wherein each camera is configured to capture an image of at least a portion of the object, wherein each camera has a predetermined fixed position relative to the one or more projector units, wherein the scanning device is configured to capture two or more images of the 3D object; and

[0463] A first processor configured to perform the following steps:

[0464] - determining image features in each image;

[0465] - a second processor configured to perform the following steps:

[0466] -determining one or more depths associated with each image feature using triangulation;

[0467] - generating image feature groups, wherein each image feature group includes one or more image features;

[0468] - generating one or more seed proposals, wherein each seed proposal includes at least a predetermined number of image feature groups, wherein generating each seed proposal comprises the following steps:

[0469] i. adding one or more image feature groups to a seed proposal based on one or more predetermined first criteria;

[0470] ii. accepting (one or more) seed proposals that include at least a predetermined number of image feature groups; and

[0471] - generating a digital representation of the 3D object by adding the seed proposal(s) to the digital representation of the 3D object based on one or more predetermined second criteria.

[0472] 118. A 3D scanning system comprising:

[0473] - an intraoral scanning device for scanning a dental object, the intraoral scanning device comprising:

[0474] - one or more projector units configured to project a pattern onto a surface of the dental object, wherein the pattern comprises a predetermined number of fiducial markers;

[0475] - two or more cameras for each projector unit, wherein the cameras are configured to acquire one or more sets of images, wherein each set of images comprises at least one image from each camera, wherein each image comprises at least a portion of the projected pattern, said portion comprising at least one of said fiducial markers;

[0476] - One or more processors configured to:

[0477] - identifying one or more fiducial markers within at least one of the acquired set of images;

[0478] - solving a correspondence problem associated with the identified fiducial markers, wherein solving the correspondence problem results in determining a point in 3D space based on the identified fiducial markers, wherein the point forms a solution to the correspondence problem; and

[0479] - calibrating the scanning device by adjusting one or more parameters of a mathematical geometric model associated with the scanning device, wherein the adjustment is based on a solution to the corresponding problem.

[0480] 119. A 3D scanning system according to item 118, wherein one or more parameters of the mathematical geometric model are selected from the following group: position of (one or more) cameras, orientation of (one or more) cameras, intrinsic parameters of (one or more) cameras.

[0481] 120. A 3D scanning system according to any of items 118-119, wherein the intrinsic parameters of the camera are selected from the following group: focal length of (one or more) cameras, offset of lenses in (one or more) cameras, distortion of one or more lenses within the scanning device and / or combinations thereof.

[0482] 121. A 3D scanning system according to any of items 118-120, wherein the one or more processors are configured to perform calibration in real time during scanning of the dental object.

[0483] 122. The 3D scanning system according to any one of items 118-121, wherein the processor is further configured to:

[0484] - mathematically casting one or more camera rays and projector rays together in 3D space, said rays being associated with the fiducial markers; and

[0485] - Minimizing the distance between a camera ray and a given associated projector ray by dynamically adjusting one or more parameters of a mathematical geometric model.

[0486] 123. A 3D scanning system according to any of items 118-122, wherein the scanning device is configured to wirelessly transmit data and / or images to an external device.

[0487] 124. A method for calibrating an intraoral scanning device, the method comprising the steps of:

[0488] - projecting a pattern onto a surface of the dental object using an intraoral scanning device, wherein the pattern comprises a predetermined number of fiducial markers;

[0489] - acquiring one or more sets of images, wherein each set of images comprises at least two images, wherein each image comprises at least a portion of the projected pattern, said portion comprising at least one of said fiducial markers;

[0490] - identifying one or more fiducial markers in the first set of acquired images;

[0491] - solving a correspondence problem associated with the fiducial markers, wherein solving the correspondence problem results in determining a point in 3D space based on the identified fiducial markers, wherein said point forms a solution to the correspondence problem; and

[0492] - calibrating the scanning device by adjusting one or more parameters of a mathematical geometric model associated with the scanning device, wherein the adjustment is based on a solution to the corresponding problem.

[0493] 125. The method of item 124, wherein the correspondence problem is solved using triangulation.

[0494] 126. A method according to any one of items 124-125, wherein one or more parameters of the mathematical geometric model are selected from the following group: the position of (one or more) cameras of the scanning device, the orientation of (one or more) cameras of the scanning device, the focal length of (one or more) cameras, the offset of the lenses in (one or more) cameras, the distortion of one or more lenses inside the scanning device and / or a combination thereof.

[0495] 127. A method according to any of items 124-126, wherein calibration is performed in real time during scanning of the dental object.

[0496] 128. The method according to any one of items 124 to 127, further comprising the following steps:

[0497] - determining one or more image features in each set of images;

[0498] - solving a second correspondence problem in each set of images using triangulation, such that a point in 3D space is determined based on the image features, wherein said point forms a solution to the second correspondence problem, wherein the second correspondence problem is solved for the set of image features; and

[0499] - generating a digital three-dimensional representation of the dental object, wherein the solution of the second correspondence problem is used to generate the three-dimensional representation.

[0500] 129. A method for calibrating an intraoral scanning device, the method comprising the steps of:

[0501] - projecting a pattern onto a surface of the dental object using an intraoral scanning device, wherein the pattern comprises a predetermined number of fiducial markers;

[0502] - acquiring one or more sets of images, wherein each set of images comprises at least two images, wherein each image comprises at least a portion of the projected pattern, said portion comprising at least one of said fiducial markers;

[0503] - identifying one or more image features in the first set of acquired images;

[0504] - solving a correspondence problem associated with the image features, wherein solving the correspondence problem results in determining a point in 3D space based on the identified image features, wherein said point forms a solution to the correspondence problem; and

[0505] - calibrating the scanning device by adjusting one or more parameters of a mathematical geometric model associated with the scanning device, wherein the adjustment is based on a solution to the corresponding problem.

[0506] Although certain embodiments have been described and illustrated in detail, the present disclosure is not limited to these details, but may also be embodied in other ways within the scope of the subject matter defined in the appended claims. Specifically, it should be understood that other embodiments may be utilized, and structural and functional modifications may be made without departing from the scope of the present disclosure. Furthermore, those skilled in the art will appreciate that, unless an embodiment is specifically presented only as an alternative, different disclosed embodiments may be combined to achieve a particular implementation, and such a particular implementation is within the scope of the present disclosure.

[0507] A claim may refer to any one of the preceding claims, and “any” should be understood as meaning “any one or more” of the preceding claims.

[0508] It should be emphasized that when the term "include / contain / comprises" is used in this specification, it is used to specify the presence of the stated features, integers, operations, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

Claims

1. A 3D scanning system for scanning a dental object, comprising: -Intra-oral 3D scanning device, the intra-oral 3D scanning device comprising: - at least one projector unit configured to project a static light pattern onto the dental object along a projector optical axis, wherein the static light pattern comprises more than 3000 pattern features; - one or more cameras, for example two or more cameras, having at least partially overlapping fields of view along different camera optical axes and along the projector optical axis, each of the camera optical axes defining an angle of at least 3 degrees with the projector optical axis, each of the cameras comprising an image sensor for acquiring one or more images; and - one or more processors configured to generate a digital three-dimensional representation of the dental object based on triangulation.

2. The 3D scanning system according to claim 1, wherein: The numerical aperture of the projector unit is between 0.0035 and 0.

015.

3. A 3D scanning system according to any one of the preceding claims, wherein: The projector unit comprises an aperture having a predetermined size such that it provides a pupil diameter between 0.2 mm and 0.7 mm.

4. A 3D scanning system according to any one of the preceding claims, wherein: The working distance of the projector unit and / or the given camera is between 15 mm and 50 mm.

5. A 3D scanning system according to any one of the preceding claims, wherein: The projector unit is configured to sequentially turn on and off a light source at a predetermined frequency, wherein the light source is turned on for a predetermined period of time.

6. The 3D scanning system according to claim 5, wherein: The time period is between 4 milliseconds (ms) and 8 milliseconds (ms), and wherein the frequency is between 25 Hz and 35 Hz, for example about 30 Hz.

7. A 3D scanning system according to any one of the preceding claims, wherein: The image sensor is a rolling shutter image sensor.

8. A 3D scanning system according to any one of the preceding claims, wherein: The static light pattern is a polygonal pattern comprising a plurality of polygons, wherein the polygons are composed of sides and corners, and wherein the pattern features correspond to the corners in the pattern.

9. The 3D scanning system according to claim 8, wherein: The polygons are selected from the group consisting of triangles, rectangles, squares, pentagons, hexagons and / or combinations thereof, and wherein the polygons are repeated throughout the pattern in a predetermined manner.

10. The 3D scanning system according to any one of the preceding claims, wherein: The pattern is a checkerboard pattern comprising alternating black and white squares, and wherein the checkerboard pattern comprises at least 100×100 squares such that the pattern comprises at least 10,000 pattern features.

11. A 3D scanning system according to any one of the preceding claims, wherein: The pattern includes a predetermined number of fiducial marks.

12. The 3D scanning system according to claim 11, wherein: The cameras are configured to acquire one or more sets of images, wherein each set of images includes at least one image from each camera, wherein each image includes at least a portion of the projected pattern.

13. The 3D scanning system according to claim 12, wherein: Images in a given set of images are acquired simultaneously, wherein each camera contributes one image to the set of images.

14. The 3D scanning system according to any one of claims 12-13, wherein: The portion includes at least one of the fiducial markers.

15. The 3D scanning system according to any one of claims 11 to 14, wherein: The pattern includes at least one fiducial mark for every 100 pattern features.

16. The 3D scanning system according to any one of claims 11 to 15, wherein: The fiducial mark is selected from the group consisting of a dot, a triangle, a rectangle, a square, a pentagon, a hexagon and / or a combination thereof.

17. The 3D scanning system according to any one of claims 11 to 16, wherein: The static light pattern is a checkerboard pattern comprising alternating black and white squares, and wherein the pattern comprises at least one fiducial mark for every 100 squares in the checkerboard pattern.

18. The 3D scanning system according to any one of claims 11 to 17, wherein: The one or more processors include a first processor configured to identify the fiducial marker in the projected pattern, wherein the first processor is configured to identify the fiducial marker using a neural network.

19. The 3D scanning system according to claim 18, wherein: The first processor is a neural processing unit (NPU).

20. The 3D scanning system according to any one of claims 11 to 19, wherein: The one or more processors are further configured to: - identifying one or more fiducial markers in at least one of the acquired sets of images; - solving a correspondence problem associated with the identified fiducial markers, wherein solving the correspondence problem results in determining a point in 3D space based on the identified fiducial markers, wherein the point forms a solution to the correspondence problem; and - calibrating the scanning device by adjusting one or more parameters of a mathematical geometrical model associated with the scanning device, wherein the adjustment is based on the solution of the corresponding problem.

21. The 3D scanning system according to claim 20, wherein: The one or more parameters of the mathematical geometric model are selected from the following group: a position of a camera, an orientation of a camera, intrinsic parameters of a camera and / or a combination thereof.

22. The 3D scanning system according to any one of claims 20-21, wherein: The one or more processors are configured to perform the calibration in real time during scanning of the dental object.

23. The 3D scanning system according to any one of claims 20 to 22, wherein: The step of calibrating the scanning device comprises the following steps: - mathematically projecting one or more camera rays and projector rays together in 3D space, said rays being associated with the fiducial markers; and - minimizing the distance between a camera ray and a given associated projector ray by dynamically adjusting one or more parameters of said mathematical geometric model.

Images