An intraoral 3D scanner that uses multiple small cameras and multiple small pattern projectors

Abstract

A method for generating a 3D image includes driving a structured light projector(s) to project a pattern of light onto the intraoral 3D surface and driving a camera(s) to capture images, each image including at least a portion of the projected pattern, each of the camera(s) including an array of pixels. A processor compares a series of images captured by each camera to determine which portions of the projected pattern can be tracked across the images. The processor constructs a 3D model of the intraoral 3D surface based at least in part on the comparison of the series of images. Other embodiments are also described.

Description

[Technical Field]

[0001] The present invention relates generally to three-dimensional imaging, and more particularly to intraoral three-dimensional imaging using structured light illumination. [Background technology]

[0002] Dental impressions of a subject's intraoral three-dimensional surfaces, such as teeth and gums, are used to plan dental procedures. Traditional dental impressions are made using a dental impression tray filled with impression material, such as PVS or alginate, onto which the subject bites. The impression material then hardens, creating a negative imprint of the teeth and gums from which a three-dimensional model of the teeth and gums can be formed.

[0003] Digital dental impressions utilize intraoral scanning to generate a three-dimensional digital model of a subject's intraoral three-dimensional surfaces. Digital intraoral scanners often use structured light three-dimensional imaging. A subject's tooth surfaces may be highly reflective and somewhat translucent, which can reduce the contrast of the structured light pattern reflected from the teeth. Therefore, when using digital intraoral scanners that utilize structured light three-dimensional imaging, the subject's teeth are often coated with an opaque powder prior to scanning to enhance the capture of the intraoral scan and promote a usable level of contrast in the structured light pattern, e.g., to make the surface more scattering. While intraoral scanners that utilize structured light three-dimensional imaging have made some progress, additional advantages may be possible. Summary of the Invention

[0004] The use of structured light 3D imaging can lead to the "correspondence problem," which requires determining correspondence between points in the structured light pattern and points seen by a camera viewing the pattern. One technique for addressing this problem is based on projecting a "coded" light pattern and imaging the illuminated scene from one or more viewpoints. Encoding the emitted light pattern makes portions of the light pattern unique and identifiable when captured by a camera system. Because the pattern is coded, it is easier to find correspondences between points in the image and points in the projected pattern. The decoded points are triangulated to recover the 3D information.

[0005] Applications of the present invention include systems and methods related to 3D intraoral scanning devices that include one or more cameras and one or more pattern projectors. For example, particular applications of the present invention may relate to intraoral scanning devices having multiple cameras and multiple pattern projectors.

[0006] Further applications of the present invention include methods and systems for decoding structured light patterns.

[0007] Yet another application of the present invention may relate to systems and methods for three-dimensional intraoral scanning that utilize uncoded structured light patterns.

[0008] For example, in some specific applications of the present invention, an intraoral scanning device is provided, the device including an elongated handheld wand having a probe at its distal end. The probe may be configured to enter the subject's oral cavity during scanning. One or more miniature structured light projectors and one or more miniature cameras are coupled to a rigid structure disposed within the distal end of the probe. Each structured light projector transmits light using a light source, such as a laser diode. In some applications, the structured light projector has an illumination field of at least 45 degrees. Optionally, the illumination field may be less than 120 degrees. Each structured light projector may further include pattern-generating optics. The pattern-generating optics may generate a light pattern using diffraction and / or refraction. In some applications, the light pattern may be a distribution of discrete, disconnected light spots. Optionally, the light pattern maintains a distribution of discrete, disconnected spots in all planes located between 1 mm and 30 mm from the pattern-generating optics when the light source (e.g., laser diode) is activated to transmit light through the pattern-generating optics. In some applications, the pattern-generating optics of each structured light projector may have a light throughput efficiency, i.e., the fraction of light falling on the pattern generator that enters the pattern, of at least 80%, e.g., at least 90%. Each camera includes a camera sensor and objective optics including one or more lenses.

[0009] Laser diode light sources and diffractive and / or refractive pattern-generating optics can offer particular advantages in some applications. For example, the use of laser diodes and diffractive and / or refractive pattern-generating optics can help maintain an energy-efficient structured light projector to prevent the probe from heating up during use. Furthermore, such components may help reduce costs by eliminating the need for active cooling within the probe. For example, current laser diodes (e.g., in contrast to current light-emitting diodes (LEDs)) can use less than 0.6 watts of power while continuously transmitting at high brightness. When pulsed in accordance with some applications of the present invention, these current laser diodes may use even less power; for example, when pulsed at a 10% duty cycle, the laser diode may use less than 0.06 watts (although in some applications, the laser diode may use at least 0.2 watts while continuously transmitting at high brightness, and may use even less power when pulsed; for example, when pulsed at a 10% duty cycle, the laser diode may use at least 0.02 watts). Furthermore, the diffractive and / or refractive pattern generating optical elements may be configured to utilize most, if not all, of the transmitted light (as opposed to, for example, a mask that stops some of the light rays from hitting the object).

[0010] In particular, diffraction- and / or refraction-based pattern-generating optics generate patterns by diffraction, refraction, or interference of light, or any combination thereof, rather than by modulation of light as occurs with transparent or transmissive masks. In some applications, this can be advantageous because the light throughput efficiency (the percentage of light falling on the pattern generator that enters the pattern) can be nearly 100%, e.g., at least 80%, e.g., at least 90%, regardless of the pattern's "area-based duty cycle." In contrast, the light throughput efficiency of transparent and transmissive mask pattern-generating optics is directly related to the "area-based duty cycle." For example, for a desired "area-based duty cycle" of 100:1, the mask-based pattern-generating optics' throughput efficiency is 1%, while the efficiency of diffraction- and / or refraction-based pattern-generating optics remains nearly 100%. Furthermore, the light collection efficiency of lasers is more than 10 times higher than that of LEDs for the same total light output, due to the laser's inherently smaller emitting area and divergence angle, resulting in a higher output illuminance per unit area. High efficiency of lasers and diffractive and / or refractive pattern generators allows for thermally efficient construction, limiting significant heating of the probe during use and potentially eliminating or limiting the need for active cooling within the probe, thereby reducing costs. While laser diodes and DOEs may be particularly preferred for some applications, they are by no means required, either alone or in combination. Other light sources, such as LEDs, and pattern generating elements, including transparent or transmissive masks, with or without active cooling, may be used for other applications.

[0011] In some applications, to improve image capture of intraoral scenes under structured light illumination without using contrast-enhancing measures such as coating the teeth with opaque powders, the inventors have realized that a light pattern such as a distribution of discrete, disconnected light spots (rather than, e.g., a line) may offer an improved balance between increasing pattern contrast while maintaining a useful amount of information. Generally, a denser structured light pattern may provide more surface sampling and higher resolution, potentially enabling better stitching of surfaces from multiple image frames. However, a structured light pattern that is too dense may complicate the correspondence problem by increasing the number of spots required to solve the correspondence problem. Furthermore, a dense structured light pattern may result in reduced pattern contrast due to the increased light in the system, which may be caused by a combination of (a) stray light that reflects off the somewhat shiny surface of the tooth and is picked up by the camera, and (b) percolation, i.e., some light incident on the tooth reflects along multiple paths within the tooth and then leaves the tooth in many different directions. As further described below, methods and systems are provided that solve the correspondence problem presented by a distribution of discrete, disconnected light spots. In some applications, the discrete, disconnected light spots from each projector may not be coded.

[0012] In some applications, the field of view of each camera may be at least 45 degrees, e.g., at least 80 degrees, e.g., 85 degrees. Optionally, the field of view of each camera may be less than 120 degrees, e.g., less than 90 degrees. In some applications, one or more of the cameras have a fisheye lens or other optical element that provides a field of view of up to 180 degrees.

[0013] In any event, the fields of view of the various cameras may or may not be the same. Similarly, the focal lengths of the various cameras may or may not be the same. As used herein, the term "field of view" of each camera refers to the diagonal field of view of each camera. Furthermore, each camera may be configured to focus at an object focal plane located between 1 mm and 30 mm, e.g., at least 5 mm and / or less than 11 mm, e.g., between 9 mm and 10 mm, from the lens farthest from the respective camera sensor. Similarly, in some applications, the illumination field of each of the structured light projectors may be at least 45 degrees, and optionally less than 120 degrees. The inventors have recognized that the large field of view achieved by combining the respective fields of view of all cameras may improve accuracy by reducing the amount of image stitching error, especially in edentulous regions where the gingival surface may be smooth and there may be fewer distinct, high-resolution 3D features. A wide field of view allows large smooth features, such as the general curve of the teeth, to appear in each image frame, improving the accuracy of stitching together surfaces obtained from multiple image frames.

[0014] In some applications, a method for generating a digital three-dimensional image of an intraoral surface is provided. It should be noted that the term "three-dimensional image" as used herein is based on a three-dimensional model, e.g., a point cloud, from which the three-dimensional image of the intraoral surface is constructed. The resulting image, while typically displayed on a two-dimensional screen, contains data related to the three-dimensional structure of the scanned object and, therefore, can typically be manipulated to display the scanned object from different perspectives or viewpoints. Furthermore, data from the three-dimensional image may be used to create a physical three-dimensional model of the scanned object.

[0015] For example, one or more structured light projectors may be driven to project a pattern of light onto the intraoral surface, such as a distribution of discrete, disconnected light spots, a pattern of intersecting lines (e.g., a grid), a checkerboard pattern, or other pattern, and one or more cameras may be driven to capture images of the projections. The images captured by each camera may include a portion of the projected pattern (e.g., at least one of the spots). In some embodiments, the one or more structured light projectors project a pattern that is spatially fixed relative to the one or more cameras.

[0016] Each camera includes a camera sensor with an array of pixels, each of which has a corresponding ray in 3D space emanating from the pixel in a direction pointing toward the object being imaged, and each point along a particular one of these rays, when imaged on the sensor, falls on a respective corresponding pixel on the sensor. The term used in this specification, including the claims, is "camera ray." Similarly, for each projected spot from each projector, there is a corresponding projector ray. Each projector ray corresponds to a path of a respective pixel on at least one camera sensor; i.e., when a camera views a feature or portion (e.g., spot) of a pattern projected by a particular projector ray, that feature or portion (e.g., spot) of the pattern is necessarily detected by pixels on the particular path of pixels corresponding to that particular projector ray. (a) The camera ray values ​​corresponding to each pixel on the camera sensor of each camera, and (b) the projector ray values ​​corresponding to each feature or portion (e.g., light spot) of the pattern projected from each projector, may be stored during a calibration process, as described below.

[0017] With respect to camera rays, in some applications, instead of storing individual values ​​for each camera ray corresponding to each pixel on the camera sensor of each camera, a smaller set of calibration values ​​is stored that can be used to represent each camera ray. For example, to define a camera ray, parameter values ​​for a parameterized camera calibration function that takes a given three-dimensional location in space and transforms it to a given pixel in the two-dimensional pixel array of the camera sensor may be stored.

[0018] With respect to projector rays, (a) in some applications, an indexed list containing the values ​​for each projector ray is stored, or (b) alternatively, in some applications, a smaller set of calibration values ​​that can be used to represent each projector ray is stored. For example, parameter values ​​of a parameterized projector calibration model that defines each projector ray for a given projector may be stored.

[0019] Based on the stored calibration values, a processor may be used to execute a correspondence algorithm to identify the three-dimensional location of each portion of a feature (e.g., a projected spot) of a projected light pattern on a surface. For a given projector ray, the processor "looks" at the corresponding camera sensor path on one of the cameras. A detected spot or other feature along that camera sensor path has a camera ray that intersects with the given projector ray. The intersection defines a three-dimensional point in space. The processor then searches through the camera sensor paths corresponding to that given projector ray on the other cameras to determine how many other cameras have detected a pattern feature (e.g., a spot) where that camera ray intersects with that three-dimensional point in space on their respective camera sensor paths corresponding to the given projector ray. As used herein throughout this application, if two or more cameras detect a portion or feature (e.g., a spot) of a pattern where their respective camera rays intersect with a given projector ray at the same point in three-dimensional space, the cameras are said to "agree" that the portion or feature (e.g., the spot) is located at that three-dimensional point. This process is repeated for additional features (e.g., spots) along the camera sensor paths, and the feature (e.g., spot) on which the most cameras "agree" is identified as the feature (e.g., spot) projected onto the surface from the given projector ray. A three-dimensional location on the surface is then calculated for that feature (e.g., the spot) of the pattern.

[0020] In some embodiments, once a location on the surface is determined for a particular feature of the pattern (e.g., a particular spot), the projector ray that projected that feature (e.g., spot) and all camera rays corresponding to that feature (e.g., spot) may be removed from consideration, and the correspondence algorithm is run again for the next projector ray.

[0021] A further application of the present invention is directed to scanning intra-oral objects by projecting a structured light pattern (e.g., parallel lines, a grid, a checkerboard, disconnected and / or uniform spots, a random spot pattern, etc.) onto the intra-oral object, capturing at least a portion of the structured light pattern projected onto the intra-oral object, and tracking the portion of the captured structured light pattern across successive images. In some embodiments, tracking the portion of the captured structured light pattern across successive images can help improve scanning speed and / or accuracy.

[0022] In a more specific example related to a structured light scanner using the projection pattern (e.g., of disconnected spots) described above, a processor may be used to compare a series of images (e.g., multiple consecutive images) captured by each camera to determine which features of the projection pattern (e.g., which of the projected spots) can be tracked across the series of images (e.g., across multiple consecutive images). The inventors have recognized that the movement of a particular detected feature or spot can be tracked across multiple images (e.g., across consecutive image frames) in the series. Thus, a correspondence relationship solved for a particular spot in any of the images or frames in which the feature or spot is tracked provides a correspondence relationship solution for that feature or spot in all of the images or frames in which the feature or spot is tracked. Because the detected features or spots that can be tracked across multiple images are those generated by the same specific projector light beam, the trajectory of the tracked feature or spot will be along a specific camera sensor path corresponding to that specific projector light beam.

[0023] In some applications, instead of or in addition to tracking detected features or spots in a two-dimensional image, the length of each projector ray may be tracked in three-dimensional space. The projector ray length is defined as the distance between the projector ray's origin, i.e., the light source, and the three-dimensional location where the projector ray intersects the intraoral surface. As explained further below, tracking the length of a particular projector ray over time can help resolve correspondence ambiguities. While some examples herein describe the above concepts of spots and ray tracing with respect to a scanner that projects disconnected spots, it should be understood that this is exemplary and in no way limiting, and that the tracking techniques are equally applicable to scanners that project other patterns (e.g., parallel lines, grids, checkerboards, disconnected and / or uniform spots, random spot patterns, etc.) onto the intraoral object.

[0024] In some embodiments, for purposes of objective scanning, an estimate of the position of the scanner relative to the object being scanned, i.e., the three-dimensional intraoral surface, may be desirable during the scan, and in some embodiments, an estimate is desirable at all times during the scan. In accordance with some applications of the present invention, the inventors have developed a method for combining visual tracking of the scanner's motion with inertial measurements of the scanner's motion to accommodate times when sufficient visual tracking may not be available. Using accumulated data of the intraoral scanner's motion relative to the intraoral surface (visual tracking) and the intraoral scanner's motion relative to the fixed coordinate system (inertial measurements), a predictive model of the intraoral surface's motion relative to the fixed coordinate system can be constructed (as described further below). When sufficient visual tracking is not available, a processor may calculate an estimated position of the intraoral scanner relative to the intraoral surface by adding (e.g., in some embodiments, subtracting) a prediction of the intraoral surface's motion relative to the fixed coordinate system from the inertial measurements of the intraoral scanner's motion relative to the fixed coordinate system (as described further below). It should be understood that the scanner position estimation concepts described herein can be used with intraoral scanners regardless of the scanning technology employed (e.g., parallel confocal scanning, focal scanning, wavefront scanning, stereo vision, structured light, triangulation, light field, and / or combinations thereof), and thus, while discussed in relation to the structured light concepts described herein, this is by way of example and not by way of limitation.

[0025] In some embodiments of the structured light scanner described herein, the stored calibration values ​​may indicate (a) camera rays corresponding to each pixel on the camera sensor of each camera, and (b) projector rays corresponding to each projected feature (e.g., light spot) from each structured light projector, each projector ray corresponding to the path of a respective pixel on at least one of the camera sensors. However, over time, at least one of the cameras and / or at least one of the projectors may move (e.g., by rotation or translation), the optics of at least one of the cameras and / or at least one of the projectors may change, or the wavelength of the laser may change, causing the stored calibration values ​​to no longer accurately correspond to the camera rays and projector rays.

[0026] For any given projector ray, if a processor collects data including the calculated three-dimensional positions on the intraoral surface of multiple detected features (e.g., spots) from that projector ray detected at different times and overlays them on a single image, the features (e.g., spots) should all fall on the camera sensor path of the pixel corresponding to that projector ray. If the calibration of the camera or projector is somehow changed, the detected feature (e.g., spot) from that particular projector ray may not appear to fall on the camera sensor path of the pixel expected according to the stored calibration values, but rather may appear to be located on the camera sensor path of the newly updated pixel. If the calibration of the camera(s) and / or projector(s) is changed, the processor may reduce the difference between the updated pixel path and the original pixel path from the calibration data by (i) changing stored calibration values ​​(e.g., stored parameter values, e.g., functions, of a parameterized camera calibration model) indicating the camera ray corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and / or (ii) changing stored calibration values ​​(e.g., stored values ​​in an indexed list of projector rays or stored parameter values ​​of a parameterized projector calibration model) indicating the projector ray r corresponding to each feature (e.g., light spot) projected from each projector of the one or more projectors.

[0027] The evaluation of the current calibration may be performed automatically, periodically (e.g., every scan, every 10 scans, monthly, every few months, etc.), or in response to certain criteria being met (e.g., in response to a threshold number of scans being performed). As a result of the evaluation, the system may determine whether the status of the calibration is accurate or inaccurate. In one embodiment, as a result of the evaluation, the system determines whether the calibration is drifting. For example, a previous calibration may still be accurate enough to produce high-quality scans, but if a detected trend continues, the system may deviate and become unable to produce accurate scans in the future. In one embodiment, the system determines a drift rate and projects the drift rate into the future to determine a predicted date and time when the calibration will no longer be accurate. In one embodiment, an automatic or manual calibration can be scheduled for that future date and time. In one example, processing logic evaluates the status of the calibration over time (e.g., by comparing the status of the calibration at multiple different points in time) and determines a drift rate from such a comparison. From the drift rate, processing logic can predict when a calibration should be performed based on trend data.

[0028] Conventional intraoral scanners are manually recalibrated by the user according to a set schedule (e.g., every six months). Conventional intraoral scanners do not have the ability to monitor or evaluate the current status of the calibration (e.g., to determine whether recalibration is required). Furthermore, conventional intraoral scanners are manually calibrated using special calibration targets. Calibrating conventional intraoral scanners is time-consuming and inconvenient for users. Therefore, the dynamic calibration performed in certain embodiments described herein can be more user-friendly and can be performed in less time than conventional intraoral scanner calibration.

[0029] In some applications, if the calibration of the camera(s) and / or projector(s) has changed, the processor may not perform a recalibration, but rather may simply determine that at least some of the stored calibration values ​​of the camera(s) and / or projector(s) are incorrect. For example, based on a determination that the stored calibration values ​​are incorrect, the user may be prompted to return the intraoral scanner to the manufacturer for maintenance and / or recalibration, or to request a new scanner.

[0030] Visual tracking of the motion of the intraoral scanner relative to the object being scanned may be obtained by stitching together respective surfaces or point clouds obtained from adjacent image frames. As described herein, in some applications, illuminating the oral cavity under near-infrared (NIR) light can increase the number of visible features that can be used to stitch together respective surfaces or point clouds obtained from adjacent image frames. In particular, because NIR light penetrates teeth, images captured under NIR light include features that are internal to the teeth, such as internal cracks, as opposed to two-dimensional color images taken under broadband illumination, which only show features that appear on the surface of the teeth. These additional subsurface features may be used to stitch together respective surfaces or point clouds obtained from adjacent image frames.

[0031] In some applications, the processor may use two-dimensional images (e.g., two-dimensional color images and / or two-dimensional monochromatic NIR images) in the 2D-3D surface reconstruction of the intraoral three-dimensional surface. As described below, using two-dimensional images (e.g., two-dimensional color images and / or two-dimensional monochromatic NIR images) can significantly improve the resolution and speed of the three-dimensional reconstruction. Therefore, as described herein, in some applications, it is useful to augment the three-dimensional reconstruction of the intraoral three-dimensional surface with a three-dimensional reconstruction from two-dimensional images (e.g., two-dimensional color images and / or two-dimensional monochromatic NIR images). In some applications, the processor calculates the three-dimensional position of each of a plurality of points on the intraoral three-dimensional surface, for example, using a correspondence algorithm described herein, and calculates the three-dimensional structure of the intraoral three-dimensional surface based on the plurality of two-dimensional images (e.g., two-dimensional color images and / or two-dimensional monochromatic NIR images) and the calculated three-dimensional positions on the intraoral surface.

[0032] According to some applications of the present invention, the calculation of the three-dimensional structure is performed by a neural network. A processor inputs (a) a plurality of two-dimensional images (e.g., two-dimensional color images) of the intraoral three-dimensional surface and (b) the calculated three-dimensional positions of a plurality of points on the intraoral three-dimensional surface to the neural network, and the neural network determines and returns respective estimated maps (e.g., depth maps, normal maps, and / or curvature maps of the intraoral three-dimensional surface captured in each of the two-dimensional images (e.g., two-dimensional color images and / or two-dimensional monochromatic NIR images)).

[0033] The inventors have recognized that when intraoral scanners are commercially produced, small manufacturing variations may exist that (a) cause the calibration of the camera(s) and / or projector(s) on each commercially produced intraoral scanner to differ slightly from the calibration of the training stage camera(s) and / or projector(s), and / or (b) cause the lighting relationship between the camera(s) and projector(s) of the commercially produced intraoral scanner to differ slightly from the lighting relationship between the training stage camera(s) and projector(s) used to train the neural network. Other manufacturing variations of the cameras and / or projectors may also exist. Some applications of the present invention provide methods that use a processor to overcome manufacturing variations of the camera(s) and / or projector(s) of the intraoral scanner and reduce the discrepancy between the estimated map and the true structure of the intraoral three-dimensional surface.

[0034] According to some applications of the present invention, one way in which manufacturing tolerances can be overcome is by rectifying, e.g., cropping and morphing, images from an in-situ intraoral scanner to obtain rectified images that match the field of view of a set of reference cameras used to train a neural network. The neural network is trained based on images received from the set of reference cameras, and the in-situ images are then rectified so that the neural network would receive these images as if they had been captured by the reference cameras. The three-dimensional structure of the intraoral three-dimensional surface is then calculated based on multiple rectified two-dimensional images of the intraoral three-dimensional surface, e.g., the neural network determines a respective estimated map of the intraoral three-dimensional surface captured in each of the multiple rectified two-dimensional images.

[0035] In some applications of the present invention, a neural network determines a respective estimated depth map for each of the intraoral three-dimensional surfaces captured in each of the two-dimensional images, and the depth maps are stitched together to obtain the three-dimensional structure of the intraoral surface. However, inconsistencies may sometimes occur between the estimated depth maps. The inventors have recognized that it is advantageous for the neural network to also determine an estimated confidence map for each of the estimated depth maps determined by the neural network, with each confidence map indicating a confidence level for each region of the respective estimated depth map. Accordingly, a method is provided herein in which multiple two-dimensional images of intraoral three-dimensional surfaces are input to a first neural network module and a second neural network module. The first neural network module determines a respective estimated depth map for each of the intraoral three-dimensional surfaces captured in each of the two-dimensional images. The second neural network module determines a respective estimated confidence map corresponding to each estimated depth map. Each confidence map indicates a confidence level for each region of the respective estimated depth map.

[0036] According to some applications of the present invention, a neural network is trained using (a) two-dimensional images of a training stage three-dimensional surface, such as a model surface and / or an intraoral surface, and (b) a corresponding true output map of the training stage three-dimensional surface, calculated based on structured light images of the training stage three-dimensional surface. For each two-dimensional image, the neural network estimates an estimated map of the intraoral three-dimensional surface captured in each two-dimensional image, and then compares each estimated image with the corresponding true map of the intraoral three-dimensional surface. Based on the difference between each estimated map and the corresponding true map, the neural network is optimized to better estimate subsequent estimated maps.

[0037] In some applications, when an intraoral three-dimensional surface is used to train a neural network, moving tissue, e.g., a subject's tongue, lips, and / or cheeks, may occlude portions of the intraoral three-dimensional surface from the field of view of one or more cameras. To prevent the neural network from "learning" based on images of moving tissue (as opposed to stationary tissue of the scanned intraoral three-dimensional surface), for two-dimensional images in which moving tissue is identified, the images may be processed to exclude at least a portion of the moving tissue before inputting the two-dimensional images into the neural network.

[0038] In some applications, to prevent cross-contamination between patients, a disposable sleeve is placed over the distal end of the intraoral scanner, e.g., over the probe, before the probe is placed in the patient's mouth. As described further herein, due to the relative positioning of the structured light projector in the probe and the adjacent camera, a portion of the projected structured light pattern may be reflected from the sleeve and reach the camera sensor of the adjacent camera. As described further herein, due to the polarization of the structured light projector's laser light, the laser may be rotated about its optical axis so that the polarization angle of the laser light relative to the sleeve is determined to reduce the degree of reflection.

[0039] In some applications of the present invention, a simultaneous localization and mapping (SLAM) algorithm is used to track the motion of a handheld wand and generate a three-dimensional image. SLAM can be performed using two or more cameras that generally view the same image but from slightly different angles. However, due to the positioning of the camera 24 within the probe 28 and the proximity of the probe 28 to the object being scanned, i.e., the intraoral three-dimensional surface, the two or more cameras within the probe often do not see substantially the same image. As described below, additional challenges to using a SLAM algorithm can be encountered when scanning intraoral three-dimensional surfaces. The inventors have devised several methods to overcome these challenges in order to use SLAM to track the motion of a handheld wand and generate a three-dimensional image of the intraoral three-dimensional surface, as further described herein.

[0040] According to some applications of the present invention, when scanning an intraoral three-dimensional surface using a handheld wand, some of the features (e.g., spots) may land on moving tissue (e.g., a patient's tongue) when a structured light projector projects a distribution of those features (e.g., a distribution of spots) onto the intraoral surface. To improve the accuracy of the three-dimensional reconstruction algorithm, features (e.g., spots) that land on moving tissue generally should not be relied upon for reconstructing the intraoral three-dimensional surface. As described herein, whether a feature (e.g., spot) is projected onto moving or stable tissue in the oral cavity may be determined based on unstructured light image frames (which may be broad-spectrum light, for example) interspersed among the structured light image frames. A confidence grading system may be used to assign a confidence grade based on a determination of whether a detected feature (e.g., spot) is projected onto stationary or moving tissue. Based on the confidence grade for each of the multiple features (e.g., spots), a processor may execute the three-dimensional reconstruction algorithm using the detected features (e.g., spots).

[0041] In one method of generating a digital three-dimensional image as provided herein, the method includes driving each light projector of one or more structured light projectors to project a pattern onto the intraoral three-dimensional surface. The method further includes driving each camera of one or more cameras to capture a plurality of images, each image including at least a portion of the projected pattern. The method further includes using the processor to compare a series of images captured by the one or more cameras, determining based on the comparison of the series of images which portions of the projected pattern can be tracked across the series of images, and constructing a three-dimensional model of the intraoral three-dimensional surface based at least in part on the comparison of the series of images. In one embodiment, the method further includes solving a correspondence algorithm for the tracked portion of the projected pattern in at least one image of the series of images, using the solved correspondence algorithm to address the tracked portion of the projected pattern in at least one image of the series of images, e.g., solving the correspondence algorithm for the tracked portion of the projected pattern in images of the series of images for which the correspondence algorithm has not been solved, and constructing a three-dimensional model using the solution of the correspondence algorithm. In one embodiment, the method further includes solving a correspondence algorithm for the tracked portion of the projected pattern based in part on the tracked position of the tracked portion in each image throughout the series of images, and constructing a three-dimensional model using the solution of the correspondence algorithm.

[0042] In one embodiment of the method, the projection pattern includes a plurality of projected light spots, and the portion of the projection pattern corresponds to a projection spot of the plurality of projected light spots. In a further embodiment, using the processor to compare a series of images based on stored calibration values ​​indicative of (a) a camera ray corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and (b) a projector ray corresponding to each projected light spot from each structured light projector of the one or more structured light projectors, each projector ray corresponding to a path of a respective pixel on at least one of the camera sensors, and determining which portion of the projection pattern can be tracked includes determining which of the projection spots s can be tracked across the series of images, each tracking spot s moving along the path of a pixel corresponding to a respective projector ray r.

[0043] In a further embodiment of the method, the step of using the processor further includes using the processor to determine, for each tracking spot s, a plurality of possible paths p of pixels on a given one of the cameras, where path p corresponds to a respective plurality of possible projector rays r. In a further embodiment, the step of using the processor further includes using the processor to execute a correspondence algorithm to perform a plurality of operations for each possible projector ray r. The plurality of operations includes identifying how many other cameras detected a respective spot q corresponding to a respective camera ray that intersects projector ray r and a camera ray of the given one of the cameras that corresponds to the tracking spot s on their respective pixel path p1 corresponding to projector ray r. The operations further include identifying a given projector ray r1 for which the most other cameras detected the respective spot q. The operations further include identifying projector ray r1 as the particular projector ray r that generated the tracking spot s.

[0044] In a further embodiment of the method, the method includes using the processor to (a) execute a correspondence algorithm to calculate a three-dimensional position of each of a plurality of detection spots on the intraoral three-dimensional surface captured in the series of images, and (b) identify, in at least one image of the series of images, the detection spot as originating from a particular projector ray r by identifying the detection spot as a tracking spot s moving along a path of pixels corresponding to the particular projector ray r.

[0045] In a further embodiment of the method, the method includes, using the processor: (a) executing a correspondence algorithm to calculate a three-dimensional position of each of a plurality of detection spots on the intraoral three-dimensional surface captured in the series of images; and (b) excluding from consideration as points on the intraoral three-dimensional surface those spots that (i) are identified as being from a particular projector ray r based on the three-dimensional positions calculated by the correspondence algorithm, and (ii) are not identified as being a tracking spot s moving along a path of a pixel corresponding to the particular projector ray r.

[0046] In a further embodiment of the method, the method includes, using the processor: (a) executing the correspondence algorithm to calculate a three-dimensional position of each of a plurality of detection spots on the intraoral three-dimensional surface captured in the series of images; and (b) for a detection spot identified as originating from two different projector rays r based on the three-dimensional positions calculated by the correspondence algorithm, identifying the spot as originating from one of the two different projector rays r by identifying the detection spot as a tracking spot s moving along one of the two different projector rays r.

[0047] In a further embodiment of the method, the method includes using the processor: (a) executing the correspondence algorithm to calculate the three-dimensional position of each of a plurality of detection spots on the intraoral three-dimensional surface captured in the series of images; and (b) identifying a weak spot whose three-dimensional position was not calculated by the correspondence algorithm as a projected spot from a specific projector ray r by identifying the weak spot as a tracking spot s moving along a path of a pixel corresponding to the specific projector ray r.

[0048] In a further embodiment of the method, the method includes using the processor to calculate respective three-dimensional positions on the intraoral three-dimensional surface at intersections of the projector ray r and the respective camera ray corresponding to the tracked spot s in each of the series of images in which the spot s was tracked.

[0049] In a further embodiment of the method, the three-dimensional model is constructed using a correspondence algorithm, which at least in part uses portions of the projected pattern that are determined to be trackable across the series of images.

[0050] In a further embodiment of the method, the method includes using the processor: (a) determining parameters of a trackable portion of a projection pattern in at least two adjacent images from the series of images, the parameters being selected from the group consisting of a size of the portion, a shape of the portion, an orientation of the portion, an intensity of the portion, and a signal-to-noise ratio (SNR) of the portion; and (b) predicting parameters of the tracked portion of the projection pattern in a subsequent image based on the parameters of the tracked portion of the projection pattern in the at least two adjacent images.

[0051] In a further embodiment of the method, the step of using the processor further comprises using the processor to search for a portion of the projection pattern in the subsequent image that substantially has the prediction parameters based on the prediction parameters of the tracked portion of the projection pattern.

[0052] In a further embodiment of the method, the parameter is a shape of the portion of the projection pattern, and the step of using the processor further includes using the processor to determine a search space in a next image in which to search for the tracking portion of the projection pattern based on the predicted shape of the tracking portion of the projection pattern.

[0053] In a further embodiment of the method, determining the search space using the processor includes using the processor to determine a search space in the next image in which to search for a tracking portion of the projection pattern, the search space having a size and aspect ratio based on a size and aspect ratio of the predicted shape of the tracking portion of the projection pattern.

[0054] In a further embodiment of the method, the parameter is a shape of the portion of the projection pattern, and the step of using the processor further includes, using the processor: (a) determining a velocity vector of the tracked portion of the projection pattern based on a direction and a distance that the tracked portion of the projection pattern has moved between the at least two adjacent images from the series of images; (b) predicting a shape of the tracked portion of the projection pattern in a subsequent image in response to a shape of the tracked portion of the projection pattern in at least one of the at least two adjacent images; and (c) determining a search space in which to search for the tracked portion of the projection pattern in the subsequent image in response to a combination of (i) the determination of the velocity vector of the tracked portion of the projection pattern and (ii) the predicted shape of the tracked portion of the projection pattern.

[0055] In a further embodiment of the method, the parameter is a shape of the portion of the projection pattern, and the step of using the processor further includes, using the processor: (a) determining a velocity vector of the tracked portion of the projection pattern based on a direction and a distance that the tracked portion of the projection pattern has moved between at least two adjacent images from the series of images; (b) predicting a shape of the tracked portion of the projection pattern in a subsequent image in response to determining the velocity vector of the tracked portion of the projection pattern; and (c) determining a search space in which to search for the tracked portion of the projection pattern in the subsequent image in response to a combination of (i) the determination of the velocity vector of the tracked portion of the projection pattern and (ii) the predicted shape of the tracked portion of the projection pattern.

[0056] In a further embodiment of the method, the step of using the processor further includes using the processor to predict a shape of the tracking portion of the projection pattern in the subsequent image in response to a combination of (i) determining the velocity vector of the tracking portion of the projection pattern and (ii) the shape of the tracking portion of the projection pattern in at least one of the two adjacent images.

[0057] In a further embodiment of the method, the step of using the processor further includes using the processor to (a) determine a velocity vector of the tracking portion of the projected pattern based on a direction and distance traveled by the tracking portion of the projected pattern between two successive images in the series of images, and (b) determine a search space in a subsequent image in response to determining the velocity vector of the tracking portion of the projected pattern.

[0058] In one embodiment of the second method for generating a digital three-dimensional image provided herein, the method comprises driving each structured light projector of one or more structured light projectors to project a pattern of light along a plurality of projector rays onto the intraoral three-dimensional surface; and driving each camera of one or more cameras to capture a plurality of images, each image comprising at least a portion of the projected pattern, and each camera of the one or more cameras comprising a camera sensor comprising an array of pixels. The second method further includes using the processor to execute a correspondence algorithm to calculate, for each of the plurality of images, three-dimensional positions on the intraoral three-dimensional surface of a plurality of detected features of the projection pattern; estimating a three-dimensional surface based on at least three features, each feature corresponding to a respective projector ray r of a plurality of projector rays, using data corresponding to the three-dimensional positions of the at least three features; estimating, for a projector ray r1 of the plurality of projector rays for which a three-dimensional position of a feature corresponding to the projector ray r1 has not been calculated, a three-dimensional position in an intersection space between the projector ray r1 and the estimated three-dimensional surface; and identifying a search space in a pixel array of at least one camera in which to search for a feature corresponding to the projector ray r1 using the three-dimensional positions in the estimated space.

[0059] In a further embodiment of the second method, the correspondence algorithm is performed based on stored calibration values ​​indicating (a) a camera ray corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and (b) a projector ray corresponding to each feature of a projected pattern from each structured light projector of the one or more structured light projectors, each projector ray corresponding to a path of each pixel on at least one of the camera sensors. Further, the search space in the data includes a search space defined by one or more thresholds.

[0060] In a further embodiment of the second method, the processor sets a threshold such that detected features below the threshold are not considered by the correspondence algorithm, and to search for a feature corresponding to projector ray r1 in the identified search space, the processor lowers the threshold to consider features not considered by the correspondence algorithm. In some embodiments, the threshold is an intensity threshold.

[0061] In a further embodiment of the second method, the pattern of light comprises a distribution of discrete spots, and each feature comprises a spot from the distribution of discrete spots.

[0062] In a further embodiment of the second method, the data corresponding to the three-dimensional position of each of the at least three features includes using data corresponding to the three-dimensional position of each of at least three features that are all captured in one of the plurality of images.

[0063] In a further embodiment of the second method, the second method further includes refining the estimate of the three-dimensional surface using data corresponding to a three-dimensional position of at least one additional feature of the projected pattern, the at least one additional feature having a three-dimensional position calculated based on another image of the plurality of images. In a further embodiment of the second method, refining the estimate of the three-dimensional surface includes refining the estimate of the three-dimensional surface such that the at least three features and the at least one additional feature all lie on the estimated three-dimensional surface.

[0064] In a further embodiment of the second method, using data corresponding to the three-dimensional position of each of the at least three features includes using data corresponding to at least three features captured in each image of the plurality of images.

[0065] In one embodiment of the third method for generating a digital three-dimensional image, the third method includes driving each structured light projector of the one or more structured light projectors to project a pattern of light onto the intraoral three-dimensional surface, and driving each camera of the plurality of cameras to capture an image, the image comprising at least a portion of the projected pattern, and each camera of the plurality of cameras comprising a camera sensor including an array of pixels. The third method further includes using a processor to execute a correspondence algorithm to calculate respective three-dimensional positions on the intraoral three-dimensional surface of a plurality of features of the projected pattern; using data from a first camera of the plurality of cameras to identify candidate three-dimensional positions of a given feature of the projected pattern that correspond to or are otherwise associated with one or more particular projector ray(s) r, wherein data from a second camera of the plurality of cameras is not used to identify the candidate three-dimensional positions; using the candidate three-dimensional positions as seen by the first camera to identify a search space on a pixel array of a second camera in which to search for features of the projected pattern from projector ray r; and if the features of the projected pattern from the projector ray r are identified within the search space, using data from the second camera to refine the candidate three-dimensional positions of the features of the projected pattern.

[0066] In a further embodiment of the third method, to identify a candidate 3D location of a given spot corresponding to a particular projector ray r, the processor uses data from at least two of the cameras, and data from a camera that is not one of the at least two cameras is not used to identify the candidate 3D location, and to identify the search space, the processor uses the candidate 3D locations seen by at least one of the at least two cameras.

[0067] In a further embodiment of the third method, the pattern of light comprises a distribution of discrete, disjoint light spots, and the projection pattern features comprise projection spots from the disjoint light spots.

[0068] In a further embodiment of the third method, the processor uses stored calibration values ​​indicating (a) a camera ray corresponding to each pixel on a camera sensor of each camera of the plurality of cameras, and (b) a projector ray corresponding to each feature of a projected light pattern from each structured light projector of the one or more structured light projectors, each projector ray corresponding to a path of each pixel of at least one of the camera sensors.

[0069] A fourth method for generating a digital three-dimensional image described herein includes activating one or more structured light projectors to project a pattern of light onto a three-dimensional intraoral surface and activating one or more cameras to capture images, the images including at least a portion of the pattern. The fourth method further includes using the processor to execute a correspondence algorithm to calculate three-dimensional positions of each of a plurality of features of the pattern on the three-dimensional intraoral surface captured in a series of images, identifying the calculated three-dimensional positions of the detected features of the imaged pattern as associated with one or more specific projector rays r in at least a subset of the series of images, and evaluating lengths associated with the one or more projector rays r in each image of the subset of images based on the three-dimensional positions of the detected features corresponding to the one or more projector rays r in the subset of images.

[0070] In a fourth method, the processor may be further used to calculate an estimated length of the one or more projector rays r in at least one image of a series of images in which the three-dimensional position of a feature projected from the one or more projector rays was not identified.

[0071] In one embodiment of the fourth method, the one or more cameras each comprise a camera sensor including a pixel array, and calculating the three-dimensional position of each of a plurality of features of a pattern on the intraoral three-dimensional surface and identifying that the calculated three-dimensional position of a detected feature of the pattern corresponds to a particular projector ray r is performed based on stored calibration values ​​indicative of (i) a camera ray corresponding to a pixel of each of the one or more camera sensors, and (ii) a projector ray corresponding to each feature of a projected light pattern from each projector of the one or more projectors, each projector ray corresponding to a path of a respective pixel on at least one of the camera sensors.

[0072] In a further embodiment of the fourth method, the step of using the processor further includes the step of using the processor to calculate an estimated length of the projector ray r in at least one image of the series of images in which the three-dimensional position of the feature projected from the projector ray r has not been identified, and determining a one-dimensional search space in which to search for the feature projected from the projector ray r in at least one image of the series of images based on the estimated length of the projector ray r in the at least one image of the series of images, the one-dimensional search space being along the path of each pixel corresponding to the projector ray r.

[0073] In a further embodiment of the fourth method, the step of using the processor further includes the steps of: using the processor to calculate an estimated length of the projector ray r in at least one image of the series of images in which the three-dimensional position of the feature projected from the projector ray r was not identified; and determining, for each pixel array based on the estimated length of the projector ray r in the at least one image of the series of images, a one-dimensional search space in the pixel array of each camera of the multiple cameras in which to search for the spot projected from the projector ray r, the one-dimensional search space being along the path of each pixel corresponding to the ray r.

[0074] In a further embodiment of the fourth method, using the processor to determine a one-dimensional search space in each pixel array of the plurality of cameras includes using the processor to determine a one-dimensional search space in each pixel array of all cameras in which to search for features projected from projector ray r.

[0075] In a further embodiment of the fourth method, the step of using the processor further includes the steps of using the processor to identify, based on the correspondence algorithm, a plurality of candidate three-dimensional locations of features projected from the projector ray r in at least one image of the series of images that are not in the subset of images, and calculating an estimated length of the projector ray r in at least one image of the series of images in which a plurality of candidate three-dimensional locations of features projected from the projector ray r have been identified.

[0076] In a further embodiment of the fourth method, the step of using the processor further comprises using the processor to determine which of the plurality of candidate three-dimensional locations is the correct three-dimensional location of the projected feature by determining which corresponds to an estimated length of a projector ray r in at least one image of the sequence of images.

[0077] In a further embodiment of the fourth method, the step of using the processor further includes the steps of: using the processor to determine a one-dimensional search space in at least one image of the series of images in which to search for a feature projected from projector ray r based on an estimated length of projector ray r in at least one image of the series of images; and determining which of the plurality of candidate three-dimensional positions of the projected feature is the correct three-dimensional position of the projected feature generated by projector ray r by determining which of the plurality of candidate three-dimensional positions corresponds to the feature generated by projector ray r detected within the one-dimensional search space.

[0078] In a further embodiment of the fourth method, the step of using the processor includes using the processor to define a curve based on the estimated lengths of projector ray r in each image of the subset of images, and removing a detected feature identified as originating from projector ray r from consideration as a point on the intraoral three-dimensional surface if the three-dimensional position of the projected feature corresponds to a length of projector ray r that is at least a threshold distance away from the defined curve.

[0079] In a further embodiment of the fourth method, the pattern comprises a plurality of spots, and each of a plurality of features of the pattern comprises one of the plurality of spots.

[0080] In a fifth method for generating a digital three-dimensional image as provided herein, the method includes driving each structured light projector of one or more structured light projectors to project a pattern of light along a plurality of projector rays onto the intraoral three-dimensional surface, and driving each camera of one or more cameras to capture a plurality of images, each image including at least a portion of the projected pattern, and each camera of the one or more cameras including a camera sensor with an array of pixels. The method further includes the steps of: using the processor to execute a correspondence algorithm to calculate, for each of a plurality of images, respective three-dimensional positions on the intraoral three-dimensional surface of a plurality of detected features of the projected pattern; using data corresponding to the respective three-dimensional positions of at least three of the detected features to estimate a three-dimensional surface based on at least three features, each feature corresponding to a respective projector ray r among a plurality of projector rays; for a projector ray r1 among the plurality of projector rays from which a plurality of candidate three-dimensional positions of a feature corresponding to projector ray r1 have been calculated, estimating a three-dimensional position in an intersection space between projector ray r1 and the estimated three-dimensional surface; and using the estimated three-dimensional position of projector ray r1 in the intersection space to select which of the plurality of candidate three-dimensional positions is the correct three-dimensional position of the feature corresponding to projector ray r1.

[0081] In a further embodiment of the fifth method, the correspondence algorithm is performed based on stored calibration values ​​indicating (a) camera rays corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and (b) projector rays corresponding to each feature of a projected pattern from each structured light projector of the one or more structured light projectors, each projector ray corresponding to a respective path on at least one pixel of the camera sensor. Further, the search space in the data includes a search space defined by one or more thresholds.

[0082] In a further embodiment of the fifth method, the pattern of light comprises a distribution of discrete spots, and each feature comprises one spot from the distribution of discrete spots.

[0083] In a further embodiment of the fifth method, the data corresponding to the three-dimensional position of each of the at least three features includes using data corresponding to the three-dimensional position of each of at least three features that are all captured in one of the plurality of images.

[0084] In a further embodiment of the fifth method, the fifth method further includes refining the estimate of the three-dimensional surface using data corresponding to a three-dimensional position of at least one additional feature of the projected pattern, the at least one additional feature having a three-dimensional position calculated based on another one of the plurality of images. In a further embodiment of the fifth method, refining the estimate of the three-dimensional surface includes refining the estimate of the three-dimensional surface such that all of the at least three features and the at least one additional feature lie on the estimated three-dimensional surface.

[0085] In a further embodiment of the fifth method, using data corresponding to the three-dimensional position of each of the at least three features includes using data corresponding respectively to at least three features captured in each one of the plurality of images.

[0086] In one method of tracking motion of an intraoral scanner provided herein, the method includes measuring the motion of the intraoral scanner relative to the intraoral surface being scanned using at least one camera coupled to the intraoral scanner, and measuring the motion of the intraoral scanner relative to the intraoral surface being scanned relative to a fixed coordinate system using at least one inertial measurement unit (IMU) coupled to the intraoral scanner. The method further includes using the processor to calculate motion of the intraoral surface relative to the fixed coordinate system based on (a) motion of the intraoral scanner relative to the intraoral surface and (b) motion of the intraoral scanner relative to the fixed coordinate system, building a predictive model of motion of the intraoral surface relative to the fixed coordinate system based on accumulated data of motion of the intraoral surface relative to the fixed coordinate system, and further calculating an estimated position of the intraoral scanner relative to the intraoral surface based on (a) the prediction of motion of the intraoral surface relative to the fixed coordinate system (derived based on the predictive model of motion) and (b) motion of the intraoral scanner relative to the fixed coordinate system (measured by an IMU). In a further embodiment of the method for tracking motion, the method further includes determining whether measurement of motion of the intraoral scanner relative to the intraoral surface using the at least one camera is disrupted, and calculating an estimated position of the intraoral scanner relative to the intraoral surface in response to determining that measurement of motion is disrupted. In a further embodiment of the method for tracking motion, calculating the motion is performed by calculating the difference between (a) the motion of the intraoral scanner relative to the intraoral surface and (b) the motion of the intraoral scanner relative to the fixed coordinate system.

[0087] One method for determining whether calibration data of an intraoral scanner is inaccurate, as provided herein, includes activating one or more light sources to project light onto an intraoral three-dimensional surface and activating one or more cameras to capture multiple images of the intraoral three-dimensional surface, the method further includes using a processor to execute a correspondence algorithm based on stored calibration data for the one or more light sources and the one or more cameras to calculate respective three-dimensional positions of a plurality of features of the projected light on the intraoral three-dimensional surface, collecting data at multiple time points, the data including the calculated respective three-dimensional positions of the plurality of features on the intraoral three-dimensional surface, and determining, based on the collected data, that at least a portion of the stored calibration data is inaccurate.

[0088] In a further embodiment of the method, the one or more light sources are one or more structured light projectors, the method comprising the steps of: driving each structured light projector of the one or more structured light projectors to project a pattern of light onto the three-dimensional intraoral surface; and driving each camera of the one or more cameras to capture a plurality of images of the three-dimensional intraoral surface, each image comprising at least a portion of the projected pattern, each camera of the one or more cameras comprising a camera sensor including an array of pixels, and the stored calibration data comprising stored calibration values ​​indicative of (a) camera rays corresponding to each pixel on the camera sensor of each camera of the one or more cameras; and (b) projector rays corresponding to each feature of the light pattern projected from each structured light projector of the one or more structured light projectors, wherein each projector ray corresponds to a path p of a respective pixel on at least one of the camera sensors.

[0089] In a further embodiment of the method, determining that at least some of the stored calibration data is inaccurate includes using the processor to define, for each projector ray r, a pixel update path p' for each of the camera sensors based on the collected data, such that all of the calculated three-dimensional positions corresponding to features generated by projector ray r correspond to positions along a pixel's respective update path p' for each camera sensor; comparing each pixel's update path p' with a pixel's path p corresponding to that projector ray r for each camera sensor from the stored calibration values; and determining that at least some of the stored calibration values ​​are inaccurate in response to the update path p' for at least one camera sensor s differing from a pixel's path p corresponding to that projector ray r from the stored calibration values.

[0090] One method of recalibration provided herein includes activating one or more light sources to project light onto a three-dimensional intraoral surface and activating one or more cameras to capture multiple images of the intraoral surface, using a processor to execute a correspondence algorithm to calculate, based on stored calibration data for the one or more light sources and the one or more cameras, respective three-dimensional positions of a plurality of features of the projected light on the intraoral surface, collecting data at multiple time points, the data including the calculated respective three-dimensional positions of the plurality of features on the intraoral surface, and recalibrating the stored calibration data using the collected data.

[0091] In a further embodiment of the recalibration method, the one or more light sources are one or more structured light projectors, the method comprising: driving each structured light projector of the one or more structured light projectors to project a pattern of light onto the intraoral three-dimensional surface; and driving each camera of the one or more cameras to capture a plurality of images of the intraoral three-dimensional surface, each image including at least a portion of the projected pattern, each camera of the one or more cameras having a camera sensor including an array of pixels. The processor performs a plurality of operations using the stored calibration data, the stored calibration data including stored calibration values ​​indicating (a) camera rays corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and (b) projector rays corresponding to each feature of the light pattern projected from each structured light projector of the one or more structured light projectors, wherein each projector ray corresponds to a path p of a respective pixel on at least one of the camera sensors. The operations include executing a correspondence algorithm to calculate respective three-dimensional positions of a plurality of features of the projection pattern on the intraoral three-dimensional surface. The operations further include collecting data at a plurality of time points, the data including the calculated respective three-dimensional positions of the plurality of features on the intraoral three-dimensional surface. The operations further include, based on the collected data, defining, for each projector ray r, a pixel update path p' for each of the camera sensors such that all of the calculated three-dimensional positions corresponding to features generated by projector ray r correspond to positions along a respective pixel update path p' for each of the camera sensors. The operations further include recalibrating the stored calibration values ​​using the update path p'.

[0092] In a further embodiment of the recalibration method, the processor performs additional operations to recalibrate the stored calibration values, including comparing each update path p' of a pixel with a path p of the pixel corresponding to that projector ray r on each camera sensor from the stored calibration values. The additional operations further include, for at least one camera sensor s, if the update path p' of the pixel corresponding to projector ray r differs from the path p of the pixel corresponding to projector ray r from the stored calibration values, reducing the difference between the update path p' of the pixel corresponding to each projector ray r and the path p of the pixel corresponding to each projector ray r from the stored calibration values ​​by changing stored calibration data selected from the group consisting of: (i) stored calibration values ​​indicating camera rays corresponding to each pixel on camera sensor s of each camera of the one or more cameras; and (ii) stored calibration values ​​indicating projector ray r corresponding to each feature projected from each structured light projector of the one or more structured light projectors.

[0093] In a further embodiment of the recalibration method, the varying the stored calibration data includes stored calibration values ​​indicative of camera rays corresponding to each pixel on a camera sensor of each camera of the one or more cameras. Further, varying the stored calibration data includes varying one or more parameters of a parameterized camera calibration function defining a camera ray corresponding to each pixel on at least one camera sensor s to reduce a difference between (i) the calculated three-dimensional positions of each of the plurality of features of the projection pattern on the intraoral three-dimensional surface and (ii) stored calibration values ​​indicative of each camera ray corresponding to each pixel on the camera sensor at which each one of the plurality of features is to be detected.

[0094] In a further embodiment of the recalibration method, the varying the stored calibration data includes stored calibration values ​​indicating projector rays corresponding to each feature of the plurality of features from each structured light projector of the one or more structured light projectors, and varying the stored calibration data includes varying (i) an indexed list that assigns each projector ray r to a pixel path p, or (ii) one or more parameters of a parameterized projector calibration model that defines each projector ray r.

[0095] In a further embodiment of the recalibration method, the step of altering the stored calibration data includes altering the indexed list by reassigning each projector ray r based on the respective update path p′ of the pixel corresponding to each projector ray r.

[0096] In a further embodiment of the recalibration method, the step of varying the stored data includes varying (i) stored calibration values ​​indicating camera rays corresponding to each pixel on a camera sensor s of each of the one or more cameras, and (ii) stored calibration values ​​indicating projector rays r corresponding to each feature of a plurality of features from each structured light projector of the one or more structured light projectors.

[0097] In a further embodiment of the recalibration method, varying the stored calibration values ​​comprises iteratively varying the stored calibration values.

[0098] In a further embodiment of the recalibration method, the method further includes driving each of one or more cameras to capture multiple images of a calibration object having predetermined parameters, and using the processor, executing a triangulation algorithm to calculate respective parameters of the calibration object based on the captured images, and executing an optimization algorithm to (a) reduce the difference between (i) an updated path p' of a pixel corresponding to projector ray r and (ii) a path p of a pixel corresponding to projector ray r from the stored calibration values, using (b) the respective parameters of the calibration object calculated based on the captured images.

[0099] In a further embodiment of the re-calibration method, the calibration object is a three-dimensional calibration object of known shape, and the step of driving each of the one or more cameras to capture a plurality of images of the calibration object includes driving each of the one or more cameras to capture an image of the three-dimensional calibration object, and the predetermined parameters of the calibration object are dimensions of the three-dimensional calibration object. In a further embodiment, with respect to the processor using each calculated parameter of the calibration object to perform an optimization algorithm, the processor further uses collected data including the calculated three-dimensional positions of each of the plurality of features on the intraoral three-dimensional surface.

[0100] In a further embodiment of the re-calibration method, the calibration object is a two-dimensional calibration object having visually distinguishable features, and driving each of the one or more cameras to capture a plurality of images of the calibration object includes driving each of the one or more cameras to capture an image of the two-dimensional calibration object, and the predetermined parameters of the two-dimensional calibration object are respective distances between each of the visually distinguishable features. In a further embodiment, with respect to the processor using the calculated respective parameters of the calibration object to perform the optimization algorithm, the processor further uses collected data including the calculated respective three-dimensional positions of the plurality of features on the intraoral three-dimensional surface.

[0101] In a further embodiment of the recalibration method, the step of driving each camera of the one or more cameras to capture an image of the two-dimensional calibration object includes the step of driving each camera of the one or more cameras to capture a plurality of images of the two-dimensional calibration object from a plurality of different viewpoints relative to the two-dimensional calibration object.

[0102] In one embodiment of an intraoral scanning device, the device includes an elongated handheld wand having a probe at its distal end, one or more illumination sources coupled to the probe, one or more near-infrared (NIR) light sources coupled to the probe, and one or more cameras coupled to the probe and configured to (a) capture images using light from the one or more illumination sources and (b) capture images using NIR light from the NIR light sources. The device further includes a processor configured to execute a navigation algorithm to determine a position of the elongated handheld wand as it moves through space, and inputs to the navigation algorithm include (a) the images captured using light from the one or more illumination sources and (b) the images captured using the NIR light.

[0103] In a further embodiment of the intraoral scanning device, the one or more illumination sources include one or more structured light sources.

[0104] In a further embodiment of the intraoral scanning device, the one or more illumination sources include one or more non-coherent light sources.

[0105] A method for tracking motion of an intraoral scanner includes illuminating an intraoral three-dimensional surface using one or more illumination sources coupled to the intraoral scanner, activating one or more NIR light sources coupled to the intraoral scanner to emit NIR light onto the intraoral three-dimensional surface, and using one or more cameras coupled to the intraoral scanner to (a) capture a first plurality of images using light from the one or more illumination sources and (b) capture a second plurality of images using the NIR light, the method further includes running a navigation algorithm using the processor to track motion of the intraoral scanner relative to the intraoral three-dimensional surface using (a) the first plurality of images captured using light from the one or more illumination sources and (b) the second plurality of images captured using the NIR light.

[0106] In one embodiment of the method for tracking motion, using one or more illumination sources comprises illuminating the intraoral three-dimensional surface.

[0107] In one embodiment of the method for tracking motion, using one or more illumination sources includes using one or more non-coherent light sources.

[0108] A sixth embodiment of a method for calculating a three-dimensional structure of an intraoral three-dimensional surface includes: activating one or more structured light projectors to project a structured light pattern onto the intraoral three-dimensional surface; activating one or more cameras to capture a plurality of structured light images, each structured light image including at least a portion of the structured light pattern; activating one or more unstructured light projectors to project unstructured light onto the intraoral three-dimensional surface; and activating the one or more cameras to capture a plurality of two-dimensional images of the intraoral three-dimensional surface. The sixth method further includes using the processor to calculate, based on the plurality of two-dimensional images of the intraoral three-dimensional surface, a three-dimensional position of each of a plurality of points on the intraoral three-dimensional surface captured in the plurality of structured light images; and calculating a three-dimensional structure of the intraoral three-dimensional surface constrained by some or all of the calculated three-dimensional positions of the plurality of points.

[0109] In some embodiments of the sixth method, the unstructured light is non-coherent light and the plurality of two-dimensional images includes a plurality of color two-dimensional images.

[0110] In some embodiments of the sixth method, the unstructured light is near-infrared (NIR) light, and the plurality of two-dimensional images comprises a plurality of monochromatic NIR images.

[0111] In a further embodiment of the sixth method, the step of driving the one or more structured light projectors includes driving the one or more structured light projectors to each project a distribution of discrete, disconnected light spots.

[0112] In a further embodiment of the sixth method, the step of calculating the three-dimensional structure includes inputting a plurality of two-dimensional images of the intraoral three-dimensional surface into a neural network, and determining, by the neural network, a respective estimated map of the intraoral three-dimensional surface captured in each of the two-dimensional images.

[0113] In a further embodiment of the sixth method, the sixth method further comprises inputting the calculated three-dimensional positions of a plurality of points on the intraoral three-dimensional surface into the neural network.

[0114] In a further embodiment of the sixth method, the sixth method further comprises using the processor to stitch together the respective maps to obtain a three-dimensional structure of the intraoral three-dimensional surface.

[0115] In a further embodiment of the sixth method, the sixth method further includes coordinating the capturing of the structured light image and the capturing of the two-dimensional image to generate an alternating sequence of one or more image frames of structured light interspersed with one or more image frames of unstructured light.

[0116] In a further embodiment of the sixth method, the determining step includes determining, by the neural network, an estimated depth map for each of the intraoral three-dimensional surfaces captured in each two-dimensional image. In one embodiment, using the processor, stitching together the estimated depth maps to obtain a three-dimensional structure of the intraoral three-dimensional surface.

[0117] In one embodiment, (a) the processor generates respective point clouds corresponding to the calculated three-dimensional positions of the plurality of points on the intraoral three-dimensional surface captured in each structured light image, the method further comprising stitching the respective estimated depth maps to the respective point clouds using the processor. In one embodiment, the method further comprises determining, by the neural network, an estimated normal map for each of the intraoral three-dimensional surface captured in each of the two-dimensional images.

[0118] In a further embodiment of the sixth method, the determining step includes determining, by the neural network, an estimated normal map for each of the intraoral three-dimensional surfaces captured in each of the two-dimensional images. In one embodiment, using the processor, the estimated normal maps are stitched together to obtain a three-dimensional structure of the intraoral three-dimensional surfaces.

[0119] In one embodiment, the method further includes interpolating three-dimensional positions on the intraoral three-dimensional surface between the calculated three-dimensional positions of the plurality of points on the intraoral three-dimensional surface captured in the plurality of structured light images based on respective estimated normal maps of the intraoral three-dimensional surface captured in each of the two-dimensional images.

[0120] In one embodiment, the method further includes coordinating the capturing of the structured light images and the capturing of the two-dimensional images to generate an alternating sequence of one or more structured light image frames interspersed with one or more unstructured light image frames. The method further includes, using the processor, (a) generating respective point clouds corresponding to the calculated three-dimensional positions of the plurality of points on the intraoral three-dimensional surface captured in each structured light image frame, and (b) stitching together, for at least a subset of the plurality of points in each point cloud, the normal to the surface at each point in the subset of points as an input for stitching, wherein for a given point cloud, the normal to the surface at at least one point in the subset of points is obtained from a respective estimated normal map of the intraoral three-dimensional surface captured in an adjacent unstructured light image frame.

[0121] In one embodiment, the method further includes using the processor to compensate for motion of the intraoral scanner between the structured light image frame and an adjacent unstructured light image frame by estimating motion of the intraoral scanner based on previous image frames.

[0122] In a further embodiment of the sixth method, the determining step includes determining, by the neural network, a curvature of the intraoral three-dimensional surface captured in each of the two-dimensional images. In one embodiment, the determining step comprises determining, by a neural network, an estimated curvature map for each of the intraoral three-dimensional surfaces captured in each of the two-dimensional images.

[0123] In one embodiment, the method further includes using the processor to evaluate a curvature of the intraoral three-dimensional surface captured in each of the two-dimensional images; and interpolating three-dimensional positions of the intraoral three-dimensional surface between the calculated three-dimensional positions of the plurality of points of the intraoral three-dimensional surface captured in the plurality of structured light images based on the evaluated curvature of the intraoral three-dimensional surface captured in each of the two-dimensional images.

[0124] In a further embodiment of the sixth method, the sixth method further includes coordinating the capture of the structured light image and the capture of the two-dimensional image to generate an alternating sequence of one or more structured light image frames interspersed with one or more unstructured light image frames.

[0125] In a further embodiment of the sixth method, the sixth method includes a step of driving the one or more cameras to capture a plurality of structured light images, which includes a step of driving each camera of two or more cameras to capture a respective one of the plurality of structured light images, and the step of driving the one or more cameras to capture a plurality of two-dimensional images includes a step of driving each camera of the two or more cameras to capture a respective one of the plurality of two-dimensional images.

[0126] In one embodiment, driving the two or more cameras includes driving each of the two or more cameras to simultaneously capture, for a given image frame, a respective two-dimensional image of a respective portion of the intraoral three-dimensional surface. Inputting the neural network includes inputting all of the respective two-dimensional images to the neural network as a single input for a given image frame, each of the respective two-dimensional images having a field of view that overlaps with at least one other of the respective two-dimensional images. Determining by the neural network includes determining, for a given image frame, an estimated depth map of the intraoral three-dimensional surface that combines the respective portions of the intraoral three-dimensional surface.

[0127] In one embodiment, driving two or more cameras to capture a plurality of structured light images includes driving each camera of three or more cameras to capture a respective one of the plurality of structured light images, and driving the two or more cameras to capture a plurality of two-dimensional images includes driving each camera of the three or more cameras to capture a respective one of the plurality of two-dimensional images. For a given image frame, each camera of the three or more cameras is driven to simultaneously capture a respective two-dimensional image of a respective portion of the intraoral three-dimensional surface. Inputting into the neural network includes, for a given image frame, inputting a subset of the respective two-dimensional images into the neural network as a single input, the subset including at least two of the respective two-dimensional images, each of the respective subsets of two-dimensional images having a field of view that overlaps with at least one other of the respective subsets of two-dimensional images. The determining step using the neural network includes determining, for a given image frame, an estimated depth map of the intraoral three-dimensional surface that combines respective portions of the intraoral three-dimensional surface captured in each subset of two-dimensional images.

[0128] In one embodiment, driving the two or more cameras includes driving each of the two or more cameras to simultaneously capture a respective two-dimensional image of a respective portion of the intraoral three-dimensional surface in a given image frame, and inputting the two or more two-dimensional images into the neural network includes inputting each of the two-dimensional images into the neural network as a separate input in a given image frame. The determining by the neural network includes determining, for a given image frame, an estimated depth map of each of the respective portions of the intraoral three-dimensional surface captured in each of the respective two-dimensional images captured in the given image frame.

[0129] In one embodiment, the method further includes using the processor to merge the respective depth maps to obtain a composite estimated depth map of the intraoral three-dimensional surface captured in a given image frame. In one embodiment, the method further includes training the neural network, wherein each input to the neural network during training includes images captured by only one camera.

[0130] In one embodiment, the method further includes determining, by the neural network, a respective estimated confidence map corresponding to each of the estimated depth maps, each of the confidence maps indicating a confidence level for each region of the respective estimated depth map. In a further embodiment, merging the respective estimated depth maps includes, using the processor, merging at least two estimated depth maps based on the confidence levels of corresponding regions as indicated by the respective confidence maps for each of the at least two estimated depth maps in response to determining a discrepancy between corresponding regions in at least two of the estimated depth maps.

[0131] In a further embodiment of the sixth method, driving one or more cameras includes driving one or more cameras of an intraoral scanner, and the method further includes training the neural network using training stage images captured by a plurality of training stage handheld wands, each training stage handheld wand including one or more reference cameras, each of the one or more cameras of the intraoral scanner corresponding to a respective one of the one or more reference cameras of each of the training stage handheld wands.

[0132] In a further embodiment of the sixth method, driving one or more structured light projectors comprises driving one or more structured light projectors of the intraoral scanner, driving one or more unstructured light projectors comprises driving one or more unstructured light projectors of the intraoral scanner, and driving one or more cameras comprises driving one or more cameras of the intraoral scanner, wherein the neural network is initially trained using training stage images captured by one or more training stage cameras of the training stage handheld wand, each of the one or more cameras of the intraoral scanner corresponding to a respective one of the one or more training stage cameras. The method then includes driving (i) the one or more structured light projectors of the intraoral scanner and (ii) the one or more unstructured light projectors of the intraoral scanner during a plurality of refinement stage scans; driving the one or more cameras of the intraoral scanner during the refinement stage scans to capture (a) a plurality of refinement stage structured light images and (b) a plurality of refinement stage two-dimensional images; calculating a three-dimensional structure of the intraoral three-dimensional surface based on the plurality of refinement stage structured light images; and refining the training of the neural network for the intraoral scanner using (a) the plurality of refinement stage two-dimensional images captured during the refinement stage scans and (b) the calculated three-dimensional structure of the intraoral three-dimensional surface calculated based on the plurality of refinement stage structured light images.

[0133] In one embodiment, the neural network includes multiple layers, and refining the training of the neural network includes constraining a subset of the layers.

[0134] In one embodiment, the method further includes selecting, from a plurality of scans, which of the plurality of scans to use as refinement stage scans based on a quality level of each scan.

[0135] In one embodiment, the method further includes using the calculated three-dimensional structure of the intraoral three-dimensional surface calculated based on the plurality of refinement stage structured light images during the refinement stage scan as a final result three-dimensional structure of the intraoral three-dimensional surface for a user of the intraoral scanner.

[0136] In a further embodiment of the sixth method, the step of driving the one or more cameras includes the step of driving one or more cameras of an intraoral scanner, each camera of the one or more cameras of the intraoral scanner corresponding to a respective one of one or more reference cameras. The method further includes using the processor to, for each camera c of one or more cameras of the intraoral scanner, crop and morph at least one of the two-dimensional images of the intraoral three-dimensional surface from camera c to obtain a plurality of cropped and morphed two-dimensional images, each cropped and morphed image corresponding to a cropped and morphed field of view of camera c, the cropped and morphed field of view of camera c matching the cropped field of view of a corresponding reference camera; inputting the plurality of two-dimensional images to the neural network includes inputting the plurality of cropped and morphed two-dimensional images of the intraoral three-dimensional surface to the neural network; and determining includes determining, by the neural network, a respective estimated map of the intraoral three-dimensional surface captured in each of the cropped and morphed two-dimensional images, the neural network having been trained using training stage images corresponding to the cropped field of view of each of the one or more reference cameras.

[0137] In a further embodiment, the cropping and morphing step includes the processor using (a) stored calibration values ​​indicating camera rays corresponding to each pixel on a camera sensor of each of the one or more cameras, and (b) reference calibration values ​​indicating (i) camera rays corresponding to each pixel on a reference camera sensor of each of the one or more reference cameras, and (ii) a cropped field of view of each of the one or more reference cameras.

[0138] In a further embodiment, the cropped field of view of each of the one or more reference cameras is 85-97% of the total field of view of each of the one or more reference cameras.

[0139] In a further embodiment, the step of using the processor further includes the step of performing reverse morphing, for each camera c, on each estimated map of each of the intraoral three-dimensional surfaces captured in each of the cropped and morphed two-dimensional images to obtain a non-morphed estimated map of each of the intraoral surfaces as seen in each of at least one two-dimensional image from camera c before morphing.

[0140] In a further embodiment, the unstructured light is non-coherent light and the plurality of two-dimensional images comprises a plurality of two-dimensional color images.

[0141] In a further embodiment, the unstructured light is near-infrared (NIR) light and the plurality of two-dimensional images comprises a plurality of monochromatic NIR images.

[0142] In a further embodiment, the step of using the processor further includes the step of performing reverse morphing, for each camera c, on each estimated map of each of the intraoral three-dimensional surfaces captured in each of the cropped and morphed two-dimensional images to obtain a non-morphed estimated map of each of the intraoral surfaces as seen in each of at least one two-dimensional image from camera c before morphing.

[0143] It should be noted that all of the above-described implementations of the sixth method relating to depth maps, normal maps, curvature maps and their use can be performed mutatis mutandis based on runtime images that have been cropped and morphed in situ.

[0144] In one embodiment, driving one or more structured light projectors includes driving one or more structured light projectors of the intraoral scanner, and driving one or more unstructured light projectors includes driving one or more unstructured light projectors of the intraoral scanner. The method further includes driving one or more structured light projectors of the intraoral scanner and one or more unstructured light projectors of the intraoral scanner during multiple refinement stage scans after the neural network has been trained using training stage images corresponding to cropped fields of view of each of the one or more reference cameras. The one or more cameras of the intraoral scanner are driven to capture (a) multiple refinement stage structured light images and (b) multiple refinement stage two-dimensional images during the multiple refinement stage structured light scans. A three-dimensional structure of the intraoral three-dimensional surface is calculated based on a plurality of refinement stage structured light images, and the training of the neural network is refined for the intraoral scanner using (a) the plurality of refinement stage two-dimensional images captured during the refinement stage scan, and (b) a calculated three-dimensional structure of the intraoral three-dimensional surface calculated based on a plurality of refinement stage structured light images. In a further embodiment, the neural network includes a plurality of layers, and refining the training of the neural network includes constraining a subset of the layers.

[0145] In one embodiment, driving one or more structured light projectors comprises driving one or more structured light projectors of the intraoral scanner; driving one or more unstructured light projectors comprises driving one or more unstructured light projectors of the intraoral scanner; and determining comprises determining, by the neural network, an estimated depth map for each of the intraoral three-dimensional surfaces captured in each of the cropped and morphed two-dimensional images. The method further includes the steps of: (a) calculating a three-dimensional structure of the intraoral three-dimensional surface based on calculated three-dimensional positions of each of a plurality of points on the intraoral three-dimensional surface captured in a plurality of structured light images; (b) calculating a three-dimensional structure of the intraoral three-dimensional surface based on estimated depth maps of each of the intraoral three-dimensional surface captured in each of the cropped and morphed two-dimensional images; and (c) comparing (i) the three-dimensional structure of the intraoral three-dimensional surface calculated based on the calculated three-dimensional positions of each of the plurality of points on the intraoral three-dimensional surface with (ii) the three-dimensional structure of the intraoral three-dimensional surface calculated based on the estimated depth maps of each of the intraoral three-dimensional surface. In response to determining a discrepancy between (i) and (ii), the method includes the steps of: activating (A) the one or more structured light projectors of the intraoral scanner and (B) the one or more unstructured light projectors of the intraoral scanner during a plurality of refinement stage scans; activating one or more cameras of the intraoral scanner during a plurality of refinement stage scans to capture (a) a plurality of refinement stage structured light images and (b) a plurality of refinement stage two-dimensional images; calculating a three-dimensional structure of the intraoral three-dimensional surface based on the plurality of refinement stage structured light images; and refining training of a neural network for the intraoral scanner using (a) the plurality of two-dimensional images captured during the refinement stage scans and (b) the calculated three-dimensional structure of the intraoral three-dimensional surface calculated based on the plurality of refinement stage structured light images.In a further embodiment, the neural network includes multiple layers, and refining the training of the neural network includes constraining a subset of the layers.

[0146] In a further embodiment of the sixth method, the sixth method further comprises the step of training the neural network, the training comprising the steps of driving one or more training stage structured light projectors to project a training stage structured light pattern onto a training stage three-dimensional surface; driving one or more training stage cameras to capture a plurality of structured light images, each image including at least a portion of the training stage structured light pattern; driving one or more training stage unstructured light projectors to project unstructured light onto the training stage three-dimensional surface; driving the one or more training stage cameras to capture a plurality of two-dimensional images of the training stage three-dimensional surface using illumination from the training stage unstructured light projectors; coordinating the capturing of the structured light images and the capturing of the two-dimensional images to produce an alternating sequence of image frames of one or more structured light images interspersed with image frames of one or more two-dimensional images; inputting a plurality of two-dimensional images into the neural network; inputting a plurality of respective 3D reconstructions of the training stage 3D surface into a neural network based on structured light images of the training stage 3D surface, the 3D reconstructions including calculated 3D positions of a plurality of points on the training stage 3D surface; interpolating a position of the one or more training stage cameras relative to the training stage 3D surface of each of the two-dimensional image frames based on the calculated 3D positions of a plurality of points on the training stage 3D surface calculated based on respective structured light image frames before and after each of the two-dimensional image frames; projecting the 3D reconstructions into respective fields of view of the one or more training stage cameras and calculating, based on the projections, a true map of the training stage 3D surface seen in each of the two-dimensional images, constrained by the calculated 3D positions of the plurality of points; comparing each estimated depth map of the training stage 3D surface with a corresponding true map of the training stage 3D surface, and based on the difference between each estimated map and the corresponding true map,and optimizing the neural network to better estimate the subsequent estimation map.

[0147] In further embodiments, the training comprises initial training of the neural network, driving the one or more structured light projectors comprises driving one or more structured light projectors of the intraoral scanner, driving the one or more unstructured light projectors comprises driving one or more unstructured light projectors of the intraoral scanner, and driving one or more cameras comprises driving one or more cameras of the intraoral scanner. The method further includes, after initial training of a neural network, activating (i) the one or more structured light projectors of the intraoral scanner and (ii) the one or more unstructured light projectors of the intraoral scanner during a plurality of refinement stage structured light scans, activating one or more cameras of the intraoral scanner during the refinement stage structured light scans to capture (a) a plurality of refinement stage structured light images and (b) a plurality of refinement stage two-dimensional images, calculating a three-dimensional structure of the intraoral three-dimensional surface based on the plurality of refinement stage structured light images, and refining the training of the neural network using (a) the plurality of two-dimensional images captured during the refinement stage scans and (b) the three-dimensional structure of the intraoral three-dimensional surface calculated based on the plurality of refinement stage structured light images. In a further embodiment, the neural network includes a plurality of layers, and refining the training of the neural network includes constraining a subset of the layers.

[0148] In a further embodiment of the sixth method, the step of driving the one or more structured light projectors to project a training stage structured light pattern includes driving the one or more structured light projectors to each project a distribution of discrete, disconnected light spots onto the training stage three-dimensional surface.

[0149] In a further embodiment of the sixth method, driving one or more training stage cameras includes driving at least two training stage cameras.

[0150] In a further embodiment of the sixth method, the unstructured light comprises broad spectrum light.

[0151] In one embodiment of an intraoral scanning device, the device includes an elongated handheld wand including a probe at a distal end thereof configured to be removably positioned within a sleeve. The device further includes at least one structured light projector coupled to the probe, the at least one structured light projector including (a) a laser configured to emit polarized laser light and (b) pattern-generating optics configured to generate a pattern of light when the laser is activated and transmits light through the pattern-generating optics. The device further includes a camera coupled to the probe, the camera including a camera sensor. The probe is configured such that light enters and exits the probe through the sleeve. Furthermore, the laser is positioned at a distance relative to the camera such that when the probe is positioned within the sleeve, a portion of the light pattern is reflected from the sleeve to reach the camera sensor. Furthermore, the laser is positioned at a rotational angle relative to its optical axis such that polarization of the light pattern causes the extent of reflection of the portion of the light pattern by the sleeve to be less than a threshold reflection for all possible rotational angles of the laser relative to its optical axis.

[0152] In a further embodiment of the intraoral scanning device, the threshold is 70% of the maximum reflection for all possible angles of rotation of said laser relative to its optical axis.

[0153] In a further embodiment of the intraoral scanning device, the laser is positioned at a rotational angle relative to its own optical axis such that the polarization of the light pattern causes the extent of reflection by the sleeve of portions of the light pattern to be less than 60% of the maximum reflection for all possible rotational angles of the laser relative to its optical axis.

[0154] In a further embodiment of the intraoral scanning device, the laser is positioned at a rotational angle relative to its own optical axis such that the polarization of the light pattern causes the extent of reflection by the sleeve of portions of the light pattern to be between 15% and 60% of the maximum reflection for all possible rotational angles of the laser relative to its optical axis.

[0155] In a further embodiment of the intraoral scanning device, when the elongated handheld wand is positioned within the sleeve, the distance between the structured light projector and the camera is 1 to 6 times the distance between the structured light projector and the sleeve.

[0156] In a further embodiment of the intraoral scanning device, the at least one structured light projector has an illumination field of at least 30 degrees and the camera has a field of view of at least 30 degrees.

[0157] In a seventh method for generating a three-dimensional image using an intraoral scanner, the seventh method includes capturing a plurality of images of the intraoral three-dimensional surface using at least two cameras rigidly connected to the intraoral scanner such that each camera has a non-overlapping field of view, the seventh method further includes using the processor to run a simultaneous localization and mapping (SLAM) algorithm using the captured images from each of the cameras for the non-overlapping portions of their respective fields of view, wherein the localization of each camera is solved based on the motion of each camera being the same as the motion of all other cameras.

[0158] In a further embodiment of the seventh method, the fields of view of a first one of the cameras and a second one of the cameras also have an overlapping portion. Further, the capturing step includes capturing multiple images of the intraoral three-dimensional surface such that features of the intraoral three-dimensional surface that are in the overlapping portion of the respective fields of view appear in the images captured by the first and second cameras. Further, the using the processor step includes performing a SLAM algorithm using features of the intraoral three-dimensional surface that appear in the images of the at least two cameras.

[0159] An eighth method for generating a three-dimensional image using an intraoral scanner, the eighth method comprising: driving one or more structured light projectors to project a structured light pattern onto the three-dimensional intraoral surface; driving one or more cameras to capture a plurality of structured light images, each structured light image including at least a portion of the structured light pattern; driving one or more unstructured light projectors to project unstructured light onto the three-dimensional intraoral surface; driving at least one camera to capture two-dimensional images of the intraoral surface using illumination from the unstructured light projectors; and coordinating the capture of structured light and unstructured light to generate an alternating sequence of one or more structured light image frames interspersed with one or more unstructured light image frames. The eighth method further comprises using the processor to calculate three-dimensional positions of each of a plurality of points on the three-dimensional intraoral surface captured in the one or more structured light image frames. An eighth method further includes using the processor to interpolate motion of at least one camera between a first unstructured light image frame and a second unstructured light image frame based on calculated three-dimensional positions of a plurality of points in each of the structured light image frames before and after the unstructured light image frame. The eighth method further includes (a) using features of the intraoral three-dimensional surface captured by the at least one camera in the first and second unstructured light image frames, and (b) performing a simultaneous localization and mapping (SLAM) algorithm constrained by the interpolated motion of the camera between the first unstructured light image frame and the second unstructured light image frame.

[0160] In one embodiment of the eighth method, driving one or more structured light projectors to project the structured light pattern comprises driving the one or more structured light projectors to respectively project a distribution of discrete, disconnected light spots onto the three-dimensional intraoral surface. In one embodiment, the unstructured light comprises broadband spectrum light, and the two-dimensional image comprises a two-dimensional color image. In one embodiment, the unstructured light comprises near-infrared (NIR) light, and the two-dimensional image comprises a two-dimensional monochromatic NIR image.

[0161] In one embodiment of a ninth method for generating a three-dimensional image using an intraoral scanner, the ninth method includes driving one or more structured light projectors to project a pattern of structured light onto the intraoral three-dimensional surface; driving one or more cameras to capture a plurality of structured light images, each structured light image comprising at least a portion of the structured light pattern; driving one or more unstructured light projectors to project unstructured light onto the intraoral three-dimensional surface; driving one or more cameras to capture two-dimensional images of the intraoral three-dimensional surface using illumination from the unstructured light projectors; and coordinating the capture of structured light and the capture of unstructured light to generate an alternating sequence of one or more structured light image frames interspersed with one or more unstructured light image frames. A ninth method further includes using the processor to calculate, based on the structured light image frames, three-dimensional positions of features on the intraoral three-dimensional surface, which features were also captured in the first unstructured light image frame and the second unstructured light image frame, and further includes calculating motion of one or more cameras between the first unstructured light image frame and the second unstructured light image frame based on the calculated three-dimensional positions of the features; and performing a simultaneous localization and mapping (SLAM) algorithm using (i) features of the intraoral three-dimensional surface whose three-dimensional positions were not calculated based on the structured light image frames captured by the one or more cameras in the first and second unstructured light image frames, and (ii) the calculated motion of the cameras between the first and second unstructured light image frames. In one embodiment, driving one or more structured light projectors to project the structured light pattern comprises driving the one or more structured light projectors to respectively project a distribution of discrete, disconnected light spots onto the intraoral three-dimensional surface. In one embodiment, the unstructured light comprises broadband spectrum light, and the two-dimensional image comprises a two-dimensional color image. In one embodiment, the unstructured light comprises near-infrared (NIR) light, and the two-dimensional image comprises a two-dimensional monochromatic NIR image.

[0162] In one method for calculating a three-dimensional structure of a three-dimensional intraoral surface within a subject's oral cavity, the method includes the steps of: (a) driving one or more structured light projectors to project a pattern of structured light onto the three-dimensional intraoral surface, the pattern including a plurality of features; (b) driving one or more cameras to capture a plurality of structured light images, each structured light image including at least one of the features of the structured light pattern; (c) driving one or more unstructured light projectors to project unstructured light onto the three-dimensional intraoral surface; (d) driving at least one camera to capture two-dimensional images of the intraoral three-dimensional surface using illumination from the one or more unstructured light projectors; and (e) coordinating the capture of structured light and the capture of unstructured light to generate an alternating sequence of one or more structured light image frames interspersed with one or more unstructured light image frames. The method further includes, using a processor, (a) determining, based on the two-dimensional image, whether one or more features of the plurality of features of the structured light pattern are projected onto moving tissue or stable tissue within the oral cavity; (b) assigning a respective confidence grade to each of the one or more features based on the determination, with high confidence assigned to stationary tissue and low confidence assigned to moving tissue; and (c) executing a three-dimensional reconstruction algorithm using the one or more features based on the confidence grade for each of the one or more features. In one embodiment, the unstructured light includes broadband spectrum light, and the two-dimensional image is a two-dimensional color image. In one embodiment, the unstructured light includes near-infrared (NIR) light, and the two-dimensional image is a two-dimensional monochromatic NIR image. In one embodiment, the plurality of features include a plurality of spots, and driving the one or more structured light projectors to project the structured light pattern includes driving the one or more structured light projectors to respectively project a distribution of discrete, disconnected light spots onto the three-dimensional intraoral surface. In one embodiment, the step of performing the three-dimensional reconstruction algorithm is performed using only a subset of the plurality of features, the subset consisting of features that have been assigned a confidence grade above a fixed tissue threshold.In one embodiment, the step of executing the 3D reconstruction algorithm includes the steps of: (a) for each feature, assigning a weight to the feature based on the respective confidence grade assigned to the feature; and (b) using the respective weight of each feature in the 3D reconstruction algorithm.

[0163] In one embodiment of a tenth method for calculating a three-dimensional structure of an intraoral three-dimensional surface, the tenth method includes activating one or more light sources of an intraoral scanner to project light onto the intraoral three-dimensional surface and activating two or more cameras of the intraoral scanner to respectively capture a plurality of two-dimensional images of the intraoral three-dimensional surface, wherein each of the two or more cameras of the intraoral scanner corresponds to a respective one of two or more reference cameras. The method includes, for each camera c of the two or more cameras of the intraoral scanner, using a processor to rectify at least one of the two-dimensional images from camera c to obtain a plurality of rectified two-dimensional images, each rectified image corresponding to a rectified field of view of camera c, wherein the rectified field of view of camera c matches the rectified field of view of a corresponding one of the reference cameras; and calculating the three-dimensional structure of the intraoral three-dimensional surface based on the plurality of rectified two-dimensional images of the intraoral three-dimensional surface.

[0164] In an embodiment of the eleventh method for calculating the three-dimensional structure of the intraoral three-dimensional surface, the eleventh method includes activating one or more light sources of the intraoral scanner to project light onto the intraoral three-dimensional surface, and activating two or more cameras of the intraoral scanner to respectively capture a plurality of two-dimensional images of the intraoral three-dimensional surface, each of the one or more cameras of the intraoral scanner corresponding to a respective one of the two or more reference cameras. The method also includes, for each camera c of the two or more cameras of the intraoral scanner, cropping and morphing at least one of the two-dimensional images from camera c to obtain a plurality of cropped and morphed two-dimensional images, each of the cropped and morphed images corresponding to a cropped and morphed field of view of camera c, wherein the cropped and morphed field of view of camera c matches the cropped field of view of a corresponding one of the reference cameras. The three-dimensional structure of the intraoral three-dimensional surface is calculated based on a plurality of cropped and morphed two-dimensional images of the intraoral three-dimensional surface by inputting the plurality of cropped and morphed two-dimensional images of the intraoral three-dimensional surface into a neural network and determining, by the neural network, a respective estimated map of the intraoral three-dimensional surface captured in each of the plurality of cropped and morphed two-dimensional images, the neural network having been trained using training stage images corresponding to a cropped field of view of each of the one or more reference cameras.

[0165] In a further embodiment of the eleventh method, the light is incoherent light and the plurality of two-dimensional images comprises a plurality of two-dimensional color images.

[0166] In a further embodiment of the eleventh method, the light is near-infrared (NIR) light and the plurality of two-dimensional images comprises a plurality of monochromatic NIR images.

[0167] In a further embodiment of the eleventh method, the light is broad spectrum light and the plurality of two-dimensional images comprises a plurality of two-dimensional color images.

[0168] In a further embodiment of the eleventh method, the cropping and morphing step includes the processor using (a) stored calibration values ​​indicating camera rays corresponding to each pixel on a camera sensor of each camera c of one or more cameras c, and (b) reference calibration values ​​indicating (i) camera rays corresponding to each pixel on a reference camera sensor of each reference camera of one or more reference cameras, and (ii) a cropped field of view of each reference camera of one or more reference cameras.

[0169] In a further embodiment of the eleventh method, the cropped field of view of each of the one or more reference cameras is between 85 and 97% of the full field of view of each of the one or more reference cameras.

[0170] In a further embodiment, the step of using the processor further includes the step of performing reverse morphing, for each camera c, on each estimated map of each of the intraoral three-dimensional surfaces captured in each of the cropped and morphed two-dimensional images to obtain a non-morphed estimated map of each of the intraoral surfaces as seen in each of at least one two-dimensional image from camera c before morphing.

[0171] It should be noted that all of the above-described implementations of the sixth method relating to depth maps, normal maps, curvature maps and their use may be performed mutatis mutandis from the eleventh method based on runtime images that are cropped and morphed in situ.

[0172] It should also be noted that all of the above-described implementations of the sixth method relating to structured light may be implemented mutatis mutandis in the context of the eleventh method and a cropped and morphed runtime two-dimensional image.

[0173] In an embodiment of the twelfth method for calculating a three-dimensional structure of an intraoral three-dimensional surface, the twelfth method includes: activating one or more light projectors to project light onto the intraoral three-dimensional surface; and activating one or more cameras to capture a plurality of two-dimensional images of the intraoral three-dimensional surface. The method also includes, using a processor, inputting the plurality of two-dimensional images of the intraoral three-dimensional surface to a first neural network module and a second neural network module; determining, by the first neural network module, a respective estimated depth map of the intraoral three-dimensional surface captured in each of the two-dimensional images; and determining, by the second neural network module, a respective estimated confidence map corresponding to each estimated depth map, wherein each confidence map indicates a confidence for each region of a respective estimated depth map.

[0174] In one embodiment of the twelfth method, the first neural network module and the second neural network module are separate modules of the same neural network.

[0175] In one embodiment of the twelfth method, each of the first and second neural network modules is not a separate module of the same neural network.

[0176] In a further embodiment of the twelfth method, the method further includes training a second neural network module to determine respective estimated confidence maps corresponding to the respective estimated depth maps determined by the first neural network module, the determining step including initially training the first neural network module to determine the respective estimated depth maps using a plurality of depth training stage two-dimensional images, and thereafter (i) inputting a plurality of confidence training stage two-dimensional images of the training stage three-dimensional surfaces into the first neural network module, and (ii) determining, by the first neural network module, the estimated depths of each of the training stage three-dimensional surfaces captured in each of the confidence training stage two-dimensional images. (iii) calculating the difference between each estimated depth map and a corresponding true depth map to obtain a respective target confidence map corresponding to each estimated depth map determined by the first neural network module; (iv) inputting a plurality of confidence training stage two-dimensional images to the second neural network module; (v) estimating, by the second neural network module, respective estimated confidence maps indicating the confidence for each region of each estimated depth map; and (vi) comparing, by the second neural network module, each estimated confidence map with the corresponding target confidence map, and optimizing the second neural network module to better estimate subsequent estimated confidence maps based on the comparison.

[0177] In one embodiment, the plurality of confidence training stage two-dimensional images are not the same as the plurality of depth training stage two-dimensional images.

[0178] In one embodiment, the plurality of confidence training stage two-dimensional images are the same as the plurality of depth training stage two-dimensional images.

[0179] In a further embodiment of the twelfth method, (a) the step of driving the one or more cameras to capture a plurality of two-dimensional images includes driving each of the two or more cameras to simultaneously capture a respective two-dimensional image of a respective portion of the intraoral three-dimensional surface, for a given image frame; and (b) the step of inputting the plurality of two-dimensional images of the intraoral three-dimensional surface to a first neural network module and a second neural network module includes, for a given image frame, driving each of the respective two-dimensional images to a separate first neural network module and a second neural network module. (c) determining by the first neural network module includes, for a given image frame, determining a respective estimated depth map for each portion of the intraoral three-dimensional surface captured in each of the two-dimensional images captured in the given image frame, and (d) determining by the second neural network module includes, for the given image frame, determining a respective estimated confidence map corresponding to each estimated depth map for each portion of the intraoral three-dimensional surface captured in each of the two-dimensional images captured in the given image frame. The method further includes using the processor to merge the respective estimated depth maps to obtain a composite estimated depth map of the intraoral three-dimensional surface captured in the given image frame. In response to determining a discrepancy between corresponding regions in at least two of the estimated depth maps, the processor merges the at least two estimated depth maps based on the confidence of the corresponding regions as indicated by a respective confidence map for each of the at least two estimated depth maps.

[0180] In an embodiment of the thirteenth method for calculating the three-dimensional structure of an intraoral three-dimensional surface, the thirteenth method includes activating one or more light sources of an intraoral scanner to project light onto the intraoral three-dimensional surface and activating one or more cameras of the intraoral scanner to capture a plurality of two-dimensional images of the intraoral three-dimensional surface, the method further includes (a) using a processor to determine, using a neural network, a respective estimated map of the intraoral three-dimensional surface captured in each of the two-dimensional images, and (b) using a processor to overcome manufacturing variations of the one or more cameras of the intraoral scanner to reduce differences between the estimated map and the true structure of the intraoral three-dimensional surface.

[0181] In an embodiment of the thirteenth method, overcoming manufacturing variations of the one or more cameras includes overcoming manufacturing variations of the one or more cameras from a reference set of one or more cameras.

[0182] In one embodiment of the thirteenth method, the intraoral scanner is one of a plurality of manufactured intraoral scanners, each manufactured intraoral scanner including one or more sets of cameras, and the step of overcoming manufacturing variations of one or more cameras of the intraoral scanner includes overcoming manufacturing variations of one or more cameras from one or more sets of cameras of at least one other of the plurality of manufactured intraoral scanners.

[0183] In a further embodiment of the thirteenth method, driving one or more cameras comprises driving two or more cameras of the intraoral scanner to respectively capture a plurality of two-dimensional images of the intraoral three-dimensional surface, each of the two or more cameras of the intraoral scanner corresponding to a respective one of two or more reference cameras, and the neural network has been trained using training stage images captured by the two or more reference cameras. Overcoming manufacturing variations comprises overcoming manufacturing variations of the two or more cameras of the intraoral scanner by using the processor to: (a) for each camera c of the two or more cameras of the intraoral scanner, rectify at least one of the two-dimensional images from camera c to obtain a plurality of rectified two-dimensional images, each rectified image corresponding to a rectified field of view of camera c, which rectified field of view of camera c matches the rectified field of view of a corresponding one of the reference cameras; and (b) determining, by the neural network, an estimated map of each of the intraoral three-dimensional surface based on the plurality of rectified two-dimensional images of the intraoral three-dimensional surface.

[0184] In a further embodiment, the light is incoherent light and the plurality of two-dimensional images comprises a plurality of two-dimensional color images.

[0185] In a further embodiment, the light is near-infrared (NIR) light and the plurality of two-dimensional images comprises a plurality of monochromatic NIR images.

[0186] In a further embodiment, the light is broad spectrum light and the plurality of two-dimensional images comprises a plurality of two-dimensional color images.

[0187] In a further embodiment, the modifying step includes cropping and morphing at least one of the two-dimensional images from camera c to obtain a plurality of cropped and morphed two-dimensional images, each cropped and morphed image corresponding to a cropped and morphed field of view of camera c, and the cropped and morphed field of view of camera c being consistent with the cropped field of view of a corresponding one of the reference cameras.

[0188] In a further embodiment, the cropping and morphing step includes the processor using (a) stored calibration values ​​indicating camera rays corresponding to each pixel on a camera sensor of each camera c of the one or more cameras c, and (b) reference calibration values ​​indicating (i) camera rays corresponding to each pixel on a reference camera sensor of each reference camera of the one or more reference cameras, and (ii) a cropped field of view of each reference camera of the one or more reference cameras.

[0189] In a further embodiment, the cropped field of view of each of the one or more reference cameras is between 85 and 97% of the full field of view of each of the one or more reference cameras.

[0190] In a further embodiment, the processor further includes performing reverse morphing on each estimated map of the intraoral three-dimensional surface captured in each cropped and morphed two-dimensional image for each camera c to obtain a respective unmorphed estimated map of the intraoral surface seen in each of the at least one two-dimensional image from camera c before morphing.

[0191] In a further embodiment of the thirteenth method, overcoming manufacturing variations of the one or more cameras of the intraoral scanner includes training the neural network using training stage images captured by a plurality of training stage intraoral scanners, each training stage intraoral scanner including one or more reference cameras, each of the one or more cameras of the intraoral scanner corresponding to a respective one of the one or more reference cameras on each training stage intraoral scanner, and the manufacturing variations of the one or more cameras are manufacturing variations of the one or more cameras from a corresponding one or more reference cameras.

[0192] In a further embodiment of the thirteenth method, the step of driving one or more cameras includes driving two or more cameras of the intraoral scanner to respectively capture a plurality of two-dimensional images of the intraoral three-dimensional surface. Overcoming manufacturing variations includes overcoming manufacturing variations of two or more cameras of the intraoral scanner by: training the neural network using training stage images, each captured by only one camera; driving the two or more cameras of the intraoral scanner to simultaneously capture respective two-dimensional images of respective portions of the intraoral three-dimensional surface in a given image frame; inputting each of the two-dimensional images as a separate input to the neural network for a given image frame; determining, by the neural network, a respective estimated depth map for each portion of the intraoral three-dimensional surface captured in each respective two-dimensional image captured in the given image frame; and merging, using the processor, the respective estimated depth maps to obtain a composite estimated depth map of the intraoral three-dimensional surface captured in the given image frame.

[0193] In a further embodiment, the determining step further comprises determining, by the neural network, a respective estimated confidence map corresponding to each estimated depth map, each confidence map indicating a confidence for each region of the respective estimated depth map.

[0194] In a further embodiment, the step of merging the respective estimated depth maps includes, in response to determining a discrepancy between corresponding respective regions in at least two of the estimated depth maps, using the processor, merging the at least two estimated depth maps based on the confidence of the corresponding respective regions as indicated by the respective confidence maps for each of the at least two estimated depth maps.

[0195] In a further embodiment of the thirteenth method, the step of overcoming manufacturing variations in one or more cameras of the intraoral scanner includes: (a) initially training the neural network using training stage images captured by one or more training stage cameras of one or more training stage handheld wands, each of the one or more cameras of the intraoral scanner corresponding to a respective one or more training stage cameras of the one or more training stage handheld wands; and (b) thereafter driving the intraoral scanner to perform multiple refinement stage scans of the intraoral three-dimensional surface and refining the training of the neural network for the intraoral scanner using the refinement stage scans of the intraoral three-dimensional surface.

[0196] In a further embodiment, the neural network includes multiple layers, and refining the training of the neural network includes constraining a subset of the layers.

[0197] In a further embodiment, the method further comprises selecting, from the plurality of scans, which of the plurality of scans to use as the refinement stage scan based on a quality level of each scan.

[0198] In a further embodiment, the step of driving the intraoral scanner to perform multiple refinement stage scans includes, during the multiple refinement stage scans, (i) driving one or more structured light projectors of the intraoral scanner to project a structured light pattern onto the intraoral three-dimensional surface; (ii) driving one or more unstructured light projectors of the intraoral scanner to project unstructured light onto the intraoral three-dimensional surface; driving one or more cameras of the intraoral scanner to capture, during the refinement stage scans, (a) multiple refinement stage structured light images using illumination from the structured light projectors and (b) multiple refinement stage two-dimensional images using illumination from the unstructured light projectors; and calculating a three-dimensional structure of the intraoral three-dimensional surface based on the multiple refinement stage structured light images.

[0199] In a further embodiment, the step of refining the training of the neural network includes refining the training of the neural network for the intraoral scanner using (a) a plurality of refinement stage two-dimensional images captured during a refinement stage scan and (b) a calculated three-dimensional structure of the intraoral three-dimensional surface calculated based on a plurality of refinement stage structured light images.

[0200] In a further embodiment, the method further includes using a calculated 3D structure of the intraoral 3D surface calculated based on the plurality of refinement stage structured light images during the refinement stage scan as a final result 3D structure of the intraoral 3D surface for a user of the intraoral scanner.

[0201] In one embodiment of the fourteenth method for training a neural network for use with an intraoral scanner, the fourteenth method includes inputting a plurality of two-dimensional images of an intraoral three-dimensional surface into a neural network; estimating, by the neural network, an estimated map of the intraoral three-dimensional surface captured in each of the two-dimensional images; calculating a true map of the intraoral three-dimensional surface seen in each of the two-dimensional images based on a plurality of structured light images of the intraoral three-dimensional surface; comparing each estimated map of the intraoral three-dimensional surface with a corresponding true map of the intraoral three-dimensional surface; and optimizing the neural network based on differences between each estimated map and the corresponding true map to improve estimation of subsequent estimated maps; wherein, for two-dimensional images in which moving tissue is identified, the image is processed to remove at least a portion of the moving tissue before inputting the two-dimensional image into the neural network.

[0202] In a further embodiment of the fourteenth method, the method includes the steps of: driving one or more structured light projectors to project a structured light pattern onto the intraoral three-dimensional surface; driving one or more cameras to capture a plurality of structured light images, each image including at least a portion of the structured light pattern; driving one or more unstructured light projectors to project unstructured light onto the intraoral three-dimensional surface; driving the one or more cameras to capture a plurality of two-dimensional images of the intraoral three-dimensional surface using illumination from the unstructured light projectors; and coordinating the capture of the structured light images and the capture of the two-dimensional images to produce an alternating sequence of image frames of one or more structured light images interspersed with image frames of one or more two-dimensional images. Further, the step of calculating a true map of the intraoral three-dimensional surface seen in each of the two-dimensional images includes the steps of: inputting a plurality of respective three-dimensional reconstructions of the intraoral three-dimensional surface into the neural network based on structured light images of the intraoral three-dimensional surface, the three-dimensional reconstructions including calculated three-dimensional positions of a plurality of points on the intraoral three-dimensional surface; interpolating positions of one or more cameras relative to the intraoral three-dimensional surface for each two-dimensional image frame based on the calculated three-dimensional positions of a plurality of points on the intraoral three-dimensional surface calculated based on respective structured light image frames before and after each two-dimensional image frame; and projecting the three-dimensional reconstructions into respective fields of view of the one or more cameras; and calculating a true map of the intraoral three-dimensional surface seen in each two-dimensional image based on the projections, the true map being constrained by the calculated three-dimensional positions of the plurality of points.

[0203] Furthermore, in accordance with some applications of the present invention, there is provided a method for generating a digital three-dimensional image, the method comprising: activating each structured light projector of the one or more structured light projectors to project a pattern of light (e.g., a distribution of discrete, disconnected light spots) onto the intraoral three-dimensional surface; activating each camera of one or more cameras to capture a plurality of images, each image including at least a portion of the projected pattern, each camera of the one or more cameras including a camera sensor including an array of pixels; and using a processor to compare multiple successive images captured by each camera to determine a portion of the captured projection pattern (e.g., projection spot s) that can be tracked across the multiple images.

[0204] In some applications, the projection pattern is a distribution of disconnected light spots, and the processor may make the determination based on stored calibration values ​​indicating (a) a camera ray corresponding to each pixel on the camera sensor of each of the one or more cameras, and (b) a projector ray corresponding to each projected light spot of each of the one or more projectors. In some embodiments, each projector ray corresponds to a path of each pixel on at least one of the camera sensors. Further, in some embodiments, the processor may determine which projected spots s can be tracked across multiple images, with each tracked spot s moving along the path of the pixel corresponding to a respective projector ray r.

[0205] In some applications, using the processor further includes using the processor to calculate, on the intraoral three-dimensional surface, respective three-dimensional positions of intersections of projector rays r and respective camera rays corresponding to tracked spot s in each of a plurality of successive images in which spot s is tracked.

[0206] In some applications, the step of using the processor further comprises using the processor to: (a) determining parameters of a tracking spot in at least two adjacent images from consecutive images, the parameters including one or more of a size of the spot, a shape of the spot, an orientation of the spot, an intensity of the spot, and a signal-to-noise ratio (SNR) of the spot; (b) predicting parameters of the tracking spot in a subsequent image based on parameters of the tracking spot in the at least two adjacent images.

[0207] In some applications, using the processor further includes using the processor to search for a spot in a subsequent image that substantially has the predicted parameters based on the predicted parameters of the tracking spot.

[0208] In some applications, the selected parameter is a spot shape, and using the processor further includes using the processor to determine a search space in a next image in which to search for the tracking spot based on the predicted shape of the tracking spot.

[0209] In some applications, determining a search space using the processor includes using the processor to determine a search space in a subsequent image in which to search for the tracking spot, the search space having a size and aspect ratio based on a size and / or aspect ratio of an expected shape of the tracking spot.

[0210] In some applications, the selected parameter is a spot shape, and the step of using the processor further comprises using the processor to: (a) determining a velocity vector of the tracking spot based on the direction and distance traveled by the tracking spot between two adjacent images from the sequence of images; (b) predicting the shape of the tracking spot in a subsequent image in response to the shape of the tracking spot in at least one of the two adjacent images; (c) determining a search space in the subsequent image in which to search for the tracking spot in response to a combination of (i) the determination of the velocity vector of the tracking spot and (ii) the predicted shape of the tracking spot.

[0211] In some applications, the selected parameter is a shape of the spot, and the step of using the processor includes using the processor to: (a) determining a velocity vector of the tracking spot based on a direction and distance traveled by the tracking spot between two adjacent images from the sequence of images; (b) predicting the shape of the tracking spot in a subsequent image in response to determining the velocity vector of the tracking spot; (c) determining a search space in the subsequent image in which to search for the tracking spot in response to a combination of (i) the determination of the velocity vector of the tracking spot and (ii) the predicted shape of the tracking spot.

[0212] In some applications, using the processor includes using the processor to predict a shape of the tracking spot in a subsequent image in response to a combination of (i) determining a velocity vector of the tracking spot and (ii) a shape of the tracking spot in at least one of two adjacent images.

[0213] In some applications, the step of using the processor includes using the processor to: (a) determining a velocity vector of the tracking spot based on the direction and distance traveled by the tracking spot between two successive images; (b) determining a search space in a subsequent image in which to search for the tracking spot in response to determining the velocity vector of the tracking spot.

[0214] In some applications, the step of using the processor further includes determining, for each tracking spot s, a plurality of possible paths p of pixels on a given one of the cameras, where the paths p correspond to a respective plurality of possible projector rays r.

[0215] In some applications, the step of using the processor further includes using the processor to execute a correspondence algorithm; (a) For each possible projector ray r, Identifying how many other cameras have detected each spot q corresponding to each camera ray that intersects the projector ray r and the camera ray of a given camera corresponding to the tracking spot s on the path p1 of each pixel of their own corresponding to the projector ray r; (b) Identify the given projector ray r1 for which the most other cameras detect the respective spot q; (c) identifying projector ray r1 as the particular projector ray r that generated said tracking spot s.

[0216] In some applications, the step of using the processor includes using the processor to: (a) executing a correspondence algorithm to calculate three-dimensional positions of each of a plurality of detection spots on the intraoral three-dimensional surface captured in a plurality of successive images; (b) identifying, in at least one of the plurality of consecutive images, the detection spot as originating from the particular projector ray r by identifying the detection spot as a tracking spot s that moves along a path of a pixel corresponding to the particular projector ray r.

[0217] In some applications, the step of using the processor further comprises using the processor to: (a) executing a correspondence algorithm to calculate three-dimensional positions of each of a plurality of detection spots on the intraoral three-dimensional surface captured in a plurality of successive images; (b) removing from consideration as points on the intraoral surface any spot that (i) is identified as originating from a particular projector ray r based on its three-dimensional position calculated by the correspondence algorithm, and (ii) is not identified as being a tracking spot s moving along the path of the pixel corresponding to the particular projector ray r.

[0218] In some applications, the step of using the processor further comprises using the processor to: (a) executing a correspondence algorithm to calculate three-dimensional positions of each of a plurality of detection spots on the intraoral three-dimensional surface captured in a plurality of successive images; (b) for a detection spot identified as originating from two different projector rays r based on the three-dimensional position calculated by the correspondence algorithm, identifying the detection spot as originating from the two different projector rays r by identifying the detection spot as a tracking spot s moving along one of the two different projector rays r.

[0219] In some applications, the step of using the processor further comprises using the processor to: (a) executing a correspondence algorithm to calculate three-dimensional positions of each of a plurality of detection spots on the intraoral three-dimensional surface captured in a plurality of successive images; (b) identifying a weak spot whose three-dimensional position has not been calculated by the correspondence algorithm as a projected spot from a specific projector ray r, thereby identifying the weak spot as a tracking spot s that moves along the path of the pixel corresponding to the specific projector ray r.

[0220] According to some applications of the present invention, there is further provided a method for generating a digital three-dimensional image, the method comprising: activating each structured light projector of the one or more structured light projectors to project a pattern of light (e.g., a distribution of discrete, disconnected light spots) onto the intraoral three-dimensional surface; activating each camera of the one or more cameras to capture an image, the image including at least a portion of the projected pattern, each camera of the one or more cameras including a camera sensor including an array of pixels; Use the processor (a) executing a correspondence algorithm to calculate the three-dimensional position of each portion of the detected pattern on the intraoral three-dimensional surface captured in a plurality of successive images; (b) identifying, in at least a subset of the plurality of consecutive images, a calculated three-dimensional location of a portion of the detected pattern as corresponding to a particular projector ray r; (c) calculating the length of the projector ray r in each image of the subset of images based on the three-dimensional position of the detected portion of the pattern that corresponds to the projector ray r in the subset of images.

[0221] In some embodiments, the light pattern may be a distribution of disconnected spots. In some embodiments, the processor may perform steps (a)-(c) based on stored calibration values ​​indicating (i) a camera ray corresponding to each pixel on a camera sensor of each of the one or more cameras, and (ii) a projector ray corresponding to each projected light spot from each of the one or more projectors. In some embodiments, each projector ray corresponds to a path of each pixel on at least one of the camera sensors.

[0222] In some applications, the step of using the processor further includes using the processor to calculate an estimated length of projector ray r in at least one of a plurality of consecutive images for which the three-dimensional position of the projection spot from projector ray r was not identified in step (b).

[0223] In some applications, the step of using the processor further includes using the processor to determine a one-dimensional search space in at least one of the multiple images to search for a projection spot from projector ray r based on an estimated length of projector ray r in at least one image of the multiple images, the one-dimensional search space being along a path of each pixel corresponding to projector ray r.

[0224] In some applications, the step of using the processor further includes using the processor to determine a one-dimensional search space in the pixel array of each of the multiple cameras to search for a projection spot from the projector ray r based on an estimated length of the projector ray r in at least one image of the multiple images, the one-dimensional search space being along a path of each pixel corresponding to the ray r.

[0225] In some applications, using a processor to determine a one-dimensional search space in each pixel array of the multiple cameras includes using a processor to determine a one-dimensional search space in which to search for a projected spot from the projector ray r in each pixel array of all cameras.

[0226] In some applications, the step of using the processor further includes calculating an estimated length of the projector ray r in at least one of the plurality of sequential images in which the plurality of candidate three-dimensional locations of the projection spot from the projector ray r were identified in step (b).

[0227] In some applications, using the processor further includes determining which of a plurality of candidate three-dimensional locations corresponds to an estimated length of the projector ray r in at least one of the plurality of images, thereby determining which is the correct three-dimensional location of the projection spot.

[0228] In some applications, the step of using the processor includes using the processor to: (a) determining a one-dimensional search space in at least one image of a plurality of images in which to search for a projected spot from a projector ray r; (b) determining which of the plurality of candidate three-dimensional positions of the projection spot is the correct three-dimensional position of the projection spot generated by projector ray r by determining which of the plurality of candidate three-dimensional positions corresponds to the spot generated by projector ray r detected in the one-dimensional search space.

[0229] In some applications, the step of using the processor further comprises using the processor to: (i) defining a curve based on the length of a projector ray r in each image of the subset of images; (ii) excluding a detection spot identified in step (b) as originating from projector ray r from consideration as a point on the intraoral surface if the three-dimensional position of the projected spot corresponds to a length of projector ray r that is at least a threshold away from the defined curve.

[0230] According to some applications of the present invention, there is further provided a method for generating a digital three-dimensional image, said method comprising: activating each structured light projector of the one or more structured light projectors to project a distribution of discrete, disconnected light spots onto the intraoral three-dimensional surface; activating each camera of one or more cameras to capture an image, said image including at least one of said spots, each camera of said one or more cameras including a camera sensor including an array of pixels; based on stored calibration values ​​indicating (a) camera rays corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and (b) projector rays corresponding to each one projected light spot from each projector of the one or more projectors, each projector ray corresponding to a respective path on at least one pixel of the camera sensor; Use the processor (a) executing a correspondence algorithm to calculate three-dimensional positions of each of a plurality of projection spots on the intraoral three-dimensional surface; (b) using data from at least two of the cameras to identify a candidate 3D location of a given spot corresponding to a particular projector ray r, and not substantially using data from another one of the cameras to identify the candidate 3D location; (c) using the candidate 3D locations seen by at least one of the two cameras to identify a search space on another one of the camera's pixel arrays in which to search for spots from the projector ray r; (d) if a spot from projector ray r is identified within the search space, using data from said other one of the cameras to refine candidate 3D locations of said spot.

[0231] According to some applications of the present invention, there is further provided a method for generating a digital three-dimensional image, said method comprising: activating each structured light projector of the one or more structured light projectors to project a distribution of discrete, disconnected light spots onto the intraoral three-dimensional surface; activating each camera of the one or more cameras to capture a plurality of images, each image including at least one of the spots, each camera of the one or more cameras including a camera sensor including a pixel array; based on stored calibration values ​​indicating (a) camera rays corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and (b) projector rays corresponding to each one projected light spot from each projector of the one or more projectors, each projector ray corresponding to a path of a respective pixel on at least one camera sensor of the camera sensors; Use the processor (a) executing a correspondence algorithm to calculate, for each of a plurality of images, respective three-dimensional positions of a plurality of detection spots on a three-dimensional intraoral surface; (b) using data corresponding to the three-dimensional positions of each of the at least three spots, each spot corresponding to a respective projector ray r, to estimate a three-dimensional surface on which all of the at least three spots are located; (c) for a projector ray r1 for which the three-dimensional position of the spot corresponding to that projector ray r1 has not been calculated in step (a), estimating the three-dimensional position in the intersection space between the projector ray r1 and the estimated surface; (d) using the estimated spatial 3D position to identify a search space in the pixel array of the at least one camera to search for a spot corresponding to the projector ray r1.

[0232] In some applications, using data corresponding to the three-dimensional position of each of the at least three spots includes using data corresponding to the three-dimensional position of each of at least three spots that were all captured in one of the plurality of images.

[0233] In some applications, the method further includes refining the estimate of the three-dimensional surface using data corresponding to a three-dimensional position of at least one additional spot, the at least one additional spot having a three-dimensional position calculated based on another image of the plurality of images such that all of the at least three spots and the at least one additional spot are on the three-dimensional surface.

[0234] In some applications, using data corresponding to the three-dimensional position of each of the at least three spots includes using data corresponding to at least three spots, each spot captured in a respective one of the multiple images.

[0235] According to some applications of the present invention, there is further provided a method for tracking motion of an intraoral scanner, the method comprising: (A) measuring motion of the intraoral scanner relative to the intraoral surface being scanned using at least one camera coupled to the intraoral scanner; (B) measuring motion of the intraoral scanner relative to a fixed coordinate system using at least one inertial measurement unit (IMU) coupled to the intraoral scanner; (C) Using a processor, (i) calculating the motion of the intraoral surface relative to the fixed coordinate system by (a) subtracting the motion of the intraoral scanner relative to the intraoral surface from (b) the motion of the intraoral scanner relative to the fixed coordinate system; (ii) constructing a predictive model of the motion of the intraoral surface relative to the fixed coordinate system based on accumulated data of the motion of the intraoral surface relative to the coordinate system; (iii) calculating an estimated position of the intraoral scanner relative to the intraoral surface by subtracting (a) a prediction of the motion of the intraoral surface relative to the coordinate system, derived based on a predictive model, from (b) the motion of the intraoral scanner relative to the coordinate system, as measured by an IMU.

[0236] For some applications, the method further includes determining whether measurement of motion of the intraoral scanner relative to the intraoral surface is impaired using at least one camera, and calculating an estimated position of the intraoral scanner relative to the intraoral surface in response to determining that measurement of the motion is impaired.

[0237] According to some applications of the present invention, there is provided a method, the method comprising: activating each structured light projector of the one or more structured light projectors to project a distribution of discrete, disconnected light spots onto the intraoral three-dimensional surface; activating each camera of one or more cameras to capture a plurality of images, each image including at least one of the spots, each camera of the one or more cameras including a camera sensor including a pixel array; based on stored calibration values ​​indicative of (a) camera rays corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and (b) projector rays corresponding to each one projected light spot from each projector of the one or more projectors, each projector ray corresponding to a respective path p on at least one pixel of the camera sensor; Use the processor (a) executing a correspondence algorithm to calculate three-dimensional positions of each of a plurality of projection spots on the intraoral three-dimensional surface; (b) collecting data at multiple time points, the data including calculated three-dimensional positions of each of the detected spots on the intraoral surface; (c) for each projector ray r, based on the collected data, defining a pixel update path p′ for each camera sensor such that all of the calculated three-dimensional positions corresponding to the spot generated by the projector ray r correspond to positions along the pixel's respective update path p′ for each camera sensor; (d) comparing each updated path p' of a pixel with the path p of the pixel corresponding to that projector ray r for each camera sensor from the stored calibration values; (e) for at least one camera sensor s, if an updated path p' of a pixel corresponding to projector ray r differs from the path p of the pixel corresponding to projector ray r from the stored calibration values, The difference between the updated path p' of the pixel corresponding to each projector ray r and the path p of each pixel corresponding to each projector ray r from the stored calibration value is reduced, and it is (i) stored calibration values ​​indicating camera rays corresponding to each pixel on a camera sensor s of each of the one or more cameras; and (ii) stored calibration values ​​indicating projector light rays r corresponding to each one projected light spot from each projector of the one or more projectors; and c) varying stored calibration data selected from the group consisting of:

[0238] In some applications, the selected stored calibration data includes stored calibration values ​​indicative of camera rays corresponding to each pixel on a camera sensor of each camera of one or more cameras; The step of varying the stored calibration data includes varying one or more parameters of a parameterized camera calibration function that defines a camera ray corresponding to each pixel on at least one camera sensor s to: (i) calculated three-dimensional positions of each of a plurality of detection spots on the intraoral surface; (ii) stored calibration values ​​indicating respective camera rays corresponding to each pixel on the camera sensor where a respective one of the plurality of detection spots should have been detected; and reducing the difference between

[0239] In some applications, the selected stored calibration data includes stored calibration values ​​indicative of projector light rays corresponding to each one of the projected light spots from each of one or more projectors, and varying the stored calibration data includes: (i) an indexed list that assigns each projector ray r to a pixel path p, or (ii) one or more parameters of a parameterized projector calibration model that defines each projector ray r; The step of varying

[0240] In some applications, altering the stored calibration data includes altering the indexed list by reassigning each projector ray r based on the update path p′ of each pixel corresponding to each projector ray r.

[0241] In some applications, varying the stored calibration data comprises: (i) stored calibration values ​​indicating camera rays corresponding to each pixel on a camera sensor s of each of the one or more cameras; (ii) stored calibration values ​​indicating projector light rays r corresponding to each one projected light spot from each projector of the one or more projectors; The step of varying

[0242] For some applications, varying the stored calibration values ​​includes iteratively varying the stored calibration values.

[0243] In some applications, the method further comprises: activating each camera of the one or more cameras to capture a plurality of images of a calibration object having predetermined parameters; Use the processor executing a triangulation algorithm to calculate respective parameters of the calibration object based on the captured images; Run the optimization algorithm (a) reducing the difference between (i) an updated path p' of a pixel corresponding to a projector ray r and (ii) a path p of a pixel corresponding to a projector ray r from the stored calibration value, (b) using the parameters of the calibration object calculated based on the captured images.

[0244] In some applications, the calibration object is a three-dimensional calibration object of known shape, and driving each of the one or more cameras to capture an image of the three-dimensional calibration object includes driving each of the one or more cameras to capture an image of the three-dimensional calibration object, and the predetermined parameter of the calibration object is a dimension of the three-dimensional calibration object.

[0245] In some applications, the calibration object is a two-dimensional calibration object having visually distinguishable features, and driving each camera of the one or more cameras to capture a plurality of images of the calibration object includes driving each camera of the one or more cameras to capture an image of the two-dimensional calibration object, and the predetermined parameter of the two-dimensional calibration object is a respective distance between each visually distinguishable feature.

[0246] According to some applications of the present invention, there is further provided a method for calculating a three-dimensional structure of an intraoral three-dimensional surface, said method comprising: scanning an intraoral surface; activating one or more uniform light projectors to project broadband spectrum light onto the intraoral three-dimensional surface; activating a camera to capture a plurality of two-dimensional color images of the intraoral three-dimensional surface; Use the processor calculating three-dimensional positions of a plurality of points on the intraoral three-dimensional surface based on the intraoral surface scan; and calculating a three-dimensional structure of the intraoral three-dimensional surface constrained by the three-dimensional positions of the plurality of points based on the plurality of two-dimensional color images of the intraoral three-dimensional surface.

[0247] In some embodiments, the intraoral surface is scanned by activating one or more structured light projectors to project structured light patterns onto the intraoral three-dimensional surface; One or more cameras are activated to capture a plurality of structured light images, each image including at least a portion of the structured light pattern.

[0248] In some applications, driving the one or more structured light projectors includes driving the one or more structured light projectors to each project a distribution of discrete, disconnected light spots.

[0249] In some applications, the step of calculating the three-dimensional structure comprises: inputting (a) a plurality of two-dimensional color images of the intraoral three-dimensional surface, and (b) the calculated three-dimensional positions of a plurality of points on the intraoral three-dimensional surface into the neural network; and determining, with the neural network, a predicted depth map for each of the intraoral three-dimensional surfaces captured in each of the two-dimensional color images.

[0250] For some applications, the method further includes using the processor to stitch together the respective depth maps to obtain a three-dimensional structure of the intra-oral three-dimensional surface.

[0251] In some applications, the method further includes coordinating the capturing of the structured light image and the capturing of the two-dimensional color image to generate an alternating sequence of one or more structured light image frames interspersed with one or more broadband spectrum light image frames.

[0252] In some applications, driving one or more cameras to capture a plurality of structured light images includes driving each camera of two or more cameras to capture a plurality of structured light images; Driving the camera to capture the plurality of two-dimensional color images includes driving each camera of the two or more cameras to capture the plurality of two-dimensional color images.

[0253] In some applications, the determining step using a neural network includes determining, for a given image frame, a respective predicted depth map of a portion of the intraoral three-dimensional surface captured in the two-dimensional color image by each of the two or more cameras; The method further includes using the processor to stitch together the respective depth maps to obtain a predicted depth map of the intraoral three-dimensional surface captured in the given image frame.

[0254] In some applications, the method further includes training the neural network, the training comprising: (a) activating one or more structured light projectors to project a training stage structured light pattern onto a training stage three-dimensional surface; (b) driving one or more training stage cameras to capture a plurality of structured light images, each image including at least a portion of the training stage structured light pattern; (c) driving one or more training stage uniform light projectors to project broadband spectrum light onto the training stage three-dimensional surface; (d) driving one or more training stage cameras to capture a plurality of two-dimensional color images of the training stage three-dimensional surface using illumination from said training stage uniform light projector; (e) coordinating the capturing of the structured light image and the capturing of the two-dimensional color image to produce an alternating sequence of image frames of one or more structured light images interspersed with image frames of one or more two-dimensional color images; (f) inputting a plurality of two-dimensional color images into the neural network; (g) inputting a respective plurality of 3D reconstructions of the training stage 3D surface into a neural network based on the structured light images of the training stage 3D surface, the 3D reconstructions including calculated 3D positions of a plurality of points on the training stage 3D surface; (h) interpolating positions of the one or more training stage cameras relative to the training stage three-dimensional surface for each two-dimensional color image frame based on the calculated three-dimensional positions of a plurality of points on the training stage surface calculated based on respective structured light image frames before and after each two-dimensional color image frame; (i) projecting the 3D reconstruction into the field of view of each of the one or more training stage cameras and estimating, based on the projections, a predicted depth map of the training stage 3D surface seen in each 2D color image, constrained by the calculated 3D positions of the plurality of points; (j) comparing each predicted depth map of the training stage three-dimensional surface with a corresponding true depth map of the training stage three-dimensional surface; (k) optimizing the neural network to better estimate subsequent predicted depth maps based on the difference between each predicted depth map and the corresponding true depth map; Includes:

[0255] In some applications, driving the one or more structured light projectors to project the training stage structured light pattern includes driving the one or more structured light projectors to each project a distribution of discrete, disconnected light spots onto the training stage three-dimensional surface.

[0256] In some applications, driving one or more training stage cameras includes driving at least two training stage cameras.

[0257] According to some applications of the present invention, there is further provided an intraoral scanning device, the device comprising: an elongated handheld wand including a probe at a distal end thereof; one or more illumination sources coupled to the probe; one or more near-infrared (NIR) light sources coupled to the probe; one or more cameras coupled to the probe and configured to (a) capture images using light from the one or more illumination sources, and (b) capture images using NIR light from the NIR light source; a processor configured to execute a navigation algorithm to determine a position of the handheld wand as it moves in space, wherein inputs to the navigation algorithm are (a) images captured using light from the one or more illumination sources, and (b) images captured using the NIR light.

[0258] In some applications, the one or more illumination sources are one or more structured light sources.

[0259] In some applications, the one or more illumination sources are one or more uniform light sources.

[0260] According to some applications of the present invention, there is further provided a method for tracking motion of an intraoral scanner, said method comprising: illuminating the intraoral three-dimensional surface using one or more illumination sources coupled to the intraoral scanner; using one or more near-infrared (NIR) light sources coupled to the intraoral scanner to drive each NIR light source of the one or more NIR light sources to emit NIR light onto the intraoral three-dimensional surface; using one or more cameras coupled to the intraoral scanner: (a) capturing a plurality of images using light from the one or more illumination sources; and (b) capturing a plurality of images using the NIR light; Use the processor running a navigation algorithm to track motion of the intraoral scanner relative to the intraoral three-dimensional surface using (a) images captured using light from the one or more illumination sources and (b) images captured using the NIR light; Includes:

[0261] For some applications, using one or more illumination sources includes illuminating the intra-oral three-dimensional surface using one or more structured light sources.

[0262] In some applications, using one or more illumination sources includes using one or more uniform light sources.

[0263] In accordance with some applications of the present invention, there is further provided an intraoral scanning device for use with the sleeve, the device comprising: an elongated handheld wand including a probe at a distal end thereof configured to be removably disposed within said sleeve; at least one structured light projector coupled to the probe, the structured light projector (a) having an illumination field of at least 30 degrees, (b) including a laser configured to emit polarized laser light, and (c) including pattern generating optics configured to generate a pattern of light when the laser diode is activated to transmit light through the pattern generating optics; at least one camera coupled to the probe, the camera comprising a camera sensor; the probe is configured so that light enters and exits the probe through a sleeve; the laser is positioned at a distance relative to the camera such that when the probe is placed within the sleeve, a portion of the light pattern is reflected from the sleeve to the camera sensor; The laser is positioned at a rotation angle relative to its own optical axis such that the polarization of the light pattern causes the extent of reflection by the sleeve of portions of the light pattern to be less than 70% of the maximum reflection for all possible rotation angles of the laser relative to its optical axis.

[0264] In some applications, the distance between the structured light projector and the camera is between 1 and 6 times the distance between the structured light projector and the sleeve when the handheld wand is placed within the sleeve.

[0265] For some applications, each camera of the at least one camera has a field of view of at least 30 degrees.

[0266] In some applications, the laser is positioned at a rotation angle relative to its own optical axis such that the polarization of the light pattern causes the extent of reflection of portions of the light pattern by the sleeve to be less than 60% of the maximum reflection for all possible rotation angles of the laser relative to its optical axis.

[0267] In some applications, the laser is positioned at a rotational angle relative to its own optical axis such that the polarization of the light pattern causes the extent of reflection of portions of the light pattern by the sleeve to be between 15% and 60% of maximum reflection for all possible rotational angles of the laser relative to its optical axis.

[0268] According to some applications of the present invention, there is further provided a method of generating a three-dimensional image using an intraoral scanner, the method comprising: (A) Using at least two cameras rigidly connected to an intraoral scanner such that the fields of view of the cameras are non-overlapping, capturing a plurality of images of an intraoral three-dimensional surface; (B) Using a processor, For the non-overlapping portions of each field of view, a simultaneous localization and mapping (SLAM) algorithm is run using the captured images from each camera, and the localization of each camera is solved based on the motion of each camera being the same as the motion of all other cameras.

[0269] In some applications, the fields of view of a first one of the cameras and a second one of the cameras also have an overlap; the capturing step includes capturing a plurality of images of the intraoral three-dimensional surface such that features of the intraoral three-dimensional surface in overlapping portions of the respective fields of view appear in the images captured by the first and second cameras; Using the processor includes running a SLAM algorithm using features of the intraoral three-dimensional surface appearing in images from the at least two cameras.

[0270] According to some applications of the present invention, there is further provided a method for generating a three-dimensional image using an intraoral scanner, the method comprising: activating one or more structured light projectors to project structured light patterns onto the intraoral three-dimensional surface; activating one or more cameras to capture a plurality of structured light images, each image including at least a portion of the structured light pattern; activating one or more uniform light projectors to project broadband spectrum light onto the intraoral three-dimensional surface; activating at least one camera to capture a two-dimensional color image of the intraoral three-dimensional surface using illumination from the uniform light projector; coordinating the capture of the structured light and the capture of the broadband spectrum light to produce an alternating sequence of one or more structured light image frames interspersed with one or more broadband spectrum light image frames; Use the processor calculating a three-dimensional position of each of a plurality of points on the intraoral three-dimensional surface captured in the plurality of structured light image frames; interpolating the motion of the at least one camera between the first broadband spectrum light image frame and the second broadband spectrum light image frame based on the calculated three-dimensional positions of a plurality of points in each of the structured light image frames before and after the broadband spectrum light image frame; (a) using intraoral three-dimensional surface features captured by the at least one camera in the first and second broadband spectrum light image frames, and (b) performing a simultaneous localization and mapping (SLAM) algorithm constrained by the interpolated motion of the camera between the first and second broadband spectrum light image frames.

[0271] In some applications, driving the one or more structured light projectors to project the structured light pattern comprises driving the one or more structured light projectors to respectively project a distribution of discrete, disconnected light spots onto the three-dimensional intra-oral surface.

[0272] According to some applications of the present invention, there is further provided a method of generating a three-dimensional image using an intraoral scanner, the method comprising: activating one or more structured light projectors to project structured light patterns onto the intraoral three-dimensional surface; activating one or more cameras to capture a plurality of structured light images, each image including at least a portion of the structured light pattern; activating one or more uniform light projectors to project broadband spectrum light onto the intraoral three-dimensional surface; activating the one or more cameras to capture a two-dimensional color image of the intraoral three-dimensional surface using illumination from the uniform light projector; coordinating the capture of the structured light and the capture of the broadband spectrum light to produce an alternating sequence of one or more structured light image frames interspersed with one or more broadband spectrum light image frames; Use the processor (a) calculating three-dimensional positions of features on the intraoral three-dimensional surface based on the structured light image frames, the features also being captured in the first broadband spectrum light image frame and the second broadband spectrum light image frame; (b) calculating motion of at least one camera between the first image frame of the broadband spectrum light and the second image frame of the broadband spectrum light based on the calculated three-dimensional positions of the features; (c) performing a simultaneous localization and mapping (SLAM) algorithm using (i) features of the intraoral three-dimensional surface for which three-dimensional positions were not calculated based on the structured light image frames captured by the at least one camera in the first and second broadband spectrum light image frames, and (ii) the calculated motion of the camera between the first and second broadband spectrum light image frames.

[0273] In some applications, driving the one or more structured light projectors to project the structured light pattern comprises driving the one or more structured light projectors to respectively project a distribution of discrete, disconnected light spots onto the three-dimensional intra-oral surface.

[0274] According to some applications of the present invention, there is further provided a method for calculating a three-dimensional structure of an intraoral three-dimensional surface within a subject's oral cavity, said method comprising: activating one or more structured light projectors to project a structured light pattern of spots onto the intraoral three-dimensional surface; activating one or more cameras to capture a plurality of structured light images, each image including at least one of said spots; activating one or more uniform light projectors to project broadband spectrum light onto the intraoral three-dimensional surface; activating at least one camera to capture a two-dimensional color image of the intraoral three-dimensional surface using illumination from the uniform light projector; coordinating the capture of the structured light and the capture of the broadband spectrum light to produce an alternating sequence of one or more structured light image frames interspersed with one or more broadband spectrum light image frames; Use the processor determining whether the spot is projected onto a moving tissue or a stable tissue in the oral cavity based on the two-dimensional color image for each of the plurality of spots; and assigning a reliability grade to each of the plurality of detected spots based on the determination, with a high reliability being assigned to the fixed tissue and a low reliability being assigned to the moving tissue. and performing a three-dimensional reconstruction algorithm using the detection spots based on a confidence grade for each of the plurality of detection spots.

[0275] In some applications, driving the one or more structured light projectors to project the structured light pattern comprises driving the one or more structured light projectors to respectively project a distribution of discrete, disconnected light spots onto the three-dimensional intra-oral surface.

[0276] In some applications, the step of performing the three-dimensional reconstruction algorithm includes a step of performing the three-dimensional reconstruction algorithm using only a subset of the detected spots, the subset consisting of spots assigned a confidence grade above a fixed tissue threshold.

[0277] In some applications, the step of executing the 3D reconstruction algorithm includes the steps of: (a) for each spot, assigning a weight to the spot based on the respective confidence level assigned to the spot; and (b) using the respective weight of each spot in the 3D reconstruction algorithm.

[0278] The present invention will be more fully understood from the following detailed description of its application taken together with the drawings. [Brief explanation of the drawings]

[0279] [Figure 1] 1 is a schematic diagram of a handheld wand having multiple structured light projectors and a camera disposed within a probe at the distal end of the handheld wand, according to some applications of the present invention; [Figure 2A] ~ [Figure 2B] 1A-1C are schematic diagrams illustrating positioning configurations for a camera and a structured light projector, respectively, according to some applications of the present invention; [Figure 2C] 10 is a chart depicting several different configurations for the position of a structured light projector and camera within a probe, according to some applications of the present invention. [Figure 2D] ~ [Figure 2E] 1A-1C are isometric views showing particular configurations of structured light projector and camera positions within a probe, according to some applications of the present invention, from two different perspectives. [Figure 3] 1 is a schematic diagram of a structured light projector according to some applications of the present invention; [Figure 4]1 is a schematic diagram of a structured light projector that projects a distribution of discrete, disconnected light spots onto multiple object focal planes, in accordance with some applications of the present invention; [Figure 5A] ~ [Figure 5B] 1 is a schematic diagram of a structured light projector including a beam-shaping optical element and an additional optical element disposed between the beam-shaping optical element and the pattern-generating optical element, according to some applications of the present invention; [Figure 6A] ~ [Figure 6B] 1A and 1B are schematic diagrams of a structured light projector projecting discrete, disconnected spots and a camera sensor detecting the spots, according to some applications of the present invention; [Figure 7] 1 is a flowchart outlining a method for generating a digital three-dimensional image, according to some applications of the present invention; [Figure 8] 8 is a flowchart outlining a method for implementing certain steps in the method of FIG. 7, according to some applications of the present invention. [Figure 9] ~ [Figure 12] 9 is a schematic diagram depicting a simplified example of the steps of FIG. 8 according to some applications of the present invention. [Figure 13] 10 is a flowchart outlining further steps in a method for generating a digital three-dimensional image, according to some applications of the present invention. [Figure 14] ~ [Figure 17] 14 is a schematic diagram depicting a simplified example of the steps of FIG. 13 according to some applications of the present invention. [Figure 18] 1 is a schematic diagram of a probe including a diffuse reflector, according to some applications of the present invention. [Figure 19A] ~ [Figure 19B] 1A-1C are schematic diagrams of a structured light projector and a cross section of a light beam transmitted by a laser diode, together with pattern generating optics positioned in the beam's optical path, according to some applications of the present invention; [Figure 20A] ~ [Figure 20E] 1 is a schematic diagram illustrating a microlens array used as a pattern-generating optical element in a structured light projector, according to some applications of the present invention; [Figure 21A] ~ [Figure 21C] 1A-1C are schematic diagrams of compound two-dimensional diffractive periodic structures used as pattern-generating optical elements in structured light projectors, according to some applications of the present invention; [Figure 22A] ~ [Figure 22B] 1A and 1B are schematic diagrams illustrating a single optical element having an aspherical first side and a planar second side opposite the first side, and a structured light projector including the optical element, according to some applications of the present invention. [Figure 23A] ~ [Figure 23B] 1A and 1B are schematic diagrams of an axicon lens and a structured light projector including an axicon lens, according to some applications of the present invention; [Figure 24A] ~ [Figure 24B] 1A and 1B are schematic diagrams illustrating an optical element having an aspherical surface on a first side and a flat surface on a second side opposite the first side, and a structured light projector including the optical element, according to some applications of the present invention. [Figure 25] 1 is a schematic diagram of a single optical element in a structured light projector, according to some applications of the present invention; [Figure 26A] ~ [Figure 26B] 1 is a schematic diagram of a structured light projector with two or more laser diodes, according to some applications of the present invention; [Figure 27A] ~ [Figure 27B] 1A-1C are schematic diagrams illustrating different ways of combining laser diodes of different wavelengths according to some applications of the present invention. [Figure 28] 1 is a flowchart outlining steps of a "spot tracking" method, in accordance with some applications of the present invention; [Figure 29]FIG. 10 is a schematic diagram depicting a simplified example of some detection spots and how a processor may determine which set of detection spots may be considered tracked, according to some applications of the present invention. [Figure 30] 1 is a flowchart outlining a method for determining tracking spots, according to some applications of the present invention. [Figure 31] ~ [Figure 32] 10 is a flowchart outlining a method for detecting tracking spots in subsequent images, according to some applications of the present invention. [Figure 33] 10 is a schematic diagram depicting an example of how spot tracking can help identify a detected spot as being projected from a particular projector beam, according to some applications of the present invention. FIG. [Figure 34A] ~ [Figure 34B] 1 is a simplified schematic diagram of a camera sensor showing two detection spots, according to some applications of the present invention. [Figure 35] ~ [Figure 36] 1 is a flowchart outlining various ways in which spot tracking may be used, according to some applications of the present invention. [Figure 37A] ~ [Figure 37B] 1A-1C are schematic diagrams illustrating points used in 3D reconstruction before and after a processor implements spot tracking, according to some applications of the present invention. [Figure 38] 1 is a flowchart outlining the steps of a method for generating a digital three-dimensional image, hereinafter referred to as "ray tracing," according to some applications of the present invention. [Figure 39] 1 is a graph showing the tracking of projector beam length over time, and showing a particular simplified view of the camera sensor corresponding to a particular image frame, according to some applications of the present invention. [Figure 40A] ~ [Figure 40B] 10 is a graph showing an experimental set of data before and after ray tracing, according to some applications of the present invention. [Figure 41]1 is a schematic diagram of multiple camera sensors and a spot projector according to some applications of the present invention; [Figure 42A] ~ [Figure 42B] 1 is a flowchart outlining a method for generating three-dimensional images, according to some applications of the present invention; [Figure 43A] ~ [Figure 43B] 10A-10C are flowcharts outlining respective methods for tracking the motion of an intraoral scanner, according to some applications of the present invention; [Figure 44A] ~ [Figure 45] FIG. 1 is a schematic diagram showing a simplified view of a camera image with multiple detection spots from a single projector beam, all captured at different times, superimposed on the same image, according to some applications of the present invention. [Figure 46A] ~ [Figure 46D] FIG. 10 illustrates a simplified scenario in which a processor identifies that it is the projector (and not any of the cameras) that should be recalibrated, in accordance with some applications of the present invention. [Figure 47A] ~ [Figure 47B] FIG. 10 illustrates a simplified scenario in which a processor identifies that it is the camera (and not any projector) that should be recalibrated, according to some applications of the present invention. [Figure 48A] ~ [Figure 48B] 10A-10C illustrate simplified scenarios in which a processor cannot reasonably assume that a shift has occurred in only the camera or only the projector, in accordance with some applications of the present invention. [Figure 49A] ~ [Figure 49B] 1A-1C are schematic diagrams illustrating three-dimensional and two-dimensional calibration objects, respectively, according to some applications of the present invention. [Figure 50] 1 is a flowchart illustrating a method for tracking the motion of a handheld wand, in accordance with some applications of the present invention. [Figure 51A] ~ [Figure 51F]1 is a flowchart depicting a method for calculating the three-dimensional structure of an intraoral three-dimensional surface, according to some applications of the present invention. [Figure 51G] ~ [Figure 51I] 1 is a schematic diagram that graphically depicts different combinations of inputs to a neural network, according to some applications of the present invention; FIG. [Figure 52A] 1 is a flowchart illustrating a method for training a neural network, according to some applications of the present invention. [Figure 52B] FIG. 1 is a block diagram of neural network training according to some applications of the present invention. [Figure 52C] 1 is a flowchart depicting how a neural network outputs a depth map and a corresponding confidence map, according to some applications of the present invention. [Figure 52D] ~ [Figure 52F] 1 is a schematic diagram depicting the training of a neural network to output a depth map as well as a corresponding confidence map, according to some applications of the present invention; [Figure 52G] 1 is a flowchart depicting how a confidence map may be used, according to some applications of the present invention. [Figure 53A] 10A-10C are schematic diagrams of a disposable sleeve that is placed over the distal end of an intraoral scanner before the probe is placed in a patient's mouth to prevent cross-contamination between patients, according to some applications of the present invention. [Figure 53B] 1 is a graph illustrating the reflectivity of polarized laser light according to Fresnel's equations and its use in accordance with some applications of the present invention. [Figure 54A] 1 is a flowchart depicting a method for generating a three-dimensional image using a handheld wand, according to some applications of the present invention. [Figure 54B] 1A and 1B are schematic diagrams illustrating the positional relationship between a projector and a camera according to some applications of the present invention. [Figure 55]1 is a flowchart illustrating a method for generating a three-dimensional image using a handheld wand, according to some applications of the present invention. [Figure 56A] 1 is a flowchart illustrating a method for generating a three-dimensional image using a handheld wand, according to some applications of the present invention. [Figure 56B] 1A-1C are schematic diagrams illustrating two image frames of unstructured light and two features of an intraoral three-dimensional surface, according to some applications of the present invention. [Figure 57] 1 is a flowchart illustrating a method for calculating the three-dimensional structure of a three-dimensional intraoral surface within a subject's oral cavity, according to some applications of the present invention. [Figure 58] ~ [Figure 59B] 1 is a schematic diagram of a neural network according to some applications of the present invention; [Figure 60] FIG. 1 illustrates one embodiment of a system for performing an intraoral scan and generating a virtual 3D model of the dental arch. [Figure 61] 1 illustrates a block diagram of an exemplary computing device according to an embodiment of the present disclosure. [Figure 62] 1 is a flowchart illustrating a method for overcoming manufacturing variations between intraoral scanners, according to some applications of the present invention. [Figure 63] 10 is a schematic diagram illustrating another method for overcoming manufacturing variations between intraoral scanners, according to some applications of the present invention. [Figure 64] FIG. 10 is a flowchart illustrating a method for testing whether the cropping and morphing of each runtime image accurately accounts for possible manufacturing variations for a given intraoral scanner, and if not, refining the training of a neural network based on local refinement stage scans for that given intraoral scanner, according to some applications of the present invention. [Figure 65] 1 is a flowchart illustrating a method for overcoming manufacturing variations between intraoral scanners, according to some applications of the present invention. [Figure 66]1 is a flowchart illustrating a method for training a neural network, according to some applications of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0280] 1, which is a schematic diagram of an elongated handheld wand 20 for intraoral scanning, in accordance with some applications of the present invention. Multiple structured light projectors 22 and multiple cameras 24 are coupled to a rigid structure 26 located at the distal end 30 of the handheld wand within a probe 28. In some applications, during an intraoral scan, the probe 28 enters the subject's oral cavity.

[0281] In some applications, the structured light projectors 22 are positioned within the probe 28 so that each structured light projector 22 faces an object 32 external to the handheld wand 20 located within its field of illumination, as opposed to placing the structured light projectors at the proximal end of the handheld wand to illuminate the object by reflecting light off a mirror and then back onto the object. Similarly, in some applications, the cameras 24 are positioned within the probe 28 so that each camera 24 faces an object 32 external to the handheld wand 20 located within its field of view, as opposed to placing the cameras at the proximal end of the handheld wand to view the object by reflecting light from a mirror back onto the camera. This positioning of the projectors and cameras within the probe 28 allows the scanner to have a large overall field of view while maintaining a low-profile probe.

[0282] In some applications, the height H1 of the probe 28 is less than 15 mm, and the height H1 of the probe 28 is measured from the lower surface 176 (sensing surface) where reflected light from the object 32 being scanned enters the probe 28 to the upper surface 178 opposite the lower surface 176. In some applications, the height H1 is between 10 and 15 mm.

[0283] In some applications, the cameras 24 each have a large field of view β (beta) of at least 45 degrees, e.g., at least 70 degrees, e.g., at least 80 degrees, e.g., 85 degrees. In some applications, the field of view may be less than 120 degrees, e.g., less than 100 degrees, e.g., less than 90 degrees. Experiments conducted by the inventors have shown that a field of view β (beta) of each camera between 80 and 90 degrees is particularly useful because it provides a good balance between pixel size, field of view and camera overlap, optical quality, and cost. The cameras 24 may include a camera sensor 58 and objective optics 60 including one or more lenses. To enable close-focus imaging, the cameras 24 may focus at an object focal plane 50 located between 1 mm and 30 mm, e.g., between 4 mm and 24 mm, e.g., between 5 mm and 11 mm, e.g., between 9 mm and 10 mm, from the lens farthest from the camera sensor. Experiments conducted by the inventors have shown that it is particularly useful to position the object focal plane 50 between 5 mm and 11 mm from the lens farthest from the camera sensor, because it is easy to scan teeth at this distance and most tooth surfaces are in high focus. In some applications, the camera 24 can capture images at a frame rate of at least 30 frames per second, e.g., at least 75 frames per second, e.g., at least 100 frames per second. In some applications, the frame rate can be less than 200 frames per second.

[0284] As explained above, the large field of view achieved by combining the respective fields of view of all cameras can improve accuracy by reducing the amount of image stitching error, especially in edentulous regions where the gingival surface may be smooth and there may be few distinct high-resolution 3D features. A large field of view allows large smooth features, such as the overall curve of the teeth, to appear in each image frame, improving the accuracy of stitching together the respective surfaces obtained from such multiple image frames.

[0285] Similarly, the structured light projectors 22 may each have a large illumination field α (alpha) of at least 45 degrees, e.g., at least 70 degrees. In some applications, the illumination field α (alpha) may be less than 120 degrees, e.g., less than 100 degrees. Further features of the structured light projectors 22 are described below.

[0286] In some applications, to improve image capture, each camera 24 has a plurality of discrete preset focus positions, at each focus position the camera focuses on a respective object focal plane 50. Each of the cameras 24 may include an autofocus actuator that selects a focus position from among the discrete preset focus positions to improve a given image capture. Additionally or alternatively, each camera 24 includes an optical aperture phase mask that extends the depth of focus of the camera such that the image formed by each camera remains in focus over all object distances between 1 mm and 30 mm, e.g., between 4 mm and 24 mm, e.g., between 5 mm and 11 mm, e.g., between 9 mm and 10 mm, from the lens furthest from the camera sensor.

[0287] In some applications, the structured light projectors 22 and cameras 24 are tightly and / or alternately coupled to the rigid structure 26 so that (a) a large portion of each camera's field of view overlaps with the field of view of an adjacent camera, and (b) a large portion of each camera's field of view overlaps with the illumination field of an adjacent projector. Optionally, at least 20%, e.g., at least 50%, e.g., at least 75%, of the projected light pattern is present in the field of view of at least one of the cameras at an object focal plane 50 located at least 4 mm away from the lens farthest from the camera sensor. Due to different possible configurations of the projectors and cameras, portions of the projected pattern may never be seen within the field of view of any camera, and portions of the projected pattern may be blocked from view by the object 32 as the scanner moves around during a scan.

[0288] Rigid structure 26 may be a non-flexible structure to which structured light projector 22 and camera 24 are coupled to provide structural stability to the optics within probe 28. Coupling all projectors and all cameras to a common rigid structure helps maintain the geometric integrity of the optics of each structured light projector 22 and each camera 24 under changing ambient conditions, such as mechanical stresses that may be induced by the subject's mouth. Furthermore, rigid structure 26 helps maintain stable structural integrity and positioning of structured light projector 22 and camera 24 relative to one another. As described further below, controlling the temperature of rigid structure 26 can help enable the geometric integrity of the optics to be maintained over a large range of ambient temperatures as probe 28 moves in and out of the subject's oral cavity or as the subject breathes during a scan.

[0289] Reference is now made to Figures 2A-B, which are schematic diagrams illustrating respective positioning configurations for cameras 24 and structured light projectors 22 according to some applications of the present invention. In some applications, the cameras 24 and structured light projectors 22 are positioned so that they do not all face the same direction as each other to improve the overall field of view and illumination of the intraoral scanner. In some applications, such as that shown in Figure 2A, the multiple cameras 24 are coupled to the rigid structure 26 such that the angle θ (theta) between the two respective optical axes 46 of at least two cameras 24 is 90 degrees or less, e.g., 35 degrees or less. Similarly, in some applications, such as that shown in Figure 2B, the multiple structured light projectors 22 are coupled to the rigid structure 26 such that the angle φ (phi) between the two respective optical axes 48 of at least two structured light projectors 22 is 90 degrees or less, e.g., 35 degrees or less.

[0290] Reference is now made to FIG. 2C, a chart depicting several different configurations for the positions of the structured light projector 22 and the camera 24 within the probe 28, according to some applications of the present invention. The structured light projector 22 is represented by a circle in FIG. 2C, and the camera 24 is represented by a rectangle in FIG. 2C. Note that rectangles are used to represent the cameras because typically, the field of view β (beta) of each camera sensor 58 and each camera 24 has an aspect ratio of 1:2. Column (a) of FIG. 2C shows bird's-eye views of various configurations of the structured light projector 22 and the camera 24. The labeled x-axis in the first row of column (a) corresponds to the central longitudinal axis of the probe 28. Column (b) shows side views of the cameras 24 from various configurations, as seen from a line of sight coaxial with the central longitudinal axis of the probe 28. Similar to FIG. 2A, column (b) of FIG. 2C shows the cameras 24 positioned with their optical axes 46 at angles of 90 degrees or less, e.g., 35 degrees or less, relative to each other. Column (c) shows side views of the camera 24 in various configurations, viewed from a line of sight perpendicular to the central longitudinal axis of the probe 28.

[0291] Typically, the distal-most (toward the positive x-direction in FIG. 2C ) and proximal-most (toward the negative x-direction in FIG. 2C ) cameras 24 are positioned so that their optical axes 46 are rotated slightly inward relative to the next-closest camera 24, e.g., by an angle of 90 degrees or less, e.g., 35 degrees or less. The more centrally located camera(s) 24, i.e., the cameras 24 that are neither the distal-most nor proximal-most cameras 24, are positioned so that they point directly outward from the probe, with their optical axes 46 substantially perpendicular to the central longitudinal axis of the probe 28. Note that in row (xi), the projector 22 is positioned at the distal-most position on the probe 28, so that its optical axis 48 points inward, resulting in more spots 33 projected from that particular projector 22 being seen by more cameras 24.

[0292] Typically, the number of structured light projectors 22 in probe 28 may range from, for example, two, as shown in line (iv) of FIG. 2C , to, for example, six, as shown in line (xii). Typically, the number of cameras 24 in probe 28 may range from, for example, four, as shown in lines (iv) and (v), to, for example, seven, as shown in line (ix). It should be noted that the various configurations shown in FIG. 2C are exemplary and not limiting, and the scope of the present invention includes additional configurations not shown. For example, the scope of the present invention includes more than five projectors 22 arranged on probe 28 and more than seven cameras arranged on probe 28.

[0293] Reference is now made to Figures 2D-E, which are isometric views of particular configurations for the positioning of structured light projectors 22 and cameras 24 within probe 28, shown from two different perspectives, according to some applications of the present invention. Figure 2D shows the same bird's-eye view as that of column (a) of Figure 2C. In some applications, six cameras 24 are evenly spaced within probe 28, with three cameras on each side of probe 28, and five structured light projectors 22 positioned within probe 28 centrally along the central longitudinal axis of probe 28 (shown by dashed line 29).

[0294] In some applications, the camera 24 and structured light projector 22 are all coupled to a flexible printed circuit board (PCB) to accommodate angular positioning of the camera 24 and structured light projector 22 within the probe 28. This angular positioning of the camera 24 and structured light projector 22 is shown in FIG. 2E . The distal-most (i.e., toward the positive x-direction) camera 24 and structured light projector 22 are positioned so that their optical axes are tilted backward toward the handheld wand 20, e.g., at an angle of 45 degrees or less, e.g., 35 degrees or less. This allows, for example, the distal-most camera to capture the posterior wall of a molar tooth in the oral cavity. The proximal-most (i.e., toward the negative x-direction) camera 24 and structured light projector 22 are positioned so that their optical axes are tilted forward toward the distal end of the probe 28, e.g., at an angle of 45 degrees or less, e.g., 35 degrees or less, to achieve improved overlap of the fields of view of the cameras 24. Furthermore, all of the structured light projectors 22 are positioned so that their optical axes are angled towards the center of the probe 28, which improves the overlap of the illumination fields of each of the structured light projectors 22. The inventors have realised that by positioning the structured light projectors 22 generally all in a row, they can all be more easily connected to the same flexible PCB.

[0295] Additionally, multiple uniform light projectors 118, multiple near-infrared (NIR) light projectors 292, and a diffractive optical element (DOE) 39 positioned above each structured light projector 22 are shown in Figures 2D-E and further described below.

[0296] Reference is now made to FIG. 3, which is a schematic diagram of a structured light projector 22 according to some applications of the present invention. In some applications, the structured light projector 22 includes a laser diode 36, beam-shaping optics 40, and pattern-generating optics 38 that generate a distribution of discrete, non-connected light spots 34 (described further below with reference to FIG. 4). In some applications, the structured light projector 22 may be configured to generate the distribution of discrete, non-connected light spots 34 at all planes located between 1 mm and 30 mm, e.g., 4 mm and 24 mm, from the pattern-generating optics 38 when the laser diode 36 transmits light through the pattern-generating optics 38. In some applications, the distribution of discrete, non-connected light spots 34 is focused at one plane located between 1 mm and 30 mm, e.g., 4 mm and 24 mm, while discrete, non-connected light spots are still present at all other planes located between 1 mm and 30 mm, e.g., 4 mm and 24 mm. While described above as using laser diodes, it should be understood that this is an exemplary and non-limiting application. Other light sources may be used in other applications. Furthermore, while described as projecting a pattern of discrete, disconnected light spots, it should be understood that this is an exemplary and non-limiting application. Other patterns or arrays of light may be used in other applications, including, but not limited to, lines, grids, checkerboards, and other arrays. In some applications, the light pattern projected by the structured light projector is spatially fixed relative to one or more cameras.

[0297] Embodiments are described herein with reference to discrete light spots and with reference to performing operations using or based on the spots. Examples of such operations include solving a correspondence algorithm to determine the positions of the light spots, tracking the light spots, mapping projector light rays to the light spots, identifying vulnerable light spots, and generating three-dimensional models based on the positions of the spots. It should be understood that such operations and other operations described with reference to spots also work for other features of other projected light patterns. Thus, discussions herein with reference to spots also apply to other features of projected light patterns.

[0298] The pattern-generating optics 38 may be configured to have a light throughput efficiency (ie, the proportion of light that falls into the pattern out of the total light that falls on the pattern-generating optics 38) of at least 80%, such as at least 90%.

[0299] In some applications, each laser diode 36 of each structured light projector 22 transmits light at a different wavelength, i.e., each laser diode 36 of at least two structured light projectors 22 transmits light at two different wavelengths. In some applications, each laser diode 36 of at least three structured light projectors 22 transmits light at three different wavelengths. For example, red, blue, and green laser diodes may be used. In some applications, each laser diode 36 of at least two structured light projectors 22 transmits light at two different wavelengths. For example, in some applications, there are six structured light projectors 22 arranged within the probe 28, three of which include blue laser diodes and three of which include green laser diodes.

[0300] Reference is now made to FIG. 4 , which is a schematic diagram of a structured light projector 22 projecting a distribution of discrete, disconnected light spots onto multiple object focal planes, according to some applications of the present invention. The scanned object 32 may be one or more teeth or other intraoral objects / tissues in a subject's mouth. The somewhat translucent and glossy characteristics of teeth can affect the contrast of the projected structured light pattern. For example, (a) some of the light striking the tooth may be scattered to other areas in the intraoral scene, causing a degree of stray light, and (b) some of the light may penetrate the tooth and then exit the tooth at some other point. Therefore, to improve image capture of intraoral scenes under structured light illumination without using contrast-enhancing measures such as coating the teeth with opaque powders, the inventors have realized that a sparse distribution 34 of discrete, disconnected light spots may provide an improved balance between reducing the amount of projected light while maintaining the amount of useful information. The sparseness of the distribution 34 may be characterized by the following ratio: (a) the illuminated area on the orthogonal plane 44 in the illumination field α (alpha), i.e., the sum of the areas of all the projected spots 33 on the orthogonal plane 44 in the illumination field α (alpha), (b) The ratio of the illumination field α (alpha) to the non-illuminated area on the orthogonal plane 44. In some applications, the sparsity / dense ratio may be at least 1:150 and / or less than 1:16 (e.g., at least 1:64 and / or less than 1:36).

[0301] In some applications, each structured light projector 22 projects at least 400 discrete, disconnected spots 33 onto the intraoral three-dimensional surface during a scan. In some applications, each structured light projector 22 projects fewer than 3000 discrete, disconnected spots 33 onto the intraoral surface during a scan. To reconstruct the three-dimensional surface from the projected sparse distributions 34, a correspondence between each projected spot 33 (or other feature of the projected pattern) and the spots (or other feature) detected by camera 24 must be determined, as further described below with reference to FIGS. 7-19.

[0302] In some applications, the pattern-generating optical element 38 is a diffractive optical element (DOE) 39 ( FIG. 3 ), and when the laser diode 36 transmits light through the DOE 39 to the target 32, it generates a distribution 34 of discrete, disconnected light spots 33. As used throughout this specification, including the claims, a light spot is defined as a small region of light having an arbitrary shape. In some applications, each DOE 39 of a different structured light projector 22 generates spots having different respective shapes, i.e., all spots 33 generated by a particular DOE 39 have the same shape, and the shape of the spots 33 generated by at least one DOE 39 is different from the shape of the spots 33 generated by at least one other DOE 39. For example, some DOEs 39 may generate circular spots 33 (as shown in FIG. 4 ), some DOEs 39 may generate square spots, and some DOEs 39 may generate elliptical spots. Optionally, some DOEs 39 may generate connected or disconnected linear patterns.

[0303] 5A-B, which are schematic diagrams of a structured light projector 22 according to some applications of the present invention, including a beam-shaping optical element 40 and an additional optical element, e.g., a DOE 39, disposed between the beam-shaping optical element 40 and the pattern-generating optical element 38. Optionally, the beam-shaping optical element 40 is a collimating lens 130. The collimating lens 130 may be configured to have a focal length of less than 2 mm. Optionally, the focal length may be at least 1.2 mm. In some applications, an additional optical element 42 disposed between the beam-shaping optical element 40 and the pattern-generating optical element 38, e.g., the DOE 39, generates a Bessel beam when the laser diode 36 transmits light through the optical element 42. In some applications, the Bessel beam is transmitted through a range of orthogonal planes 44 (e.g., orthogonal planes located between 1 mm and 30 mm from the DOE 39, e.g., between 4 mm and 24 mm from the DOE 39, etc.) such that all discrete, disconnected light spots 33 maintain a small diameter (e.g., less than 0.06 mm, e.g., less than 0.04 mm, e.g., less than 0.02 mm) through the DOE 39. The diameter of the spots 33, in the context of this patent application, is defined by the full width at half maximum (FWHM) of the spot intensity.

[0304] Despite the above statement that all spots are smaller than 0.06 mm, some spots with diameters near the upper end of these ranges (e.g., only slightly smaller than 0.06 mm, or 0.02 mm), near the edge of the illumination field of projector 22, may be elongated when intersecting a geometric plane orthogonal to DOE 39. In such cases, it is useful to measure their diameters when intersecting the inner surface of a geometric sphere centered at DOE 39 and having a radius between 1 mm and 30 mm, corresponding to the distance of each orthogonal plane located between 1 mm and 30 mm from DOE 39. As used throughout this application, including the claims, the term "geometric" is considered to relate to an abstract geometric construct (such as a plane or sphere) and not to be part of any physical device.

[0305] In some applications, when a Bessel beam is transmitted through DOE 39, spots 33 with diameters greater than 0.06 mm are generated in addition to spots with diameters less than 0.06 mm.

[0306] In some applications, optical element 42 is an axicon lens 45, shown in FIG. 5A and further described below with reference to FIGS. 23A-B. Alternatively, optical element 42 may be an annular aperture ring 47, as shown in FIG. 5B. Maintaining a small spot diameter improves 3D resolution and accuracy throughout the depth of focus. Without optical element 42, such as an axicon lens 45 or annular aperture ring 47, the size of spot 33 may change, e.g., grow, as one moves farther away from the plane of best focus due to diffraction and defocusing.

[0307] Reference is now made to FIGS. 6A-B, which are schematic diagrams of structured light projectors 22 projecting discrete, disconnected spots 33 and camera sensors 58 detecting spots 33′ according to some applications of the present invention. In some applications, a method is provided for determining correspondences between projected spots 33 on the intraoral surface and detected spots 33′ on each camera sensor 58. As previously described, this method also applies to determining correspondences between other projected features on the intraoral surface and detected features on each camera sensor. Once the correspondences are determined, a three-dimensional image of the surface is reconstructed. Each camera sensor 58 has an array of pixels, for each of which there is a corresponding camera ray 86. Similarly, for each projected spot 33 from each projector 22, there is a corresponding projector ray 88. Each projector ray 88 corresponds to a path 92 of a respective pixel on at least one of the camera sensors 58. Thus, when the camera views a spot 33' projected by a particular projector ray 88, that spot 33' will necessarily be detected by pixels on a particular path 92 of pixels corresponding to that particular projector ray 88. With particular reference to FIG. 6B, the correspondence between each projector ray 88 and each camera sensor path 92 is shown. Projector ray 88' corresponds to camera sensor path 92', projector ray 88" corresponds to camera sensor path 92", and projector ray 88" corresponds to camera sensor path 92". For example, if a particular projector ray 88 projects a spot into a dust-filled space, a line of dust in the air will be illuminated. The line of dust detected by camera sensor 58 will follow the same path on camera sensor 58 as the camera sensor path 92 corresponding to the particular projector ray 88.

[0308] During the calibration process, calibration values ​​are stored based on camera rays 86 corresponding to pixels on camera sensor 58 of each of cameras 24 and projector rays 88 corresponding to projected spots 33 (or other features) of light from each structured light projector 22. For example, calibration values ​​may be stored for (a) multiple camera rays 86 corresponding to respective multiple pixels on camera sensor 58 of each of cameras 24, and (b) multiple projector rays 88 corresponding to respective multiple projected spots 33 of light from each structured light projector 22. As used throughout this application, including the claims, stored calibration values ​​indicating camera rays corresponding to each pixel on camera sensor 58 of each camera refer to (a) values ​​assigned to each camera ray, or (b) parameter values ​​of a parameterized camera calibration model, e.g., a function. As used throughout this application, including the claims, stored calibration values ​​indicative of projector rays corresponding to each projected spot of light (or other projected feature) from each structured light projector means (a) values ​​assigned to each projector ray, e.g., in an indexed list, or (b) parameter values ​​of a parameterized projector calibration model, e.g., a function.

[0309] As an example, the following calibration process may be used: A high-precision dot target, e.g., a black dot on a white background, is illuminated from below, and images of the target are captured by all cameras. The dot target is then moved vertically toward the camera, i.e., along the z-axis, to the target surface. For every dot at every position along the z-axis, the dot center is calculated, creating a three-dimensional grid of dots in space. Then, using the distortion and camera pinhole model, pixel coordinates for each three-dimensional location of each dot center are determined, thereby defining, for each pixel, a camera ray as a ray emanating from that pixel toward the corresponding dot center in the three-dimensional grid. Camera rays corresponding to pixels between grid points may be interpolated. The camera calibration procedure described above is repeated for every wavelength of each laser diode 36, so that the camera ray 86 corresponding to each pixel on each camera sensor 58 for each wavelength is included in the stored calibration values. Alternatively, the stored calibration values ​​are parameter values ​​of the distortion and camera pinhole model, indicating the value of the camera ray 86 corresponding to each pixel on each camera sensor 58 for each wavelength.

[0310] After the cameras 24 are calibrated and all camera ray 86 values ​​are stored, the structured light projector 22 may be calibrated as follows: A flat, featureless target is used, and the structured light projectors 22 are turned on one at a time. Each spot (or other feature) is placed on at least one camera sensor 58. Now that the cameras 24 are calibrated, the three-dimensional spot position of each spot (or other feature) is calculated by triangulation based on images of the spot (or other feature) on multiple different cameras. The above process is repeated with the featureless target positioned at multiple different z-axis positions. Each projected spot (or other feature) on the featureless target will define a projector ray in space originating from the projector.

[0311] Reference is now made to FIG. 7, a flowchart outlining a method for generating a digital three-dimensional image according to some applications of the present invention. Steps 62 and 64 of the method outlined in FIG. 7 respectively drive each structured light projector 22 to project a pattern of light (e.g., a distribution 34 of discrete, disconnected light spots 33) onto the intraoral three-dimensional surface and drive each camera 24 to capture an image including at least a portion of the pattern (e.g., one of the spots 33). A correspondence algorithm is executed in step 66 using a processor 96 (FIG. 1), further described below with reference to FIGS. 8-12, based on stored calibration values ​​indicating (a) camera rays 86 corresponding to each pixel on the camera sensor 58 of each camera 24 and (b) projector rays 88 corresponding to each projected spot 33 of light from each structured light projector 22. In some embodiments, processor 96 is a processor located in elongated handheld wand 20. In some embodiments, the processor 96 is located in a computing device, such as that described below with reference to FIGS. 60-61 , which may be operatively connected (e.g., via a wired or wireless connection) to the elongated handheld wand 20. In some embodiments, multiple processors are used, where one or more processors may be located in the elongated handheld wand and / or one or more processors may be located in the computing device. Once the correspondences are solved, three-dimensional locations on the intraoral surface are calculated in step 68 and used to generate a digital three-dimensional image of the intraoral surface. Furthermore, using multiple cameras 24 to capture the intraoral scene provides an improvement in signal-to-noise in the capture as a factor of the square root of the number of cameras.

[0312] Reference is now made to FIG. 8, a flowchart outlining the correspondence algorithm of step 66 of FIG. 7, according to some applications of the present invention. Based on the stored calibration values, all projector rays 88 and all camera rays 86 corresponding to all detection spots 33′ are mapped (step 70), and all intersection points 98 (FIG. 10) between at least one camera ray 86 and at least one projector ray 88 are identified (step 72). FIGS. 9 and 10 are schematic diagrams illustrating simplified examples of steps 70 and 72 of FIG. 8, respectively. As shown in FIG. 9, three projector rays 88 are mapped, along with eight camera rays 86 corresponding to a total of eight detection spots 33′ on the camera sensor 58 of the camera 24. As shown in FIG. 10, sixteen intersection points 98 are identified.

[0313] In steps 74 and 76 of Figure 7, processor 96 determines a correspondence between the projected spots 33 and the detected spots 33' to identify the three-dimensional location of each projected spot 33 on the surface. Figure 11 is a schematic diagram depicting steps 74 and 76 of Figure 8 using the simplified example described in the immediately preceding paragraph. For a given projector ray i, processor 96 "sees" a corresponding camera sensor path 90 on one camera sensor 58 of camera 24. Each detected spot j along camera sensor path 90 will have a camera ray 86 that intersects with the given projector ray i at an intersection point 98. The intersection point 98 defines a three-dimensional point in space. The processor 96 then "looks" at the camera sensor path 90' corresponding to the given projector ray i on each of the other cameras 24's camera sensors 58' and determines how many other cameras 24 have similarly detected a respective spot k on their respective camera sensor paths 90' corresponding to the given projector ray i, where that camera ray 86' intersects with the same three-dimensional point in space defined by the intersection point 98. This process is repeated for all detected spots j along the camera sensor paths 90, and the spot j on which the most cameras 24 "agree" is identified as the spot 33 ( FIG. 12 ) projected on the surface from the given projector ray i. That is, the projector ray i is identified as the particular projector ray 88 that generated the detected spot j where the most other cameras detected the respective spot k. The three-dimensional location on the surface is calculated for that spot 33. The same process may be performed to calculate the three-dimensional location on the surface of other features of the projection pattern.

[0314] 11, all four cameras detect a respective spot on their respective camera sensor paths corresponding to projector ray i where their respective camera ray intersects with projector ray i at intersection point 98, where intersection point 98 is defined as the intersection point of projector ray i with camera ray 86 corresponding to detected spot j. Therefore, all four cameras are said to "agree" that spot 33 projected by projector ray i exists at intersection point 98. However, when this process is repeated for the next spot j', no other camera detects the respective spot on its respective camera sensor path corresponding to projector ray i where its respective camera ray intersects projector ray i at intersection point 98', which was defined as the intersection point of camera ray 86" (corresponding to detected spot j') and projector ray i. Thus, only one camera can be said to "agree" that spot 33 (or other feature) projected by projector ray i is located at intersection point 98', while all four cameras can be said to "agree" that spot 33 (or other feature) projected by projector ray i is located at intersection point 98'. Therefore, projector ray i is identified as the particular projector ray 88 that produced detected spot j by projecting spot 33 (or other feature) onto the surface at intersection point 98 (FIG. 12). As in step 78 of FIG. 8 and as shown in FIG. 12, the three-dimensional location 35 on the intraoral surface is calculated at the intersection point 98.

[0315] Reference is now made to FIG. 13, a flowchart outlining further steps of the correspondence algorithm, according to some applications of the present invention. Once location 35 on the surface has been determined, projector ray i that projected spot j, and all camera rays 86 and 86' corresponding to spot j and the respective spot k, are removed from consideration (step 80), and the correspondence algorithm is run again for the next projector ray i (step 82). FIG. 14 depicts the simplified example described above after removing the particular projector ray i that projected spot 33 at location 35. As per step 82 of the flowchart in FIG. 13, the correspondence algorithm is then run again for the next projector ray i. As shown in FIG. 14, the remaining data indicates that the three cameras "agree" that spot 33 at intersection 98 of camera ray 86 corresponds to detected spot j and projector ray i, with intersection 98 being defined by the intersection of camera ray 86 and projector ray i corresponding to detected spot j. Thus, as shown in FIG. 15, a three-dimensional position 37 is calculated at the intersection point 98 .

[0316] As shown in FIG. 16 , once the 3D location 37 on the surface is determined, the projector ray i that projected spot j and all camera rays 86 and 86′ corresponding to spot j and the respective spot k are removed from consideration. The remaining data indicates the spot 33 projected by projector ray i at intersection point 98, and the 3D location 41 on the surface is calculated at intersection point 98. As shown in FIG. 17 , according to a simplified example, the three projected spots 33 of the three projector rays 88 of the structured light projector 22 are now located at 3D locations 35, 37, and 41 on the surface. In some applications, each structured light projector 22 projects between 400 and 3000 spots 33. Once the correspondences have been solved for all projector rays 88, a reconstruction algorithm can be used to reconstruct a digital image of the surface using the calculated 3D locations of the projected spots 33.

[0317] Reference is now made to FIG. 28, a flowchart outlining the steps of a method, hereinafter referred to as "spot tracking," for generating a digital three-dimensional image in accordance with some applications of the present invention. While this method is called "spot tracking," it may be equally applied to tracking other types of features in a projected pattern. Motion of the handheld intraoral scanner relative to the intraoral surface during scanning causes the projected points to move across the intraoral surface. The inventors have realized that if the movement of a particular detection spot (or other feature) can be tracked in successive image frames, then a correspondence relationship solved for that particular spot (or other feature) in any of the frames in which the spot (or other feature) is tracked will provide a solution for the correspondence relationship for that spot (or other feature) in all frames in which the spot (or other feature) is tracked. That is, if in a frame the processor 96 determines that a given detection spot 33' was projected by a given projector ray 88, and spot tracking causes the processor 96 to determine that the detection spot 33' in the next image is the same spot, then the processor 96 automatically determines that the same projector ray 88 generated the detected tracking spot in the next image.

[0318] Because the detection spot 33′ that can be tracked across successive images is generated by the same specific projector ray, the trajectory of the tracked spot will follow the specific camera sensor path 90 that corresponds to that specific projector ray 88. Once a correspondence is found for the detection spot 33′ at a point along the specific camera sensor path 90, the three-dimensional positions on the surface for all points along the camera sensor path 90 at which that spot 33′ was detected can be calculated; that is, the processor can calculate the respective three-dimensional positions on the intraoral three-dimensional surface at the intersections of the specific projector ray 88 that generated the detection spot 33′ and each camera ray 86 that corresponds to the tracked spot, in each of multiple successive images along which the spot 33′ was tracked. This can be particularly useful for situations where a particular detection spot is only seen by one camera (or a small number of cameras) in a particular image frame. If that particular detection spot was seen by other cameras 24 in previous consecutive image frames, and correspondences were resolved for that particular detection spot in those previous image frames, then even in image frames in which a particular detection spot was seen by only one camera 24, the processor knows which projector ray 88 produced that spot and can determine the spot's three-dimensional location on the intraoral three-dimensional surface.

[0319] For example, hard-to-reach areas within the oral cavity may be imaged by only a single camera 24. In this case, if the detection spot 33′ on the camera sensor 58 of the single camera 24 can be tracked through multiple previous sequential images, the three-dimensional position on the surface of the spot (even though it is only seen by the single camera 24) can be calculated based on the information gained from the tracking, i.e., information about (a) which camera sensor path the spot is moving along and (b) which projector ray generated the tracked spot 33′.

[0320] In step 180 of the method outlined in FIG. 28, each structured light projector 22 is driven to project a pattern of light, which in one embodiment is a distribution 34 of discrete, disconnected light spots 33, onto the intraoral three-dimensional surface, and in step 182, each camera 24 is driven to capture an image including at least a portion of the projected pattern (e.g., at least one of the spots 33). In one embodiment, based on stored calibration values ​​indicating (a) the camera ray 86 corresponding to each pixel on the camera sensor 58 of each camera 24 and (b) the projector ray 88 corresponding to each projected spot 33 of light from each structured light projector 22, in step 184, the processor 96 is used to compare a series of images (e.g., multiple consecutive images) captured by each camera 24 to determine which features of the projected pattern (e.g., which of the projected spots 33) can be tracked across the multiple images, with each tracked feature (e.g., spot 33s′) moving along a pixel path p corresponding to a particular camera sensor path 90 corresponding to a respective projector ray, e.g., projector ray 88. In some applications, in step 186, the processor 96 calculates the three-dimensional position of each tracked feature (e.g., spot 33s′) on the intraoral three-dimensional surface in the series of images (e.g., in each consecutive image).

[0321] In one embodiment, each structured light projector of the one or more structured light projectors is driven to project a pattern onto the intraoral three-dimensional surface. Furthermore, each camera of the one or more cameras is driven to capture multiple images, each image including at least a portion of the projected pattern. The projected pattern may include multiple projected light spots, and portions of the projected pattern may correspond to projected spots of the multiple projected light spots. The processor 96 then compares the series of images captured by the one or more cameras, determines which portions of the projected pattern can be tracked across the series of images based on the comparison of the series of images, and builds a three-dimensional model of the intraoral three-dimensional surface based at least in part on the comparison of the series of images. In one embodiment, the processor solves a correspondence algorithm for the tracked portion of the projected pattern in at least one image of the series of images, and uses the solved correspondence algorithm to address the tracked portion of the projected pattern in at least one image of the series of images, e.g., solves a correspondence algorithm for the tracked portion of the projected pattern in images of the series of images for which the correspondence algorithm has not been solved, and builds the three-dimensional model using the solution of the correspondence algorithm. In one embodiment, the processor solves a correspondence algorithm for the tracked portion of the projected pattern based on the position of the tracked portion in each image throughout the sequence of images and constructs a three-dimensional model using the solution of the correspondence algorithm. In one embodiment, the processor compares the sequence of images based on stored calibration values ​​that indicate (a) a camera ray corresponding to each pixel on the camera sensor of each camera of the one or more cameras, and (b) a projector ray corresponding to each one projected light spot from each structured light projector of one or more structured light projectors, each projector ray corresponding to a path of a respective pixel on at least one camera sensor of the camera sensors, wherein each tracking spot s moves along the path of the pixel corresponding to a respective projector ray r.

[0322] Reference is now made to Figure 29, which is a schematic diagram illustrating several detection spots 33' (specifically, 33a' and 33b') and illustrating how processor 96 may determine which set of detection spots 33' may be considered tracked (step 184 of Figure 28) according to some applications of the present invention. Spots 33a' are detection spots 33' in the previous image, and spots 33b' are detection spots 33' in the current image. Processor 96 searches within a search radius to find possible matches between spots 33a' and 33b' that may be considered the same tracked spot 33' between the two images.

[0323] The inventors have identified that there are typically three factors that can affect how much a spot moves between frames. 1. How far the spot moves between frames is typically inversely proportional to the camera frame rate. That is, if the camera frame rate is very fast, the spot will appear to move a relatively small distance between successive frame pairs, and if the camera frame rate is slow, the spot will appear to move farther between successive frame pairs. How fast the wand is moving relative to the intraoral surface will also affect how far the spot moves between frames; the faster the wand is moving, the farther the spot will appear to move between successive frame pairs. 2. How far the spot moves between frames typically depends on the degree of tilt of the intraoral surface being scanned, relative to the projector 22 and / or camera 24. If the spot is projected onto a tilted surface and the spot is moving in the direction of the tilt, the corresponding detected spot on the camera sensor will move faster, and therefore the tracked spot will move farther between successive sets of frames. 3. How far the spot has moved between frames, as seen from the camera's field of view, typically depends on the distance between the scan surface and the projector. If the surface is close to the projector, even a small movement of the projector will cause a large movement of the tracking spot on the sensor 58 of the camera 24 between successive pairs of frames. In contrast, if the surface is far away, the same movement of the projector will cause a smaller movement of the tracking spot on the sensor 58 of the camera 24 between successive pairs of frames. For example, if the surface approaches an infinite distance from the projector, the movement of the projector will cause almost zero movement of the tracking spot on the sensor 58 of the camera 24.

[0324] In some applications, processor 96 searches within a fixed search radius of at least 3 pixels and / or less than 10 pixels (e.g., 5 pixels). In some applications, processor 96 calculates the search radius taking into account parameters such as the level of spot position error, which may be determined during calibration. For example, the search radius may be 2 * (spot position error) or 3 * (spot position error).

[0325] In the simplified example shown in FIG. 29, spots 33a' and 33b' in each set 112 are deemed close enough to each other to be considered the same projected spot 33 moving through the two images. That is, for each set 112, the two detected spots 33a' and 33b' are deemed to be the same tracking spot 33s'. In contrast, spots 33a' and 33b' in set 114 are too far away to be considered tracked. Spots 33a' and 33b' in set 116 are close enough, but are not considered tracked because more than one match is found. As explained further below, in set 116, there are pairs of tracking spots, and continuing to analyze more images can help determine which spots are actually tracking spots.

[0326] In one embodiment, to generate a digital three-dimensional image, the intraoral scanner drives each structured light projector of one or more structured light projectors to project a pattern of light onto the intraoral three-dimensional surface. The intraoral scanner further drives each of a plurality of cameras to capture an image, the image including at least a portion of the projected pattern, each camera of the plurality of cameras including a camera sensor including a pixel array. The intraoral scanner further uses a processor to execute a correspondence algorithm to calculate the three-dimensional positions of each of a plurality of features of the projected pattern on the intraoral three-dimensional surface. The processor uses data from a first camera of the plurality of cameras, e.g., data from at least two cameras, to identify candidate three-dimensional positions of a given feature of the projected pattern corresponding to a particular projector light ray r, while data from a second camera of the plurality of cameras, e.g., a camera other than one of the at least two cameras, is not used to identify the candidate three-dimensional positions. The processor further uses the candidate three-dimensional positions seen by the first camera to identify a search space on the pixel array of the second camera in which to search for features of the projected pattern from the projector light ray r. If a feature of the projected pattern from projector light ray r is identified within the search space, the processor refines the candidate three-dimensional locations of the feature of the projected pattern using data from the second camera. In one embodiment, the light pattern includes a distribution of discrete, disconnected light spots, and the feature of the projected pattern includes a projection spot from the disconnected light spots. In one embodiment, the processor uses stored calibration values ​​that indicate (a) a camera light ray corresponding to each pixel on a camera sensor of each camera of the multiple cameras, and (b) a projector light ray corresponding to each feature of the feature of the projected light pattern from each structured light projector of the one or more structured light projectors, where each projector light ray corresponds to a path of a respective pixel on at least one of the camera sensors.

[0327] Reference is now made to FIG. 30, a flowchart outlining a method for determining tracking features (e.g., tracking spots 33s′, etc.) according to some applications of the present invention. While FIG. 30 is discussed with reference to tracking spots, it applies equally to other types of tracking features. In some applications, in addition to searching for tracking spots by monitoring the proximity of spots in successive images, processor 96 may search for tracking spots 33s′ based on the parameters(s) of detected spots 33′, hereinafter referred to as “parametric tracking.” Processor 96 determines the parameters of detected spots 33′ in a first one of the successive images (step 188) and in an adjacent image. Processor 96 then uses the determined parameters of detected spots 33′ in two adjacent images to predict the same parameters of spots in subsequent images, e.g., the next image (and subsequent images) (step 190). Processor 96 searches for spots in subsequent images, e.g., the next image, that have substantially the predicted parameters (step 192). For example, two particular detection spots 33 a′ and 33 b′ may be determined to both originate from the same projector ray in two adjacent frames via a correspondence algorithm as described above or via proximity tracking as described in the immediately preceding two paragraphs. Once processor 96 determines that detection spots 33 a′ and 33 b′ in two adjacent frames are caused by the same projector ray 88, processor 96 can determine parameters of the spots and predict parameters of the spots in a later image, e.g., the next image, based on the parameters of the spots in the two adjacent images.

[0328] In some applications, the spot parameter is the spot size, the spot shape, such as the spot aspect ratio, the spot orientation, the spot intensity, and / or the spot signal-to-noise ratio (SNR). For example, if the determined parameter is the shape of the tracking spot 33s', the processor 96 predicts the shape of the tracking spot 33s' in a later image, e.g., the next image, and determines a search space in the later image in which to search for the tracking spot 33s' based on the predicted shape of the tracking spot 33s' in the later image, e.g., a search space having a size and aspect ratio based on (e.g., within a factor of 2) the size and aspect ratio of the predicted shape of the tracking spot 33s'. In some applications, the spot shape may refer to the aspect ratio of an elliptical spot.

[0329] Referring again to Figure 29, in some applications, parametric tracking may be useful in resolving ambiguities such as those shown in set 116 of spots in Figure 29. As explained above, spots 33a' and 33b' in set 116 are close enough to be considered tracking spots, but more than one match is found. Based on the parametric tracking, processor 96 may then be able to determine which spot in set 116 is actually the tracking spot.

[0330] Reference is now made to Fig. 31, a flowchart outlining a method for detecting a tracking spot 33s' in a subsequent image, according to some applications of the present invention. Fig. 31 also applies to detecting other tracking features in a subsequent image. In some applications, based on the direction and distance that the tracking spot 33s' has moved between two images (e.g., between two consecutive images), the processor 96 determines a velocity vector of the tracking spot 33s' (step 194). The processor 96 then uses the velocity vector to determine a search space in the subsequent image, e.g., the next image, in which to search for the tracking spot 33s' (step 196).

[0331] In some applications, the search space in the subsequent image may be determined by estimating the new position of the tracking spot 33s' using a predictive filter, for example a Kalman filter.

[0332] Reference is now made to FIG. 32, a flowchart outlining a method for detecting a tracking spot 33s' in a subsequent image, according to some applications of the present invention. FIG. 32 also applies to detecting other tracking features in subsequent images. Determining the velocity vector for the tracking spot 33s' may be used to help determine the search space in which to look for the tracking spot; i.e., the faster the spot is moving, the farther it has moved between successive frames, allowing the processor 96 to define a larger search space in which to look for the tracking spot. The inventors have recognized that the shape of the tracking spot 33s' and the direction in which it is moving can indicate the velocity of the spot. For example, in some applications, if a spot projected as a circle appears elliptical, it is likely that the spot is descending on a surface that is inclined relative to the projector 22 and / or camera 24. Similarly, as discussed above, a spot moving along an inclined surface in the direction of the incline will move faster than a spot not moving in the direction of the incline; the steeper the surface slope, the faster the spot will move. Furthermore, if a spot appears elongated into an ellipse by a tilted surface, it is likely that it will appear elongated in the tilt direction, i.e., the major axis of the ellipse will appear to be in the tilt direction. Thus, moving an elliptical spot along its major axis indicates that the spot is tilted, moving up or down, and therefore faster than moving an elliptical spot along its minor axis (which may indicate that the spot is not moving in the tilt direction, even though it is projected onto the tilted surface).

[0333] Thus, in some applications, after determining the shape of the tracking spot 33s' (step 198), the processor 96 may determine a velocity vector of the tracking spot 33s' based on the direction and distance the tracking spot 33s' has moved between two successive images (step 200). The processor 96 may then use the determined velocity vector and / or the shape of the tracking spot 33s' to predict the shape of the tracking spot 33s' in a subsequent image, e.g., the next image (step 202). Following predicting the shape of the tracking spot 33s', the processor 96 may use a combination of the velocity vector and the predicted shape of the tracking spot 33s' to determine a search space in the subsequent image, e.g., the next image, in which to search for the tracking spot 33s'. Referring again to the above example of an elliptical spot, if the shape of the spot is determined to be elliptical and the spot is determined to be moving along its major axis, a larger search space will be specified compared to if the elliptical spot is moving along its minor axis.

[0334] Reference is now made to FIG. 33, which is a schematic diagram illustrating an example in which spot tracking, according to some applications of the present invention, is useful for identifying a detected spot 33′ as projected from a particular projector ray 88. This is useful when the correspondence algorithm does not provide a solution for a particular detected spot 33′, such as when the detected spot 33′ in a particular frame is seen by only one camera 24. In such a case, once the detected spot 33′ is identified as a tracked spot 33s′ moving along a particular camera sensor path 90 of the pixel corresponding to the particular projector ray 88, it can be assumed that that particular projector ray 88 projected the spot, as explained above. Based on the solved correspondences for the tracked spot 33s′ in previous frames, correspondences can be solved for frames in which only one camera detected the spot 33′. Thus, in some applications, if, after running a correspondence algorithm such as the correspondence algorithm described above in connection with Figures 7-17, it is determined that the detected spot 33' is a tracking spot 33s' that moves along a particular path 90 on the camera sensor 58 that corresponds to the particular projector ray 88, then the particular projector ray 88 may be assumed to have generated the detected spot 33'. That is, the processor 96 may identify the detected spot 33' as originating from the particular projector ray 88 by identifying the detected spot 33' as a tracking spot 33s' that moves along the path 90 of the pixels of the camera sensor 58 that correspond to the particular projector ray 88.

[0335] In the example shown in FIG. 33 , in two consecutive frames captured at times 1 and 2, two camera sensors 58 each detected a projected spot 33′ from projector light ray 88. The correspondence algorithm described above determined that projector light ray 88 generated detected spot 33′ in frames 1 and 2 and solved the correspondence between detected spot 33′ in frames 1 and 2. However, in the third frame, only one camera detects spot 33′. In this example, it is assumed that the correspondence algorithm was unable to solve the correspondence for detected spot 33′ in frame 3. However, processor 96 determines that detected spot 33′ in frame 3 is a tracking spot 33s′ moving along the same projector light ray 88 that generated spot 33′ in frames 1 and 2. Therefore, processor 96 identifies detected spot 33′ in frame 3 as being generated by projector light ray 88.

[0336] Reference is now made to FIGS. 34A-B, which are simplified schematic diagrams of camera sensor 58 showing two detection spots 33c' and 33d', according to some applications of the present invention. In FIG. 34A, for each of the detection spots, there is ambiguity as to which projector ray generated that spot, i.e., to which pixel of camera sensor path 90 the detection spot corresponds. In some applications, processor 96 may resolve these ambiguities using spot tracking. Thus, in some applications, if, after performing a correspondence algorithm (such as the correspondence algorithm described above with respect to FIGS. 7-17), detection spot 33' is identified as originating from two different candidate projector rays 88 and 88' based on the three-dimensional position calculated by the correspondence algorithm, processor 96 may identify detection spot 33' as originating from only one of the two different candidate projector rays 88 and 88' by identifying it as a tracking spot 33s' moving along either path 90 or path 90'.

[0337] An example of such ambiguity is represented by detection spot 33c' in FIG. 34A. Spot 33c' is located at the intersection of two different paths 90c and 90c'. The correspondence algorithm may know that such detection spot 33c' is generated by both projector ray 88 corresponding to path 90c and projector ray 88' corresponding to path 90c'. Another type of such ambiguity is represented by detection spot 33d' in FIG. 34A. Spot 33d' is very close to two different paths 90d and 90d' but is not at the intersection between paths 90 and 90d. Due to signal noise, it may have been unclear during the correspondence algorithm whether spot 33d' was generated by projector ray 88 corresponding to path 90d or projector ray 88' corresponding to path 90d'.

[0338] As shown in Figure 34B, processor 96 may identify which projector beam generated spots 33c' and 33d' by identifying spots 33c' and 33d' as tracking spots 33s' that move along a particular one of the paths. Detection spot 33c' is identified as tracking spot 33s' that moves along path 90c', and thus detection spot 33c' is identified as having been generated by projector beam 88' that corresponds to path 90c'. Detection spot 33d' is identified as tracking spot 33s' that moves along path 90d', and thus detection spot 33d' is identified as having been generated by projector beam 88' that corresponds to path 90d'.

[0339] Reference is now made to FIG. 35, a flowchart outlining an additional or alternative method in which spot tracking may be used, according to some applications of the present invention. The concepts illustrated in FIG. 35 also apply to methods using other feature tracking. In some applications, processor 96 may use spot tracking to eliminate false positive spots 33′ from consideration as points on the intraoral three-dimensional surface. In some applications, after performing a correspondence algorithm (step 206), such as the correspondence algorithm described above with reference to FIGS. 7-17, processor 96 may identify detected spots 33′ as originating from a particular projector ray 88 based on the correspondence algorithm (step 208). Step 206 typically follows step 186 of the method outlined in the flowchart of FIG. 28. Furthermore, processor 96 may identify a series of detected spots across multiple successive images as tracked spots 33′ that all move along a pixel path 90 corresponding to the same particular projector ray 88. If the detection spot 33′ is one of the tracking spots 33s′, as indicated by decision diamond 210, then the detection spot 33′ may be considered to be a point on the intraoral three-dimensional surface (step 212). However, if the detection spot 33′ is not identified as the tracking spot 33s′ that moves along the path 90 of the pixel corresponding to that particular projector ray 88, then the detection spot 33′ may be assumed to have been a false positive spot, and the detection spot 33′ may be excluded from consideration as a point on the intraoral three-dimensional surface (step 214).

[0340] Reference is now made to FIG. 36, a flowchart outlining an additional or alternative method in which spot tracking may be used, according to some applications of the present invention. The concepts illustrated in FIG. 36 also apply to methods using other feature tracking. To reduce the incidence of camera 24 detecting a large number of false positive spots, processor 96 may set an intensity threshold, and any detected spots 33′ below the threshold will not be included as candidate spots in the correspondence algorithm. However, this may also result in false positive spots, i.e., spots that may have provided useful information, not being considered because they are weak spots (having intensities below the threshold). For example, in difficult-to-capture areas of an intraoral scene, some of the projected spots 33 may appear below the intensity threshold. Thus, in some applications, after running a correspondence algorithm (step 216), such as the correspondence algorithm described herein with reference to FIGS. 7-17, processor 96 may identify weak spots 33′ whose three-dimensional locations were not calculated by the correspondence algorithm (step 218), for example, by lowering the intensity threshold and considering spots 33′ not considered by the correspondence algorithm. As indicated by decision diamond 220, if a weak spot 33' is identified as a tracking spot 33s' that moves along the path 90 of pixels corresponding to a particular projector ray 88, then the weak spot 33' is identified as being projected from that particular projector ray 88 and is considered to be a point on the intraoral three-dimensional surface (step 222). If a weak spot is not identified as a tracking spot 33s', then the weak spot is excluded from being considered as a point on the intraoral three-dimensional surface (step 224).

[0341] In some applications, for a tracking spot 33s', the processor 96 may determine multiple possible camera sensor paths 90 of the pixel through which the tracking spot 33s' is moving, with the multiple paths 90 corresponding to respective multiple possible projector rays 88. For example, multiple projector rays 88 may closely correspond to the paths 90 of pixels on the camera sensor 58 of a given camera. The processor 96 may run a correspondence algorithm to identify which of the possible projector rays 88 generated the tracking spot 33s' in order to calculate the three-dimensional position on the surface for each position of the tracking spot 33s'.

[0342] For a given camera sensor 58, for each of multiple possible projector rays 88, a 3D point in space exists at the intersection of the possible projector ray 88 and the camera ray corresponding to the tracking spot 33s' detected at the given camera sensor 58. For each possible projector ray 88, the processor 96 considers the camera sensor paths 90 corresponding to the possible projector ray 88 at each of the other camera sensors 58 and determines how many other camera sensors 58 whose camera ray intersects with that 3D point in space also detected a spot 33s' on their respective camera sensor paths 90 corresponding to that possible projector ray 88, i.e., how many other cameras agree that the tracking spot 33s' is projected by that projector ray 88. This process is repeated for all possible projector rays 88 corresponding to the tracking spot 33s'. The projector ray 88 that most other cameras are likely to agree on is determined to be the particular projector ray 88 that generated the tracking spot 33s'. Once the particular projector ray 88 for the tracking spot 33s' has been determined, the camera sensor path 90 along which the spot is moving is known, and the three-dimensional position on the surface of each intersection of the particular projector ray 88 corresponding to the tracking spot 33s' with each camera ray is calculated for each successive image along which the spot 33s' is tracked.

[0343] Reference is now made to FIGS. 37A-B, which are schematic diagrams illustrating points used in a 3D reconstruction before and after processor 96 performs spot tracking, according to some applications of the present invention. The size of each data point represents how many cameras were used to resolve that point; i.e., the larger the point, the more cameras that viewed that spot. Note that the size of the data points is used in the diagrams only to distinguish how many cameras viewed a given point, and does not represent the size of the projected spot on the surface. In FIG. 37A, there are many small peripheral spots that do not appear to be points on the intraoral surface. These small spots represent detected spots that were seen by very few cameras, e.g., only one, but were assigned a 3D location in space based on the correspondence algorithm. After performing the correspondence algorithm, processor 96 performs spot tracking and may therefore determine that these light peripheral spots are actually false positives (by determining that they are not tracked spots). Thus, as shown in FIG. 37B, after spot tracking, most of the spots that spot tracking determined to be false positives have been removed from consideration as points on the intraoral surface.

[0344] Reference is now made to Figure 38, a flowchart outlining the steps of a method for generating a digital three-dimensional image, referred to herein as "ray tracing," according to some applications of the present invention. In some applications, instead of or in addition to tracking detection spots and / or other features in a two-dimensional image (as described above), the length of each projector ray 88 may be tracked in three-dimensional space. The length of a projector ray 88 is defined as the distance between the origin of the projector ray 88, i.e., the light source, and the three-dimensional location where the projector ray 88 intersects the intraoral surface.

[0345] In step 226 of the method outlined in FIG. 38, each structured light projector 22 is driven to project a pattern of light, such as a distribution 34 of discrete, disconnected light spots 33, onto the intraoral three-dimensional surface, and in step 228, each camera 24 is driven to capture multiple images, each image including at least one feature of the projected pattern (e.g., at least one of the spots 33). While the method is described with reference to spots, other types of features will also work. In step 230, the processor 96 is used to run a correspondence algorithm, such as the one described above with reference to FIGS. 7-17, based on stored calibration values ​​indicating (a) the camera ray 86 corresponding to each pixel on the camera sensor 58 of each camera 24 and (b) the projector ray 88 corresponding to each projected spot 33 of light from each structured light projector 22. As a result of the correspondence algorithm, each resolved projector ray 88 in each image frame results in a reconstructed three-dimensional point in space, which in turn defines the length of the resolved projector ray 88 in that frame.

[0346] Thus, in step 232, in at least a subset of the captured images, e.g., in a series of images or multiple consecutive images, processor 96 identifies the calculated three-dimensional position of detection spot 33′ (calculated from the correspondence algorithm) as corresponding to a particular projector ray 88. In step 234, based on each three-dimensional position corresponding to projector ray 88 in the subset of images, processor 96 estimates, e.g., calculates, the length of projector ray 88 in each image of the subset of images. Because camera 24 captures images at a relatively high frame rate, e.g., approximately 100 Hz, the geometry of the spot seen by each camera does not change significantly from frame to frame. Therefore, when the estimated, e.g., calculated, length of projector ray 88 is tracked and plotted versus time, the data points will follow a relatively smooth curve, although some discontinuities may occur, as discussed further below. Thus, the projector ray length over time forms a relatively smooth univariate function versus time. As mentioned above, the detection spot 33' corresponding to the projector ray 88 across multiple successive images will appear to move along a one-dimensional line that is the path 90 of the camera sensor pixel corresponding to the projector ray 88.

[0347] In one embodiment, a method for generating a digital three-dimensional image includes activating each structured light projector of one or more structured light projectors to project a pattern onto a three-dimensional intraoral surface and activating each camera of one or more cameras to capture images, the images including at least a portion of the pattern. The method further includes using the processor to execute a correspondence algorithm to calculate three-dimensional positions of each of a plurality of features of the pattern on the three-dimensional intraoral surface captured in a series of images. The processor further identifies, in at least a subset of the series of images, the calculated three-dimensional positions of detected features of the imaged pattern as corresponding to one or more specific projector rays r. Based on the three-dimensional positions of the detected features corresponding to the one or more projector rays r in the subset of images, the processor estimates, e.g., calculates, lengths associated with one or more projector rays r in each image of the subset of images. In one embodiment, the processor calculates estimated lengths of one or more projector rays r in at least one of the series of images in which the three-dimensional positions of features projected from the one or more projector rays r were not determined. In one embodiment, each camera of the one or more cameras comprises a camera sensor including a pixel array, and calculating the three-dimensional positions of each of a plurality of features of a pattern on the three-dimensional intraoral surface and identifying that the calculated three-dimensional positions of the detected features of the pattern correspond to a particular projector ray r is performed based on stored calibration values ​​indicative of (i) camera rays corresponding to each pixel of the camera sensor of each of the one or more cameras, and (ii) projector rays corresponding to each feature of a projected light pattern from each projector of the one or more projectors, each projector ray corresponding to a path of a respective pixel on at least one of the camera sensors. In one embodiment, the pattern includes a plurality of spots, and each of a plurality of features of the pattern includes one of the plurality of spots.

[0348] Reference is now made to FIG. 39, which is a graph showing tracking of the length of a projector light ray 88 over time and a particular simplified view of the camera sensor 58 corresponding to a particular image frame, according to some applications of the present invention. The inventors have realized several use cases for the ray tracing described above. In some applications, there may be at least one image from multiple consecutive images in which the three-dimensional location of the projected spot 33 (or other feature) from a particular projector light ray 88 was not identified in step 232 of the method shown in FIG. 38. For example, the projected spot (or other feature) in a particular frame may be below an intensity threshold and not be considered by the correspondence algorithm, or there may have been a false detection of a spot in a particular frame. However, because the projector light ray length is tracked over time, the processor 96 may calculate an estimated length of a particular projector light ray 88 in that image.

[0349] For example, in the exemplary graph shown in FIG. 39 , for scan frame s1 captured at time t1, the three-dimensional position of the projected spot 33 was not determined based on the correspondence algorithm, and therefore, there is no data point corresponding to the length of the projector light ray 88 for scan frame s1, as illustrated by dashed circle 236. However, because the length of the light ray is tracked through multiple consecutive images, an estimated length L1 of the projector light ray 88 can be calculated for scan frame s1, for example, by interpolation. As described above, all projected points from a particular projector light ray 88 appear on a particular path 90 of pixels on camera sensor 58 corresponding to that particular projector light ray 88. Therefore, for scan frame s1 for which the three-dimensional position of the spot 33 corresponding to a particular projector light ray 88 was not determined in step 232, processor 96 may determine a one-dimensional search space 238 within scan frame s1 in which to search for the projected spot from that particular projector light ray 88. The one-dimensional search space 238 is along the path 90 of each pixel corresponding to a particular projector ray 88. This is in contrast to spot tracking algorithms such as those described above, where the processor 96 searches two-dimensionally within the image for spots that are close enough to each other from one frame to the next to be considered tracked spots generated by the same projector ray in each of the image frames.

[0350] In some applications, based on the estimated length L1 of the projector ray 88 in at least one image of the multiple images, the processor 96 may determine a one-dimensional search space in each pixel array, e.g., camera sensor 58, of the multiple cameras 24, e.g., all of the cameras 24. For each respective pixel array, the one-dimensional search space is along the path 90 of each pixel corresponding to the projector ray 88 in that particular pixel array, e.g., camera sensor 58. The length L1 of the projector ray 88 corresponds to a three-dimensional point in space, which corresponds to a two-dimensional position on the camera sensor 58. All other cameras 24 also have respective two-dimensional positions on the camera sensors 58 that correspond to the same point in three-dimensional space. Thus, the length of the projector ray in a particular frame may be used to define a one-dimensional search space in the multiple camera sensors 58, e.g., all of the camera sensors 58, for that particular frame.

[0351] In some applications, there may be at least one image in a plurality of consecutive images in which multiple candidate three-dimensional locations are calculated for a projected spot 33 (or other feature) from a particular projector beam 88, i.e., a false positive of a projected spot 33 (or other feature) has occurred, as opposed to a false positive in which an expected spot (or other feature) was not detected. For example, in the exemplary graph shown in FIG. 39 , based on the correspondence algorithm, for scan frame s2 taken at time t2, two candidate detection spots 33′ and 33″ were calculated to both originate from a particular projector ray 88, and therefore two candidate three-dimensional positions of the projected spot 33 were calculated, and processor 96 calculated two candidate lengths of the projector ray 88 corresponding to each of the candidate three-dimensional positions for that frame. For example, processor 96 calculates that for candidate detection spot 33′, the candidate length of the projector ray 88 is L2 (represented by data point 242 in FIG. 39 ), and for candidate detection spot 33″, the candidate length of the projector ray 88 is L3 (represented by data point 244 in FIG. 39 ). Due to the length of the projector ray 88 being tracked across multiple consecutive images, when the ray length data for scan frame s2 are added, it becomes clear which candidate length, L2 or L3, is the estimated length of the projector ray 88 for scan frame s2. In this manner, processor 96 can determine which of multiple candidate three-dimensional positions for projection spot 33 corresponds to the estimated length of projector light ray 88 for that image, thereby determining which is the correct three-dimensional position for projection spot 33.

[0352] Based on the estimated length of projector beam 88 in at least one of the multiple images, for example, scan frame s2, processor 96 may determine a one-dimensional search space 246 in scan frame s2. Processor 96 may then determine which of the multiple candidate three-dimensional positions of projected spot 33′ is the correct three-dimensional position of projected spot 33 generated by projector beam 88 by determining which of the multiple candidate three-dimensional positions of projected spot 33′ corresponds to spot 33′ generated by projector beam 88 and is found within one-dimensional search space 246. Before the additional information provided by ray tracing, camera sensor 58 for scan frame s2 would have shown both candidate detection spots 33' and 33" on the path 90 of pixels corresponding to projector ray 88. Processor 96 calculates an estimated length of projector ray 88 based on the length of the ray traced across multiple successive images, enabling processor 96 to determine a one-dimensional search space 246 and determine that candidate detection spot 33' is, in fact, the correct spot. Candidate detection spot 33" is then excluded from being considered to be a point on the three-dimensional intraoral surface.

[0353] In some applications, processor 96 may define curve 248 based on the estimated, e.g., calculated, lengths of projector ray 88 in each image of a subset of images, e.g., a plurality of consecutive images. We may reasonably assume that any detected point whose three-dimensional location based on the correspondence algorithm corresponds to a length of projector ray r that is at least a threshold distance away from defined curve 248 may be considered a false positive and may be excluded from being considered a point on the three-dimensional intraoral surface.

[0354] Reference is now made to Figures 40A-B, which are graphs showing experimental sets of data before and after ray tracing, according to some applications of the present invention. In Figure 40A, the length of a particular projector ray 88 is plotted for every spot and / or other feature calculated by the correspondence algorithm. Before ray tracing is applied, ray lengths corresponding to spots and / or other features that appear far from the general curve defined by projector rays 88 are included in the data. Figure 40B shows what the data looks like after ray tracing has been applied and used to determine which false positive spots and / or other features should be removed from being considered points on the three-dimensional intraoral surface.

[0355] Reference is now made to Figure 41, which is a schematic diagram illustrating multiple camera sensors and projectors projecting spots or other features, according to some applications of the present invention. In some applications, after running a correspondence algorithm, such as the correspondence algorithms described above with reference to Figures 7-17, processor 96 can determine with some certainty a potential three-dimensional location of projected spot 33 (or other feature) based on how many cameras 24 detect a given projected spot 33 (or other feature) on their respective pixel arrays (i.e., camera sensors 58). The potential three-dimensional location of projected spot 33 (or other feature) in Figure 41 is indicated by dashed circle 250.

[0356] The more cameras 24 that view the projected spot 33 (or other feature), the more certain the candidate three-dimensional location 250. Thus, using data from at least two of the cameras 24, the processor 96 can identify a candidate three-dimensional location 250 for a given spot 33 (or other feature) that corresponds to a particular projector beam 88. Assuming that the identification of the candidate three-dimensional location 250 is substantially determined without using data from at least another camera 24′, there may be some error in the candidate three-dimensional location 250, and the candidate three-dimensional location 250 may be refined if the processor 96 had data from another camera 24′.

[0357] Thus, after obtaining the correspondence, assuming that at least two cameras 24 saw the projected spot 33 (or other feature), the processor 96 now knows (a) which projector ray 88 generated the projected spot 33 (or other feature) and (b) the candidate three-dimensional location 250 of the spot (or other feature). Combining (a) and (b), the processor 96 can determine a one-dimensional search space 252 of the pixel array, i.e., camera sensor 58', of another camera 24' in which to search for the spot (or other feature) from the projector ray 88. The one-dimensional search space 252 is along a path 90 of pixels on the camera sensor 58' of the other camera 24', and may be along a particular segment of the path 90 that corresponds to the candidate three-dimensional location 250. If a spot 33' (or other feature) resulting from projector ray 88, e.g., a false positive spot 33' that was not considered by the correspondence algorithm (e.g., due to intensity below a threshold), is identified within that one-dimensional search space 252, data from the other camera 24' obtained this time may be used to refine the candidate three-dimensional location 250 of spot 33 (or other feature) into a refined three-dimensional location 254.

[0358] Reference is now made to FIGS. 42A-B, which illustrate flowcharts outlining a method for generating a three-dimensional image according to some applications of the present invention. In some applications, once three-dimensional positions have been identified for at least three projected spots 33 (or other features) from three different projector rays 88, a three-dimensional surface may be estimated such that all three of the identified three-dimensional positions lie on the estimated surface. For another projector ray 88′ whose three-dimensional position was not determined because the projected spot 33 (or other feature) had an intensity below a threshold, for example, a candidate three-dimensional position may be calculated at the intersection of the other projector ray 88′ with the estimated three-dimensional surface. Similar to the above description with reference to FIG. 41, the processor 96 may use a combination of (a) identifying a particular projector ray, i.e., identifying the path 90 on the sensor to be viewed, and (b) identifying the candidate three-dimensional positions to determine a one-dimensional search space in at least one camera sensor 58 in which to search for detected spots 33′ from other projector rays 88′.

[0359] 42A-B, each structured light projector 22 is driven to project a structured light pattern, e.g., a distribution 34 of discrete, disconnected light spots 33, onto the intraoral three-dimensional surface in step 256, and each camera 24 is driven to capture a plurality of images, each image including at least one of the spots, in step 258. In step 260, processor 96 is used to execute a correspondence algorithm (e.g., the correspondence algorithm described above with reference to FIGS. 7-17) to calculate, for each of the plurality of images, the three-dimensional positions of each of the plurality of detection spots 33′ on the intraoral three-dimensional surface based on stored calibration values ​​indicating (a) camera rays 86 corresponding to each pixel on camera sensor 58 of each camera 24 and (b) projector rays 88 corresponding to each projected spot 33 of light from each structured light projector 22. In step 262, using data corresponding to the three-dimensional positions of each of the at least three detection spots 33′, each corresponding to a respective projector ray 88, processor 96 estimates the three-dimensional surface on which all of the at least three detection spots 33′ are located. Processor 96 then considers another projector ray 88′. As indicated by decision hexagon 266, if the correspondence algorithm already identified the three-dimensional position of one of the detection spots 33′ from the other projector ray 88′ in step 260, that detection spot 33′ from the other projector ray 88′ is considered to be a point on the intraoral surface at that three-dimensional position (step 268). However, for another projector ray 88′ for which the three-dimensional position of the spot 33 corresponding to the other projector ray 88′ was not calculated in step 260, processor 96 may estimate the three-dimensional position within the intersection space between the other projector ray 88′ and the estimated three-dimensional surface (step 270). In step 272, the processor 96 uses the estimated three-dimensional spatial position to identify a search space (e.g., a one-dimensional search space) in which to search for detection spots 33' corresponding to other projector rays 88' in the pixel array of at least one camera 24 (e.g., camera sensor 58).

[0360] As mentioned above, to reduce the occurrence of camera 24 detecting many false positive spots, processor 96 may set a threshold, e.g., an intensity threshold, such that any detected feature, e.g., spot 33', below the threshold is not considered by the correspondence algorithm. Thus, for example, the three-dimensional position of spot 33 corresponding to other projector light beam 88' may not have been calculated in step 260 because detected spot 33' was below the threshold intensity. In step 272, to search for features, e.g., detected spot 33', the processor may lower the threshold to consider features not originally considered by the correspondence algorithm.

[0361] As used throughout this application, including the claims, when a search space is specified to search for a detected feature, for example, a detection spot 33', it may be the following case. (a) a falsely mis-detected spot (e.g., a spot below the threshold that was not initially considered by the correspondence algorithm, or a spot blocked by moving tissue), in which case processor 96 may lower the threshold to re-search that particular region, i.e., the identified search space, for a detected spot 33′; or (b) a false positive spot, i.e., multiple candidate three-dimensional locations for a spot identified by the correspondence algorithm, in which case the processor 96 may determine which spot is the correct spot based on a re-search of that particular region, i.e., the identified search space, for the detected spot 33'.

[0362] In some applications, in step 260, the correspondence algorithm may identify multiple candidate three-dimensional locations for the detection spot 33' from the projector ray 88'. For projector rays 88' for which multiple candidate three-dimensional locations for the detection spot 33' have been identified, as indicated by decision hexagon 267, processor 96 may estimate a three-dimensional location in the intersection space of the projector ray 88' and the estimated three-dimensional surface (step 269). In step 271, processor 96 selects which of the candidate three-dimensional locations for the detection spot 33' from the projector ray 88' is the correct location based on the three-dimensional location of the intersection of the projector ray 88' with the estimated three-dimensional surface.

[0363] In some applications, in step 262, processor 96 uses data corresponding to the three-dimensional position of each of at least three detection spots 33′ that were all captured in one of the multiple images. Further, after the three-dimensional surface has been estimated, the estimate may be refined by adding data points from a subsequent image, i.e., using data corresponding to the three-dimensional position of at least one additional spot whose three-dimensional position was calculated based on another image of the multiple images, such that all spots (the three used in the original estimate and the at least one additional spot) are on the refined estimated three-dimensional surface. In some applications, in step 262, processor 96 uses data corresponding to the three-dimensional position of each of at least three detection spots 33′ that were each captured in another image, i.e., each image of the multiple images.

[0364] It should be noted that the above discussion refers to false positive projection spots. It is also possible that a false positive projection spot is a true false positive, where a spot corresponding to a particular projector beam was not actually projected onto the intraoral surface due to occlusion by moving tissue, such as the patient's tongue, the patient's cheek, or the practitioner's finger.

[0365] In one embodiment, a method for generating a digital three-dimensional image includes driving each structured light projector of one or more structured light projectors to project a pattern of light onto a three-dimensional intraoral surface and driving each camera of one or more cameras to capture a plurality of images, each image including at least a portion of the projected pattern, each camera including a camera sensor including an array of pixels. The method further includes using a processor to execute a correspondence algorithm to calculate, for each of the plurality of images, a respective three-dimensional position on the intraoral three-dimensional surface of a plurality of detected features of the projected pattern. The processor uses data corresponding to the respective three-dimensional positions of at least three features to estimate a three-dimensional surface on which all of the at least three features are located, each feature corresponding to a respective projector ray r. For projector rays r1 for which no three-dimensional position of a feature corresponding to the projector ray r1 was calculated or for which multiple three-dimensional positions of a feature corresponding to the projector ray r1 were calculated, the processor estimates a three-dimensional position within an intersection space between the projector ray r1 and the estimated three-dimensional surface. The processor then uses the estimated three-dimensional location in space to identify a se...

Claims

1. 1. A method for generating a digital three-dimensional image, comprising the steps of: activating each structured light projector of the one or more structured light projectors to project a pattern onto an intra-oral three-dimensional surface; driving each camera of the one or more cameras to capture a respective series of images, each image in each series of images including at least a portion of the projection pattern; Using the processor, for each camera, comparing images in the sequence of images captured by said camera; determining, for each camera, what portions of the projection pattern may be tracked across the sequence of images captured by the camera based on the comparison of the images in the sequence of images, where portions of the projection pattern that meet one or more matching criteria may be tracked across multiple images in the sequence of images for the camera; constructing a three-dimensional model of the intraoral three-dimensional surface based at least in part on the comparison of the images in the respective series of images in each camera of the one or more cameras; The method includes:

2. The step of using the processor may further include using the processor to: solving a correspondence algorithm for the tracked portion of the projected pattern in at least one image of at least one sequence of images; using the solved correspondence algorithm for the tracked portion in the at least one image of the at least one sequence of images to solve a correspondence algorithm for the tracked portion of the projected pattern in at least another image of the at least one sequence of images, and constructing the three-dimensional model using the solution of the correspondence algorithm; 2. The method of claim 1, comprising:

3. The step of using the processor may further include using the processor to:

2. The method of claim 1 , wherein solving a correspondence algorithm for the tracked portion of the projected pattern based on a position of the tracked portion in each image throughout at least one sequence of images and constructing the three-dimensional model comprises constructing the three-dimensional model using the solution of the correspondence algorithm.

4. The method of claim 1 , wherein the one or more structured light projectors project a spatially fixed pattern onto the one or more cameras.

5. The method according to claim 1 , wherein the projection pattern comprises a plurality of projected light spots, and the portion of the projection pattern corresponds to a projection spot s of the plurality of projected light spots.

6. The step of comparing the images of the series of images using the processor comprises: comparing said images of the sequence of images based on stored calibration values ​​indicative of (a) a camera light ray corresponding to each pixel on a camera sensor of each camera of the one or more cameras, and (b) a projector light ray corresponding to each projected light spot of the projected light spots from each structured light projector of the one or more structured light projectors, each projector light ray corresponding to a path of a respective pixel on at least one of the camera sensors; 6. The method of claim 5, wherein determining which portions of the projected pattern can be tracked comprises determining which of the projected spots s can be tracked across the sequence of images, each tracking spot s moving along a path of pixels corresponding to a respective projector ray r.

7. 7. The method of claim 6, wherein the step of using the processor further comprises the step of using the processor to determine, for each tracking spot s, a path p of a plurality of possible pixels on a given one of the cameras, where the path p corresponds to a respective plurality of possible projector rays r.

8. The step of using the processor further comprises: executing a correspondence algorithm using said processor; For each possible projector ray r of the plurality of possible projector rays r, Identifying how many other cameras have detected respective spots q corresponding to respective camera rays that intersect a projector ray r and the camera ray of a given one of the cameras corresponding to the tracking spot s on their respective pixel paths p1 corresponding to the projector ray r; Identifying a given projector ray r1 for which the most other cameras have detected a respective spot q; and identifying projector ray r1 as the particular projector ray r that generated the tracking spot s.

9. The step of using the processor may further include using the processor to: executing a correspondence algorithm to calculate a three-dimensional position of each of a plurality of detection spots on the intra-oral three-dimensional surface captured in at least one series of images; 7. The method of claim 6, comprising identifying in at least one image of the sequence of images that a detection spot originates from a particular projector ray r by identifying the detection spot as a tracking spot s moving along a path of the pixel corresponding to the particular projector ray r.

10. The step of using the processor may further include using the processor to: executing a correspondence algorithm to calculate a three-dimensional position of each of a plurality of detection spots on the intra-oral three-dimensional surface captured in at least one series of images; and (ii) excluding from consideration as points on the intraoral three-dimensional surface those spots that are identified as originating from a particular projector ray r based on the three-dimensional positions calculated by the correspondence algorithm, and that are not identified as a tracking spot s moving along a path of the pixel corresponding to the particular projector ray r.

11. The step of using the processor may further include using the processor to: executing a correspondence algorithm to calculate a three-dimensional position of each of a plurality of detection spots on the intra-oral three-dimensional surface captured in at least one series of images; and for a detection spot identified as originating from two different projector rays r based on the three-dimensional position calculated by the correspondence algorithm, identifying the detection spot as originating from one of the two different projector rays r by identifying the detection spot as a tracking spot s moving along one of the two different projector rays r.

12. The step of using the processor may further include using the processor to: executing a correspondence algorithm to calculate a three-dimensional position of each of a plurality of detection spots on the intra-oral three-dimensional surface captured in at least one series of images; and identifying a weak spot whose three-dimensional position was not calculated by the correspondence algorithm as a projected spot originating from a particular projector ray r by identifying the weak spot as a tracking spot s that moves along a path of the pixel corresponding to the particular projector ray r.

13. 7. The method of claim 6, wherein using the processor further comprises using the processor to calculate respective three-dimensional positions on the intra-oral three-dimensional surface of intersections of the projector ray r and the respective camera ray corresponding to the tracked spot s in each of the sequence of images in which spot s was tracked.

14. 5. The method of claim 1, wherein the three-dimensional model is constructed using a correspondence algorithm that uses, at least in part, the portions of the projected pattern that are determined to be trackable across at least one sequence of images.

15. The step of using the processor may further include using the processor to: determining parameters of a tracked portion of the projection pattern in at least two adjacent images from at least one sequence of images, the parameters being selected from the group consisting of a size of the portion, a shape of the portion, an orientation of the portion, an intensity of the portion, and a signal-to-noise ratio (SNR) of the portion; 5. The method according to claim 1 , further comprising predicting parameters of the tracking portion of the projection pattern in a subsequent image based on parameters of the tracking portion of the projection pattern in the at least two adjacent images.

16. 16. The method of claim 15, wherein using the processor further comprises using the processor to search for a portion of the projection pattern in the subsequent image having substantially the predicted parameters based on the predicted parameters of the tracked portion of the projection pattern.

17. 16. The method of claim 15, wherein the selected parameter is a shape of the portion of the projection pattern, and wherein using the processor further comprises using the processor to determine a search space in a next image in which to search for the tracking portion of the projection pattern based on a predicted shape of the tracking portion of the projection pattern.

18. 20. The method of claim 17, wherein determining the search space using the processor comprises using the processor to determine a search space in the subsequent image in which to search for the tracking portion of the projection pattern, the search space having a size and aspect ratio based on a size and aspect ratio of the predicted shape of the tracking portion of the projection pattern.

19. The selected parameter is the shape of the portion of the projection pattern, and the step of using the processor includes using the processor to: determining a velocity vector of the tracking portion of the projected pattern based on a direction and a distance that the tracking portion of the projected pattern has moved between the at least two adjacent images from at least one sequence of images; predicting a shape of the tracking portion of the projection pattern in a subsequent image in response to a shape of the tracking portion of the projection pattern in at least one of the at least two adjacent images; 16. The method of claim 15, comprising determining a search space in the subsequent image in which to search for the tracking portion of the projection pattern in response to a combination of (i) the determination of the velocity vector of the tracking portion of the projection pattern and (ii) the predicted shape of the tracking portion of the projection pattern.

20. The parameter is a shape of the portion of the projection pattern, and the step of using the processor further comprises using the processor to: determining a velocity vector of the tracking portion of the projected pattern based on a direction and a distance that the tracking portion of the projected pattern has moved between the at least two adjacent images from at least one sequence of images; predicting the shape of the tracking portion of the projection pattern in a subsequent image in response to determining the velocity vector of the tracking portion of the projection pattern; 16. The method of claim 15, comprising: determining a search space in the subsequent image in which to search for the tracking portion of the projection pattern in response to a combination of: (i) the determination of the velocity vector of the tracking portion of the projection pattern; and (ii) the predicted shape of the tracking portion of the projection pattern.

21. The step of using the processor includes using the processor to 21. The method of claim 20, comprising predicting the shape of the tracking portion of the projection pattern in the subsequent image in response to a combination of (i) the determination of the velocity vector of the tracking portion of the projection pattern and (ii) the shape of the tracking portion of the projection pattern in at least one of the two adjacent images.

22. The step of using the processor includes using the processor to determining a velocity vector of the tracked portion of the projected pattern based on a direction and a distance traveled by the tracked portion of the projected pattern between two successive images in at least one sequence of images; and determining a search space in a subsequent image in which to search for the tracked portion of the projected pattern in response to said determining the velocity vector of the tracked portion of the projected pattern.

23. An intraoral scanning system configured to perform the method of any one of claims 1 to 22, comprising: an intraoral scanner comprising the one or more structured light projectors and the one or more cameras; the processor; An intraoral scanning system comprising:

Images