Intraoral scanner system and method for superimposing two-dimensional images on three-dimensional model

Abstract

According to an embodiment, a handheld intraoral scanner system is disclosed. The hand-held intraoral scanner system includes a hand-held intraoral scanner configured to acquire light information reflected from a three-dimensional dental object during a scanning session, and further includes one or more processors operably connected to the hand-held intraoral scanner device. The one or more processors are configured to determine a position of a two-dimensional image of the dental object relative to a virtual three-dimensional model of the dental object using the first, second and third neural networks and an algorithm. The one or more processors are further configured to superimpose a two-dimensional image of a dental object on a corresponding dental object of the virtual three-dimensional model, or vice versa. The one or more processors are further configured to display, on the display, a two-dimensional image superimposed on the virtual three-dimensional model, or vice versa. The one or more processors are further configured to display diagnostic and / or health information from the two-dimensional image and the virtual three-dimensional model. According to an embodiment, a method for superimposing a two-dimensional image of a dental object on a virtual three-dimensional model of a dental object or vice versa is also disclosed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to determining the position of a two-dimensional image of a dental object relative to a virtual three-dimensional model of the dental object, and in particular to overlaying a two-dimensional image of a dental object on a virtual three-dimensional model of the dental object and simultaneously displaying information from the two-dimensional image and the virtual three-dimensional model. BACKGROUND

[0002] It can occasionally be necessary for a dentist to view diagnostic or health information from different information sources simultaneously, such as bite-wing X-rays, panoramic X-rays, ultrasound images, infrared images or virtual three-dimensional models. Such simultaneous viewing allows the dentist to efficiently and conveniently obtain more details about the dental condition of a patient. It is beneficial to be able to view diagnostic or health information associated with each type of information source simultaneously.

[0003] It would therefore be more efficient and convenient for a dentist to simultaneously view a bite-wing film (an X-ray image of a portion of a dental arch) of a patient and a virtual three-dimensional model of the dental arch of the patient.

[0004] It would even be more beneficial in some cases to view an image associated with a dental disease, such as a bite-wing film, overlaid on a virtual three-dimensional model of the dental arch of a patient suffering from the dental disease, or vice versa, such that the virtual three-dimensional model is overlaid on the bite-wing film.

[0005] The expression "overlaying" should be understood as "placing on top of or above something", so that overlaying an X-ray image of a tooth on a virtual three-dimensional model of a dental arch comprising the tooth should be understood as placing the X-ray image of the tooth on top of the same tooth (corresponding tooth) of the virtual three-dimensional model.

[0006] It is a common practice to overlay medical two-dimensional images, such as bite-wing images, panoramic X-ray images, CBCT scan images and infrared images, on virtual three-dimensional models of a dental arch or a portion of a dental arch, with the aim of allowing a dentist to view diagnostic or health information from the medical two-dimensional images and the virtual three-dimensional model.

[0007] Overlaying an image on a virtual three-dimensional model requires knowing the image taking angle (for example when the image is a bite-wing film) or the data points of the image (for example when the image is from a CBCT scan) in order to overlay the image on the virtual three-dimensional model in the corresponding position.

[0008] However, in some cases, information about the image taking angle or data point is not available. For example, when a dental practitioner receives an X-ray image (e.g. bite wing) of a patient taken at an earlier date and at another place, and if the dental practitioner wishes to see the X-ray image superimposed on a virtual three-dimensional model of the patient's dentition in its corresponding position, such superimposition can not be possible because such image does not show or contain explicit information about the image taking angle or data point. In such a case, superimposing such an image on a virtual three-dimensional model in a corresponding position with sufficient accuracy for practical use can not be possible. SUMMARY

[0009] One aspect of the present disclosure is to allow superimposition of an arbitrary image of a dental object onto a virtual three-dimensional model of the dental object.

[0010] Another aspect of the present disclosure is to allow a dental practitioner to efficiently and conveniently view an arbitrary image of a dental object superimposed on a virtual three-dimensional model of the dental object, and at the same time view diagnostic or health information from the image and the virtual three-dimensional model.

[0011] According to an aspect, a hand-held intraoral scanner system configured to determine a position of a segmented two-dimensional image of a dental object relative to a segmented virtual three-dimensional model of the dental object is disclosed. The hand-held intraoral scanner system can include a hand-held intraoral scanner that can be configured to acquire light information reflected from a three-dimensional dental object during a scan session, wherein the scan session is a period of time in which scanning is performed using the hand-held intraoral scanner. The light information can be intraoral scan data and can be configured to be used to generate or update a virtual three-dimensional model of the dental object. The dental object can be a tooth, a portion of a tooth, a plurality of teeth, a maxilla or a mandible or a portion thereof, an entire dentition or a portion thereof, and / or a gum or a portion thereof.

[0012] The hand-held intraoral scanner system can further include a two-dimensional image of the dental object. The two-dimensional image can be an arbitrary image showing the dental object or a portion thereof. The two-dimensional image can be, for example, an X-ray image (e.g. bite wing or two-dimensional panoramic dental image), an infrared or near-infrared image, an ultrasound image, a photographic image, etc. The two-dimensional image can also be obtained by the hand-held intraoral scanner, a camera, a second hand-held intraoral scanner, or an extraoral scanner. The two-dimensional image can be obtained by an infrared or near-infrared image capturing device, such as a hand-held infrared scanner or a hand-held near-infrared scanner. The system can further include a memory unit configured to load the two-dimensional image of the dental object.

[0013] The handheld intraoral scanner system can further comprise one or more processors. The one or more processors can be operatively connected to the handheld intraoral scanner. The one or more processors can be configured to determine surface information from the light information or intraoral scan data in real time and generate a virtual three-dimensional model (three-dimensional surface model) of the dental object using the surface information. The one or more processors can comprise a processor having one or more processor cores, such as a CPU (Central Processing Unit). The one or more processors can comprise more than one processor, such as a plurality of CPUs, e.g. a processing cluster, wherein each of the plurality of CPUs comprises one or more processor cores. The handheld intraoral scanner and the one or more processors can be separate entities, which allows for processing of the light information or intraoral scan data outside of the intraoral scanner and can thereby allow for the use of remote resources or can allow for cloud-based processing.

[0014] The one or more processors may, for example, be located entirely or partially inside the handheld intraoral scanner, can be located entirely or partially in a laptop computer, a desktop computer, a tablet computer, a smartphone or a smart television, or can be located entirely or partially remotely in a server for cloud-based computing and be operatively connected to the handheld intraoral scanner by a cable, via a router over a wireless network or over the Internet. The one or more processors can also be distributed between two or more of the above-mentioned locations. For example, some of the one or more processors can be located in the handheld intraoral scanner in the form of a CPU (Central Processing Unit), while others of the one or more processors can be located in a desktop computer next to the handheld intraoral scanner, e.g. in the same dental practice, while others of the one or more processors can be located in a remote server in the "cloud" as a cloud-based computing service and be connected to the handheld intraoral scanner or the desktop computer over the Internet.

[0015] The one or more processors can also be configured to segment the two-dimensional image of the dental object. The system can include a first neural network trained to segment any dental object of a two-dimensional image. The one or more processors are configured to segment the two-dimensional image using the first neural network. The segmentation can be performed using the first neural network, which can be trained to segment any dental object in an image. Such training can allow the first neural network to recognize dental objects depicted in an image, and can allow the first neural network to identify any dental object in any image and segment that image. The system can also be configured to identify the dental object by determining a tooth number or tooth numbers in the segmented two-dimensional image according to a universal tooth numbering system.

[0016] The first neural network can segment the two-dimensional image by any image segmentation method known in the art, such as thresholding, clustering, edge detection, or an image segmentation neural network, such as an impulse-coupled neural network or a convolutional neural network, such as a U-net.

[0017] Segmenting the image can be beneficial in that a pattern of similar dental objects in the image can be identified.

[0018] For example, the first neural network can segment an X-ray image of 4 teeth of the upper jaw and 4 teeth of the lower jaw using any of the methods described above, resulting in a segmentation of the X-ray image. Such segmentation can be beneficial in that it can allow each tooth in the X-ray image to be recognized and identified.

[0019] The one or more processors can also be configured to obtain a first set of two-dimensional information of the segmented two-dimensional image that has been segmented using the first neural network. The first set of two-dimensional information can be data points. The data points can include two-dimensional coordinates. The first set of two-dimensional information can be pixels. Thus, the one or more processors can be configured to determine data points or pixels from the segmented two-dimensional image of the dental object, and for each pixel determine which portion of the image the pixel is associated with. Such determination can be made by the first neural network by recognizing the outline of the dental object in the two-dimensional image and determining which pixels exist within the dental object as defined by the recognized outline.

[0020] Such a determination is beneficial in allowing an element in a two-dimensional image to be selected separately and independently from other elements in the two-dimensional image, and can allow subsequent modification of the element.

[0021] The one or more processors can be configured to determine pixels from the segmented X-ray image and determine for each pixel which tooth the pixel is associated with. Thus, a set of pixels depicting a particular tooth can be selected and modified, thereby selecting and modifying the tooth separately and independently from the rest of the teeth shown in the X-ray image, for example by modifying the position, orientation and / or size of a single pixel, a set of pixels, a tooth, a portion of a tooth, multiple teeth, a whole dentition or a portion thereof.

[0022] The one or more processors can be configured to segment a virtual three-dimensional model of a dental object. The system can further comprise a second neural network which can be trained to segment any dental object of a virtual three-dimensional model. The one or more processors can be configured to segment the virtual three-dimensional model using the second neural network. Segmentation can be performed using the second neural network trained to segment any dental object in a virtual three-dimensional model. The second neural network can be trained by feeding a large number of virtual three-dimensional models of dental objects to the algorithm of the second neural network and feeding to the algorithm of the second neural network values or information of each of the virtual three-dimensional models and / or dental objects in the virtual three-dimensional models, for example teeth, portions of teeth, gums, portions of gums, upper jaw, lower jaw, whole dentition, dental appliances such as braces or aligners, etc., such as whether a tooth, teeth, portions of teeth, portions of dentition, dentition, dental appliances such as braces or aligners, or portions of gums are located in the virtual three-dimensional model. Such training can allow the second neural network to recognize dental objects represented in a virtual three-dimensional model and can allow the second neural network to identify any dental object in a virtual three-dimensional model and segment the virtual three-dimensional model. The system can further be configured to identify a dental object by determining one tooth number or multiple tooth numbers in the virtual three-dimensional model according to a universal tooth numbering system.

[0023] The second neural network can segment a virtual three-dimensional model of a dental object using any known three-dimensional segmentation method, for example polygon triangulation, spatial scanning, surface decomposition, etc.

[0024] Segmenting a virtual three-dimensional model of a dental object is beneficial in that a virtual three-dimensional model containing one or more portions similar to a dental object can be identified and divided into individual portions. Thus, each individual portion of a dental object in a virtual three-dimensional model can be identified.

[0025] For example, a virtual three-dimensional model of a complete dentition comprising a portion of a maxilla, a mandible, teeth, and gingiva can be segmented by the second neural network using any of the above methods, resulting in a segmentation of the virtual three-dimensional model. Such segmentation can be beneficial in that it can allow each tooth and gingiva to be individually determined, identified, and identified in the virtual three-dimensional model.

[0026] Such determination, identification, and identification can be beneficial in that it allows an element in the virtual three-dimensional model of the dental object to be individually selected and distinguished from the rest of other elements in the virtual three-dimensional model, and can allow subsequent modification of the element.

[0027] The one or more processors can be configured to determine an initial position of the segmented two-dimensional image of the dental object relative to the segmented virtual three-dimensional model of the dental object. The system can further comprise a third neural network trained to determine an initial position of any segmented two-dimensional image of a dental object relative to a corresponding dental object of a segmented virtual three-dimensional model. The one or more processors can be configured to determine the initial position of the segmented two-dimensional image relative to the segmented virtual three-dimensional model using the third neural network.

[0028] The initial position of the segmented two-dimensional image can be determined relative to a corresponding dental object in the segmented virtual three-dimensional model corresponding to the dental object of the segmented two-dimensional image.

[0029] The initial position can be determined using a third neural network trained to determine an initial position of any segmented two-dimensional image of a dental object relative to a segmented virtual three-dimensional model of the dental object.

[0030] The third neural network can be trained by feeding a large number of segmented two-dimensional images of dental objects and a large number of segmented virtual three-dimensional models of dental objects to an algorithm of the third neural network.

[0031] Information can also be fed to the third neural network regarding a location in each of a large number of segmented virtual three-dimensional models containing each dental object in each of a large number of segmented two-dimensional images to which each dental object belongs.

[0032] The one or more processors can be configured to arrange the segmented two-dimensional image relative to the segmented virtual three-dimensional model by performing a positioning of a two-dimensional image plane relative to the segmented virtual three-dimensional model, or vice versa, by using the third neural network. The two-dimensional image plane can be a plane in a space containing the segmented two-dimensional image.

[0033] The third neural network can perform a first guess of an initial position relative to the segmented virtual three-dimensional model, which can be based on an approximate position of the dental object of the segmented two-dimensional image relative to a corresponding dental object corresponding to the dental object present in the segmented two-dimensional image. The third neural network can be trained to perform the first guess to determine an initial position of any dental object of the segmented two-dimensional image relative to any corresponding dental object of the segmented virtual three-dimensional model.

[0034] Determining the initial position of the segmented two-dimensional image of the dental object or the dental object in the segmented two-dimensional image relative to the virtual three-dimensional model of the dental object can allow for a more efficient arrangement, positioning, superimposition, etc. as the one or more processors can position the segmented two-dimensional image of the dental object or the dental object of the segmented two-dimensional image in a position relatively close to and in the vicinity of the corresponding dental object in the virtual three-dimensional model of the dental object using the third neural network.

[0035] The expressions “relatively close” and “in the vicinity” are to be understood such that the segmented two-dimensional image or the dental object in the segmented two-dimensional image can be positioned closer to the maxilla or mandible in which the corresponding dental object is located in the virtual three-dimensional model than to another maxilla or mandible and also closer to the right or left side of the virtual three-dimensional model in which the corresponding dental object is located than to another right or left side of the virtual three-dimensional model.

[0036] Thereby, a superimposition requiring less processing steps or processing power can be achieved compared to an initial positioning of a non-segmented two-dimensional image relative to a virtual three-dimensional model and vice versa.

[0037] The expression “corresponding dental object” is to be understood as referring to the same dental object in the virtual three-dimensional model as in the two-dimensional image of the dental object and vice versa. The expression can also be understood as referring to the position of the dental object in the virtual three-dimensional model being the same as the position of the dental object in the two-dimensional image and vice versa.

[0038] For example, the dental object in the two-dimensional image can be the first molar of the maxilla and right side of the patient. The corresponding dental object in the virtual three-dimensional model of the dental object (corresponding to the dental object described above) is also the first molar of the maxilla and right side of the patient in the virtual three-dimensional model.

[0039] In case the first molar of the maxilla and right side of the patient in the two-dimensional image of the example described above is extracted after the two-dimensional image is taken and replaced with, for example, a dental implant, the corresponding dental object in the virtual three-dimensional model subsequently obtained will be the dental implant.

[0040] In another case, the first molar of the above example is extracted after the two-dimensional image is taken, but is not subsequently replaced with a dental implant or a dental prosthesis, leaving a blank space in the virtual three-dimensional model, the corresponding dental object in the subsequently obtained virtual three-dimensional model refers to the location where the first molar should or would have been.

[0041] In yet another case, the dental object in the two-dimensional image is on the left side of the mandible, and the corresponding dental object in the virtual three-dimensional model is also on the left side of the mandible in the virtual three-dimensional model.

[0042] For example, given a segmented X-ray image of a second molar on the left upper side of a patient’s dentition, and a segmented virtual three-dimensional model of the patient’s upper jaw that includes the second molar, the one or more processors can be configured to determine, using a third neural network, an initial location of the second molar of the segmented X-ray image relative to the segmented virtual three-dimensional model by performing a first guess. The third neural network can determine that what is depicted in the segmented X-ray image is a second molar on the left side of the dentition. The third neural network may, for example, further determine that the second molar is on the patient’s upper jaw by identifying the surrounding contours of the second molar, such as other teeth or gums. The third neural network can then determine a second molar on the left upper side of the virtual three-dimensional model, and determine a location near the second molar.

[0043] The one or more processors can also be configured to obtain a two-dimensional projection of the segmented virtual three-dimensional model of the dental object. The two-dimensional projection can contain two-dimensional information from the segmented virtual three-dimensional model of the dental object, which is referred to as a second set of two-dimensional information in the disclosure below. The one or more processors can also be configured to obtain the second set of two-dimensional information of the two-dimensional projection of the segmented virtual three-dimensional model of the dental object. The one or more processors can be configured to obtain the two-dimensional projection using an algorithm. The algorithm can also be configured to obtain the two-dimensional projection of the segmented virtual three-dimensional model of the dental object. The algorithm can also be configured to obtain or determine the second set of two-dimensional information. The second set of two-dimensional information can be the two-dimensional information of the two-dimensional projection.

[0044] Obtaining the two-dimensional projection of the segmented virtual three-dimensional model of the dental object from the initial location can allow obtaining a second set of two-dimensional information associated with the segmented virtual three-dimensional model from a location near the corresponding dental object in the segmented virtual three-dimensional model, which can be used in subsequent processes that can enable a faster and more efficient superimposition process.

[0045] The two-dimensional projection can be an instant image of the segmented virtual three-dimensional model of the dental object from the initial position. Thus, the two-dimensional projection can be a screenshot of the segmented virtual three-dimensional model of the dental object viewed from the initial position. Thus, the two-dimensional projection can contain two-dimensional information, which can be pixels.

[0046] The one or more processors can be configured to use an algorithm to take an instant image, e.g. a screenshot, of the segmented virtual three-dimensional model of the dental object from the initial position determined using the third neural network. The algorithm can also be configured to obtain a second set of two-dimensional information, which can be pixels of the screenshot (instant image).

[0047] For example, the one or more processors can forward information about the determined initial position obtained using the third neural network to the algorithm. The one or more processors can then use the algorithm to take a screenshot (instant two-dimensional image) of the segmented virtual three-dimensional model of the dental object viewed from the determined initial position, thereby obtaining a two-dimensional projection (screenshot) of the segmented virtual three-dimensional model, which contains two-dimensional information (second set of two-dimensional information), which is pixels of the screenshot (instant image, two-dimensional projection).

[0048] The two-dimensional projection can be a set of two-dimensional information, which can be a set of two-dimensional data points of the projection of the segmented virtual three-dimensional model onto the projection plane at the determined initial position.

[0049] The one or more processors can be configured to forward information about the determined initial position obtained using the third neural network to the algorithm. The algorithm can be configured to use the forwarded information to arrange the projection plane at the determined initial position. The algorithm can also be configured to extract two-dimensional data points from the segmented virtual three-dimensional model of the dental object and arrange them in corresponding positions on the projection plane. The algorithm can also be configured to assign a depth value to each arranged two-dimensional data point on the projection plane, such that the projection plane contains two-dimensional data points with two-dimensional coordinates and depth values. The one or more processors can be configured to obtain the two-dimensional projection of the segmented virtual three-dimensional model on the projection plane by taking another two-dimensional image of the segmented virtual three-dimensional model from the determined initial position.

[0050] For example, the one or more processors can forward information about the determined initial position obtained using the third neural network to the algorithm. The algorithm can use the forwarded information to arrange the projection plane at the determined initial position. The algorithm can then extract two-dimensional data points from the segmented virtual three-dimensional model of the dental object and arrange them in corresponding positions in the projection plane. The algorithm can then assign a depth value to each arranged data point on the projection plane.

[0051] The one or more processors can be configured to align the first set of two-dimensional information of the segmented two-dimensional image with a second set of two-dimensional information of the two-dimensional projection of the segmented virtual three-dimensional model. The alignment can be performed using the algorithm. The algorithm can be configured to arrange the first set of two-dimensional information on the second set of two-dimensional information. The algorithm can also be configured to arrange the segmented two-dimensional image on the two-dimensional projection. The algorithm can also be configured to align the first set of two-dimensional information with the second set of two-dimensional information by changing the position, orientation and / or size of the first set of two-dimensional information of the two-dimensional image. The one or more processors can be configured to modify the position, orientation and / or size of the segmented virtual three-dimensional model using the algorithm.

[0052] Aligning the first set of two-dimensional information with the second set of two-dimensional information or aligning the segmented two-dimensional image with the two-dimensional projection of the segmented virtual three-dimensional model can allow to determine which two-dimensional information (pixels or data points) between the segmented two-dimensional image and the two-dimensional projection coincide with each other.

[0053] The one or more processors can be configured to determine a first number of coinciding two-dimensional information between the first set of two-dimensional information and the second set of two-dimensional information. The one or more processors can be configured to determine the first number of coinciding two-dimensional information using the algorithm. The algorithm can be configured to determine the first number of coinciding two-dimensional information.

[0054] Determining the first number of coinciding two-dimensional information between the segmented two-dimensional image and the two-dimensional projection can allow to determine whether the initial position is acceptable for superimposing the segmented two-dimensional image on the segmented virtual three-dimensional model or whether a different position of the segmented two-dimensional image relative to the segmented virtual three-dimensional model is needed.

[0055] The expression “different position” or “another position” is to be understood in the present disclosure in relation to another position, orientation, size, shape or direction.

[0056] The algorithm can be configured to determine the first number of coinciding two-dimensional information. The first number of coinciding two-dimensional information can be two-dimensional information from the first set of two-dimensional information and two-dimensional information from the second set of two-dimensional information which are identical, similar, have the same color, have the same size, have the same graphical content, have the same position and / or orientation, or are arranged within the same segmented dental object, for example within the same dental object contour.

[0057] For example, the algorithm can obtain a segmented two-dimensional image of the tooth, where a first set of pixels is within the outline of the tooth; and another image, which is a screenshot of a virtual three-dimensional model of the tooth taken from an initial position, and where a second set of pixels is within the outline of the tooth in the screenshot. The algorithm determines or identifies the pixels belonging to the first set of pixels in the segmented two-dimensional image and the pixels belonging to the second set of pixels in the screenshot. Then, the algorithm determines the number of coinciding pixels by determining how many pixels of the two sets of pixels (the first set of pixels and the second set of pixels) are within the tooth depicted in both images (the segmented two-dimensional image and the screenshot).

[0058] The algorithm can be configured to determine the coinciding pixels or segmented objects in different images using one of a plurality of known methods, for example, an “intersection over union” (IoU) method, in which the degree of overlap between objects in two images is detected, or the probability that objects in two segmented images intersect is determined.

[0059] The one or more processors can also be configured to repeat determining another number of coinciding two-dimensional information between the first set of two-dimensional information and the second set of two-dimensional information. The other number of coinciding two-dimensional information can be determined using different positions of the segmented two-dimensional image relative to the segmented virtual three-dimensional model. The one or more processors can be configured to repeat determining the other number of coinciding two-dimensional information until the other number of coinciding two-dimensional information reaches a predetermined number of coinciding two-dimensional information.

[0060] The one or more processors can repeat determining the other number of coinciding two-dimensional information and determining different positions using the algorithm. The algorithm can be configured to determine the different positions. The algorithm can also be configured to repeat determining the other number of coinciding two-dimensional information.

[0061] Thus, the algorithm can be configured to obtain a two-dimensional projection and a second set of two-dimensional information, align the first set of two-dimensional information with the second set of two-dimensional information, and determine a first number of coinciding two-dimensional information, and repeat determining another number of coinciding two-dimensional information between the first set of two-dimensional information and the second set of two-dimensional information using different positions of the segmented two-dimensional image relative to the segmented virtual three-dimensional model until the other number of coinciding two-dimensional information reaches a predetermined number of coinciding two-dimensional information.

[0062] For example, the algorithm can determine four different positions, one to the right of the initial position, one to the left of the initial position, one above the initial position, and one below the initial position. For each of the four different positions, the above process of determining another number of coinciding two-dimensional information is repeated, thereby determining four sets of another number of coinciding two-dimensional information. The algorithm can then determine which of the four sets of another number of coinciding two-dimensional information is the largest number, and based on the largest number determine where to place the next different position, such that a larger number of coinciding two-dimensional information can be obtained for each repetition.

[0063] The predetermined number can be a fixed number, a number based on user input, or a calculated number, such as a ratio. The one or more processors or algorithm are configured to terminate the process of determining the another number of coinciding two-dimensional information when the another number of coinciding two-dimensional information reaches or exceeds the predetermined number. The predetermined number can be a number of pixels, an exponent, or a ratio.

[0064] The one or more processors can further be configured to overlay the segmented two- dimensional image on the segmented virtual three-dimensional model. The one or more processors can be configured to overlay the segmented two-dimensional image on the segmented virtual three-dimensional model at a position where the another number of coinciding two-dimensional information has reached the predetermined number of coinciding two-dimensional information.

[0065] The overlaying can be performed by placing the segmented two-dimensional image of the dental object on the segmented virtual three-dimensional model of the dental object, or vice versa, where the segmented virtual three-dimensional model of the dental object is placed on the segmented two-dimensional image of the dental object. In another embodiment, only a portion of the segmented two-dimensional image is placed on the segmented virtual three-dimensional model, or vice versa, where only a portion of the segmented virtual three-dimensional model is placed on the segmented two-dimensional image.

[0066] For example, only one dental object of the segmented two-dimensional image of the dental object is placed on the corresponding dental object of the segmented virtual three-dimensional model of the dental object, or vice versa.

[0067] The overlaying can further mean that the segmented two-dimensional image overlaid on the segmented virtual three-dimensional model can be made partially transparent, such that the dental object in the segmented two-dimensional image is visible and the dental object of the segmented virtual three-dimensional model is visible, or vice versa, where the dental object of the segmented virtual three-dimensional model overlaid on the segmented two-dimensional image can be made partially transparent, such that the dental object of the segmented virtual three-dimensional model can be visible and the dental object of the segmented two-dimensional image can be visible.

[0068] Superimposing the segmented two-dimensional image on the segmented virtual three-dimensional model at the location where the amount of coinciding two-dimensional information has reached the predetermined amount of coinciding two-dimensional information can allow superimposition with an optimal amount of coinciding two-dimensional information, and thus can be a sufficiently accurate superimposition.

[0069] In instances where the two-dimensional information is pixels and the predetermined amount of coinciding pixels is a first particular amount, the one or more processors determine, using a third neural network, another position of the segmented two-dimensional image relative to the segmented virtual three-dimensional model. In a subsequent process, the one or more processors use the algorithm to obtain, from the determined another position, a new screenshot of the segmented virtual three-dimensional model, obtain a second set of pixels from the screenshot, align the segmented two-dimensional image on the screenshot, and determine a second particular amount of coinciding pixels between the segmented two-dimensional image and the screenshot, where the second particular amount of coinciding pixels is greater than the first particular amount. The one or more processors use the algorithm to determine that the amount of coinciding pixels is greater than the predetermined amount of coinciding pixels, and superimpose the segmented two-dimensional image on the segmented virtual three-dimensional model at the determined another position using the determined another position.

[0070] The system can include a display. The system can also be configured to display, on the display, the superimposed segmented two-dimensional image on the segmented virtual three-dimensional model, or the superimposed segmented virtual three-dimensional model on the segmented two-dimensional image. The system can also be configured to display, on the display, diagnostic and / or health data from the two-dimensional image and from the intraoral scan data (or virtual three-dimensional model) simultaneously.

[0071] The diagnostic and / or health data can include caries (cavities and bad teeth), gum disease (gingivitis, periodontal disease, and peri-implant disease), bone loss, tooth wear, tooth cracks and fractures, plaque, oral disease (cancer), etc.

[0072] A dentist, for example, can view, on the display, a bite-wing film (X-ray image) of a patient’s dentition superimposed on a virtual three-dimensional model of the patient’s dentition. The dentist can view gum infection from the bite-wing film, and can view tooth wear on the patient’s teeth from the virtual three-dimensional model. Thus, the dentist is able to efficiently and conveniently view multiple diagnostic and health information associated with the patient’s dentition simultaneously without having to switch between bite-wing film view and virtual three-dimensional model view. Furthermore, the dentist does not need to acquire a bite-wing film that contains information indicative of a bite-wing film angle, but can use any bite-wing film of the patient’s dentition.

[0073] According to this aspect, there is provided a method for determining a position of a segmented two-dimensional image of a dental object relative to a segmented virtual three-dimensional model of the dental object.

[0074] The two-dimensional image can be any image showing the dental object or a part thereof. For example, the two-dimensional image can be an X-ray image (e.g. a bite wing or a two-dimensional panoramic dental image), an infrared (IR) image, a near infrared (NIR) image, an ultrasound image, a photographic image or any arbitrary image.

[0075] The dental object can be a tooth, a part of a tooth, a plurality of teeth, a maxilla or a mandible or a part thereof, a whole dentition or a part thereof, and / or a gum or a part thereof.

[0076] The virtual three-dimensional model can be generated based on light information of a three-dimensional scanner. The three-dimensional scanner can be a lab scanner configured to scan a dental impression or can be a handheld intraoral scanner. The light information can be scanning data, e.g. intraoral scanning data.

[0077] The two-dimensional image of the dental object can also be obtained by a handheld intraoral scanner, a camera, a second handheld intraoral scanner or an extraoral scanner. On the other hand, the two-dimensional image of the dental object can be acquired by loading the two-dimensional image from a memory unit.

[0078] The method can comprise a step of determining, using one or more processors, a position of the segmented two-dimensional image of the dental object relative to the segmented virtual three-dimensional model of the dental object.

[0079] The one or more processors can be operatively connected to the handheld intraoral scanner. The one or more processors can be configured to determine surface information in real-time from the light information or the intraoral scanning data and to generate a virtual three-dimensional model (three-dimensional surface model) of the dental object using the surface information. The one or more processors can comprise one processor having one or more processor cores, e.g. a CPU (Central Processing Unit). The one or more processors can comprise more than one processor, e.g. a plurality of CPUs, e.g. a processing cluster, wherein each of the plurality of CPUs comprises one or more processor cores. The handheld intraoral scanner and the one or more processors can be separate entities, which allows for processing of the light information or the intraoral scanning data outside the intraoral scanner and thereby can allow for using remote resources or can allow for cloud-based processing.

[0080] From this, the one or more processors can be located entirely or partially inside the handheld intraoral scanner, can be located entirely or partially in a laptop computer, a desktop computer, a tablet computer, a smartphone or a smart television, or can be located entirely or partially remotely in a server for cloud-based computing and operatively connected to the handheld intraoral scanner by a cable, via a router through a wireless network or through an internet connection. The one or more processors can also be distributed between two or more of the above-mentioned locations. For example, some of the one or more processors can be located in the handheld intraoral scanner in the form of CPUs (Central Processing Units), some of the one or more processors can be located in a desktop computer next to the handheld intraoral scanner, for example in the same dentist's office, and some of the one or more processors can be located in a remote server in the "cloud" as cloud-based computing and connected to the handheld intraoral scanner or to the desktop computer through an internet connection.

[0081] The method can comprise a step of acquiring a virtual three-dimensional model of the dental object based on light information of the dental object, for example intraoral scan data of the dental object. The virtual three-dimensional model of the dental object can be acquired by loading the virtual three-dimensional model from a file, by downloading the virtual three-dimensional model from a network such as the internet, by performing an intraoral three-dimensional scan of the dental object or by performing a laboratory scan of a dental impression comprising the dental object. Acquiring the virtual three-dimensional model can be performed using the one or more processors.

[0082] The method can further comprise a step of acquiring a two-dimensional image of the dental object. The two-dimensional image can be an X-ray image (for example a bite-wing or panoramic X-ray), an infrared (IR) image, a near-infrared (NIR) image, an ultrasound image, a photographic image or any other two-dimensional image. The two-dimensional image of the dental object can be acquired by loading the two-dimensional image from a file, by downloading the two-dimensional image from a network such as the internet or by taking an instant image of the dental object for example with a camera or by a screen capture.

[0083] The method can further comprise a step of segmenting the two-dimensional image of the dental object. The two-dimensional image can be segmented using any first neural network trained to segment images of any dental object. The first neural network can be trained by feeding a large number of images of dental objects (e.g. teeth, parts of teeth, gums, and parts of gums) to the algorithm of the first neural network, and feeding the algorithm of the first neural network values or information of each image and / or dental object in the image, regardless of what is in the image, a tooth, multiple teeth, parts of teeth, parts of dentition, dentition, dental appliances such as braces or mouth guards, or parts of gums, etc. Such training can allow the first neural network to recognize dental objects depicted in the image, and can allow the first neural network to identify any dental object in any image and segment the image. The dental object can be identified by determining a tooth number or multiple tooth numbers according to a universal tooth numbering system.

[0084] The first neural network can segment the two-dimensional image by any image segmentation method known in the art, such as thresholding, clustering, edge detection, or image segmentation neural networks, such as pulse coupled neural networks or convolutional neural networks, such as U-nets.

[0085] Segmenting the image is beneficial in that it can allow identification of similar dental objects in the image.

[0086] For example, the first neural network can segment an X-ray image of 4 teeth of the upper jaw and 4 teeth of the lower jaw using the above described method without the need for numerical information of dental significance, thereby resulting in segmentation of the X-ray image. Such segmentation is beneficial in that it can allow identification and labeling of each tooth in the X-ray image.

[0087] The method can further comprise a step of obtaining a first set of two-dimensional information of the segmented two-dimensional image.

[0088] The first set of two-dimensional information can be obtained using the one or more processors.

[0089] The first set of two-dimensional information can be data points. The data points can comprise two-dimensional coordinates. In another aspect, the first set of two-dimensional information can be pixels. Thus, the one or more processors can be configured to determine data points or pixels from the segmented two-dimensional image of the dental object, and for each pixel or data point determine which part of the image the pixel or data point is associated with. Such determination can be made by the first neural network by recognizing the outline of the dental object in the two-dimensional image and determining which pixels or data points are present within the dental object as defined by the recognized outline.

[0090] Such a determination is beneficial in allowing the selection of an element in a two-dimensional image separately and independently from the rest of the other elements in the two-dimensional image, and can allow the modification of the element.

[0091] Continuing with the above example, the one or more processors can be configured to determine pixels from the segmented X-ray image, and for each pixel determine which tooth the pixel is associated with. Thus, a set of pixels depicting a particular tooth can be selected and modified, thereby selecting and modifying the tooth separately and independently from the rest of the teeth shown in the X-ray image, for example by modifying the position, orientation and / or size of a tooth, a portion of a tooth, multiple teeth, a whole dentition or a portion thereof, a single pixel or a set of pixels.

[0092] The method can comprise a step of segmenting the virtual three-dimensional model of the dental object. The virtual three-dimensional model can be segmented using a second neural network trained to segment any dental object of the virtual three-dimensional model. The second neural network can be trained by feeding a large number of virtual three-dimensional models of dental objects to the algorithm of the second neural network, and feeding to the algorithm of the second neural network the value or information of each of the virtual three-dimensional models and / or dental objects, such as teeth, portions of teeth, gums, portions of gums, upper jaw, lower jaw, whole dentition, dental appliances such as tooth aligners or braces, etc., regardless of the content of the virtual three-dimensional model being a tooth, multiple teeth, a portion of a tooth, a portion of a dentition, a dentition, a dental appliance such as a tooth aligner or braces, or a portion of a gum. Such training can allow the second neural network to recognize dental objects present in a virtual three-dimensional model, and can allow the second neural network to identify any dental object in any virtual three-dimensional model and segment the virtual three-dimensional model.

[0093] The second neural network can segment the virtual three-dimensional model of the dental object using any known three-dimensional segmentation method, such as polygonal triangulation, spatial scanning, surface decomposition, etc.

[0094] Segmenting the virtual three-dimensional model of the dental object is beneficial in that a virtual three-dimensional model containing one or more portions similar to a dental object can be recognized, identified and divided into individual portions. Thus, each individual portion of a dental object in a virtual three-dimensional model can be recognized.

[0095] For example, the second neural network can segment a virtual three-dimensional model of a complete dentition containing portions of an upper jaw, a lower jaw, teeth and gums, resulting in the segmentation of the virtual three-dimensional model. Such segmentation is beneficial in that each tooth and gum can be determined, recognized and identified in the virtual three-dimensional model.

[0096] A benefit of such determination, identification and labelling is to allow an element in the virtual three-dimensional model of the dental object to be selected individually and separated from the rest of the other elements in the virtual three-dimensional model, and can allow modification of this element.

[0097] Continuing with the above example of the virtual three-dimensional model, the one or more processors can be configured to segment the virtual three-dimensional model using a second neural network, and can be configured to modify a position, orientation and / or size of the segmented virtual three-dimensional model using an algorithm.

[0098] The method can further comprise a step of determining an initial position of the segmented two-dimensional image of the dental object relative to the segmented virtual three-dimensional model of the dental object.

[0099] The dental object can be a tooth, a plurality of teeth, a portion of a tooth, a portion of a dentition, a dentition, a dental appliance such as an orthodontic appliance or a mouthpiece, or a portion of a gum.

[0100] The initial position can be determined using a third neural network.

[0101] The determination of the initial position can be performed using a third neural network, which can be trained to determine an initial position of any segmented two-dimensional image of a dental object relative to a segmented virtual three-dimensional model of the dental object, or vice versa. The third neural network can be further trained to determine an initial position of a segmented two-dimensional image of a dental object relative to any corresponding dental object of a segmented virtual three-dimensional model, or vice versa. The third neural network can be trained by feeding its algorithm with a large number of segmented two-dimensional images of dental objects and a large number of segmented virtual three-dimensional models of dental objects, and can further be fed with information about which position of the segmented virtual three-dimensional model the segmented dental object in the two-dimensional image belongs to. The third neural network can perform a first guess of the initial position relative to the segmented virtual three-dimensional model, which can be based on an approximate position relative to a corresponding dental object that corresponds to the dental object present in the segmented two-dimensional image. Thus, the third neural network can be configured to determine an initial position of the dental object of the segmented two-dimensional image relative to the corresponding dental object of the segmented virtual three-dimensional model by performing the first guess. Thus, the method can comprise a step of determining an initial position of the segmented two-dimensional image relative to the corresponding dental object of the segmented virtual three-dimensional model that corresponds to the dental object of the segmented two-dimensional image.

[0102] Determining the initial position of the segmented two-dimensional image or the dental object in the segmented two-dimensional image relative to the virtual three-dimensional model of the dental object can allow for a more efficient arrangement, positioning, superimposition, etc. as the third neural network can position the segmented two-dimensional image or the dental object of the segmented two-dimensional image in relative close proximity to the position of the corresponding dental object in the virtual three-dimensional model and in its vicinity, or vice versa, such that the dental object of the segmented virtual three-dimensional model is in relative close proximity to the position of the corresponding dental object of the segmented two-dimensional image and in its vicinity.

[0103] The expressions "in relative close proximity" and "in vicinity" are to be understood such that the segmented two-dimensional image or the dental object in the segmented two-dimensional image can be positioned in closer proximity to the maxilla or mandible in which the corresponding dental object is located in the virtual three-dimensional model than to the other maxilla or mandible, and also in closer proximity to the right or left side of the virtual three-dimensional model in which the corresponding dental object is located than to the other right or left side of the virtual three-dimensional model.

[0104] Thereby, the superimposition can be achieved with less processing than with an initial positioning of a two-dimensional image not segmented relative to the virtual three-dimensional model, and vice versa.

[0105] The expression "corresponding dental object" is to be understood as referring to the same dental object in the virtual three-dimensional model as in the two-dimensional image of the dental object, or vice versa. The expression can also be understood as referring to the position of the dental object in the virtual three-dimensional model being the same as the position of the dental object in the two-dimensional image, or vice versa.

[0106] For example, the dental object in the two-dimensional image can be the first molar on the right side of the maxilla of the patient. The corresponding dental object in the virtual three-dimensional model of the dental object (corresponding to the dental object described above) is the first molar on the right side of the maxilla of the patient in the virtual three-dimensional model.

[0107] In the case where the first molar on the right side of the maxilla of the patient in the two-dimensional image of the above example is extracted after the two-dimensional image is taken, and replaced with e.g. a dental implant, said corresponding dental object is in this case the dental implant.

[0108] In another case, the first molar is extracted after the two-dimensional image is taken, but is not subsequently replaced with e.g. a dental implant or a dental prosthesis, leaving a blank space in the virtual three-dimensional model, said corresponding dental object refers to the position in which the first molar would have been in the virtual three-dimensional model.

[0109] In yet another case, the dental object in the two-dimensional image is on the left side of the mandible, the corresponding dental object in the virtual three-dimensional model is on the left side of the mandible in the virtual three-dimensional model of the dental object. In yet another case, the dental object in the two-dimensional image is on the left side of the mandible, the corresponding dental object in the virtual three-dimensional model is on the left side of the mandible in the virtual three-dimensional model of the dental object.

[0110] For example, given a segmented X-ray image of a second molar on the upper left side of a patient's dentition, and a segmented virtual three-dimensional model of the patient's maxilla including the second molar, the third neural network can determine an initial position of the second molar of the segmented X-ray image relative to the segmented virtual three-dimensional model by performing a first guess. The third neural network can determine that what is depicted in the segmented X-ray image is a second molar on the left side of the dentition. The third neural network may, for example, further determine that the second molar is on the patient's maxilla by identifying the surrounding profile of the second molar, such as other teeth or gums. The third neural network can then determine a second molar on the upper left side of the virtual three-dimensional model, and determine a position near that second molar.

[0111] The method can include a step of obtaining a two-dimensional projection of the segmented virtual three-dimensional model. The two-dimensional projection can be on a projection plane. Obtaining the two-dimensional projection can be performed by taking another two-dimensional image of the segmented virtual three-dimensional model viewed from the determined initial position.

[0112] The method can further include a step of obtaining a second set of two-dimensional information of the two-dimensional projection of the segmented virtual three-dimensional model.

[0113] The method can include a step of obtaining, using the one or more processors, a two-dimensional projection of the segmented virtual three-dimensional model of the dental object. The two-dimensional projection can contain two-dimensional information from the segmented virtual three-dimensional model of the dental object. The method can include a step of obtaining, using the one or more processors, a second set of two-dimensional information of the two-dimensional projection of the segmented virtual three-dimensional model of the dental object. The one or more processors can be configured to obtain the two-dimensional projection using an algorithm. The algorithm can be further configured to obtain the two-dimensional projection of the segmented virtual three-dimensional model of the dental object. The algorithm can be further configured to obtain or determine the second set of two-dimensional information. The second set of two-dimensional information can be the two-dimensional information of the two-dimensional projection.

[0114] Obtaining the two-dimensional projection of the segmented virtual three-dimensional model of the dental object from the initial position can allow for obtaining a second set of two-dimensional information associated with the segmented virtual three-dimensional model from a position near the segmented virtual three-dimensional model, which can be used in subsequent processes that can enable a faster and more efficient superimposition process.

[0115] The two-dimensional projection can be an instant image of the segmented virtual three-dimensional model of the dental object from the initial position. Thus, the two-dimensional projection can be a screenshot of the segmented virtual three-dimensional model of the dental object from the initial position. Thus, the two-dimensional projection can contain two-dimensional information, which can be pixels.

[0116] The one or more processors can be configured to use the algorithm to take a snapshot (e.g. a screenshot) of the segmented virtual three-dimensional model of the dental object from the determined initial position using the third neural network. The algorithm can also be configured to obtain a second set of two-dimensional information, which can be the pixels of the screenshot (snapshot).

[0117] For example, the one or more processors can forward the information obtained using the third neural network about the determined initial position to the algorithm. The one or more processors can then use the algorithm to obtain a screenshot (snapshot two-dimensional image) of the segmented virtual three-dimensional model of the dental object viewed from the determined initial position, thereby obtaining a two-dimensional projection (screenshot) of the segmented virtual three-dimensional model containing two-dimensional information, which is the pixels of the screenshot (snapshot, two-dimensional projection).

[0118] Alternatively, the two-dimensional projection can be a set of two-dimensional information, which can be a set of two-dimensional data points of the projection of the segmented virtual three-dimensional model onto the projection plane at the determined initial position.

[0119] The one or more processors can be used to forward the information obtained using the third neural network about the determined initial position to the algorithm. The algorithm can use the forwarded information to arrange the projection plane at the determined initial position. The algorithm can also be configured to extract two-dimensional data points from the segmented virtual three-dimensional model of the dental object and arrange them in corresponding positions on the projection plane. The algorithm can also be configured to assign a depth value to each arranged two-dimensional data point on the projection plane, such that the projection plane contains two-dimensional data points with two-dimensional coordinates and depth values.

[0120] For example, the one or more processors can forward the information obtained using the third neural network about the determined initial position to the algorithm. The algorithm can use the forwarded information to arrange the projection plane at the determined initial position. The algorithm can then extract two-dimensional data points from the segmented virtual three-dimensional model of the dental object and arrange them in corresponding positions in the projection plane. The algorithm can then assign a depth value to each arranged data point on the projection plane.

[0121] The method can also comprise a step of aligning the first set of two-dimensional information of the segmented two-dimensional image with the second set of two-dimensional information of the two-dimensional projection of the segmented virtual three-dimensional model.

[0122] The method can also comprise a step of determining a first number of coinciding two-dimensional information between the first set of two-dimensional information and the second set of two-dimensional information.

[0123] The one or more processors can align the first set of two-dimensional information with the second set of two-dimensional information using the algorithm. Thus, aligning the first set of two-dimensional information with the second set of two-dimensional information can be performed by the algorithm. The method can perform the alignment of the first set of two-dimensional information with the second set of two-dimensional information using the algorithm changing the position, orientation and / or size of the first set of two-dimensional information of the segmented two-dimensional image.

[0124] The algorithm can be configured to arrange the first set of two-dimensional information on the second set of two-dimensional information. The algorithm can also be configured to arrange the segmented two-dimensional image on the two-dimensional projection.

[0125] Aligning the first set of two-dimensional information with the second set of two-dimensional information or aligning the segmented two-dimensional image with the two-dimensional projection can allow determining which two-dimensional information between the segmented two-dimensional image and the two-dimensional projection coincide with each other.

[0126] A first number of coinciding two-dimensional information between the first set of two-dimensional information and the second set of two-dimensional information can be determined by the one or more processors. The one or more processors can be configured to determine the first number of coinciding two-dimensional information using the algorithm. The algorithm can be configured to determine the first number of coinciding two-dimensional information.

[0127] Determining the first number of coinciding two-dimensional information between the segmented two-dimensional image and the two-dimensional projection can allow determining whether the initial position is acceptable for superimposing the segmented two-dimensional image on the virtual three-dimensional model or whether a different position of the segmented two-dimensional image with respect to the segmented virtual three-dimensional model is needed.

[0128] The algorithm can be configured to determine the first number of coinciding two-dimensional information. The first number of coinciding two-dimensional information can be two-dimensional information from the first set of two-dimensional information and two-dimensional information from the second set of two-dimensional information that are identical, similar, have the same color, have the same size, have the same graphical content, have the same position and / or orientation, or are arranged within the same segmented dental object, for example within the same dental object contour.

[0129] For example, the algorithm can obtain a segmented two-dimensional image of a tooth, wherein a first set of pixels is within a contour of the tooth. The algorithm can also obtain another image, which is a screenshot of a virtual three-dimensional model of the tooth taken from an initial position, wherein a second set of pixels is within a contour of the tooth in the screenshot. The algorithm determines or identifies pixels belonging to the first set of pixels in the segmented two-dimensional image and pixels belonging to the second set of pixels in the screenshot. Then, the algorithm determines the number of coinciding pixels by determining how many pixels of these two sets of pixels (the first set of pixels and the second set of pixels) are within the tooth depicted in both images (the segmented two-dimensional image and the screenshot).

[0130] The method can use one of a variety of known methods to determine coinciding pixels or objects in different images, such as the "Intersection over Union" (IoU) method, in which the degree of overlap between objects in two images is detected, or the probability of objects in two segmented images intersecting is determined. The method can use the algorithm to perform the above-mentioned methods.

[0131] The method can further comprise repeating the step of determining another number of coinciding two-dimensional information between the first set of two-dimensional information and the second set of two-dimensional information. The other number of coinciding two-dimensional information can be determined using different positions of the segmented two-dimensional image relative to the segmented virtual three-dimensional model, or vice versa. The step of repeating the determination of the other number of coinciding two-dimensional information can be performed repeatedly until the other number of coinciding two-dimensional information reaches a predetermined number of coinciding two-dimensional information.

[0132] The step of repeating the determination of the other number of coinciding two-dimensional information can be performed by the one or more processors using the algorithm.

[0133] The one or more processors can use the algorithm to repeat the determination of the other number of coinciding two-dimensional information and determine different positions. The algorithm can be configured to determine different positions. The algorithm can be further configured to repeat the determination of the other number of coinciding two-dimensional information. The algorithm can be configured to repeat the determination of the other number of coinciding two-dimensional information until the other number of coinciding two-dimensional information reaches a predetermined number of coinciding two-dimensional information.

[0134] For example, four different positions can be determined using the algorithm, one to the right of the initial position, one to the left of the initial position, one above the initial position, and one below the initial position. For each of the four different positions, the above-mentioned process of determining the other number is repeated, thereby determining four sets of the other number of coinciding two-dimensional information. The algorithm can then determine which of the four sets of the other number of coinciding two-dimensional information is the largest number, and determine where to place the next different position based on the largest number, such that a larger number of coinciding two-dimensional information can be obtained for each repetition.

[0135] The predetermined number can be a fixed number, a number based on user input, or a calculated number, such as a ratio. The one or more processors or the algorithm are configured to terminate the process of determining the other number of coinciding two-dimensional information when the other number of coinciding two-dimensional information reaches or exceeds the predetermined number. The predetermined number can be a number of pixels, an exponent, or a ratio.

[0136] For a more detailed example, see the description below of Figures 3A-3B .

[0137] The method can further comprise the step of superimposing the segmented two- dimensional image on the segmented virtual three-dimensional model at locations where the number of coinciding two-dimensional information has reached a predetermined number of coinciding two-dimensional information. The method can further comprise the step of simultaneously displaying on a display the superimposed segmented two-dimensional image on the segmented virtual three-dimensional model and diagnostic and / or health data from the two-dimensional image and from the intraoral scan data (virtual three-dimensional model), or vice versa, so that the segmented virtual three-dimensional model can be superimposed on the segmented two-dimensional image.

[0138] Displaying the superimposed segmented two-dimensional image on the segmented virtual three-dimensional model and displaying diagnostic and / or health data can allow the dentist to more efficiently and conveniently obtain details about the patient's dental condition.

[0139] For example, the dentist can view a bite-wing film (X-ray image) showing the status of hard tissue (teeth and bone) superimposed on a virtual three-dimensional model of the patient's dentition showing the status of soft tissue (gingiva) and hard tissue in the patient's dentition. Thereby, the dentist is able to efficiently and conveniently view a variety of diagnostic and health data simultaneously without having to switch between bite-wing film view and virtual three-dimensional model view. Furthermore, the dentist does not need to acquire a bite-wing film containing information indicative of the bite-wing film angle of the bite-wing film, but can use any bite-wing film of the patient's dentition.

[0140] Furthermore, the virtual three-dimensional model of the dental object can be superimposed on the two-dimensional image of the dental object.

[0141] For example, the virtual three-dimensional model of the first, second and third molar of the lower jaw of the patient's dentition can be superimposed on a panoramic X-ray image of the patient's dentition, including the same lower jaw and the same first, second and third molar (this case is shown in the disclosure Figure 4

[0142] The dentist viewing the thus displayed virtual three-dimensional model superimposed on the panoramic X-ray image can simultaneously get an overview of diagnostic and / or health information of a variety of dental conditions obtained from the virtual three-dimensional model and the panoramic X-ray image, and thereby can more efficiently perform the diagnosis of the patient's dental condition. Such information can be, for example, bone level (from the panoramic X-ray image) related to gum disease, and the status of the gingiva (gingival margin level and inflammation) in the patient's dentition (from the virtual three-dimensional model).

[0143] Other aspects of the application will be appreciated by those skilled in the art upon reading and understanding the attached description.

[0144] ​The handheld intraoral scanner system can be configured to determine a position of a segmented two-dimensional image of a dental object relative to a segmented virtual three-dimensional model of the dental object. The system can include a handheld intraoral scanner configured to acquire intraoral scan data of the dental object to generate or update a virtual three-dimensional model of the dental object, a two-dimensional image of the dental object. The system further includes one or more processors configured to segment the two-dimensional image of the dental object and obtain a first set of two-dimensional information of the segmented two-dimensional image, segment the virtual three-dimensional model of the dental object, determine an initial position of the segmented two-dimensional image relative to the segmented virtual three-dimensional model, obtain a two-dimensional projection of the segmented virtual three-dimensional model on a projection plane of the determined initial position, and obtain a second set of two-dimensional information of the two-dimensional projection of the segmented virtual three-dimensional model, and align the first set of two-dimensional information of the segmented two-dimensional image with the second set of two-dimensional information of the two-dimensional projection of the segmented virtual three-dimensional model until a determined first amount of coinciding two-dimensional information between the first set of two-dimensional information and the second set of two-dimensional information has reached a predetermined amount of coinciding two-dimensional information.

[0145] The alignment of the first set of two-dimensional information and the second set of two-dimensional information can be performed based on an image alignment algorithm, which is feature-based and comprises one of the following algorithms:

[0146] • keypoint detector (e.g. DoG, Harri, GFFT, etc.)

[0147] • local invariant descriptor (e.g. SIFT, SURF, ORB, etc.)

[0148] • keypoint matching (e.g. Ransac and its variants)

[0149] • similarity measure based on cross-correlation or subset of squared intensity differences, and

[0150] • deep learning BRIEF DESCRIPTION OF DRAWINGS

[0151] Various aspects of the disclosure can be best understood from the following detailed description when read with the accompanying drawings. The drawings are schematic and simplified for clarity, and they merely provide illustrations of the principles of the application. Other features, aspects, and / or techniques contrary to, or not specifically described herein can be employed. Like reference numerals are used to indicate like parts throughout the specification. Each aspect can be used independently or in combination with any or all other aspects. These and other aspects, features, and / or techniques will be apparent from the following description when taken in conjunction with the accompanying drawings, described below. In the drawings:

[0152] Figure 1A handheld intraoral scanner system is shown, comprising a handheld intraoral scanner, a first, second and third neural network, an algorithm, a two-dimensional image of a dental object, a virtual three-dimensional model of the dental object, and the two-dimensional image superimposed on the virtual three-dimensional model;

[0153] Figure 2 A segmented two-dimensional image and a segmented virtual three-dimensional model are shown;

[0154] Figure 3A An exemplary process of determining another initial position of the two-dimensional image relative to the virtual three-dimensional model is schematically shown;

[0155] Figure 3B An exemplary continuation of the process of Figure 3A is schematically shown, and also the two-dimensional image superimposed on the virtual three-dimensional model;

[0156] Figure 4 is a screenshot of a graphical user interface of a software, showing a virtual three-dimensional model superimposed on an X-ray image containing health information; and

[0157] Figure 5 An overview table of a method for a handheld intraoral scanner system is shown, the method comprising the step of superimposing a two-dimensional image of a dental object on a virtual three-dimensional model of the dental object. DETAILED DESCRIPTION

[0158] The detailed description set forth below, in connection with the appended drawings, is intended as a description of various configurations and is not intended to limit the scope of the concepts. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. It will be apparent to those skilled in the art, however, that these concepts can be practiced without these specific details. Numerous aspects of devices, systems, media, programs, and methods are illustrated by various blocks, functional units, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as “elements”). These elements can be implemented using electronic hardware, computer programs, or any combination thereof, depending upon the particular application, design constraints, or other reasons.

[0159] Electronic hardware can include microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionalities described in this disclosure. Computer programs should be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0160] The scanning for providing extra-oral scan data and / or intra-oral scan data can be performed by a dental scanning system, which can comprise an intra-oral scanning device, such as a TRIOS series scanner from 3Shape A / S or a lab-based scanner, such as an E series scanner from 3Shape A / S. The dental scanning system can comprise a wireless capability provided by a wireless interface, such as a wireless network unit. The scanning device can employ a scanning principle, such as triangulation-based scanning, confocal scanning, focus scanning, ultrasound scanning, X-ray scanning, stereo vision, motion structure, optical coherence tomography (OCT), or any other scanning principle. In an embodiment, the scanning device is operated by projecting a pattern and translating a focal plane along an optical axis of the scanning device and capturing a plurality of two-dimensional images at different focal plane positions, such that each series of captured two-dimensional images corresponding to each focal plane forms a stack of two-dimensional images, which enables the scanning device to obtain surface information. The obtained two-dimensional images are also referred to herein as raw two-dimensional images, wherein "raw" means herein that the images are not image-processed. The focal plane positions are preferably moved along the optical axis of the scanning system such that the two-dimensional images captured at a plurality of focal plane positions along the optical axis form said stack of two-dimensional images (also referred to herein as a sub-scan) for a given view of the object, i.e. for a given arrangement of the scanning system relative to the object. After moving the scanning device relative to the object or imaging the object in a different view, a new stack of two-dimensional images for this view can be captured. The focal plane positions can be changed by means of at least one focusing element, such as a moving focusing lens. During a scanning session, the scanning device is typically moved and tilted relative to the dentition such that at least some groups of sub-scans at least partially overlap in order to enable the reconstruction of a digital dental three-dimensional model by stitching together the overlapping sub-scans in real-time and to display the progress of the virtual three-dimensional model on a display as feedback to the user. The result of the stitching is a digital three-dimensional representation of a surface that is larger than the surface that can be captured by a single sub-scan, i.e. larger than the field of view of the three-dimensional scanning device. The stitching is also referred to as registration and fusion, which works by identifying overlapping regions of the three-dimensional surface in the individual sub-scans and transforming the sub-scans to a common coordinate system such that the overlapping regions match, resulting in the digital three-dimensional model. An Iterative Closest Point (ICP) algorithm can be used for this purpose. Another example of a scanning device is a triangulation scanner, in which a time-varying pattern is projected onto the dental object and a sequence of images of different pattern configurations is acquired by one or more cameras arranged at an angle relative to the projector unit.

[0161] The color texture of the dental object can be obtained by illuminating the object with different monochromatic light, such as red, green and blue, or with polychromatic light, such as white light. The two-dimensional images can be acquired during white light flickering.

[0162] Typically, the process of obtaining surface information of a dental object to be scanned in real time requires the scanning device to illuminate the surface and acquire a large number of two-dimensional images. Typically, a high-speed video camera is used at a frame rate of 300-2000 two-dimensional frames per second, depending on the technology and two-dimensional image resolution. The scanning device needs to process the large amount of image data either to forward the raw image data stream directly to an external processing device or to perform some image processing before transferring the data to an external device or display. This process requires multiple electronic components within the scanner to operate at a high workload, thus having a high current demand.

[0163] The scanning device comprises one or more light projectors configured to generate an illumination pattern to be projected onto the three-dimensional dental object during scanning. The light projector(s) preferably comprises a light source, a mask having a spatial pattern, and one or more lenses, such as a collimating lens or a projection lens. The light source can be configured to generate light of a single wavelength or a combination of wavelengths (monochromatic or polychromatic). The combination of wavelengths can be generated using a light source configured to generate light comprising different wavelengths, such as white light. Alternatively, the light projector(s) can comprise multiple light sources, such as LEDs generating light of different wavelengths, respectively, such as red, green and blue, which can be combined to form light comprising different wavelengths. Thus, the light generated by the light source can be defined by a wavelength defining a specific color, or by a range of different wavelengths defining a combination of colors, such as white light. In an embodiment, the scanning device comprises a light source configured for exciting fluorescent materials of the teeth to obtain fluorescence data from the dental object. Such a light source can be configured to generate a narrow range of wavelengths. In another embodiment, the light from the light source is infrared (IR) light, which is able to penetrate tooth tissue. The light projector(s) can be a DLP light projector using a micro-mirror array to generate a time-varying pattern, or a diffractive optical element (DOF), or a backlit mask light projector, where a light source is placed behind a mask having a spatial pattern, whereby the light projected on the surface of the dental object is patterned. The backlit mask light projector can comprise a collimating lens for collimating the light from the light source, which is placed between the light source and the mask. The mask can have a checkerboard pattern, such that the generated illumination pattern is a checkerboard pattern. Alternatively, the mask can have other patterns, such as lines or dots, etc.

[0164] The scanning device preferably further comprises optical components for directing the light from the light source towards the surface of the dental object. The specific arrangement of the optical components depends on whether the scanning device is a focus scanning device, a scanning device using triangulation, or any other type of scanning device. The same applicant further describes a focus scanning device in EP 2442 720 B1, the entire content of which is incorporated herein by reference.

[0165] The optical components of the scanning device direct light reflected from the dental object in response to the illumination of the dental object towards the image sensor(s). The image sensor(s) are configured to generate a plurality of images based on the incident light received from the illuminated dental object. The image sensor can be a high speed image sensor, for example an image sensor configured for acquiring images with an exposure shorter than 1 / 1000 of a second or a frame rate exceeding 250 frames per second (fps). For example, the image sensor can be a rolling shutter (CCD) or a global shutter sensor (CMOS). The image sensor(s) can be a monochrome sensor comprising a color filter array, for example a Bayer filter and / or a further filter which can be configured to substantially eliminate one or more color components from the reflected light before the reflected light is converted into an electrical signal and only retain the other, non-eliminated components. For example, such a further filter can be used to eliminate a certain portion of the white light spectrum, for example the blue component, of the signal generated in response to the excitation of the fluorescent material of the teeth and only retain the red and green components.

[0166] The wireless network unit is configured to wirelessly connect the intraoral scanning system to a network comprising a plurality of network elements, including at least one network element configured to receive the processed data.

[0167] The dental scanning system preferably further comprises a processor configured to generate scanning data (e.g. extra-oral scanning data and / or intra-oral scanning data) by processing two-dimensional (2D) images acquired by the scanning device. The processor can be part of the scanning device. For example, the processor can comprise a field programmable gate array (FPGA) and / or an advanced reduced instruction set computer (ARM) and / or an x86 processor and / or a combination of FPGA, ARM and / or x86 processor located on the scanning device. The scanning data comprises information related to the three-dimensional dental object. The scanning data can comprise any of the following information: two-dimensional images, three-dimensional point clouds, depth data, texture data, intensity data, color data and / or combinations thereof. For example, the scanning data can comprise one or more point clouds, wherein each point cloud comprises a set of three-dimensional points describing the three-dimensional dental object. As another example, the scanning data can comprise images, each image comprising image data, e.g. described by image coordinates and a timestamp (x,y,t), wherein depth information can be inferred from the timestamp. The image sensor(s) of the scanning device can acquire a plurality of raw two-dimensional images of the dental object in response to illuminating the dental object using one or more light projectors. The plurality of raw two-dimensional images can also be referred to herein as a stack of two-dimensional images. The two-dimensional images can subsequently be provided as input to the processor, which processes the two-dimensional images to produce the scanning data. The processing of the two-dimensional images can comprise a step of determining which part of each two-dimensional image is in focus, to infer / generate depth information from the images. The depth information can be used to generate a three-dimensional point cloud comprising a set of three-dimensional points in space, e.g. described by Cartesian coordinates (x,y,z). The three-dimensional point cloud can be generated by the processor or another processing unit. Each two / three-dimensional point can further comprise a timestamp indicating when the two / three-dimensional point was recorded, i.e. which image in the stack of two-dimensional images the point originates from. The timestamp is associated with the Z-coordinate of the three-dimensional point, i.e. the Z-coordinate can be inferred from the timestamp. Thus, the output of the processor is the scanning data, and this scanning data can comprise image data and / or depth data, e.g. described by image coordinates and a timestamp (x,y,t) or otherwise described as (x,y,z). The scanning device can be configured to also send other types of data than the scanning data. Examples of data include three-dimensional information, texture information, e.g. infrared (IR) images, fluorescence images, reflective color images, X-ray images and / or combinations thereof.

[0168] Figure 1A handheld intraoral scanner system 100 is shown, comprising a handheld intraoral scanner 10, comprising one or more processors 5, a two-dimensional image 2 of teeth 1, shown as an X-ray image, a virtual three-dimensional model 3 of the teeth 1, an image 4 of the two-dimensional image 2 superimposed on the virtual three-dimensional model 3, a first neural network 6a trained to segment dental objects 1 in the two-dimensional image 2, a second neural network 6b trained to segment dental objects 1 in the virtual three-dimensional model 3, a third neural network 6c trained to determine an initial position 20a (Fig. 3) of any corresponding dental object 1 of the segmented two-dimensional image 2a Figure 2 ) relative to the segmented virtual three-dimensional model 3a( Figure 2 ) and an algorithm 7 configured to superimpose the segmented two-dimensional image 2a on the segmented virtual three-dimensional model 3a of the dental objects 1. The algorithm 7 and the first, second and third neural networks 6a-6c can be located completely or partially in the handheld intraoral scanner 10, partially or completely in a computer, or partially or completely in a server, for example in a cloud system. The one or more processors 5 are configured to execute and use the first, second and third neural networks 6a-6c and the algorithm 7.

[0169] Figure 2 A two-dimensional image 2 segmented by the first neural network 6a and a virtual three-dimensional model 3 segmented by the second neural network 6b are shown. The segmented two-dimensional image 2a shown depicts dental objects 1, which are shown as three teeth 1, and the pixels constituting the dental objects 1 are represented by a checkerboard pattern on the three teeth 1 and are referred to as a first group of pixels 22. The first group of pixels 22 can be determined by the algorithm 7. Since the two-dimensional image 2a is segmented, the first group of pixels 22 constituting the dental objects 1 and the contours of the dental objects 1 are defined and can be separated from the rest of the segmented two-dimensional image 2a. In the segmented virtual three-dimensional model 3a, the dental objects 1 are shown as three teeth 1. Figure 2 The segmented virtual three-dimensional model 3a of the dental objects 1, shown as three teeth 1 in the middle, is further shown segmented using triangulation. The segmentation is represented by waves on the dental objects 1.

[0170] Figures 3A-3B A series of illustrations 300-309 is shown, which illustrates the process of obtaining an optimal position for superimposing the segmented two-dimensional image 2a on the segmented virtual three-dimensional model 3a.

[0171] Figures 300 show the initial position 20a relative to the segmented virtual three- dimensional model 3a. The initial position 20a is determined by the third neural network 6c, which is trained to determine an initial position 20a of a segmented two-dimensional image 2a of a dental object 1 relative to a corresponding dental object 1 of a segmented virtual three-dimensional model 3a. The third neural network 6c recognizes the three teeth 1 shown in the segmented two-dimensional image 2a based on the data it is trained on, and determines an approximate initial position 20a in space (three dimensions) relative to the segmented virtual three-dimensional model 3a, which is in close proximity to the corresponding teeth 1 in the segmented virtual three-dimensional model 3a. Figure 301 shows a view on the segmented virtual three-dimensional model 3a as seen from the determined initial position 20a. Figure 302 shows a subsequent process of obtaining a screenshot 222 of the segmented virtual three-dimensional model 3a as seen from the determined initial position 20a. Figure 302 also shows a second set of pixels 22a, which is shown as a checkerboard pattern on the three teeth shown on the screenshot 222. The second set of pixels 222 can be determined by the algorithm 7. A subsequent process is shown in figure 303, in which the segmented two-dimensional image 2a is aligned with the screenshot 222. In this process, the algorithm 7 lays out the segmented two-dimensional image 2a on the screenshot 222 and determines the number of coinciding pixels 30. If the determined number of coinciding pixels 30 is less than a predetermined number of coinciding pixels, a different position 20b is determined using the algorithm 7 (figure 304), and the process of obtaining a screenshot 222 from this different position 20b, aligning the segmented two-dimensional image 2a on the screenshot 222 and determining the number of coinciding pixels 30 is repeated, as shown in subsequent figures 304-309.

[0172] The number of coinciding pixels 30 is a measure of an acceptable fit, and represents a criterion for superimposing the segmented two-dimensional image 2a on the segmented virtual three-dimensional model 3a from the position where the closest number of coinciding pixels is determined.

[0173] The different position 20b is determined by the algorithm 7, wherein the algorithm 7 can determine the number of coinciding pixels 30 at positions in space to the right, left, above, below, in front of and behind the initial position 20a, and determine the different position 20b based on the highest determined number of coinciding pixels 30. For example, if the number of coinciding pixels 30 to the right is greater than to the left, it is an indication that a better alignment is achieved when the different position 20b is placed to the right relative to the initial position 20a.

[0174] Fig. 304 shows a subsequent process in which a different position 20b in three-dimensional space relative to the segmented virtual three-dimensional model 3a is determined. This illustration shows that the different position 20b has been determined to be located to the right and above the initial position 20a shown in illustration 300 and is further shown in an orientation perpendicular to the segmented virtual three-dimensional model 3a. Subsequent illustration 305 shows another screenshot 222a taken from the different position 20b. This illustration further shows the pixels of the screenshot represented by the checkerboard pattern.

[0175] Fig. 306 shows the process of algorithm 7 aligning the segmented two-dimensional image 2a on the other screenshot 222a of illustration 305 from the different position 20b of illustration 304. The previously explained process of determining the number of coinciding pixels 30 is subsequently repeated and another number of coinciding pixels 30a is determined for the process shown in illustration 306.

[0176] Algorithm 7 subsequently determines a new different spatial position 20c relative to the segmented virtual three-dimensional model 3a based on the other number of coinciding pixels 30a as previously described, as shown in illustration 307. Algorithm 7 subsequently obtains a further screenshot 222b of the segmented virtual three-dimensional model 3a from the new different position 20c and determines a new second set of pixels 22c, as shown by the checkerboard pattern in illustration 308. Algorithm 7 subsequently aligns the segmented two-dimensional image 2a on the further screenshot 222b by aligning the first set of pixels 22 with the new second set of pixels 22c at the new different position 20c, as shown in illustration 309. Algorithm 7 subsequently determines a new further number of coinciding pixels 30b. When the final number of coinciding pixels reaches the predetermined number of coinciding pixels, the process of determining the number of coinciding pixels ends and the segmented two-dimensional image 2a is superimposed on the segmented virtual three-dimensional model 3a from the final different position used to determine the most recent number of coinciding pixels and reach the predetermined number of coinciding pixels.

[0177] Figure 4 An image of a graphical user interface 9 of software is shown, which graphical user interface 9 is configured to display a virtual three-dimensional model 3 of a dental object 1, a two-dimensional image 2 of the dental object 1 and a superimposed image 400 of the virtual three-dimensional model 3 superimposed on the two-dimensional image 2 or vice versa. In Figure 4In this example, a portion of the virtual three-dimensional model 3 is shown superimposed 400 on the two-dimensional image 2 (shown in the left window 200). The graphical user interface 9 is shown with two windows 200, 300, i.e. a right window 300 and a left window 200. The right window 300 shows a virtual three-dimensional model 3 of a lower jaw of a dentition including teeth and surrounding gingiva. The left window 200 shows a panoramic X-ray image 2 of the same dentition, including the lower jaw, teeth and surrounding gingiva. The left window 200 also shows the virtual three-dimensional model 3 from the right window 300 superimposed 400 on the corresponding (same) teeth in the panoramic X-ray image 200. The image 400 in the left window also shows diagnostic and health information in the form of bone levels provided by the panoramic X-ray image 2, 400, and gingival status (gingival margin levels, inflammation) provided by the superimposed virtual three-dimensional model 3, 400. In one example, a dentist can have performed an intraoral scan of a patient’s teeth using a handheld intraoral scanner 10, thereby obtaining a virtual three-dimensional model 3 of the patient’s dentition. The virtual three-dimensional model 3 is displayed in the graphical user interface 9 on the display. The dentist then selects a region of interest on the virtual three-dimensional model 3 of the dentition, for example the lower jaw, as shown in the right window 300. The dentist then selects a two-dimensional image 2 of the same dentition including the lower jaw that was previously captured, for example a panoramic X-ray image 2 shown in the left window 200. The dentist then initiates superimposition 400 of the virtual three-dimensional model 3 of the lower jaw onto the panoramic X-ray image 2, for example by clicking a button in the graphical user interface 9 on the display. The one or more processors 5 then perform the superimposition 400 using the first, second and third neural networks 6a-6c and the algorithm 7, as previously described in the present disclosure. The result of the superimposition 400 of the virtual three-dimensional model 3 of the lower jaw on the corresponding lower jaw in the panoramic X-ray image 2 is then displayed in the graphical user interface 9 on the display, as shown in the left window 200, 400. Figure 4

[0178] Figure 5 An overview table of a method for a handheld intraoral scanner system is shown, the method comprising a step of superimposing a two-dimensional image 2 of a dental object 1 on a virtual three-dimensional model 3 of the dental object 1.

[0179] In step 5a, a virtual three-dimensional model 3 of the dental object 1 is obtained and segmented. In one example, the virtual three-dimensional model 3 can be obtained by the handheld intraoral scanner 10 by scanning a physical three-dimensional model of a dental impression, or by loading a previously scanned virtual three-dimensional model 3 into a storage unit 4 of the handheld intraoral scanner system 100, for example into a memory of a computer.

[0180] ​When the virtual three-dimensional model 3 of the dental object 1 is acquired, the virtual three-dimensional model 3 is segmented. The segmentation can be performed using the second neural network 6b, which is trained to segment any virtual three-dimensional model of a dental object, as previously described in the present disclosure.

[0181] In step 5b, a two-dimensional image 2 of the dental object 1 is acquired and segmented. The two-dimensional image 2 can be acquired by the handheld intraoral scanner 10 or a camera, and can thus be a color image or an infrared image, for example. In another example, the two-dimensional image 2 can be acquired by extraoral means such as an X-ray generator or an ultrasound scanner, and can thus be an X-ray, panoramic X-ray, bite-wing or ultrasound image. In yet another example, the two-dimensional image 2 can be acquired by loading the two-dimensional image into the memory unit 4 of the handheld intraoral scanner system 100, for example into the memory of a computer. When the two-dimensional image 2 of the dental object 1 is acquired, the two-dimensional image 2 is segmented. The segmentation can be performed using the first neural network 6a, which is trained to segment any two-dimensional image of a dental object, as previously described in the present disclosure. The algorithm 7 can obtain a first set of pixels 22 of the dental object in the segmented two-dimensional image 2a.

[0182] In step 5c, an initial position 20a of the segmented two-dimensional image 2a relative to the segmented virtual three-dimensional model 3a is determined. The one or more processors 5 can determine the initial position 20a by using the third neural network 6c, which is trained to determine the position of a segmented two-dimensional image of a dental object relative to a corresponding dental object of a segmented virtual three-dimensional model.

[0183] In step 5d, the algorithm 7 obtains a two-dimensional projection 222 of the segmented virtual three-dimensional model 3a from the determined initial position 20a. The algorithm 7 can obtain a second set of two-dimensional information 22b. The second set of two-dimensional information 22b can be two-dimensional data points 22b, which contain the two-dimensional coordinates of the segmented virtual three-dimensional model 3a arranged on a plane of the determined initial position 20a. Alternatively, the second set of two-dimensional information 22b can be a second set of pixels 22b. The second set of pixels can be obtained by obtaining a screenshot 222 of the segmented virtual three-dimensional model 3a from the determined initial position 20a.

[0184] In step 5e, the segmented two-dimensional image 2a is aligned with the two-dimensional projection 222. The one or more processors 5 can use the algorithm 7 to perform the alignment. The algorithm 7 can align the two-dimensional information 22 of the segmented two-dimensional image 2a, which can be the first set of pixels 22, with the two-dimensional information 22b of the two-dimensional projection 222, which can be the second set of pixels 22b.

[0185] In step 5f, the one or more processors 5 can use algorithm 7 to determine the number of coinciding two-dimensional information 30 between the segmented two-dimensional image 2a and the two-dimensional projection 222. Algorithm 7 can use the "intersection over union" method to determine the number of coinciding pixels 30 between the first set of pixels 22 and the second set of pixels 22b.

[0186] In step 5g, the one or more processors 5 repeat steps 5d-5f using different positions 20b until the number of coinciding two-dimensional information 30 reaches a predetermined number of coinciding two-dimensional information. Algorithm 7 can determine the number of coinciding pixels from different positions around the determined initial position, and based on the different positions reaching the maximum number of coinciding pixels, determine the different position used in step 5g when repeating steps 5d-5f using different positions.

[0187] In step 5h, the one or more processors 5 can use algorithm 7 to overlay the segmented two-dimensional image 2a on the segmented virtual three-dimensional model 3a from the determined position that is used when the number of coinciding pixels reaches the predetermined number of coinciding pixels.

[0188] While several embodiments have been disclosed, a few examples of the disclosure have been represented, the disclosure is not limited to those details; the subject matter of the claims is not limited to the scope of the disclosure as set forth in the following claims. In particular, it is understood that there can be other embodiments and structural and functional modifications without departing from the scope of the disclosure.

[0189] The benefits, other advantages, and solutions to problems have been described herein with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any piece of information that might cause any benefit, advantage, or solution, or contribute to any benefit, advantage, or solution, to be realized, should not be construed as being a critical, required, or essential feature or part of the claims. The scope of the application should be accorded the broadest interpretation so as to encompass all

[0190] The structural features of the apparatus described in the detailed description and / or claims can be combined with the steps of the method when appropriately replaced by the corresponding process.

[0191] The singular forms "a," "an," and "the" as used herein, are intended to include plural forms (i.e., meaning "at least one") unless otherwise expressly stated. It will be further understood that the terms "includes," "including," "includes," and / or "including" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It will be further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present, unless expressly stated otherwise. Moreover, "connected" or "coupled" as used herein can include wireless connection or coupling. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. The steps of any disclosed methods are not limited to the exact order in which they are described herein, unless otherwise expressly stated.

[0192] It is to be understood that a reference to "one implementation" or "one aspect" or a feature in this specification means that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation of the disclosure. Moreover, the specific features, structures, or characteristics can be combined in any suitable

[0193] The claims are not limited to the aspects shown in the figures, but are commensurate with the full scope of the claims as construed in accordance with the language of the claims and any equivalents thereto, unless otherwise explicitly stated in the claims. Except as otherwise expressly stated in the claims, an element referred to in the singular is not intended to mean "one and only one" unless explicitly so stated. The term "some" refers to one or more unless otherwise explicitly stated in the claims.

[0194] Item

[0195] 1. A handheld intraoral scanner system (100) configured to determine a position of a segmented two-dimensional image (2a) of a dental object (1) relative to a segmented virtual three-dimensional model (3a) of the dental object (1), wherein the system (100) comprises:

[0196] - a handheld intraoral scanner (10) configured to acquire intraoral scan data of the dental object (1) to generate or update a virtual three-dimensional model (3) of the dental object (1),

[0197] - a two-dimensional image (2) of the dental object (1), and

[0198] - one or more processors (5) configured to:

[0199] • segment the two-dimensional image (2) of the dental object (1) and obtain a first set (22) of two-dimensional information of the segmented two-dimensional image (2a),

[0200] • segment a virtual three-dimensional model (3) of the dental object (1),

[0201] • determine an initial position (20a) of the segmented two-dimensional image (2a) with respect to the segmented virtual three-dimensional model (3a),

[0202] • obtain a two-dimensional projection (222) of the segmented virtual three-dimensional model (3a) on a projection plane at the determined initial position (20a) and obtain a second set (22b) of two-dimensional information of the two-dimensional projection (222) of the segmented virtual three-dimensional model (3a), and

[0203] • align the first set (22) of two-dimensional information of the segmented two-dimensional image (2a) with the second set (22b) of two-dimensional information of the two-dimensional projection (222) of the segmented virtual three-dimensional model (3a) until a determined first number of coinciding two-dimensional information (30) between the first set (22) of two-dimensional information and the second set (22b) of two-dimensional information has reached a predetermined number of coinciding two-dimensional information.

Claims

1. A handheld intraoral scanner system (100) configured to determine the position of a segmented two-dimensional image (2a) of a dental object (1) relative to a segmented virtual three-dimensional model (3a) of the dental object (1), wherein the system (100) comprises: - A handheld intraoral scanner (10) configured to acquire intraoral scan data of the dental object (1) to generate or update a virtual three-dimensional model (3) of the dental object (1). - A two-dimensional image (2) of the dental object (1), and - One or more processors (5), which are configured to: • Segment the two-dimensional image (2) of the dental object (1) and obtain the first set (22) of two-dimensional information of the segmented two-dimensional image (2a). • A virtual 3D model (3) of the segmented dental object (1), • Determine the initial position (20a) of the segmented 2D image (2a) relative to the segmented virtual 3D model (3a). • Obtain the two-dimensional projection (222) of the segmented virtual 3D model (3a) on the projection plane at the determined initial position (20a), and obtain the second set (22b) of two-dimensional information of the two-dimensional projection (222) of the segmented virtual 3D model (3a). Align the first set (22) of two-dimensional information of the segmented two-dimensional image (2a) with the second set (22b) of two-dimensional information of the two-dimensional projection (222) of the segmented virtual three-dimensional model (3a), and determine a first number of overlapping two-dimensional information (30) between the first set (22) of two-dimensional information and the second set (22b) of two-dimensional information. Using the segmented two-dimensional image (2a) at different positions (20b) relative to the segmented virtual three-dimensional model (3a), another number of overlapping two-dimensional information (30a) between the first group (22) of two-dimensional information and the second group (22b) of two-dimensional information is repeatedly determined until the other number of overlapping two-dimensional information (30a) reaches a predetermined number of overlapping two-dimensional information.

2. The system according to claim 1, wherein, The one or more processors (5) are also configured to overlay the segmented two-dimensional image (2a) onto the segmented virtual three-dimensional model (3a) at the position where another number of overlapping two-dimensional information (30a) has reached the predetermined number of overlapping two-dimensional information.

3. The system of claim 2 further includes a display, and the system is further configured to display a superimposed segmented two-dimensional image (2a) on the segmented virtual three-dimensional model (3a) on the display, and is further configured to simultaneously display diagnostic and / or health data from the two-dimensional image (2) and from the intraoral scan data on the display.

4. The system according to any one of the preceding claims, wherein, The system also includes: - A first neural network (6a), which is trained to segment any dental object (1) in a two-dimensional image (2), - A second neural network (6b), trained to segment any dental object (1) of a virtual 3D model (3), and - A third neural network (6c) is trained to determine the initial position (20a) of any corresponding dental object (1) in a segmented two-dimensional image (2a) of the dental object (1) relative to a segmented virtual three-dimensional model (3a). And among them: - The two-dimensional image (2) is segmented by the first neural network (6a). - The virtual 3D model (3) is segmented by the second neural network (6b), and - The initial position (20a) of the segmented two-dimensional image (2a) relative to the segmented virtual three-dimensional model (3a) is determined by the third neural network (6c).

5. The system according to any one of the preceding claims, wherein, The system (100) also includes an algorithm (7) configured to: - Obtain the two-dimensional projection (222) and the second set (22b) of two-dimensional information. - Align the first group (22) of two-dimensional information with the second group (22b) of two-dimensional information, and determine the first number of overlapping two-dimensional information (30), and Using the segmented two-dimensional image (2a) at different positions (20b) relative to the segmented virtual three-dimensional model (3a), another number of overlapping two-dimensional information (30a) between the first group (22) of two-dimensional information and the second group (22b) of two-dimensional information is repeatedly determined until the other number of overlapping two-dimensional information (30a) reaches a predetermined number of overlapping two-dimensional information.

6. The system according to claim 5, wherein, The algorithm (7) is also configured to align the first set (22) of two-dimensional information with the second set (22b) of two-dimensional information by changing the position, orientation and / or size of the two-dimensional information of the two-dimensional image (2).

7. The system according to any one of the preceding claims, wherein, The one or more processors (5) are configured to obtain a two-dimensional projection (222) of the segmented virtual three-dimensional model (3a) on the projection plane by acquiring another two-dimensional image (222) of the segmented virtual three-dimensional model (3a) from a determined initial position (20a).

8. The system according to any one of the preceding claims, wherein, This two-dimensional information is in pixels.

9. The system according to any one of the preceding claims, wherein, The two-dimensional image (2) is one of an X-ray image, an infrared (IR) image, a near-infrared (NIR) image, or an ultrasound image.

10. The system according to any one of the preceding claims, wherein, The dental object (1) is a tooth, part of a tooth, or multiple teeth.

11. A method for determining the position of a segmented two-dimensional image (2a) of a dental object (1) relative to a segmented virtual three-dimensional model (3a) of the dental object (1), the method comprising the steps of: -A virtual three-dimensional model (3) of the dental object (1) is obtained based on the intraoral scan data of the dental object (1). - Obtain a two-dimensional image (2) of the dental object (1). - Segment the two-dimensional image (2) of the dental object (1) and obtain the first set (22) of two-dimensional information of the segmented two-dimensional image (2a). -The virtual three-dimensional model (3) of the dental object (1) is segmented. - Determine the initial position (20a) of the segmented 2D image (2a) relative to the segmented virtual 3D model (3a), - Obtain the two-dimensional projection (222) of the segmented virtual 3D model (3a) on the projection plane at the determined initial position (20a), and obtain the second set (22b) of two-dimensional information of the two-dimensional projection (222) of the segmented virtual 3D model (3a). - Align the first set (22) of the segmented two-dimensional image (2a) with the second set (22b) of the two-dimensional projection (222) of the segmented virtual three-dimensional model (3a), and determine a first number of overlapping two-dimensional information (30) between the first set (22) and the second set (22b). Using the segmented two-dimensional image (2a) at different positions (20b) relative to the segmented virtual three-dimensional model (3a), another number of overlapping two-dimensional information (30a) between the first group (22) of two-dimensional information and the second group (22b) of two-dimensional information is repeatedly determined until the other number of overlapping two-dimensional information (30a) reaches a predetermined number of overlapping two-dimensional information.

12. The method of claim 11, further comprising the step of: - The segmented two-dimensional image (2a) is superimposed on the segmented virtual three-dimensional model (3a) at the position where another number of overlapping two-dimensional information (30a) has reached the predetermined number of overlapping two-dimensional information.

13. The method according to any one of claims 11-12, wherein: -The two-dimensional image (2) of any dental object (1) trained to segment a two-dimensional image (2) is segmented using a first neural network (6a). -The virtual 3D model (3) of any dental object (1) is segmented using a second neural network (6b) trained to segment the virtual 3D model (3), and - The initial position (20a) of the segmented two-dimensional image (2a) relative to the segmented virtual three-dimensional model (3a) is determined using a third neural network (6c) trained to determine the initial position (20a) of the segmented two-dimensional image (2a) of the dental object (1) relative to any corresponding dental object (1) of the segmented virtual three-dimensional model (3a).

14. The method according to any one of claims 11-13, wherein, The step of obtaining a two-dimensional projection (222) of the segmented virtual three-dimensional model (3a) on the projection plane is performed by acquiring another two-dimensional image (222) of the segmented virtual three-dimensional model (3a) from a determined initial position (20a), wherein the two-dimensional information is pixels.

15. The method according to any one of claims 11-14, wherein, The two-dimensional image (2) is one of an X-ray image, an infrared (IR) image, a near-infrared (NIR) image, or an ultrasound image.

Images