Rendering tooth model

By estimating lighting and calculating shadows, combining color space matching and camera effect simulation, the 'unny valley theory' problem of real display of dental models in dental augmented reality is solved, and real-time real rendering and illumination effects in dental augmented reality applications are achieved. .

CN110992464BActive Publication Date: 2024-11-01IVOCLAR VIVADENT AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910709552.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-09-28
Filing Date
2019-08-01
Publication Date
2024-11-01
Estimated Expiration
2039-08-01

AI Technical Summary

Technical Problem

In dental augmented reality applications, prior art is difficult to display three-dimensional dental models in face images or videos, especially the problem of 'uncanny valley theory' due to the lack of matching lighting information, and the ray tracing method is efficient in real-time applications Lower.

Method used

By estimating the lighting situation in the face image, determining the boundary of the mouth area and calculating the shadow, calculating the lighting of the tooth model based on the three-dimensional face geometry and lighting situation, combining color space matching and camera effect simulation, real rendering and lighting of the tooth model can be achieved. Bright.

Benefits of technology

It realizes real-time rendering and illuminating tooth models under limited computing power and live video conditions, overcomes the problem of 'Underworld Valley Theory' and provides real-life display effects in augmented reality applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110992464B_ABST
    Figure CN110992464B_ABST
Patent Text Reader

Abstract

The invention relates to a method for realistically displaying a three-dimensional virtual tooth model (1) in a two-dimensional image (20) of a face, the method comprising estimating or assuming lighting conditions in the face image, determining the boundaries of a mouth region in the image, calculating shadows in the mouth region based on the boundaries of the mouth region, based on a three-dimensional face geometry and based on the lighting conditions, calculating the lighting of the tooth model (1) based at least on the calculated shadows, and displaying the tooth model (2) and the two-dimensional image (20), wherein at least the mouth region in the image overlaps with the display (21) of the tooth model with the calculated lighting.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for realistically displaying a three-dimensional model of a denture or other dental prosthesis (tooth model) in an image or image stream of a human face. In particular, the described method allows realistic display of a tooth model in a tooth virtual model application such as a dental augmented reality application.

[0002] In the context of the present invention, the term "denture" is not necessarily limited to a full denture, but also includes a partial denture or correction of an orthodontic condition / adjustment or a dental restoration such as a dental prosthesis, including crowns, crown layers, veneers, inlays and onlays, bridges, dental implants, implant restorations. Accordingly, the term "dental model" includes all models of dental prostheses used for oral restorations, such as models of full dentures or partial dentures. The dental model may also include appliances attached to the denture or real teeth, such as caps and retainers or mouth guards for various purposes. Background Art

[0003] For dentists and patients, it is of interest to obtain a visual impression of the appearance of the patient's face with the adjusted dental situation, i.e. to display the adjusted dental situation in an image of the patient's face. Likewise, the appearance during or after the dental treatment is important for the patient before deciding to undergo this treatment. For this purpose, a virtual preview (virtual model) of the dentition adjusted by the dental treatment is helpful for the dentist and can also be used during the interactive adjustment of the treatment plan to obtain the most favorable aesthetic result.

[0004] In dental virtual model applications, virtual teeth are shown in a picture or video of the patient's face or part of it. Realistic display is important in this field because people are very sensitive to deviations from real human faces. This is a well-known problem called the "uncanny valley theory".

[0005] In order to overcome this "uncanny valley" problem when displaying virtual teeth in a face picture or video, realistic lighting of the three-dimensional (3D) rendered content is key. Thus, the lighting of the rendered virtual content should match the basic lighting of the real scene, i.e. the face. So far, such lighting has not existed in applications for rendering 3D virtual dentition in dentistry.

[0006] In computer games that render virtual characters, the three-dimensional facial geometry of the character is used to create lighting and shading effects. However, images or videos of faces are two-dimensional (2D) and thus cannot provide three-dimensional facial geometry.

[0007] Another common solution to this problem is ray tracing. This allows realistic rendering by taking many physical effects into account. However, this method is often too slow for real-time applications. Summary of the invention

[0008] It is therefore an object of the present invention to provide a computer-implemented method which allows realistic rendering and lighting of virtual three-dimensional dental models, in particular displaying complete or partial dentures.

[0009] Another object is to provide a method wherein the resulting rendered virtual dental model is adapted to a camera image of the face.

[0010] Another aim is to provide methods that could allow overcoming the problems of the uncanny valley theory.

[0011] Another aim is to provide methods that allow realistic rendering and lighting of models in real time.

[0012] Another object is to provide methods that allow realistic rendering and lighting of models in augmented reality (AR) applications.

[0013] Another object is to provide a method that can be executed (especially in real time) on mobile devices with limited computing power and / or capable of providing live video.

[0014] Another object is to provide a method that is performed fully or semi-automatically.

[0015] Yet another object is to provide a handheld mobile device for performing this method.

[0016] At least one of these objects is achieved by the method of claim 1 , the mobile device of claim 14 and / or one of the dependent claims of the present application.

[0017] A first aspect of the invention relates to a method for realistically displaying a virtual dental model in a two-dimensional image of a face, the image comprising at least the mouth region of the face. The dental model comprises a three-dimensional display of a dental prosthesis (or a part thereof). For example, the model may be a model of a complete denture or a partial denture. In particular, the dental model is a digital three-dimensional model of one or more teeth, in particular a plurality of teeth, for example six or more teeth, with or without gums (gingiva).

[0018] The method comprises estimating or assuming lighting conditions in a facial image and determining the boundaries of a mouth region in the image, in particular wherein the boundaries are defined by the inner boundaries of the lips of the face. The lighting conditions include one or more light sources, in particular also the position of the light sources, image-based or global lighting, spherical harmonic lighting or directional lighting.

[0019] Shadows in the mouth region (or spatially in the region behind the mouth region) are calculated based on the boundary of the mouth region, based on the three-dimensional face geometry and based on the lighting conditions. At least based on the calculated shadows, the lighting of the dental model is calculated. Finally, the three-dimensional dental model is displayed together with the two-dimensional image, and the display of the dental model (e.g., the two-dimensional display of the dental model) has the calculated lighting. Specifically, at least the mouth region in the image overlaps with the display of the dental model.

[0020] According to an embodiment of the method, the 3D face geometry refers to the geometry of the face, ie the same face shown in the image, wherein the face geometry is acquired using a depth sensor. In particular, the 3D face geometry can be acquired together with the 2D image or as a distance image.

[0021] According to an alternative embodiment of the method, the three-dimensional face geometry at least approximates the geometry of a human face (or at least a mouth region), wherein an image plane including at least a boundary of the mouth region is projected onto the three-dimensional geometry. In particular, the three-dimensional face geometry may include at least a portion of a side surface of a cylinder or a prism, and the image plane including at least an image portion of the mouth region is projected onto the portion of the side surface.

[0022] According to one embodiment of the method, calculating the illumination includes

[0023] - generating a first light ray reaching each of a plurality of virtual point light sources for a plurality of surface points on the surface of the virtual tooth model (such as vertices of the virtual tooth model),

[0024] - generating for each first ray a second ray reaching the virtual camera, the intersection point of the first ray with the three-dimensional face geometry being used as the starting point of the second ray, and

[0025] - the intersection point of the second ray with the image plane is used as the observation position for determining the illumination of the corresponding surface point, wherein if the second ray intersects the image plane at the mouth region, the surface point is assumed to be illuminated by a virtual point light source,

[0026] According to another embodiment of the method, displaying the dental model in the two-dimensional image comprises adapting the coloring of the dental model based on the coloring of the image.

[0027] In one embodiment, adapting the coloring includes

[0028] - identifying a reference color of at least one of teeth, gums and lips in the image. For example, the reference color may refer to an average color, and identifying the reference color may include automatically segmenting image pixels in the image that are associated with teeth, gums and / or lips. Using one or more reference colors to estimate a color space of the image. Using the estimated color space of the image to convert the color space of the tooth model. In particular, the estimated color space of the image is used as the color space of the tooth model.

[0029] For example, the color space of the tooth model may be converted to match the color space of the image.

[0030] According to another embodiment of the method, the lighting of the tooth model is calculated based on an estimated lighting situation. The lighting situation may be estimated based on the lighting of the face in the image.

[0031] In one embodiment, estimating lighting conditions includes

[0032] - estimate low frequency lighting, especially with spherical harmonic lighting and virtual point lights, and / or

[0033] - Detect reflections in the eyes of the face to estimate the location of light sources, especially high frequency lighting.

[0034] According to a further embodiment of the method, a soft or blurred edge of the mouth region is calculated from the determined boundary, wherein the shadow is calculated based on the soft or blurred edge in order to avoid harsh shadow edges.

[0035] According to another embodiment of the method, calculating the lighting of the dental model comprises applying lighting and shading effects, said effects comprising at least one of specular reflection, color or ambient occlusion.

[0036] According to another embodiment of the method, displaying the dental model includes simulating camera effects to make the appearance of the dental model close to the appearance of the face in the image, the simulated camera effects including at least one of dynamic range, contrast, saturation, noise, lens distortion, white balance, defocus and motion blur.

[0037] According to another embodiment of the method, the image is part of an image stream, and the method is performed in real time at least for a plurality of frames of the image stream.

[0038] According to one embodiment, displaying the dental model in the two-dimensional image comprises adjusting the color of the dental model based on the colors of multiple frames of the image stream, such as, for example, existing images and previous images comprising the image stream, in particular wherein the coloring is adapted using an average value from multiple consecutive frames of images.

[0039] According to another embodiment, the method further comprises acquiring the image stream by means of a camera and displaying the dental model and the two-dimensional image to a user on a display device, in particular wherein the face is a face of a user.

[0040] According to another embodiment, the camera and the display device are parts of the same mobile device, wherein the method is performed by means of one or more algorithms installed on a computing unit of the mobile device.

[0041] According to another embodiment of the method, the virtual tooth model is in the form of a polygonal mesh and includes a plurality of vertices. Calculating the lighting of the tooth model optionally includes calculating shading. The calculation of shading can be performed in a vertex shader and / or performed locally in each vertex or each fragment.

[0042] A second aspect of the invention relates to a mobile device comprising a camera and a display, the display being arranged such that an image of a user's face can be captured by the camera when the user is looking at the display, i.e., for example, a smartphone with a "selfie camera". The device comprises a computing unit configured with at least one algorithm adapted to perform the method according to the first aspect. In particular, an image with an overlay display of a model can be displayed on the display.

[0043] Another aspect of the invention relates to a computer program product comprising a computer code stored in a machine-readable medium or implemented by an electromagnetic wave comprising program code segments and having computer executable instructions for performing the method according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention will be described in detail below with reference to exemplary embodiments accompanied by the accompanying drawings, in which:

[0045] Figure 1 An example embodiment of a computer-implemented method for realistically displaying a 3D tooth model in a face image is shown;

[0046] Figure 2 showing the mouth region identified in the image;

[0047] Figure 3 Determining the real lighting of a tooth model according to a first exemplary embodiment of the method is shown;

[0048] Figure 4 Determining the real lighting of a tooth model according to a second exemplary embodiment of the method is shown; and

[0049] Figure 5 Determining the real lighting of a tooth model according to a third exemplary embodiment of the method is shown. DETAILED DESCRIPTION

[0050] Figure 1 A method for realistically displaying a three-dimensional (3D) tooth model in a face image implemented by a computer according to an exemplary embodiment of the present invention is shown.

[0051] According to the illustrated embodiment, a face image 20 is captured by means of a camera 4. In this embodiment, the camera 4 is a component of a handheld mobile device 10, such as a smartphone. The device 10 comprises a display 12, which is capable of displaying the image 20 in real time to a user, i.e. the person whose image is captured. The three-dimensional (three-dimensional) data of the virtual dental model 1 are provided in a data memory of the mobile device 10. An algorithm provided in a computing unit of the mobile device 10 is adapted to match the dental model 1 into the image to provide an augmented reality (AR) image 25 on the display 12, wherein a rendered dental model 21 (e.g. a two-dimensional (2D) display of the model) is displayed in the mouth area of ​​the face. The processing can be performed offline on an image or video or in real time on a live camera image, thereby providing a live augmented reality application.

[0052] In some embodiments, the dental model 1 may include appliances of the dental model 1, such as braces and retainers or mouth guards for various purposes, which are either temporarily or permanently attached to the denture or real teeth. In some embodiments, the dental model 1 includes only these appliances or only models of the appliances are shown in the image.

[0053] In the field of artificial dentition, the lack of realistic lighting information for rendering digital tooth models 1 leads to unsatisfactory results. Realistic display is important in this field, because people are very sensitive to even small deviations from the reality of the human face. In order to overcome this "uncanny valley" problem of virtually displaying teeth in a photo or video of the face, realistic lighting of the three-dimensional rendered content is critical. According to the invention disclosed herein, realism can be achieved by rendering shadows and optionally also by applying color correction, so that the lighting of the rendered virtual content should match the lighting of the real scene, i.e. essentially the face.

[0054] In one embodiment, a given or constant lighting situation can be simply assumed for calculating shadows. For a more realistic effect, the approximate lighting situation in the image can be detected instead. The face can be used as a light probe to get the approximate lighting of the face to estimate the lighting. Both low-frequency lighting (such as with spherical harmonic lighting or virtual point light sources) and high-frequency lighting for reflections (such as from the eyes) can be estimated.

[0055] According to the invention, shading can be calculated even without a detailed 3D model of the face. According to some embodiments of the invention, a 2D image of the mouth region 22 (e.g., a binary 2D mask image) is used as a 3D shading filter. In these embodiments, Figure 2As shown, adding real lighting to the dental model 1 includes identifying the boundaries of the mouth area 22 in the imaged face. Figure 2 In the example embodiment of the present invention, the mouth region 22 includes the portion of the imaged face located inside the lips. The mouth region 22 may be defined by a user (e.g., by selecting an area within the image 20) or automatically identified by a feature recognition algorithm.

[0056] Having identified the boundaries of the mouth region 22, these can be used to calculate the lighting and shadows in the area behind the mouth region 22, which is actually the area corresponding to where the dentition of the virtual model 1 will be arranged in reality. The method uses the three-dimensional mouth contour to cast shadows. Optionally, color correction is then applied to match camera image properties, such as light balance, brightness, saturation, contrast, etc. This is described below. Figure 3 , 4 and 5 are shown.

[0057] If the acquired two-dimensional image also carries depth information, for example acquired using a depth sensor, then the effective three-dimensional information of the face can be used to provide the required three-dimensional mouth contour.

[0058] If only a two-dimensional (2D) mouth region can be obtained from a two-dimensional image, then the mouth region and surrounding parts of the face need to be introduced into three dimensions. Figure 3 In the embodiment of the method shown in , at least the mouth area or its shape is projected onto a three-dimensional geometry 2 close to the geometry of a human face. In particular, the geometry 2 may be the side of a cylinder or a prism. Alternatively, a three-dimensional face model of an explicit face geometry may be generated to cast shadows. The known two-dimensional boundaries of the mouth area 22 that have been projected into three dimensions may then be used to calculate three-dimensional lighting and shadows.

[0059] Having assumed or estimated the position of the light source 5 relative to the face in the image, the position of each point of the projection 3 of the mouth area can be determined. The model 1 is positioned behind the mouth area, at the position where the real dentition is arranged in the real oral cavity. It is assumed that the three-dimensional pose of the virtual dental model 1 is known, for example, based on a deep learning method or a facial marker with three-dimensional head pose estimation, given the camera parameters for rendering the three-dimensional model 1. Accordingly, the model 1 can be positioned behind the projection with the correct pose.

[0060] For a plurality of points 6 of the model 1, it is then determined whether the intersection point 7 of the light from the light source 5 and the three-dimensional geometry 2 is located within the mouth region. If so, the point 6 is illuminated by the light source 5, and if not, the point is obscured by the face. If the virtual dental model 1 is in the form of a polygonal mesh, the point can be a vertex or a fragment. The lighting of the virtual dental model 1 can be calculated locally to take into account both the estimated lighting and the shadows from outside the mouth region.

[0061] The border of the defined mouth area can be optionally implemented as a soft or blurred edge to avoid unrealistic hard shadow edges, especially when using point-like virtual light sources. In addition, other lighting and shading can be applied, such as specular reflection, color or ambient occlusion. Optionally, further camera effects can be simulated to display the rendered model more realistically. The camera effects include, for example, dynamic range, contrast, saturation, noise, lens distortion, white balance, defocus or motion blur.

[0062] Figure 4 Another embodiment of the present invention is shown, which shows a dental model 1 and a three-dimensional geometric body 2 in a cross-sectional view. For a point 6 on the surface of the model 1, the intersection point 7 of the three light rays originating from the three light sources 5a-c is detected at a position where it is located inside or outside the mouth area. Here, the intersection point 7 of the light rays from the light sources 5a and 5b is located inside the mouth area, but the intersection point 7 with respect to the light source 5c is not located inside the mouth area. This means that the point 6 is illuminated by only two of the three light sources, namely the light sources 5a and 5b.

[0063] The model 1 has four different brightness levels. This depends on how many of the three light sources 5a-c illuminate a point or part of the model 1. The maximum brightness is located in the center of the model, where all three light sources 5a-c illuminate the surface of the dental model 1. Point 6 is located in an area with a second brightness level. The third brightness level is illuminated by only one of the light sources 5a-c, and the fourth brightness level is completely hidden by the face when projected on a three-dimensional geometry.

[0064] It is important to provide the correct brightness for each of the multiple brightness levels, i.e. not to make the rendered model or parts thereof appear too dark or too bright to look realistic. Thus, the method may optionally include the step of detecting the lighting conditions of the oral cavity background in the image, for example by how much darker the inside of the oral cavity is than the outside of the face. To this end, the darkest point of the oral cavity background may be determined for use as a reference for shadows. Additionally or alternatively, the contrast level of the camera image may be detected.

[0065] Figure 5 Another embodiment of determining the real lighting of the dental model 1 is shown. This embodiment comprises performing the lighting calculations in the vertex shader. A ray is generated for each vertex 6 to each virtual point light source 5. Its intersection 7 with the face substitute geometry 2 (e.g. a cylinder) is used to create a ray to the (virtual) camera 4. Its projection 8 in the image plane 3 is the observation position to determine whether the light is visible through the mouth area, i.e. not blocked by the lips or facial tissue.

[0066] The lighting information has been added to the model, and the rendered 3D model only overlaps the defined mouth area on the photograph. Likewise, the colors can be adjusted so that the 3D rendered content matches the camera picture. One method of matching colors is to automatically find the average tooth color and / or the average gum or lip color in the image. This can be achieved by automatically dividing the tooth pixels and the gum and lip pixels. These average colors are used as reference colors and can be used to convert the color space into the color space estimated from the camera image. Of course, instead of using the average color, the reference color can also be obtained using the minimum and / or maximum values ​​or histograms.

[0067] If a video is rendered instead of a single image, in order to avoid flickering over time, a reference color derived from a previous frame may be taken into account, for example by temporally smoothing the value.

[0068] In an example embodiment, a method for realistically displaying a virtual tooth model in a facial image is provided. The method includes acquiring a virtual tooth model. The virtual tooth model includes a three-dimensional display of a tooth object to be displayed, such as a denture or other dental prosthesis, a brace, a mouth guard, a tooth ornament, etc. An assumed, measured, or approximate lighting condition of the image is determined. For example, the method may include determining the position of at least one image light source used to illuminate the face of the image. A mouth area in the image is determined, wherein the mouth area is defined by the inner boundary of the lips of the face. In some embodiments, the mouth area is determined using a processor. The processor is used to calculate shadows in the mouth area based on facial geometry and determined lighting conditions (such as the position of at least one light source). The facial geometry may be, for example, the approximate geometry of the face. The lighting of the tooth model is calculated using a processor, wherein the lighting is based on shadows calculated in the mouth area. The tooth model and the calculated applied lighting are displayed on a display, and wherein the tooth model is shown in the mouth area of ​​the image.

[0069] In the context of the present invention, the term "denture" is not necessarily limited to a complete denture, but also includes a partial denture or an orthodontic situation / adjustment or a dental restoration such as a dental prosthesis, including crowns, copings, veneers, inlays and onlays, bridges, dental implants, implant restorations. Accordingly, the term "dental model" includes all models of dental prostheses for oral restoration or of a patient situation that may be partially or completely edentulous, such as models of a complete denture or a partial denture.

[0070] In some embodiments, the present invention is implemented using a system having a camera, a processor, an electronic data storage unit, and a display. The camera may be a standard camera, an infrared dot projection detector, a flood illumination camera, a structured light three-dimensional scanner, a standard infrared detector, an ultrasonic imaging device, a Doppler detector, or any other suitable display system capable of collecting information about a patient's teeth. The processor may be a single processor having one or more cores or a plurality of processors connected by a bus, network, or other data link. The electronic data storage unit may be any form of non-transitory computer-readable storage medium suitable for storing data generated by the system. The display may be any display suitable for displaying electronic color or grayscale images.

[0071] In some embodiments, the camera, memory, electronic data storage unit, and digital display may be part of a single device. The single device may be a smartphone, tablet computer, laptop computer, personal digital assistant, or other computing device.

[0072] In some embodiments, the processor communicates with an external processor and / or a network-connected electronic data storage unit for performing one or more computational steps via a network that may be wired or wireless. In some embodiments, the present invention employs cloud computing to remotely perform one or more computational steps and / or employs remote storage for remotely enabling data storage for collaborative or remote analysis. In some embodiments, the system includes multiple graphical user interfaces to allow multiple users to view and analyze the same data.

[0073] In some embodiments, the operating system provides one or more users with a display of a virtual dental model of the patient's teeth that can be adjusted to display the effects of one or more dental or orthodontic adjustments. In some embodiments, this allows one or more users to view a "before" dentition image, i.e., how the patient's dentition would have appeared prior to dental or orthodontic surgery, and an "after" dentition image, i.e., a display of the desired appearance of the patient's dentition after a proposed dental or orthodontic surgery.

[0074] In some embodiments, the system operates by acquiring information about the patient's dentition using a camera, creating a model of the patient's dentition on a processor, matching the model of the proposed adjusted dentition to the patient's dentition on the processor, coloring the proposed adjusted dentition to match the desired real adjusted color, and displaying the matching model of the proposed adjusted dentition in the patient's real dentition or showing the patient's real facial features on a display. The information about the patient's dentition, the model of the patient's dentition, and the model of the proposed adjusted dentition can be stored on an electronic data storage unit. In some embodiments, the operations are performed in real time.

[0075] In some embodiments, the user interface is configured so that the user can view both a "before" dentition image and an "after" dentition image simultaneously, either side-by-side, or panoramically, or partially overlapping.

[0076] Among them, the term "non-transitory" used in this article is a limitation on the computer-readable storage medium itself, that is, it is tangible rather than one-time, which is contrary to the limitation on the persistence of data storage. Non-transitory computer-readable storage media do not necessarily store information permanently. Random access memory (which may be volatile, non-volatile, dynamic, static, etc.), read-only memory, flash memory, buffer memory or any other tangible, computer-readable storage medium (synchronous or asynchronous) can achieve permanent storage of information.

[0077] Although the present invention has been described above in part with reference to some preferred embodiments, it must be understood that various features of the embodiments may be modified and combined in many ways. All such modifications are within the scope of the appended claims.

Claims

1. A method for realistically displaying a virtual tooth model (1) in a two-dimensional image (20) of a face, the image comprising at least a mouth region (22) of the face, the tooth model (1) comprising a three-dimensional display of a denture or other dental prosthesis, the method comprising: - setting a lighting situation in the facial image (20), the lighting situation comprising one or more light sources (5), - determining the boundaries of said mouth region (22) in the image, wherein said boundaries are defined by the inner boundaries of the lips of the face, - the border of the mouth region (22) is projected onto a three-dimensional face geometry (2) that approximates the geometry of a human face, - calculating the shadows in the mouth region (22) based on the projected boundaries of the mouth region (22), based on the three-dimensional face geometry (2) and based on the lighting conditions, - calculating the lighting of the tooth model (1) at least based on the calculated shadows, and - displaying the dental model (1) and the two-dimensional image (20), wherein the display (21) of the dental model (1) has a calculated illumination.

2. The method according to claim 1, characterized in that: The three-dimensional face geometry (2) comprises at least a portion of the side surface of a cylinder or a prism, and an image plane (3) comprising at least the image portion of the mouth region is projected onto the portion of the side surface.

3. The method according to any one of the preceding claims, characterized in that Calculate the lighting, including - generating a first light beam reaching each of a plurality of virtual point light sources (5) for a plurality of surface points (6) on the surface of the virtual tooth model (1), - generating for each first ray a second ray reaching the virtual camera, the intersection point (7) of the first ray with the three-dimensional face geometry (2) being used as the starting point of the second ray, and the intersection point (8) of the second ray of light with the image plane (3) is used as the observation position for determining the illumination of the corresponding surface point (6), wherein if the second ray of light intersects the image plane (3) in the mouth region (22), the surface point (6) is assumed to be illuminated by a virtual point light source (5), A plurality of the surface points (6) are vertices of the virtual tooth model (1).

4. The method according to claim 1, characterized in that: Displaying the tooth model (1) in the two-dimensional image (20) includes adapting the coloring of the tooth model (1) based on the coloring of the image (20), wherein adapting the coloring includes - identifying a reference color of at least one of teeth, gums and lips in said image (20), wherein - the reference color is an average color, and / or - identifying the reference color comprises automatically segmenting image pixels in the image (20) associated with teeth, gums and / or lips, - using a reference color or a plurality of reference colors to estimate the color space of the image (20), and - using the color space of the image (20) to convert the color space of the tooth model (1), wherein the color space of the image (20) is used for the color space of the tooth model (1).

5. The method according to claim 1, characterized in that: The illumination of the tooth model (1) is calculated based on an estimated illumination condition, wherein the illumination condition is estimated based on the illumination of the face in the image (20), wherein estimating the illumination condition comprises - estimate low frequency lighting with spherical harmonic lighting or virtual point lights, and / or - Detecting reflections in the eyes of the face to estimate the position of the light source.

6. The method according to claim 1, characterized in that: A soft or blurred edge of the mouth region (22) is calculated from the determined boundary, wherein the shadow is calculated based on the soft or blurred edge in order to avoid harsh shadow edges.

7. The method according to claim 1, characterized in that: - calculating the lighting of the dental model (1) comprises applying lighting and shading effects, the effects comprising at least one of specular reflection, color or ambient occlusion; and / or - displaying the dental model includes simulating camera effects to make the appearance of the dental model (1) close to the appearance of the face in the image (20), the simulated camera effects including at least one of dynamic range, contrast, saturation, noise, lens distortion, white balance, defocus and motion blur.

8. The method according to claim 1, characterized in that: The image (20) is part of an image stream, and the method is performed in real time for at least a plurality of frames of images (20) in the image stream.

9. The method according to claim 8, characterized in that: Displaying the dental model (1) in the two-dimensional image (20) includes adapting the coloring of the dental model (1) based on the coloring of multiple frames (20) of the image stream, including current images and previous images of the image stream, wherein an average value from multiple consecutive frames is used for adapting the coloring.

10. The method according to claim 8 or claim 9, characterized in that: The method also includes acquiring the image stream by means of a camera (4) and displaying the dental model (1) and the two-dimensional image (20) to a user on a display device (12), wherein the face is the face of the user.

11. The method according to claim 10, characterized in that: The camera (4) and the display device (12) are components of the same mobile device (10), wherein the method is performed by means of one or more algorithms installed on a computing unit of the mobile device (10).

12. The method according to claim 1, characterized in that: The virtual tooth model (1) is in the form of a polygonal mesh and comprises a plurality of vertices, wherein calculating the lighting of the tooth model comprises calculating shading, wherein the calculation of the shading is performed in a vertex shader and / or locally at each vertex or each fragment.

13. A mobile device (10) comprising a camera (4) and a display (12), wherein the display (12) is arranged so that an image (20) of a user's face can be captured by the camera (4) when the user looks at the display (12), wherein: The device comprises a computing unit equipped with at least one algorithm suitable for executing the method according to any of the preceding claims.

14. A computer program product comprising a computer code stored in a machine-readable medium and having computer-executable instructions for performing the method according to any one of claims 1 to 12.