Method for determining occlusion relation of three-dimensional digital models of upper jaw teeth and lower jaw teeth based on tooth photos

Abstract

The invention provides a computer-implemented method for determining the occlusion relationship of three-dimensional digital models of maxillary and mandibular dentitions based on tooth photos, which comprises the following steps: acquiring a first photo of teeth of a patient in an occlusion state, the first photo comprising a plurality of maxillary teeth and a plurality of mandibular teeth; obtaining a first three-dimensional digital model and a second three-dimensional digital model respectively representing the upper jaw dentition and the lower jaw dentition of the patient; rendering the first three-dimensional digital model and the second three-dimensional digital model to obtain a first rendering graph; predicting an optical flow between the first photo and the first rendering graph by using a trained optical flow neural network based on the first photo and the first rendering graph; and based on the optical flow, calculating a first spatial position transformation relationship which can transform the relative position relationship between the first and second three-dimensional digital models to a state closer to the bite state shown by the first photograph, with a solution to solve the PnP problem, and calculating a second spatial position transformation relationship which can transform the relative position relationship between the first and second three-dimensional digital models to a state closer to the bite state shown by the first photograph.

Description

Technical Field

[0001] The present application generally relates to a method for determining the occlusal relationship of a three-dimensional digital model of upper and lower teeth based on dental photographs. Background Art

[0002] With the rapid development of computer technology, dental diagnosis and treatment are increasingly relying on computer technology, among which the use of three-dimensional digital models of dentition is particularly widespread.

[0003] The three-dimensional digital model of the dentition is generally obtained by performing an intraoral scan on the patient or scanning the patient's impression or a physical model of the patient's dentition (for example, a plaster model). The three-dimensional digital models of the upper and lower dentition obtained in this way are independent of each other.

[0004] In some cases, dental treatment requires the actual occlusal relationship between the three-dimensional digital models of the upper and mandibular dentition. Currently, there are two methods to determine the actual occlusal relationship between the three-dimensional digital models of the upper and mandibular dentition. One is that dental professionals manually adjust the relative positional relationship between the three-dimensional digital models of the upper and mandibular dentition through a computer operating interface so that they are in an actual occlusal relationship. The other is to use a three-dimensional scanning device to scan the patient's upper and mandibular dentition in an occlusal state, and then adjust the relative position of the three-dimensional digital models of the patient's upper and mandibular dentition based on the scan results so that they are in an actual occlusal relationship. However, the first method mentioned above requires dental professionals to undergo special training and is time-consuming and labor-intensive. The second method mentioned above requires a three-dimensional scanning device, which is usually only available in professional dental institutions. In order to determine the occlusal relationship, patients are required to make an additional visit to a professional dental institution, which increases the burden on patients.

[0005] In view of the above, it is necessary to provide a new method for determining the occlusal relationship of the three-dimensional digital models of the maxillary and mandibular dentition. Summary of the Invention

[0006] One aspect of the present application provides a computer-implemented method for determining the occlusal relationship of three-dimensional digital models of upper and lower dentition based on dental photographs, comprising: obtaining a first photograph of the patient's teeth in an occlusal state, the first photograph including a plurality of upper teeth and a plurality of lower teeth; obtaining a first and a second three-dimensional digital model representing the patient's upper dentition and mandibular dentition, respectively; rendering the first and second three-dimensional digital models to obtain a first rendering image; using a trained optical flow neural network to predict the optical flow between the two based on the first photograph and the first rendering image; and based on the optical flow, using a solution to the PnP problem, calculating a first spatial position transformation relationship, which can transform the relative position relationship between the first and second three-dimensional digital models to an occlusal state closer to that shown in the first photograph.

[0007] In some embodiments, the calculation of the first spatial position transformation relationship is performed through multiple iterations, and the first rendering in each iteration is obtained by rendering the first and second three-dimensional digital models in an updated relative position relationship, wherein the updated relative position relationship is calculated based on the first spatial position transformation relationship calculated in the previous iteration.

[0008] In some embodiments, the method further includes: obtaining a first set of shooting parameters; rendering the first or second three-dimensional digital model with the first set of shooting parameters to obtain a second rendering; using the trained optical flow neural network to predict the optical flow between the first photo and the second rendering based on the first photo and the second rendering; calculating a second spatial position transformation relationship based on the optical flow using a solution to the PnP problem; and calculating a second set of shooting parameters based on the first set of shooting parameters and the second spatial position transformation relationship, the second set of shooting parameters being closer to the shooting parameters of the first photo than the first set of shooting parameters, wherein the first rendering is rendered based on the second set of shooting parameters.

[0009] In some embodiments, the calculation of the second spatial position transformation relationship is performed through multiple iterations, and the second rendering in each iteration is obtained by rendering the first or second three-dimensional digital model using the updated second set of shooting parameters, wherein the updated second set of shooting parameters is calculated based on the second spatial position transformation relationship calculated in the previous iteration.

[0010] In some embodiments, the calculation of the first spatial position transformation relationship includes: using the trained optical flow neural network to predict the optical flow LF1 based on the first photograph and the maxillary dentition part in the first rendering; based on the optical flow LF1, calculating the spatial position transformation relationship T1 with a solution to the PnP problem; using the trained optical flow neural network to predict the optical flow LF2 based on the first photograph and the mandibular dentition part in the first rendering; based on the optical flow LF2, calculating the spatial position transformation relationship T2 with a solution to the PnP problem; and calculating the first spatial position transformation relationship based on the spatial position transformation relationships T1 and T2.

[0011] In some embodiments, the method further includes: obtaining a second photograph of the patient's teeth in an occluded state, the second photograph being taken at a different angle than the first photograph, the second photograph including a plurality of maxillary teeth and a plurality of mandibular teeth; rendering the first and second three-dimensional digital models to obtain a third rendering; using a trained optical flow neural network to predict the optical flow between the second photograph and the third rendering; and calculating the first spatial position transformation relationship based on the optical flow between the first photograph and the first rendering and the optical flow between the second photograph and the third rendering to solve the PnP problem.

[0012] In some embodiments, the input of the trained optical flow neural network can be one of the following: a tooth mask map generated based on the first photograph and a mask map obtained by rendering the first and second three-dimensional digital models; a tooth RGB image generated based on the first photograph and a mask map obtained by rendering the first and second three-dimensional digital models; a tooth mask map generated based on the first photograph and a mask map obtained by rendering the first and second three-dimensional digital models; and a tooth RGB image generated based on the first photograph and an RGB image obtained by rendering the first and second three-dimensional digital models, wherein, in the mask map, the attributes of the areas of each two adjacent teeth are different.

[0013] In some embodiments, the first and second three-dimensional digital models are roughly registered so that their relative positional relationship is close to the occlusal relationship shown in the first photograph.

[0014] In some embodiments, the optical flow is dense optical flow.

[0015] In some embodiments, the output of the optical flow neural network further includes an occlusion mask for filtering mutually invisible portions of the two two-dimensional images input into the optical flow neural network.

[0016] In some embodiments, the output of the optical flow neural network also includes the confidence of the optical flow of each point pair. In the calculation of the solution based on optical flow to solve the PnP problem, the optical flow of each point pair is assigned a weight according to its confidence, and the higher the confidence, the higher the weight.

[0017] In some embodiments, the optical flow neural network is trained using simulated data in a supervised learning manner, and then trained using real data in a self-supervised learning manner.

[0018] In some embodiments, in the self-supervised learning training, consistency loss is used as a loss function, which includes key point loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features of the present application will be further described below in conjunction with the accompanying drawings and detailed description thereof. It should be understood that these drawings only illustrate several exemplary embodiments of the present application and should not be considered to limit the scope of protection of the present application. Unless otherwise specified, the drawings are not necessarily to scale, and similar reference numerals represent similar components.

[0020] Figure 1 This is a schematic flow chart of a method for determining the occlusal relationship of upper and lower mandibular teeth three-dimensional digital models based on dental photographs in one embodiment of the present application;

[0021] Figure 2 For example, a photo of a patient's teeth;

[0022] Figure 3 for Figure 2 a grayscale image of the portion of the tooth in the shown dental photograph; and

[0023] Figure 4 for Figure 2 Grayscale rendering of the 3D digital model of the maxillary and mandibular dentition of the same patient as the dental photograph shown. DETAILED DESCRIPTION

[0024] The following detailed description refers to the drawings that form a part of this specification. The illustrative embodiments mentioned in the specification and drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. In light of this application, those skilled in the art will understand that many other embodiments can be adopted and various changes can be made to the described embodiments without departing from the subject matter and scope of protection of this application. It should be understood that the various aspects of the present application described and illustrated herein can be arranged, replaced, combined, separated and designed according to many different configurations, all of which are within the scope of protection of this application.

[0025] One aspect of the present application provides a computer-implemented method for determining the occlusal relationship of a three-dimensional digital model of the upper and lower mandibular dentition based on a dental photograph. It will be understood that determining the occlusal relationship of the three-dimensional digital model of the upper and lower mandibular dentition refers to determining the relative positional relationship of the three-dimensional digital model of the upper and lower mandibular dentition in an occlusal state, such that the occlusal relationship of the three-dimensional digital model of the upper and lower mandibular dentition is consistent with the occlusal relationship of the upper and lower mandibular dentition in the dental photograph.

[0026] In one embodiment, the patient's dental photos can be taken by a non-dental professional, such as the patient himself or her family or friends. The patient provides the dental photos to a professional institution, which uses a computer to determine the occlusal relationship of the patient's upper and lower jaw dentition three-dimensional models based on the photos.

[0027] Based on the three-dimensional digital model of the patient's upper and lower dentition in the occlusal state before orthodontic treatment, an orthodontic treatment plan can be designed for the patient. Based on the three-dimensional digital model of the patient's upper and lower dentition in the occlusal state during orthodontic treatment, the orthognathic progress of orthodontic treatment can be monitored. In light of this application, it can be understood that in addition to the above uses, the occlusal relationship of the three-dimensional digital model of the upper and lower dentition determined using the method of this application can also be used for other purposes, which are not listed here one by one.

[0028] Please refer to Figure 1 , is a schematic flowchart of a method 100 for determining the occlusal relationship of the upper and lower mandibular dentition three-dimensional digital models based on dental photographs executed by a computer in one embodiment of the present application.

[0029] In 101 , at least one photograph of a patient's teeth in a biting state is obtained.

[0030] The dental photographs are two-dimensional, and each of the dental photographs includes a plurality of maxillary teeth and a plurality of mandibular teeth.

[0031] In one embodiment, only one dental photograph may be obtained, and the occlusal relationship of the three-dimensional digital model of the upper and lower dentition may be determined based on this one dental photograph. In another embodiment, two dental photographs taken from different angles may be obtained, for example, the left side and the front, or the right side and the front, or the left side and the right side, and the occlusal relationship of the three-dimensional digital model of the upper and lower dentition may be determined based on these two photographs. In another embodiment, three dental photographs taken from different angles may be obtained, namely the left side, the front and the right side, and the occlusal relationship of the three-dimensional digital model of the upper and lower dentition may be determined based on these three photographs. In some cases, determining the occlusal relationship of the three-dimensional digital model of the upper and lower dentition based on two or more dental photographs taken at different angles is conducive to reducing the error between the determined occlusal relationship and the actual occlusal relationship. The following describes the present application in detail by taking the determination of the occlusal relationship of the three-dimensional digital model of the upper and lower dentition based on a single dental photograph as an example.

[0032] In one embodiment, the patient's teeth photos can be taken by the patient himself, without the need to return to a professional dental institution, thereby reducing the burden on the patient.

[0033] Currently, there are devices that can allow non-dental professionals to take dental photos, for example, the dental photography device disclosed in Chinese Patent No. ZL202220617972.5 or Chinese Patent No. ZL202223452359.1.

[0034] Please refer to Figure 2 , is an example of a frontal dental photo taken by the patient himself.

[0035] Since only the tooth portion in the tooth photo is needed in subsequent operations, the tooth portion in the tooth photo can be segmented. In one embodiment, a trained deep neural network can be used to segment the tooth portion from the tooth photo.

[0036] In one embodiment, to improve accuracy and robustness, the tooth images segmented from the dental photograph can be converted into grayscale images, and the three-dimensional digital models of the patient's upper and lower dentition can be rendered to obtain grayscale images. Subsequently, optical flow prediction can be performed based on these two grayscale images. In another embodiment, both can also be rendered into images of other colors.

[0037] Please refer to Figure 3 , for the general Figure 2 Grayscale image obtained by rendering the tooth part in the image.

[0038] In 103, first and second three-dimensional digital models representing the patient's maxillary dentition and mandibular dentition, respectively, are obtained.

[0039] In one embodiment, the first and second three-dimensional digital models can be obtained by intraoral scanning. In another embodiment, the first and second three-dimensional digital models can be obtained by scanning a physical model of the patient's jaw, such as a plaster model. In another embodiment, the first and second three-dimensional digital models can be obtained by scanning an impression of the patient.

[0040] The first and second three-dimensional digital models are independent of each other. In one embodiment, to more quickly determine the occlusal relationship between the two, the relative positional relationship between the two can be adjusted to roughly approximate the actual occlusal relationship. In one embodiment, a trained deep neural network can be used to adjust the relative positional relationship between the first and second three-dimensional digital models to roughly approximate the actual occlusal relationship.

[0041] In 105, the trained optical flow neural network is used to predict the optical flow between the two based on the tooth photo and the renderings of the first and second three-dimensional digital models, and the relative position relationship between the first and second three-dimensional digital models in the occlusal state is calculated based on the predicted optical flow.

[0042] In one embodiment, PyRender may be used to render the first and second three-dimensional digital models to obtain the rendering images. It is understood that, in addition to PyRender, any other applicable method may also be used to render the first and second three-dimensional digital models.

[0043] The intrinsic parameters of the camera used to capture the dental photograph are known, while the extrinsic parameters can be roughly estimated based on the angle of the dental photograph. Therefore, in one embodiment, the intrinsic parameters and the estimated extrinsic parameters can be used as camera parameters for rendering the first and second three-dimensional digital models.

[0044] Please refer to Figure 4 , shows the grayscale images obtained by rendering the first and second three-dimensional digital models with given camera parameters in an example.

[0045] Since the above-mentioned camera parameters may not be close enough to the shooting parameters of the dental photograph, in order to improve the prediction accuracy of the upper and lower jaw dentition postures in the occlusal state, the trained optical flow neural network can be used to optimize the given shooting parameters.

[0046] Since the relative positional relationship between the first and second three-dimensional digital models may be inconsistent with that in the dental photograph, when optimizing the given shooting parameters, optical flow prediction can be performed based only on the maxillary dentition or the mandibular dentition.

[0047] In a preferred embodiment, the optical flow may be a dense optical flow.

[0048] For example, the tooth photograph converted into a grayscale image and the rendering of the three-dimensional digital model of the maxillary dentition can be input into the trained optical flow neural network to predict the optical flow between the two. Then, based on the predicted optical flow, a spatial position transformation relationship, for example, a spatial position transformation matrix, is calculated using the solution to the PnP (Perspective-n-Point) problem. Finally, the optimized shooting parameters are calculated based on the spatial position transformation relationship and the given shooting parameters. Generally, in the occluded state, the maxillary dentition partially covers the mandibular dentition. Therefore, it is preferred to optimize the shooting parameters based on the maxillary dentition.

[0049] The input of the optical flow neural network is two two-dimensional images, one corresponding to the dental photograph and the other obtained by rendering the first and second three-dimensional digital models.

[0050] In one embodiment, the spatial position transformation relationship can be calculated using a DLT (Direct Linear Transformation) solution. In light of this application, it can be understood that in addition to the DLT solution, any other applicable solution can be used to solve the PnP problem, such as a gradient descent method or Newton's method.

[0051] After obtaining the optimized shooting parameters, the first and second three-dimensional digital models can be re-rendered using them to obtain an updated rendering. Then, the optical flow neural network is used to predict the optical flow of the maxillary dentition and the mandibular dentition between the dental photo and the updated rendering. Based on the two predicted optical flows, the first and second spatial position transformation relationships are calculated using the solution to the PnP problem, which respectively represent the spatial position transformation relationship between the first dentition and the second dentition between the updated rendering and the dental photo. Finally, based on the first and second spatial position transformation relationships, a third spatial position transformation relationship is calculated, which is used to adjust the relative position between the first and second three-dimensional digital models to be close to the occlusal state shown in the dental photo.

[0052] After a single adjustment, the relative positional relationship between the first and second three-dimensional digital models may not be close enough to the occlusal state shown in the dental photograph. In one embodiment, the relative positional relationship between the first and second three-dimensional digital models can be adjusted to be close enough to the occlusal state shown in the dental photograph through iteration.

[0053] In one embodiment, each iteration may only optimize the relative positional relationship between the first and second three-dimensional digital models. In another embodiment, each iteration may first optimize the shooting parameters and then optimize the relative positional relationship between the first and second three-dimensional digital models. In yet another embodiment, the shooting parameters may first be iteratively optimized to be sufficiently close to the shooting parameters of the dental photograph, and then the relative positional relationship between the first and second three-dimensional digital models may be iteratively optimized.

[0054] In one embodiment, the number of iterations may be pre-set, and the relative positional relationship between the first and second three-dimensional digital models obtained after the iterations is used as the final relative positional relationship.

[0055] In one embodiment, when predicting the optical flow of a single dentition (e.g., maxillary dentition or mandibular dentition), the dental photograph and a rendering of a three-dimensional digital model of the single dentition may be input into the optical flow neural network. In another embodiment, when predicting the optical flow of a single dentition, the portion of the dental photograph corresponding to the single dentition and a rendering of the three-dimensional digital model of the single dentition may also be input into the optical flow neural network. In one embodiment, a trained deep neural network may be used to segment the maxillary and mandibular dentition in the dental photograph.

[0056] The optical flow neural network can be any applicable deep neural network capable of predicting optical flow, including but not limited to FlowNet network, LiteFlowNet network and PWC-Net network.

[0057] In one embodiment, the dental photo and the rendering are input into the optical flow neural network, which outputs the optical flow (i.e., the correspondence between each pixel in the dental photo and the rendering), the optical flow confidence (i.e., the confidence of each correspondence), and the occlusion mask (including the tooth parts that are visible in the dental photo but not in the rendering, and the tooth parts that are visible in the rendering but not in the dental photo). In particular, when calculating the spatial position transformation relationship based on the predicted optical flow using the solution to the PnP problem, the optical flow of each point pair is assigned a corresponding weight according to its confidence. The higher the confidence, the greater the weight, which helps to improve the robustness of the matching. The occlusion mask can remove the occlusion problem caused by the rigid motion of the three-dimensional object.

[0058] In one embodiment, the optical flow neural network can be trained in two steps. First, supervised training is performed on the optical flow neural network using simulated data to enable it to learn how to predict optical flow. Then, self-supervised training is performed on the optical flow neural network using real data to enable it to learn how to generate optical flow based on real images and rendered images.

[0059] In one example, data from 2,000 patients were collected for training the optical flow neural network, and the data for each patient included 1 to 3 dental photos (for example, left-side dental photos, right-side dental photos, and front dental photos), a three-dimensional digital model of the maxillary dentition, and a three-dimensional digital model of the mandibular dentition, wherein the relative positional relationship between the three-dimensional digital models of the upper and mandibular dentitions in each patient's occlusal state was known.

[0060] For the supervised training, the training data required for one training session can be generated by the following method. A first set of shooting parameters is used to render the three-dimensional digital model of the upper and lower jaws of a first patient in a first relative position relationship to obtain a first rendering image. A second set of shooting parameters is used to render the three-dimensional digital model of the upper and lower jaws of the first patient in a second relative position relationship to obtain a second rendering image. The first and second sets of shooting parameters may be different, and the first and second relative position relationships are different. Since the correspondence between the pixels in the first and second rendering images is known, the optical flow between the two can be calculated based on this. The first and second rendering images are used as inputs to the deep neural network, and the calculated optical flow is used as the ground truth.

[0061] In one embodiment, noise may be added when generating renderings for training to improve the robustness of the network.

[0062] For details on how to generate simulated data for training optical flow neural networks and how to train optical flow neural networks in a supervised learning manner, please refer to the paper "FlowNet: Learning Optical Flow with Convolutional Networks" published by Philipp Fischer et al. on arXiv in 2015.

[0063] For the self-supervised training, the training data required for a single training session can be generated using the following method: A dental photograph of the first patient is used to render the three-dimensional digital models of the upper and lower jaws of the first patient using parameters that are substantially similar to those of the dental photograph to obtain a rendered image. The dental photograph and rendered image are used as input to the optical flow neural network.

[0064] In one embodiment, consistency loss can be used as a loss function for the self-supervised training based on real data. The loss function can be defined by the following equation (1):

[0065] Loss = D(warp(I0,F),I1) Equation (1)

[0066] Among them, I0 and I1 represent two images input into the optical flow neural network, F represents the optical flow predicted by the optical flow neural network based on the input I0 and I1, the warp function transforms the image I0 based on the predicted optical flow F to obtain the image I1', and D measures the similarity between I1 and I1', which can include pixel-level error, deep feature error (perceptual loss) and key point error. Among them, the key point error is used to solve the mismatching problem caused by the lack of tooth texture, which can make the optical flow learning more accurate.

[0067] In another embodiment, the optical flow neural network may be trained in a reinforcement learning manner based on real data.

[0068] In the above specific embodiment, the occlusal relationship of the upper and lower mandibular dentition 3D digital models was determined based on a single dental photograph. For multiple dental photographs, the spatial position transformation relationship can be jointly solved using the PnP problem solution based on the optical flow obtained for each dental photograph.

[0069] In one embodiment, a trained deep neural network can be used to segment each tooth in a dental photograph, obtain a mask for each tooth, and generate a segmentation mask image. In the segmentation mask image, the regions corresponding to each two adjacent teeth have different attributes to distinguish the regions of the two adjacent teeth. The attributes can be colors or other assigned attributes, such as codes or other labels.

[0070] The teeth of the first and second three-dimensional digital models can be segmented using a trained deep neural network and then rendered to obtain a mask image. Similarly, the attributes of the region corresponding to each two adjacent teeth in the segmentation mask image are different, so as to distinguish the regions of the two adjacent teeth.

[0071] The segmentation mask image corresponding to the tooth photo and the rendered segmentation mask image are used as inputs of the optical flow neural network to predict the optical flow between the two.

[0072] In another embodiment, the segmentation mask image corresponding to the dental photograph and the RGB image obtained by rendering the first and second three-dimensional digital models, such as a grayscale image, can also be used as input of the optical flow neural network to predict the optical flow between the two.

[0073] In another embodiment, the RGB image corresponding to the dental photograph and the segmentation mask image obtained by rendering the first and second three-dimensional digital models may be used as inputs of an optical flow neural network to predict the optical flow between the two.

[0074] It is understandable that for different input images, the optical flow neural network needs to be adjusted accordingly.

[0075] Although various aspects and embodiments of the present application are disclosed herein, other aspects and embodiments of the present application will be readily apparent to those skilled in the art in light of this disclosure. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to be limiting. The scope and subject matter of this application are determined solely by the appended claims.

[0076] Similarly, various diagrams may illustrate exemplary architectures or other configurations of the disclosed methods and systems that aid in understanding the features and functionality that may be included in the disclosed methods and systems. The claimed content is not limited to the exemplary architectures or configurations shown, and the desired features may be implemented using a variety of alternative architectures and configurations. Furthermore, for flow charts, functional descriptions, and method claims, the order of blocks presented herein should not limit various embodiments to being implemented in the same order to perform the described functionality, unless the context clearly dictates otherwise.

[0077] Unless otherwise expressly stated, the terms and phrases used herein and their variations should be interpreted as open ended rather than restrictive. In some instances, the appearance of broad words and phrases such as "one or more," "at least," "but not limited to," or other similar terms should not be understood as intending or requiring a narrowing of the context in which such broad terms may not be used.

Claims

1. A computer-implemented method for determining the occlusal relationship of a three-dimensional digital model of upper and lower dentition based on a dental photograph, comprising: Acquire a first photograph of the patient's teeth in an occlusal state, the first photograph including a plurality of maxillary teeth and a plurality of mandibular teeth; obtaining first and second three-dimensional digital models representing the patient's maxillary dentition and mandibular dentition, respectively; Rendering the first and second three-dimensional digital models to obtain a first rendering; Using a trained optical flow neural network, based on the first photo and the first rendering, predict an optical flow between the two; as well as Based on the optical flow, a first spatial position transformation relationship is calculated using a solution to the PnP problem. It is capable of transforming the relative positional relationship between the first and second three-dimensional digital models to be closer to the occlusal state shown in the first photograph.

2. The method according to claim 1, wherein The calculation of the first spatial position transformation relationship is performed through multiple iterations, and the first rendering in each iteration is obtained by rendering the first and second three-dimensional digital models in an updated relative position relationship, wherein the updated relative position relationship is calculated based on the first spatial position transformation relationship calculated in the previous iteration.

3. The method according to claim 1, wherein It also includes: Get the first set of shooting parameters; Rendering the first or second three-dimensional digital model using a first set of shooting parameters to obtain a second rendering; Using the trained optical flow neural network, based on the first photo and the second rendering, predict an optical flow between the two; Based on the optical flow, a second spatial position transformation relationship is calculated using a solution to the PnP problem; and A second set of shooting parameters is calculated based on the first set of shooting parameters and the second spatial position transformation relationship, and the second set of shooting parameters is closer to the shooting parameters of the first photo than the first set of shooting parameters, wherein the first rendering is rendered based on the second set of shooting parameters.

4. The method according to claim 3, wherein The calculation of the second spatial position transformation relationship is performed through multiple iterations, and the second rendering image in each iteration is obtained by rendering the first or second three-dimensional digital model using the updated second set of shooting parameters, wherein the updated second set of shooting parameters is calculated based on the second spatial position transformation relationship calculated in the previous iteration.

5. The method according to claim 1, wherein The calculation of the first spatial position transformation relationship includes: Using the trained optical flow neural network, predicting an optical flow LF1 based on the first photograph and the maxillary dentition portion in the first rendering; Based on the optical flow LF1, a solution to the PnP problem is used to calculate the spatial position transformation relationship T1; Using the trained optical flow neural network, predicting an optical flow LF2 based on the first photograph and the mandibular dentition portion in the first rendering; Based on the optical flow LF2, a solution to the PnP problem is used to calculate the spatial position transformation relationship T2; and A first spatial position transformation relationship is calculated based on the spatial position transformation relationships T1 and T2.

6. The method according to claim 1, wherein It also includes: Obtaining a second photograph of the patient's teeth in an occlusal state, wherein the second photograph is taken at a different angle than the first photograph, and the second photograph includes a plurality of maxillary teeth and a plurality of mandibular teeth; Rendering the first and second three-dimensional digital models to obtain a third rendering; Using the trained optical flow neural network, based on the second photo and the third rendering, predict the optical flow between the two; and The first spatial position transformation relationship is calculated based on the optical flow between the first photo and the first rendering and the optical flow between the second photo and the third rendering using a solution to the PnP problem.

7. The method according to claim 1, wherein The input of the trained optical flow neural network may be one of the following: a tooth mask generated based on the first photograph and a mask obtained by rendering the first and second three-dimensional digital models; a tooth RGB image generated based on the first photograph and a mask obtained by rendering the first and second three-dimensional digital models; a tooth mask generated based on the first photograph and a mask obtained by rendering the first and second three-dimensional digital models; and a tooth RGB image generated based on the first photo and an RGB image obtained by rendering the first and second three-dimensional digital models, wherein in the mask image, the attributes of the regions of every two adjacent teeth are different.

8. The method according to claim 1, wherein The first and second three-dimensional digital models are roughly registered so that their relative positional relationship is close to the occlusal relationship shown in the first photograph.

9. The method according to claim 1, wherein The optical flow is a dense optical flow.

10. The method according to claim 1, wherein The output of the optical flow neural network also includes an occlusion mask, which is used to filter out mutually invisible parts of the two two-dimensional images input into the optical flow neural network.

11. The method according to claim 1, wherein The output of the optical flow neural network also includes the confidence of the optical flow of each point pair. In the calculation of the solution based on optical flow to solve the PnP problem, the optical flow of each point pair is assigned a weight according to its confidence. The higher the confidence, the higher the weight.

12. The method according to claim 1, wherein The optical flow neural network is trained using simulated data in a supervised learning manner, and then trained using real data in a self-supervised learning manner.

13. The method according to claim 12, wherein: In the self-supervised learning training, consistency loss is adopted as the loss function, which includes key point loss.

Images