A method for 3D reconstruction of tooth crowns using sparse views

Abstract

This invention discloses a method for 3D reconstruction of dental crowns using sparse views, comprising: S1, data preprocessing, segmenting the data using a SAM model and obtaining mask images, then removing outliers and noise using the Otsu algorithm to obtain a dental crown photograph; S2, initialization of 3D information of the dental crown, estimating the camera pose under sparse views, and generating corresponding new views, normal maps, and depth maps using a conditional view diffusion model, providing the necessary data foundation for the 3D information of the dental crown, and constructing the preliminary 3D outline of the dental crown using visual shell technology and 3DGS; S3, the 3D model optimization stage of the dental crown, employing a sparse view generation strategy to optimize the view conditional diffusion model and the initial camera pose, and proposing a 3D Gaussian geometry optimization method for the dental crown to further refine the preliminary reconstructed dental crown shape. This invention significantly improves the reconstruction quality of 3D dental crowns and enhances detail representation through the new views, normal maps, and depth maps generated by the conditional diffusion model.

Description

Technical Field

[0001] This invention belongs to the field of dental crown image processing technology, and particularly relates to a method for three-dimensional reconstruction of dental crowns from sparse views. Background Technology

[0002] With the rapid development of oral medicine and artificial intelligence technologies, 3D crown reconstruction technology plays an increasingly important role in orthodontic treatment and restoration. Although some 3D reconstruction methods have been proposed, they often face insufficient accuracy when handling sparse viewpoint input. Some methods rely on dense image input and complex equipment configurations, which are limited in practical orthodontic treatment. While deep learning-based methods have made some progress, there is still room for improvement in the model's generalization ability and detail reconstruction.

[0003] In image-based 3D reconstruction tasks, Structure from Motion (SfM) is a widely used classical method. However, its performance degrades significantly when faced with image sparsity or occlusion. In 2D photographs, the surface of tooth crowns typically lacks texture information, and the spatial position of teeth is difficult to infer directly from the image, making 3D reconstruction of image targets exceptionally challenging. To address these issues, researchers have proposed a method combining shadow shape algorithms and statistical shape priors for 3D tooth reconstruction. Mostafa et al. further developed a model-based SFS tooth restoration method by introducing a coupled statistical model by linking the coefficients of 2D texture models, 3D shape models, and spherically harmonic projection image models. However, the statistical SFS framework for tooth reconstruction still has limitations when handling missing or occluded teeth. For 3D shape reconstruction, 3D deformable models are commonly used. However, this method has not received widespread attention in restoring the 3D shape of teeth.

[0004] In recent years, 3D Geometric Shape (3DGS) analysis technology has made significant progress in computer vision and graphics, bringing revolutionary changes to the task of 3D object reconstruction. 3DGS not only provides an accurate description of the surface morphology of an object but also contains rich spatial structural information, which is particularly important for the complex and varied structures of dental crowns. Combined with deep learning technology, more and more researchers are using 3DGS for target 3D reconstruction; however, its application to dental crown 3D reconstruction is still relatively limited. Although 3DGS demonstrates powerful synthesis capabilities in novel views, it relies on the amount of images and sparse point clouds for rendering inference when synthesizing new viewpoints. When the data volume is insufficient, artifacts not belonging to the specified viewpoint may appear on the reconstructed 3D model, affecting the reconstruction quality. Based on current research, the task of 3D reconstruction of dental crowns using sparse views faces three challenges. The first challenge is how to obtain sufficient information from sparse views for high-quality 3D reconstruction of dental crowns. The second challenge is how to construct multiple views from highly sparse input. Figure 1 Consistency. The third problem lies in how to reconstruct a reasonable, high-quality three-dimensional geometric representation of the crown from missing or compressed crown information. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of quality, consistency, and information omission in existing sparse views.

[0006] To achieve the above objectives, the present invention provides a method for three-dimensional reconstruction of a tooth crown from sparse views, comprising:

[0007] S1. Data preprocessing: The acquired image data is segmented using SAM to obtain a mask image. Then, the Otsu algorithm is used to remove outliers and noise to obtain the crown image. The brightness, size, and sharpness of the crown image are adjusted.

[0008] S2. Initialization of 3D information of the crown: COLMAP is used to estimate the camera pose under sparse view, and the corresponding new crown image I′, crown normal map n′ and crown depth map s′ are generated with the help of the conditional view diffusion model F; the crown contour image is obtained by using the findContours function in the OpenCV image processing library on the new crown image I′.

[0009] S3. 3D model construction of the crown: Based on the outline image and mask image of the crown from multiple perspectives, a visual shell of the crown is created as the geometric framework for initializing the 3D Gaussian, thereby constructing the initial 3D outline of the crown.

[0010] A visual shell for the crown is created based on the contour images and mask images of the crown from multiple perspectives, serving as the geometric framework for initializing the 3D Gaussian. The processed 3D Gaussian points are projected onto the input image plane and preserved within the intersection of all mask image spaces. The color of the 3D points is calculated from the pixels projected from the input image using bilinear interpolation and the average value C(x,y) is taken.

[0011]

[0012] Where C(x,j) and C(x,j+1) are two points on the x-axis; the position of each 3D point is set to μ, and its color is converted to sh; the average distance between adjacent points is set to β, the rotation q is set to a unit quaternion by default, and the opacity is... Set to a constant value; parameterize the initialized visual shell model as a set of Gaussian distributions G, realize the conversion of 3D points into 3D Gaussian representation, and use 3D Gaussian points to display the visual shell model of the crown.

[0013] S4. Optimization of the 3D model of the crown: Based on the crown normal map n′ and crown depth map s′ from step S2, the view condition diffusion model F is optimized, and the backpropagation of the view condition diffusion model is used to refine the initially estimated camera pose π. i Furthermore, the Gaussian set is optimized by combining 3D Gaussian with crown depth maps and crown normal maps.

[0014] Furthermore, the specific implementation method in step S1 is as follows:

[0015] S101. The SAM model is used to segment the crown region in the exit photograph and obtain its mask image. Then, the Otsu algorithm is used to remove outliers and noise to obtain an image containing only the crown.

[0016] S102. Use the Augmentor and Imgaau image data augmentation libraries to augment the crown image data; based on the interference factors on the three-dimensional reconstruction of crowns in the real clinical environment, use the AdditiveGaussianNoise function in the Augmentor and Imgaau image data augmentation libraries to add noise, use the Cutout function to crop the input crown image of arbitrary size to a size of 256*256, and use the Brightness function to adjust the brightness.

[0017] Furthermore, the specific implementation method of step S2 is as follows:

[0018] S21. Initialize the camera pose P = {I1, ..., I2} for the n crown images input to the network. n}, where P is the set of crown images, P i ∈RH×W×3 This is represented as P, where each image in the set has three channels. i A data point in the set of dental crown images P;

[0019] S22. Estimate I for each pair of images using the COLMAP method. i I j Relative attitude π i→j A rough estimate of the distribution is used as the camera rotation matrix R. i→j ∈R 3×3 Translation matrix T i→j ∈R 3 The distribution of the camera rotation matrix and translation matrix is ​​then used to calculate the maximum likelihood estimate based on l(π;I)=logL(π;I), thereby obtaining the initial camera pose π. i Where l(π; I) represents the log-likelihood of image I as a specific sample under camera pose π; L is the likelihood function, representing the probability of observing image I under camera pose π; I is a crown image;

[0020] S23. Adjust the single-image 3D reconstruction channel of the view conditional diffusion model F to a multi-image 3D reconstruction channel. Then, input the crown image I and the relative camera pose π as conditional signals into the view conditional diffusion model F. After training, generate a new crown image I′, crown normal map n′ and crown depth map s′ under multiple camera pose transformations.

[0021] Furthermore, the specific steps for estimating the camera pose of the two images using the COLMAP method in step S22 are as follows:

[0022] S221. Input two images I i I j The SIFT feature detection algorithm is used to extract local feature points and their corresponding descriptors from each input image.

[0023] S222. Use brute force matching to match the extracted feature descriptors between image pairs to establish the point feature correspondence between image pairs;

[0024] S223. Using the matched feature point pairs, the camera pose is estimated using the default preset basis matrix. The camera pose includes the camera rotation matrix, translation matrix, and camera relative transformation matrix.

[0025] Furthermore, the specific implementation method of the visual shell technique for the crown in step S3 is as follows:

[0026] (1) Obtain the crown image and the mask image, and at the same time obtain the camera pose calculated in step S2;

[0027] (2) The Canny edge detection algorithm is used to perform edge detection and contour extraction on each image, and the processed image is converted into a point set or contour line;

[0028] (3) Based on the camera pose parameters, the two-dimensional information of the contour is used to generate the corresponding ray and map it to the three-dimensional space to generate the view frustum for each viewpoint;

[0029] (4) Use the three-dimensional geometric intersection algorithm to calculate the intersection of volumes and determine the visible shell of the object;

[0030] (5) Using the Delaunay triangulation algorithm, the visible shell volume boundary is used as the constraint of the object surface to generate a polygonal 3D mesh crown model.

[0031] Further, the specific implementation of step S4 is as follows:

[0032] S41. Optimize the text-to-image diffusion model into a text-to-normal and depth diffusion model; using the rotation matrix R of the camera parameters. i→j Convert the crown normal map n′ and crown depth map s′ from world coordinates to camera coordinates;

[0033] The camera pose π is added as a condition to the view condition diffusion model, and the model is fine-tuned. The fine-tuning process uses the same optimization objective as the original diffusion model.

[0034]

[0035] Where π is the camera pose, t is the time step; ∈ represents noise, and c φ It is a camera attitude adjustment network; σ t and α t These are parameters of the diffusion scheduler; This is a view condition diffusion model, where n is the normal map, s is the depth map, and E is the depth map. π,t,∈ The expected value of the parameter;

[0036] S42. During the fine-tuning of the view condition diffusion model, the initial estimated camera pose π is also refined through backpropagation of the model. i The view condition diffusion model F is initialized with the camera pose π. i A new view of the crown is synthesized conditionally from the crown image, and the probability distribution P(I′|I,π) of the new crown view I′ is modeled. According to Bayesian rules, the view conditional diffusion model F implicitly simulates the camera pose distribution of the input image I and the new crown view I′. The objective of optimizing the camera pose is:

[0037]

[0038] in Image I i The noise latent coding, π i→j For the camera pose of image i changing to j, I i For the i-th image;

[0039] S43. Combine the depth map and normal map generated by the 3D Gaussian distribution with the view conditional diffusion model. When the Gaussian distribution is located on the same local plane in 3D space, the 2D projection is also located in the adjacent region and exhibits similar geometric properties; each input model's view has a camera extrinsic function [W,t]∈R. 3×4 The center A of the initialized crown Gaussian point cloud i Projected onto the camera coordinate system, as A′ i The depth value of the 3D Gaussian;

[0040]

[0041] Where x i y i and z i Let z be the three-dimensional coordinates of a point i on the x, y, z axes, where z i It refers to the Gaussian depth from the current viewpoint, where W is an external function of the camera.

[0042] In a Gaussian distribution G, its covariance matrix is ​​expressed as: Rotation matrix R i Determine its three orthogonal eigenvectors, scaling matrix S i ∈R 3×3 Determine the scaling factor along the eigenvector direction; the covariance matrix ∑x of the 3D Gaussian sphere is approximately represented as an ellipsoid shape, where the eigenvectors correspond to the axes of the ellipsoid, and the scaling factor refers to the length of the axes; according to the optimization process of the Gaussian shader, the 3D Gaussian sphere will gradually flatten and approach a plane; for calculating the normal value of the Gaussian point cloud of a tooth crown, the direction of its shortest axis is approximately the Gaussian normal direction B. i The direction is:

[0043] B i =R i [r,:], r = argmin(s1,s2,s3)

[0044] Where diag(s1,s2,s3) = S iargmin(·) is the operation to find the minimum index; the crown depth map and crown normal map generated by the optimized view condition diffusion model at the current viewpoint are checked using multi-view geometric consistency to filter out inaccurate results, resulting in filtered crown depth maps and crown normal maps; the relative difference between the filtered depth and normal values ​​and the depth and normal values ​​of the crown 3D Gaussian mapping are calculated. For regions where the relative difference exceeds the threshold, the initialized crown Gaussian distribution cannot accurately reconstruct the region in 3D; the pixels in the region are projected back into 3D space to render the crown depth map and crown normal map at the current viewpoint, initialized as 3D Gaussian and added to the existing Gaussian distribution, and then iteratively optimized.

[0045] Beneficial effects:

[0046] 1. By combining 2D and 3D information, the color map, normal map, and depth map generated by the conditional diffusion model significantly improve the reconstruction quality of 3D crowns and enhance the detail representation; the sparse view generation strategy is adopted and improved, and the initial camera pose of the view conditional diffusion model and sparse view is optimized, which is conducive to effective view synthesis.

[0047] 2. By using an efficient sparse view reconstruction method, the need for high-density image data is reduced, the time and cost of image acquisition and processing are lowered, and the overall resource utilization efficiency is improved.

[0048] 3. This invention can provide more personalized solutions that meet the needs of patients, achieve rapid and accurate model generation, and help promote digital technology in the dental field. Attached Figure Description

[0049] Figure 1 This is a data preprocessing flowchart provided by the present invention;

[0050] Figure 2 This is a schematic diagram of the initialization of the dental crown provided by the present invention;

[0051] Figure 3 This is a flowchart of the 3D model of a dental crown provided by the present invention;

[0052] Figure 4 This is a schematic diagram of crown reconstruction provided by the present invention. Figure 1 ;

[0053] Figure 5 This is a schematic diagram of crown reconstruction provided by the present invention. Figure 2 ;

[0054] Figure 6 This is a schematic diagram of crown reconstruction provided by the present invention. Figure 3 ;

[0055] Figure 7This is a schematic diagram of crown reconstruction provided by the present invention. Figure 4 . Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0057] The application principle of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0058] Example 1:

[0059] This invention proposes a 3D reconstruction method for dental crowns using sparse views. First, in the data preprocessing stage, various models are used to extract the crown region from the acquired crown images. Next, in the initialization stage of the crown's 3D information, this method first estimates the camera pose under the sparse view and uses a conditional view diffusion model to generate corresponding color maps (new views), normal maps, and depth maps, providing the necessary data foundation for the crown's 3D information. Then, visual shell technology and 3DGS are used to construct the preliminary 3D outline of the crown. Finally, in the crown 3D model optimization stage, this method adopts and optimizes a sparse view generation strategy, aiming to optimize the view conditional diffusion model and the initial camera pose. Simultaneously, a crown 3D Gaussian geometry optimization method is proposed to further refine the preliminary reconstructed crown shape.

[0060] S1, Data Preprocessing

[0061] The image data collected by dentists typically combines the crown and gingiva. Therefore, the SAM model is first used to segment the crown region in the intraoral photograph and obtain its mask image. Then, the Otsu algorithm is used to remove outliers and noise, resulting in an image containing only the crown. Due to the limited amount of acquired tooth image data, data augmentation methods are used to process the crown images in terms of brightness, size, and sharpness, enriching the training data and making it better suited for model training. Specifically, the Augmentor and Imgaau image augmentation libraries are used to augment the crown image data. Based on the interference factors in 3D crown reconstruction in real clinical environments, the AdditiveGaussianNoise function from the Augmentor and Imgaau image augmentation libraries is used to add noise. The Cutout function is used to crop the input crown image of arbitrary size to 256*256 pixels, and the Brightness function is used to adjust the brightness. The results are shown below. Figure 1 As shown.

[0062] The SAM model mainly consists of three parts: an image encoder, a cue encoder, and a mask decoder. The input image first passes through the image encoder, whose task is to extract feature representations from the input image. The image encoder is typically a pre-trained convolutional neural network model. Through multiple layers of convolutional neural networks, the encoder extracts low-level features (edges and textures) and high-level features (object shape and category) of the image layer by layer. The encoder then uses downsampling (max pooling or convolution) to reduce spatial dimensionality and enhance the semantic information of the feature map. After passing through the encoder, the input image outputs a multi-scale feature representation containing rich contextual information. Next, the decoder is responsible for converting the features extracted by the encoder into a segmentation mask. The decoder gradually restores the spatial resolution of the image through upsampling operations, combining the features from the encoder to generate a complete segmentation map. In each step of decoding, the decoder fuses the feature map with the encoder features of the corresponding layer, ensuring that details are preserved. Finally, the decoder outputs a probability map, where each pixel corresponds to a probability value, representing the probability that the pixel belongs to the target object. Finally, the mask generator generates the final segmentation mask from the probability map. The mask generator applies a threshold to the probability map, typically choosing an optimal threshold (0.5) to distinguish between the foreground and background, generating a binary mask image. In the binary mask, the foreground region is marked as 1, and the background region is marked as 0, thus ultimately outputting a binary mask image.

[0063] The core idea of ​​the Otsu algorithm is to analyze the gray-level histogram of the image to find a segmentation threshold that maximizes the inter-class variance between the segmented foreground and background, thereby removing outliers and noise. The specific processing steps are as follows:

[0064] (1) Convert the input image to a grayscale image, count the number of pixels corresponding to each grayscale value in the grayscale image, and obtain the grayscale histogram.

[0065] (2) Traverse each gray value in the gray histogram and use it as a threshold.

[0066] (3) For each threshold, the image pixels are divided into foreground (grayscale value greater than the threshold) and background (grayscale value less than or equal to the threshold). The number of pixels, average grayscale value, and total number of pixels for each foreground and background are calculated. Based on the number of pixels, average grayscale value, and total number of pixels for the foreground and background, the inter-class variance at the current threshold is calculated. The formula for calculating the inter-class variance is:

[0067] σ 2 =ω0×(μ0-μ) 2 +ω1×(μ1-μ) 2

[0068] Where ω0 and ω1 are the proportions of foreground and background pixels to the total number of pixels, respectively; μ0 and μ1 are the average gray levels of the foreground and background, respectively; and μ is the total average gray level of the image.

[0069] (4) After traversing all gray values, find the gray value that maximizes the inter-class variance and use it as the optimal segmentation threshold.

[0070] (5) Binarize the image using the optimal segmentation threshold. Set pixels with gray values ​​greater than the threshold to white to represent the foreground; set pixels with gray values ​​less than or equal to the threshold to black to represent the background.

[0071] (6) In the binarized image, outliers and noise usually appear as small black or white areas. These outliers and noise can be further removed by morphological processing (erosion, dilation, opening, closing operations) and connected component analysis.

[0072] S2, Initialization of crown 3D information

[0073] S21. First, this invention uses the RelPose++ model to initialize the camera pose of the n crown images input to the network, P = {I1, ..., I...} n}, where P is the set of crown images, P i ∈R H×W×3 This indicates that each image in set P is three-channel, and P... i This is a data point in the set of dental crown images P.

[0074] S22. First, the COLMAP method is used to estimate I for each pair of images. i I j Between camera pose π i→j A rough estimate of the distribution is used as the camera rotation matrix R. i→j ∈R 3×3 Translation matrix T i→j ∈R 3 The distribution of the camera rotation matrix and translation matrix is ​​then determined. The maximum likelihood estimate is calculated based on l(π;I) = logL(π;I), thus finding a set of initialization parameters for the camera pose π. i Where l is the log-likelihood function, representing the log-likelihood of image I as a specific sample under camera pose π. π is the camera pose; L is the likelihood function, representing the probability of observing image I under camera pose π; I is the crown image.

[0075] The specific steps for estimating the camera pose of two images using the COLMAP method are as follows:

[0076] (1) Input two images and use the SIFT feature detection algorithm to extract local feature points and their corresponding descriptors from each input image.

[0077] (2) Use brute force matching to match the extracted feature descriptors between image pairs to establish the point feature correspondence between image pairs.

[0078] (3) Using the matched feature point pairs, the camera pose is estimated by the default preset basis matrix, which includes the camera rotation matrix, translation matrix and camera relative transformation matrix.

[0079] S23. Since 3D reconstruction from sparse images is an ill-posed problem, this invention relies on learning 3D prior information about the crown from a large-scale image dataset. A Zero-1-to-3 model is used as the view conditional diffusion model F in this study. The Zero-1-to-3 model is a pioneering method for novel view synthesis based on a diffusion model, consisting of a camera pose adjustment network c. Φ and learning noise reduction ε θ The composition was fine-tuned for the stable diffusion model to achieve the synthesis of new views.

[0080] The Zero-1 to-3 model first incorporates the camera relative transformation matrix [R|T] corresponding to the viewpoint of each crown image into the text-image contrast pre-trained model (CLIP), and maps it to the original CLIP dimension to form a global pose-CLIP. This pose-CLIP interacts with the UNet layer through cross-attention to generate rich high-level semantic information. Simultaneously, channel concatenation is performed between the original image and the preprocessed image to help the model preserve the morphology and details of the synthesized target.

[0081] This application first adjusts the single-image 3D reconstruction channel of the view conditional diffusion model F into a multi-image 3D reconstruction channel. Then, the crown image I and the camera pose π are input as conditional signals into the view conditional diffusion model F. After training, a new crown image I′, a crown normal map n′, and a crown depth map s′ are generated under multiple camera pose transformations. The crown contour image is obtained by processing the new crown image I′ using the findContours function from the OpenCV image processing library.

[0082] S3. Constructing the initial 3D outline of the crown.

[0083] To better utilize the crown structure information in the limited input images, a visual shell for the crown is created based on the contour images and mask images of the crown from multiple viewpoints (different angles, such as 0° frontal view and 90° lateral view). This visual shell serves as the geometric framework for initializing 3D Gaussians. The visual shell is represented in STL (Structured Layer Transform) patch form, which is subsequently converted to 3D Gaussians using visual shell technology. After creating the 3D crown visual shell, it is transformed into a 3D Gaussian representation. Specifically, the visual shell technology randomly initializes 3D Gaussian points within the crown visual shell using a random sampling method. Compared to the limited number of SfM points in an extremely sparse setting, the crown visual shell technology provides more structural prior information. Mean filtering removes unreasonable distributions of Gaussian points to help establish multi-view... Figure 1 Consistency. This application projects the processed 3D Gaussian points onto the input image plane and preserves them within the intersection of all mask image spaces. The color of the 3D points is calculated using bilinear interpolation from the pixels projected from the input image, and the average value is obtained as C(x,y).

[0084]

[0085] Where C(x,j) and C(x,j+1) are two points on the x-axis. The position of each 3D point is set to μ, and its color is converted to sh. The average distance between adjacent points is set to β, the rotation q is set to a unit quaternion by default, and the opacity is... Set to a constant value. As described above, the initialized visual shell model can be parameterized as a set of Gaussian distributions G, thereby converting 3D points into a 3D Gaussian representation, using these points to represent the crown visual shell model. Although this initialized crown visual shell may contain inaccuracies, subsequent optimization processes can build upon it for more detailed 3D crown reconstruction. The initialization effect is as follows... Figure 2 As shown.

[0086] Specific implementation methods of visual shell technology for dental crowns:

[0087] (1) Obtain the crown contour image and mask image, and at the same time obtain the camera pose calculated in step S2.

[0088] (2) The Canny edge detection algorithm is used to perform edge detection and contour extraction on each image, and the processed image is converted into a point set or contour line.

[0089] (3) Based on the camera pose parameters, the two-dimensional information of the contour is used to generate the corresponding ray mapping to the three-dimensional space to generate the view frustum for each viewpoint.

[0090] (4) Use the three-dimensional geometric intersection algorithm to calculate the intersection of volumes and determine the visible shell of the object.

[0091] (5) The Delaunay triangulation algorithm is used to use the visible shell volume boundary as the constraint of the object surface to generate a polygonal 3D mesh model, namely the visual shell of the crown.

[0092] S4, Crown 3D Model Optimization

[0093] S41. First, optimize the view conditional diffusion model F. Several challenges arise when constructing a conditional diffusion model for high-quality generation of normal and depth maps. The scarcity of existing 3D crown datasets results in a limited number of normal and depth maps available for training. Therefore, this invention proposes a sparse view generation strategy for crowns. This strategy first optimizes the text-to-image diffusion model into a text-to-normal and depth diffusion model. Then, addressing the problem of overfitting or underfitting caused by significant changes in the rendered normal and depth maps under different viewpoints, the rotation matrix R of the camera parameters is used... i→j The normal map n′ and depth map s′ are transformed from world coordinates to camera coordinates to mitigate the effects of overfitting or underfitting and allow the view conditional diffusion model to focus on generating details of perceptual geometry. Finally, the camera pose π is added as a condition to the view conditional diffusion model, and the model is fine-tuned.

[0094] The fine-tuning process uses the same optimization objective as the original diffusion model:

[0095]

[0096] Where π represents the camera pose and t is the time step. ∈ represents noise, and c φ It is a camera pose adjustment network. σ t and α t These are the parameters of the diffusion scheduler. This is a view condition diffusion model, where n is the normal map, s is the depth map, and E is the depth map. π,t,∈ The expected value of the parameter.

[0097] S42. During the fine-tuning of the above view conditional diffusion model, the initially estimated camera pose π is further refined through backpropagation of the model. The view conditional diffusion model F synthesizes a new view of the crown based on the initialized camera pose π and the crown image. It models the probability distribution P(I′|I,π) of the new view I′ of the input crown image I. According to Bayes' rule, the view conditional diffusion model F implicitly simulates the camera pose distribution of the input image I and the generated image I′. Therefore, while fine-tuning the view conditional diffusion model F, the initially estimated camera pose is optimized by backpropagating the gradient to the parameters in the model. Formally, the goal of optimizing the camera pose is:

[0098]

[0099] in Image I i The noise latent coding, π i→j For the camera pose of image i changing to j, I i Let i be the i-th image.

[0100] S43. To address the problem of 3D Gaussian point clouds struggling to perceive geometric connectivity, this application proposes a 3D Gaussian geometric representation optimization strategy. This strategy combines 3D Gaussian points with depth maps and normal maps generated by a view-conditional diffusion model. The depth map and normal map guide the growth of the crown Gaussian point cloud, aiding in dense reconstruction of the Gaussian point cloud while efficiently determining the neighborhood information of each Gaussian point and propagating the crown's geometric information between them. Specifically, when a Gaussian distribution lies on the same local plane in 3D space, its 2D projection should also lie in adjacent regions and exhibit similar geometric characteristics, namely depth and normal. For calculating the depth value of the crown Gaussian point cloud, each input model's view has a camera extrinsic function [W,t]∈R. 3×4 The center A of the initialized crown Gaussian point cloud i Projected onto the camera coordinate system, as A′ i This method is used to calculate the depth value of a 3D Gaussian.

[0101]

[0102] x i y i and z i Let z be the three-dimensional coordinates of a point i on the x, y, z axes, where z i This refers to the Gaussian depth from the current viewpoint, where W is the camera's external function. In a Gaussian distribution G, its covariance matrix is ​​expressed as... Rotation matrix R i Determine its three orthogonal eigenvectors, scaling matrix S i ∈R 3×3 Determine the scaling factor along the eigenvector direction. The covariance matrix ∑x of a 3D Gaussian sphere can be approximated as an ellipsoid shape, where the eigenvectors correspond to the axes of the ellipsoid, and the scale refers to the length of the axes. During the Gaussian optimization process, the 3D Gaussian sphere gradually flattens and approaches a plane. Therefore, for calculating the normal values ​​of the crown Gaussian point cloud, the direction of its shortest axis can be approximated by the Gaussian normal direction B. i This direction is

[0103] B i =R i[r,:], r = argmin(s1,s2,s3)

[0104] Where diag(s1,s2,s3) = S i `argmin(·)` is the operation to find the minimum index. The depth map and normal map generated by the optimized view condition diffusion model at the current viewpoint are checked using multi-view geometric consistency, filtering out inaccurate results to obtain filtered depth and normal maps. The relative difference between the filtered depth and normal values ​​and the depth and normal values ​​of the crown 3D Gaussian mapping is calculated. For regions where the relative difference exceeds a threshold (0-1), it is considered that the initialized crown Gaussian distribution cannot accurately reconstruct these regions in 3D. The difference is specifically obtained by subtracting the depth and normal values ​​of the crown 3D Gaussian mapping from the filtered depth and normal values. Therefore, the pixels in these regions are projected back into 3D space to render the depth and normal maps at the current viewpoint, initialized as 3D Gaussians, and added to the existing Gaussian distribution for iterative optimization. The overall flowchart is as follows: Figure 3 As shown in the diagram. The final result of the crown reconstruction is illustrated below. Figure 4-7 As shown.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of a tooth crown from sparse views, characterized in that, include: S1. Data preprocessing: The acquired image data is segmented using SAM to obtain a mask image. Then, the Otsu algorithm is used to remove outliers and noise to obtain the crown image. The brightness, size, and sharpness of the crown image are adjusted. S2. Initialization of 3D information of the crown: COLMAP is used to estimate the camera pose under sparse view, and the corresponding new crown image I′, crown normal map n′ and crown depth map s′ are generated with the help of the conditional view diffusion model F; the crown contour image is obtained by using the findContours function in the OpenCV image processing library on the new crown image I′. S3. 3D model construction of the crown: Based on the outline image and mask image of the crown from multiple perspectives, a visual shell of the crown is created as the geometric framework for initializing the 3D Gaussian, thereby constructing the initial 3D outline of the crown. A visual shell for the crown is created based on the contour images and mask images of the crown from multiple perspectives, serving as the geometric framework for initializing the 3D Gaussian. The processed 3D Gaussian points are projected onto the input image plane and preserved within the intersection of all mask image spaces. The color of the 3D point is calculated from the pixels projected from the input image using bilinear interpolation, and the average value C(x,y) is taken. Where C(x,j) and C(x,j+1) are two points on the x-axis; the position of each 3D point is set to μ, and its color is converted to sh; the average distance between adjacent points is set to β, the rotation q is set to a unit quaternion by default, and the opacity is... Set to a constant value; parameterize the initialized visual shell model as a set of Gaussian distributions G, realize the conversion of 3D points into 3D Gaussian representation, and use 3D Gaussian points to display the visual shell model of the crown. S4. Optimization of the 3D model of the crown: Based on the crown normal map n′ and crown depth map s′ from step S2, the view condition diffusion model F is optimized, and the backpropagation of the view condition diffusion model is used to refine the initially estimated camera pose π. i Furthermore, the Gaussian set is optimized by combining 3D Gaussian with crown depth maps and crown normal maps.

2. The method for three-dimensional reconstruction of a tooth crown from sparse views according to claim 1, characterized in that, Specific implementation method in step S1: S101. The SAM model is used to segment the crown region in the exit photograph and obtain its mask image. Then, the Otsu algorithm is used to remove outliers and noise to obtain an image containing only the crown. S102. Use the Augmentor and Imgaau image data augmentation libraries to augment the crown image data; based on the interference factors on the three-dimensional reconstruction of crowns in the real clinical environment, use the AdditiveGaussianNoise function in the Augmentor and Imgaau image data augmentation libraries to add noise, use the Cutout function to crop the input crown image of arbitrary size to a size of 256*256, and use the Brightness function to adjust the brightness.

3. The method for three-dimensional reconstruction of a tooth crown from sparse views according to claim 1, characterized in that, The specific implementation method of step S2 is as follows: S21. Initialize the camera pose P = {I1, ..., I2} for the n crown images input to the network. n }, where P is the set of crown images, P i ∈R H×W×3 This is represented as P, where each image in the set has three channels. i A data point in the set of dental crown images P; S22. Estimate I for each pair of images using the COLMAP method. i I j Relative attitude π i→j A rough estimate of the distribution is used as the camera rotation matrix R. i→j ∈R 3×3 Translation matrix T i→j ∈R 3 The distribution of the camera rotation matrix and translation matrix is ​​then used to calculate the maximum likelihood estimate based on l(π;I)=logL(π;I), thereby obtaining the initial camera pose π. i Where l(π; I) represents the log-likelihood of image I as a specific sample under camera pose π; L is the likelihood function, representing the probability of observing image I under camera pose π; I is a crown image; S23. Adjust the single-image 3D reconstruction channel of the view conditional diffusion model F to a multi-image 3D reconstruction channel. Then, input the crown image I and the relative camera pose π as conditional signals into the view conditional diffusion model F. After training, generate a new crown image I′, crown normal map n′ and crown depth map s′ under multiple camera pose transformations.

4. The method for three-dimensional reconstruction of a tooth crown from sparse views according to claim 3, characterized in that, The specific steps for estimating the camera pose of the two images using the COLMAP method in step S22 are as follows: S221. Input two images I i I j The SIFT feature detection algorithm is used to extract local feature points and their corresponding descriptors from each input image. S222. Use brute force matching to match the extracted feature descriptors between image pairs to establish the point feature correspondence between image pairs; S223. Using the matched feature point pairs, the camera pose is estimated using the default preset basis matrix, which includes the camera rotation matrix, translation matrix, and camera relative transformation matrix.

5. The method for three-dimensional reconstruction of a tooth crown from sparse views according to claim 1, characterized in that, Specific implementation method of the visual shell technique for crowns in step S3: (1) Obtain the crown image and the mask image, and at the same time obtain the camera pose calculated in step S2; (2) The Canny edge detection algorithm is used to perform edge detection and contour extraction on each image, and the processed image is converted into a point set or contour line; (3) Based on the camera pose parameters, the two-dimensional information of the contour is used to generate the corresponding ray and map it to the three-dimensional space to generate the view frustum for each viewpoint; (4) Use the three-dimensional geometric intersection algorithm to calculate the intersection of volumes and determine the visible shell of the object; (5) Using the Delaunay triangulation algorithm, the visible shell volume boundary is used as the constraint of the object surface to generate a polygonal 3D mesh crown model.

6. The method for three-dimensional reconstruction of a tooth crown from sparse views according to claim 1, characterized in that, Detailed implementation of step S4: S41. Optimize the text-to-image diffusion model into a text-to-normal and depth diffusion model; using the rotation matrix R of the camera parameters. i→j Convert the crown normal map n′ and crown depth map s′ from world coordinates to camera coordinates; The camera pose π is added as a condition to the view condition diffusion model, and the model is fine-tuned; the fine-tuning process uses the same optimization objective as the original diffusion model: Where π is the camera pose, t is the time step; ∈ represents noise, and c φ It is a camera attitude adjustment network; σ t and α t These are parameters of the diffusion scheduler; This is a view condition diffusion model, where n is the normal map, s is the depth map, and E is the depth map. π,t,∈ The expected value of the parameter; S42. During the fine-tuning of the view condition diffusion model, the initial estimated camera pose π is also refined through backpropagation of the model. i ; The view condition diffusion model F is initialized with the camera pose π. i A new view of the crown is synthesized conditionally from the crown image, and the probability distribution P(I′|I,π) of the new crown view I′ is modeled. According to Bayesian rules, the view conditional diffusion model F implicitly simulates the camera pose distribution of the input image I and the new crown view I′. The objective of optimizing the camera pose is: in Image I i The noise latent coding, π i→j For the camera pose of image i changing to j, I i For the i-th image; S43. Combine the depth map and normal map generated by the 3D Gaussian distribution with the view conditional diffusion model. When the Gaussian distribution is located on the same local plane in 3D space, the 2D projection is also located in the adjacent region and exhibits similar geometric properties; each input model's view has a camera extrinsic function [W,t]∈R. 3×4 The center A of the initialized crown Gaussian point cloud i Projected onto the camera coordinate system, as A′ i The depth value of the 3D Gaussian; Where x i y i and z i Let z be the three-dimensional coordinates of a point i on the x, y, z axes, where z i It refers to the Gaussian depth from the current viewpoint, where W is an external function of the camera. In a Gaussian distribution G, its covariance matrix is ​​expressed as: Rotation matrix R i Determine its three orthogonal eigenvectors, scaling matrix S i ∈R 3×3 Determine the scaling factor along the direction of the feature vector; The covariance matrix ∑x of a 3D Gaussian sphere is approximately represented by an ellipsoid shape, where the eigenvectors correspond to the axes of the ellipsoid, and the scaling factor refers to the length of the axes. According to the optimization process of the Gaussian shader, the 3D Gaussian sphere gradually flattens and approaches a plane. For calculating the normal values ​​of the Gaussian point cloud of a dental crown, the direction of its shortest axis approximates the Gaussian normal direction B. i The direction is: B i =R i [r,:],r=argmin(s1,s2,s3) Where diag(s1,s2,s3) = S i argmin(·) is the operation to find the index of the minimum value; The crown depth map and crown normal map generated by the optimized view condition diffusion model at the current viewpoint are checked using multi-view geometric consistency to filter out inaccurate results, resulting in filtered crown depth maps and crown normal maps. The relative difference between the filtered depth and normal values ​​and the depth and normal values ​​of the crown 3D Gaussian mapping is calculated. For regions where the relative difference exceeds a threshold, the initialized crown Gaussian distribution cannot accurately reconstruct the region in 3D. The pixels in this region are projected back into 3D space to render the crown depth map and crown normal map at the current viewpoint, initialized as 3D Gaussian and added to the existing Gaussian distribution, and then iteratively optimized.

Images