Dental crown three-dimensional reconstruction method of sparse view

Abstract

The invention discloses a sparse view dental crown three-dimensional reconstruction method, which comprises the following steps: S1, data preprocessing: segmenting data through an SAM model and obtaining a mask image, and removing outliers and noise by using an Otsu algorithm to obtain a dental crown photo; s2, initializing 3D information of the dental crown, estimating a camera attitude under a sparse view angle, generating a corresponding new view, a normal map and a depth map by means of a conditional view angle diffusion model, providing a necessary data basis for the 3D information of the dental crown, and constructing a preliminary 3D contour of the dental crown by using a visual shell technology and 3DGS; and S3, in a dental crown 3D model optimization stage, a sparse view angle generation strategy is adopted to optimize a view condition diffusion model and an initial camera attitude, and meanwhile, a dental crown 3D Gaussian geometric optimization method is proposed to further refine the primarily reconstructed dental crown appearance. According to the method, the new view, the normal graph and the depth map are generated through the conditional diffusion model, so that the reconstruction quality of the 3D dental crown is remarkably improved, and the detail representation is enhanced.

Description

Technical Field

[0001] The invention belongs to the technical field of dental crown image processing, and in particular relates to a dental crown three-dimensional reconstruction method of a sparse view. Background Art

[0002] With the rapid development of stomatology and artificial intelligence technology, 3D crown reconstruction technology plays an increasingly important role in orthodontic treatment and restoration. Although some 3D reconstruction methods have been proposed, they often face the problem of insufficient accuracy when dealing with sparse view input. Some methods rely on dense image input and complex equipment configuration, which is limited in actual orthodontic treatment. Although deep learning-based methods have made some progress, there is still room for improvement in the generalization ability of the model and detail reconstruction.

[0003] In the image-based 3D reconstruction task, Structure from Motion (SfM) is a widely used classic method. However, the performance of this method will be significantly reduced when faced with image sparsity or occlusion. In 2D photos, the crown surface usually lacks texture information, and the spatial position of the tooth is difficult to infer directly from the image, which makes the 3D reconstruction task of the image target extremely difficult. To address these problems, researchers proposed a method that combines the shading shape algorithm and the statistical shape prior to achieve 3D reconstruction of teeth. Mostafa et al. introduced a coupled statistical model by linking the coefficients of the 2D texture model, 3D shape model, and spherical harmonic projection image model, and further developed the model-based SFS tooth restoration method, but the statistical SFS framework of tooth reconstruction still has limitations when dealing with missing or occluded teeth. For the reconstruction of 3D shape, the commonly used method is the 3D deformable model. However, this method has not received widespread attention in restoring the 3D shape of teeth.

[0004] In recent years, 3D Geometric Shape (3DGS) analysis technology has made significant progress in the fields of computer vision and graphics, bringing revolutionary changes to the task of 3D reconstruction of objects. 3DGS not only provides an accurate description of the surface morphology of objects, but also contains rich spatial structural information, which is particularly important for complex and diverse crown structures. Combined with deep learning technology, more and more researchers are using 3DGS to study target 3D reconstruction. However, there are still few studies on its use for 3D reconstruction of crowns at this stage. Although 3DGS shows strong synthesis capabilities in novel views, it relies on the number of images and sparse point clouds for rendering reasoning when synthesizing new viewpoints. When the amount of data is insufficient, artifacts that do not belong to the specified viewpoint will appear on the reconstructed 3D model, which will affect the reconstruction quality. According to the current research results, there are three problems in the task of 3D reconstruction of crowns using sparse views. The first problem is how to obtain enough information in sparse views for high-quality 3D reconstruction of crowns. The second problem is how to construct multi-view from highly sparse input. Figure 1 The third problem is how to reconstruct a reasonable high-quality 3D geometric representation of the crown from the missing or compressed crown information. Summary of the invention

[0005] A method for three-dimensional reconstruction of a tooth crown based on a sparse view, comprising: S1. Data preprocessing: SAM is used to segment the collected image data and obtain a mask image. The Otsu algorithm is used to remove outliers and noise, obtain a crown image, and adjust the brightness, size and clarity of the crown image. S2, initialization of the crown 3D information, using COLMAP to estimate the camera pose under sparse view, and using the conditional view diffusion model F to generate the corresponding new view, crown normal map n′ and crown depth map s′; using the findContours function in the OPENCV image processing library on the new crown image I′ to obtain the crown contour image; S3, construction of the 3D model of the crown, creating a crown visual shell based on the contour images and mask images of the crown at multiple viewing angles as a geometric framework for initializing the 3D Gaussian, thereby constructing a preliminary 3D contour of the crown; The crown visual shell is created based on the contour image and mask image of the crown at multiple viewing angles as the geometric framework for initializing the 3D Gaussian. The processed 3D Gaussian points are projected onto the input image plane and retained within the intersection of all mask image spaces. The color of the 3D point is calculated from the pixels projected from the input image using bilinear interpolation and the average value is taken. ; in and Two points on the x-axis; the position of each 3D point is set to , converting its color to ; The average distance between adjacent points is set to , rotation q is set to unit quaternion by default, opacity Set to a constant value; parameterize the initialized visual shell model as a set of Gaussian distributions , thereby 3D points can be converted into a representation of 3D Gaussians, and the points of 3D Gaussians can be used to display the visual shell model of the crown; S4, crown 3D model optimization, according to the crown normal map n' and crown depth map s' in step S2, the view conditional diffusion model F is optimized, and the back propagation of the view conditional diffusion model is used to refine the initial estimated camera pose , and optimize the Gaussian set by combining the 3D Gaussian with the depth map and the normal map.

[0006] Furthermore, the specific implementation method in step S1 is: S101, using the SAM model to segment the crown area in the photo inside the mouth and obtain its mask image, and then using the Otsu algorithm to remove outliers and noise to obtain a picture of only the crown; S102. Use Augmentor and Imgaug image data enhancement libraries to amplify crown image data. According to the interference factors for crown 3D reconstruction in real clinical environments, use the AdditiveGaussianNoise function in the Augmentor and Imgau image data enhancement libraries to add noise, use the Cutout function to crop the input crown image of any size to a size of 256*256, and use the Brightness function to adjust the brightness.

[0007] Further, the specific implementation method of step S2 is: S21. Initialize the camera posture of n crown images input to the network , where P is the set of crown images, It means that each image in the set has three channels. is a data of the crown image set P; S22. Estimate each pair of images using the COLMAP method , Relative posture A rough estimate of the distribution, taking it as the camera rotation and pan distribution; then under the camera rotation and translation distribution, according to Calculate its maximum likelihood estimate to obtain the initial camera pose ;in is the log-likelihood function, which means that Below is the log-likelihood value of image I as a specific sample; For relative posture; is the likelihood function, which means that The probability of a given observed image I; For the crown image; S23. The single-image 3D reconstruction channel of the view conditional diffusion model F is adjusted to a multi-image 3D reconstruction channel, and then the crown image I and the relative camera pose are input into the view conditional diffusion model F as conditional signals. After training, a new crown image I′, a crown normal map n′ and a crown depth map s′ are generated under multiple camera pose transformations.

[0008] Furthermore, the specific steps of estimating the camera poses of the two images using the COLMAP method in step S22 are as follows: S221, input two images , , the feature detection algorithm SIFT is used to extract local feature points and their corresponding descriptors from each input image; S222, using brute force matching between the extracted feature descriptors between the image pairs to establish a point feature correspondence relationship between the image pairs; S223. Using the matched feature point pairs, the camera pose is estimated through a default preset basic matrix. The camera pose includes a camera rotation matrix, a translation matrix, and a camera relative transformation matrix.

[0009] Furthermore, the specific implementation method of the crown visual shell technology in step S3 is: (1) Obtain the crown image and mask image, and at the same time obtain the camera pose calculated above; (2) Use the Canny edge detection algorithm to perform edge detection and contour extraction on each image, and convert the processed image into a point set or contour line; (3) Based on the camera attitude parameters, the two-dimensional information of the contour is mapped to the corresponding ray into the three-dimensional space to generate the viewing cone of each viewing angle; (4) Using a three-dimensional geometric intersection algorithm to calculate the intersection of volumes and determine the visible shell of the object; (5) The Delaunay triangulation algorithm is used to take the visible shell volume boundary as a constraint on the object surface to generate a polygonal 3D mesh crown model.

[0010] Further, the specific implementation of step S4 is: S41. Optimize the text-to-image diffusion model to a text-to-normal and depth diffusion model; through the rotation matrix of the camera parameters Convert the normal map n′ and depth map s′ from world coordinates to camera coordinates; Set the camera posture It is added as a condition to the view-conditional diffusion model and the model is fine-tuned; the fine-tuning process uses the same optimization objective as the original diffusion model: in is the camera pose and t is the time step. is noise, and It is a camera pose adjustment network. and are the parameters of the diffusion scheduler; is the view-conditional diffusion model, is the normal map, is the depth map, is the expected value of the parameter;

[0011] S42. Model fine-tuning process In the above view condition expansion, the initial estimated camera pose is also refined through the back propagation of the model ; The view-conditioned diffusion model F is initialized with the camera pose and the crown image as the condition to synthesize a new view of the crown, and the probability distribution of the new view I′ of the crown Modeling; According to the Bayesian rule, the view conditional diffusion model F implicitly simulates the camera pose distribution of the input image I and the new view image I′. The goal of optimizing the camera pose is: in is an image The noisy latent code of is the camera pose from image i to image j, is the i-th image.

[0012] S43. Combine 3D Gaussians with depth and normal maps generated by a view-conditioned diffusion model. When the Gaussian distributions are located on the same local plane in 3D space, the 2D projections are also located in adjacent areas and exhibit similar geometric properties, namely depth and normals. Each view of the input model has a camera external function. , the center of the initialized crown Gaussian point cloud Projected into the camera coordinate system, as The depth value of the 3D Gaussian; in , and is the three-dimensional coordinate value of a point i on the x, y, and z axes, where refers to the Gaussian depth at the current viewing angle, is the camera external function; In Gaussian distribution G, its covariance matrix is ​​expressed as ; Rotation matrix Determine its three orthogonal eigenvectors and scale the matrix Determine the scaling along the eigenvector direction; covariance matrix of 3D Gaussian The approximate representation is an ellipsoid shape, where the eigenvectors correspond to the axes of the ellipsoid and the scaling refers to the length of the axes; according to the optimization process of the Gaussian shader, the 3D Gaussian sphere will gradually become flat and approach a plane; for calculating the normal value of the crown Gaussian point cloud, the direction of its shortest axis is approximately the direction of the Gaussian normal , the direction is in , It is an operation to find the minimum index; use multi-view geometric consistency to check the depth map and normal map generated by the optimized view conditional diffusion model under the current viewpoint, filter out inaccurate results, and obtain filtered depth map and normal map; calculate the relative difference between the filtered depth value and normal value and the depth value and normal value of the crown 3D Gaussian mapping. For areas where the relative difference value exceeds the threshold, the initialized crown Gaussian distribution cannot accurately reconstruct the area in three dimensions; project the pixels in the area back to the 3D space to render the depth map and normal map under the current viewpoint, initialize them as 3D Gaussians and add them to the existing Gaussian distribution for iterative optimization.

[0013] Beneficial effects: 1. Combining 2D and 3D information, the color map, normal map and depth map generated by the conditional diffusion model significantly improve the reconstruction quality of the 3D crown and enhance the detail expression; the sparse view generation strategy is adopted and improved, and the view conditional diffusion model and the initial camera pose of the sparse view are optimized, which is conducive to effective view synthesis.

[0014] 2. Through the efficient sparse view reconstruction method, the demand for high-density image data is reduced, the time and cost of image acquisition and processing are reduced, and the overall resource utilization efficiency is improved.

[0015] 3. The present invention can provide personalized solutions that better meet the needs of patients, achieve fast and accurate model generation, and help promote digital technology in the field of dentistry. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a data preprocessing flow chart provided by the present invention; Figure 2 is a schematic diagram of initializing a dental crown provided by the present invention; Figure 3 It is a flow chart of the 3D model of the dental crown provided by the present invention; Figure 4 This is a schematic diagram of the crown reconstruction provided by the present invention. Figure 1 ; Figure 5 This is a schematic diagram of the crown reconstruction provided by the present invention. Figure 2 ; Figure 6 This is a schematic diagram of the crown reconstruction provided by the present invention. Figure 3 ; Figure 7 This is a schematic diagram of the crown reconstruction provided by the present invention. Figure 4 . DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] The application principle of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0019] Embodiment 1: The present invention proposes a method for three-dimensional reconstruction of dental crowns with sparse views. The first part is the data preprocessing part, in which multiple models are used to extract the dental crown area from the collected dental crown images. The next part is the initialization part of the dental crown 3D information. The method first estimates the camera posture under the sparse view, and generates the corresponding color map (new view), normal map and depth map with the help of the conditional view diffusion model, which provides the necessary data basis for the 3D information of the dental crown. Then, the visual shell technology and 3DGS are used to construct the preliminary 3D contour of the dental crown. The last part is the dental crown 3D model optimization stage. The method adopts and optimizes a sparse view generation strategy, which aims to optimize the view conditional diffusion model and the initial camera posture. At the same time, a dental crown 3D Gaussian geometry optimization method is proposed to further refine the preliminary reconstructed crown shape.

[0020] S1. Data preprocessing Generally, the image data collected by dentists is a fusion of the crown and the gums. For this reason, the SAM model is first used to segment the crown area in the photo taken inside the mouth and obtain its mask image. The Otsu algorithm is then used to remove outliers and noise to obtain a picture of only the crown. Since the collected tooth image data is limited, the data enhancement method is used to process the crown image in terms of brightness, size, and clarity to enrich the training data, and to amplify it to make it better as data for model training. Specifically, the Augmentor and Imgaug image data enhancement libraries are used to amplify the crown image data. According to the interference factors for the three-dimensional reconstruction of the crown in the real clinical environment, the AdditiveGaussianNoise function in the Augmentor and Imgau image data enhancement library is used to add noise, and the Cutout function is used to crop the input crown image of any size to a size of 256*256, and the Brightness function is used to adjust the brightness. The effect is as follows: Figure 1 shown.

[0021] The SAM model mainly consists of three parts: image encoder, hint encoder and mask decoder. The input image first passes through the image encoder, whose task is to extract the feature representation of the input image. The image encoder is usually a pre-trained convolutional neural network model. Through multiple layers of convolutional neural networks, the encoder extracts low-level features (edges and textures) and high-level features (shape and category of objects) of the image layer by layer. The encoder then uses downsampling (max pooling or convolution) to reduce the spatial dimension and enhance the semantic information of the feature map. After the input image passes through the encoder, the output is a multi-scale feature representation containing rich contextual information. Next, the decoder is responsible for converting the features extracted by the encoder into a segmentation mask. The decoder gradually restores the spatial resolution of the image through upsampling operations and combines the features from the encoder to generate a complete segmentation map. At each step of decoding, the decoder fuses the feature map with the encoder features of the corresponding layer to ensure that the details are preserved. The decoder finally outputs a probability map, with each pixel corresponding to a probability value, indicating the probability that the pixel belongs to the target object. Finally, the mask generator generates the final segmentation mask from the probability map. The mask generator applies a threshold to the probability map, usually selecting an optimal threshold (0.5) to distinguish the foreground from the background, and generates a binary mask map. In the binary mask, the foreground area is marked as 1 and the background area is marked as 0, thus finally outputting a binary mask image.

[0022] The core idea of ​​the Otsu algorithm is to find a segmentation threshold by analyzing the grayscale histogram of the image, so that the inter-class variance of the segmented foreground and background is maximized under the threshold, thereby removing outliers and noise. The specific processing steps are:

[0023] (1) Convert the input image into a grayscale image, count the number of pixels corresponding to each grayscale value in the grayscale image, and obtain a grayscale histogram.

[0024] (2) Traverse each grayscale value in the grayscale histogram and use it as the threshold.

[0025] (3) For each threshold, the image pixels are divided into foreground (grayscale value greater than the threshold) and background (grayscale value less than or equal to the threshold), and the number of pixels, grayscale average and total number of pixels of the foreground and background are calculated. Based on the number of pixels, grayscale average and total number of pixels of the foreground and background, the inter-class variance under the current threshold is calculated. The formula for calculating the inter-class variance is:

[0026] in and are the ratios of foreground and background pixels to the total number of pixels, and are the average grayscale values ​​of the foreground and background, is the overall average grayscale of the image.

[0027] (0) After traversing all grayscale values, find the grayscale value that maximizes the inter-class variance and use it as the optimal segmentation threshold.

[0028] (1) Use the optimal segmentation threshold to binarize the image. Set the pixels with grayscale values ​​greater than the threshold to white, indicating the foreground; set the pixels with grayscale values ​​less than or equal to the threshold to black, indicating the background.

[0029] (2) In the binarized image, outliers and noise usually appear as small black or white areas. These outliers and noise can be further removed through morphological processing (erosion, dilation, opening and closing operations) and connected region analysis methods.

[0030] S2. Initialization of crown 3D information S21. First, the present invention uses the idea of ​​RelPose++ model to initialize the camera pose of n crown images input to the network. , where P is the set of crown images, Indicates that each image in the set P has three channels, A data of the crown image set P.

[0031] S22, first use the COLMAP method to estimate each pair of images , Relative posture A rough estimate of the distribution, taking it as the camera rotation and pan Then under the distribution of camera rotation and translation, according to Calculate its maximum likelihood estimate to find a set of initial camera poses .in is the log-likelihood function, which means that Below, image I is taken as the log-likelihood value of a specific sample. For relative posture; is the likelihood function, which means that The probability of a given observed image I; This is an image of a tooth crown.

[0032] The specific steps of estimating the camera pose of two images using the COLMAP method are as follows: (3) Input two images and use the feature detection algorithm SIFT to extract local feature points and their corresponding descriptors from each input image.

[0033] (4) Use brute force matching between the extracted feature descriptors between image pairs to establish the point feature correspondence between the image pairs.

[0034] (5) Using the matched feature point pairs, the camera pose is estimated through the default preset basic matrix. The camera pose includes the camera rotation matrix, translation matrix and camera relative transformation matrix.

[0035] S23. Since 3D reconstruction from sparse images is an ill-posed problem, the present invention relies on learning 3D prior information of the crown from a large-scale image dataset. The Zero-1-to-3 model is used as the view-conditioned diffusion model F in this study. The Zero-1-to-3 model is a pioneering method for novel view synthesis based on a diffusion model, which is conditioned by the camera pose network. and learning denoiser Composition,the stable diffusion model is fine-tuned to achieve new view synthesis.

[0036] The Zero-1-to-3 model first adds the camera relative transformation matrix [R|T] of each crown image corresponding to the perspective in the text-image comparison pre-trained model (CLIP) and maps it to the original CLIP dimension to form a global pose-CLIP, which interacts with the UNet layer through cross-attention to generate rich high-level semantic information. At the same time, the original image is channel-connected with the preprocessed image to help the model retain the morphology and details of the synthetic target.

[0037] In this application, the single-image 3D reconstruction channel of the view conditional diffusion model F is first adjusted to a multi-image 3D reconstruction channel, and then the crown image I and the relative camera pose are combined. As a conditional signal, it is input into the view conditional diffusion model F. After training, a new crown image I′, crown normal map n′ and crown depth map s′ are generated under multiple camera posture transformations. The findContours function in the OPENCV image processing library is used on the new crown image I′ to obtain the crown contour image.

[0038] S3. Construct the preliminary 3D outline of the crown In order to better utilize the crown structure information in the limited input image, the crown visual shell is created according to the contour image and mask image of the crown under multiple viewing angles (different angles, such as 0° from the front and 90° from the side) as the geometric framework for initializing the 3D Gaussian, where the crown visual shell is represented in the form of STL patches, and the STL patches are subsequently converted into 3D Gaussians using the visual shell technology. With the created 3D crown visual shell, the visual shell is converted into the representation of 3D Gaussians. Specifically, the visual shell technology randomly initializes 3D Gaussian points within the crown visual shell by random sampling. Compared with the limited number of SfM points in the extremely sparse setting, the crown visual shell technology provides more structural prior information and helps establish multi-view by removing the unreasonable distribution of Gaussian points through mean filtering. Figure 1 The application projects the processed 3D Gaussian points onto the input image plane and keeps them in the intersection of all mask image spaces. The color of the 3D point is calculated from the projected pixels of the input image using bilinear interpolation and averaged. .

[0039] in and Two points on the x-axis. The position of each 3D point is set to , converting its color to The average distance between adjacent points is set to , rotation q is set to unit quaternion by default, opacity Set to a constant value. The above description can be used to parameterize the initialized visual shell model as a set of Gaussian distributions , which can convert 3D points into 3D Gaussians, and use 3D Gaussians points to display the crown visual shell model Although this initialized crown visual shell may be inaccurate, the subsequent optimization process can be used to perform a more detailed three-dimensional reconstruction of the crown based on it. Figure 2 shown.

[0040] Specific implementation method of crown visual shell technology: (1) Obtain the crown contour image and mask image, and at the same time obtain the camera pose calculated above.

[0041] (2) Use the Canny edge detection algorithm to perform edge detection and contour extraction on each image, and convert the processed image into a point set or contour line.

[0042] (3) Based on the camera posture parameters, the two-dimensional information of the contour is mapped to the corresponding ray into the three-dimensional space to generate the viewing cone of each viewing angle.

[0043] (4) Use a three-dimensional geometric intersection algorithm to calculate the intersection of volumes and determine the visible shell of the object.

[0044] (5) The Delaunay triangulation algorithm is used to take the visible shell volume boundary as a constraint on the object surface to generate a polygonal 3D mesh model, namely the visual shell of the crown.

[0045] S4. Optimization of crown 3D model S41. First, optimize the view conditional diffusion model F. There are several challenges in constructing a conditional diffusion model for generating high-quality normal maps and depth maps. Due to the scarcity of existing 3D crown datasets, the number of normal maps and depth maps used for training is limited. Therefore, the present invention proposes a crown sparse view generation strategy. The strategy first optimizes the text-to-image diffusion model to a text-to-normal and depth diffusion model. Then, in order to solve the problem that the rendered normal map and depth map change significantly under different viewing angles, resulting in overfitting or underfitting, the rotation matrix of the camera parameters is used to generate the sparse view of the crown. The normal map n′ and the depth map s′ are transformed from world coordinates to camera coordinates to alleviate the effects of overfitting or underfitting and allow the view-conditioned diffusion model to focus on generating the details of the perceived geometry. Finally, the camera pose Added as a condition to the view-conditional diffusion model and fine-tuned the model.

[0046] The fine-tuning process adopts the same optimization objective as the original diffusion model: in is the camera pose and t is the time step. is noise, and It is a camera pose adjustment network. and is a parameter of the diffusion scheduler. is the view-conditional diffusion model, is the normal map, is the depth map, is the expected value of the parameter.

[0047] S42. In the process of fine-tuning the above view conditional diffusion model, the initial estimated camera pose is also refined by back propagation of the model. The view-conditioned diffusion model F is initialized with the camera pose The probability distribution of the new view I′ of the input crown image I is The view-conditional diffusion model F implicitly models the distribution of camera poses of the input image I and the generated image I′ according to the Bayesian rule. Therefore, while fine-tuning the view-conditional diffusion model F, the initial estimated camera pose is optimized by back-propagating the gradient to the parameters in the model. Formally speaking, the goal of optimizing the camera pose is:

[0048] in is an image The noisy latent code of is the camera pose from image i to image j, is the i-th image.

[0049] S43. In order to solve the problem that 3D Gaussian point clouds are difficult to perceive the connectivity of geometric shapes, this application proposes a 3D Gaussian geometric representation optimization strategy. This strategy combines 3D Gaussian with the depth map and normal map generated by the view-conditional diffusion model, where the depth map and normal map are used to guide the growth of the crown Gaussian point cloud and help the dense reconstruction of the Gaussian point cloud. At the same time, it can efficiently determine the neighborhood information of each Gaussian point and propagate the geometric information of the crown between them. Specifically, when the Gaussian distribution is located on the same local plane in 3D space, its 2D projection should also be located in adjacent areas and exhibit similar geometric properties, namely depth and normal. For calculating the depth value of the crown Gaussian point cloud, each view of the input model has a camera extrinsic function , the center of the initialized crown Gaussian point cloud Projected into the camera coordinate system, as , the depth value of 3D Gaussian is calculated in this way.

[0050] , and is the three-dimensional coordinate value of a point i on the x, y, and z axes, where refers to the Gaussian depth at the current viewing angle, is the camera external function. In Gaussian distribution G, its covariance matrix is ​​expressed as The rotation matrix Determine its three orthogonal eigenvectors and scale the matrix Determines the scaling along the eigenvector direction. Covariance matrix of 3D Gaussian It can be approximated as an ellipsoid shape, where the eigenvectors correspond to the axes of the ellipsoid and the scale refers to the length of the axes. According to the Gaussian optimization process, the 3D Gaussian sphere will gradually become flat and approach a plane. Therefore, for calculating the normal value of the crown Gaussian point cloud, the direction of its shortest axis can be approximated to the direction of the Gaussian normal. , the direction is

[0051] in , is the operation of finding the minimum index. The depth map and normal map generated by the optimized view conditional diffusion model under the current viewpoint are checked by using multi-view geometric consistency, and inaccurate results are filtered out to obtain the filtered depth map and normal map. The relative difference between the filtered depth value and normal value and the depth value and normal value of the crown 3D Gaussian map is calculated. For areas where the relative difference value exceeds the threshold, where the threshold value ranges from 0 to 1, it is considered that the initialized crown Gaussian distribution cannot accurately reconstruct these areas in three dimensions, where the difference value is specifically obtained by subtracting the depth value and normal value of the crown 3D Gaussian map from the calculated filtered depth value and normal value. Therefore, the pixels in these areas are projected back to the 3D space to render the depth map and normal map under the current viewpoint, initialize them as 3D Gaussians and add them to the existing Gaussian distribution for iterative optimization. The overall flow chart is as follows Figure 3 The final result of crown reconstruction is shown in Figure 4-7 shown.

[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sparse view dental crown three-dimensional reconstruction method, characterized in that: include: S1. Data preprocessing: SAM is used to segment the collected image data and obtain a mask image. The Otsu algorithm is used to remove outliers and noise, obtain a crown image, and adjust the brightness, size and clarity of the crown image. S2, initialization of the crown 3D information, using COLMAP to estimate the camera pose under sparse view, and using the conditional view diffusion model F to generate the corresponding new view, crown normal map n′ and crown depth map s′; using the findContours function in the OPENCV image processing library on the new crown image I′ to obtain the crown contour image; S3, construction of the 3D model of the crown, creating a crown visual shell based on the contour images and mask images of the crown at multiple viewing angles as a geometric framework for initializing the 3D Gaussian, thereby constructing a preliminary 3D contour of the crown; A crown visual shell is created based on the contour image and mask image of the crown at multiple viewing angles as a geometric framework for initializing the 3D Gaussian. The processed 3D Gaussian points are projected onto the input image plane and retained within the intersection of all mask image spaces. The color of the 3D point is calculated from the projected pixels of the input image using bilinear interpolation and the average value is taken. ; in and Two points on the x-axis; the position of each 3D point is set to , converting its color to ; The average distance between adjacent points is set to , rotation q is set to unit quaternion by default, opacity Set to a constant value; parameterize the initialized visual shell model as a set of Gaussian distributions , thereby 3D points can be converted into a representation of 3D Gaussians, and the points of 3D Gaussians can be used to display the visual shell model of the crown; S4, crown 3D model optimization, according to the crown normal map n' and crown depth map s' in step S2, the view conditional diffusion model F is optimized, and the back propagation of the view conditional diffusion model is used to refine the initial estimated camera pose , and optimize the Gaussian set by combining the 3D Gaussian with the depth map and the normal map.

2. The method for three-dimensional reconstruction of a tooth crown based on sparse views according to claim 1, characterized in that: Specific implementation method in step S1: S101, using the SAM model to segment the crown area in the photo inside the mouth and obtain its mask image, and then using the Otsu algorithm to remove outliers and noise to obtain a picture of only the crown; S102. Use Augmentor and Imgaug image data enhancement libraries to amplify crown image data. According to the interference factors for crown 3D reconstruction in real clinical environments, use the AdditiveGaussianNoise function in the Augmentor and Imgau image data enhancement libraries to add noise, use the Cutout function to crop the input crown image of any size to a size of 256*256, and use the Brightness function to adjust the brightness.

3. The method for three-dimensional reconstruction of tooth crowns based on sparse views according to claim 1, characterized in that: The specific implementation method of step S2 is: S21. Initialize the camera posture of n crown images input to the network , where P is the set of crown images, It means that each image in the set has three channels. is a data of the crown image set P; S22. Estimate each pair of images using the COLMAP method , Relative posture A rough estimate of the distribution, taking it as the camera rotation and pan distribution; then under the camera rotation and translation distribution, according to Calculate its maximum likelihood estimate to obtain the initial camera pose ;in is the log-likelihood function, which means that Below is the log-likelihood value of image I as a specific sample; For relative posture; is the likelihood function, which means that The probability of a given observed image I; For the crown image; S23. The single-image 3D reconstruction channel of the view conditional diffusion model F is adjusted to a multi-image 3D reconstruction channel, and then the crown image I and the relative camera pose are input into the view conditional diffusion model F as conditional signals. After training, a new crown image I′, a crown normal map n′ and a crown depth map s′ are generated under multiple camera pose transformations.

4. The method for 3D reconstruction of tooth crowns based on sparse views according to claim 3, characterized in that: The specific steps of estimating the camera pose of the two images using the COLMAP method in step S22 are as follows: S221, input two images , , the feature detection algorithm SIFT is used to extract local feature points and their corresponding descriptors from each input image; S222, using brute force matching between the extracted feature descriptors between the image pairs to establish a point feature correspondence relationship between the image pairs; S223. Using the matched feature point pairs, the camera pose is estimated through a default preset basic matrix. The camera pose includes a camera rotation matrix, a translation matrix, and a camera relative transformation matrix.

5. The method for 3D reconstruction of tooth crowns based on sparse views according to claim 1, characterized in that: Specific implementation method of the crown visual shell technology in step S3: (1) Obtain the crown image and mask image, and at the same time obtain the camera pose calculated above; (2) Use the Canny edge detection algorithm to perform edge detection and contour extraction on each image, and convert the processed image into a point set or contour line; (3) Based on the camera attitude parameters, the two-dimensional information of the contour is mapped to the corresponding ray into the three-dimensional space to generate the viewing cone of each viewing angle; (4) Using a three-dimensional geometric intersection algorithm to calculate the intersection of volumes and determine the visible shell of the object; (5) The Delaunay triangulation algorithm is used to take the visible shell volume boundary as a constraint on the object surface to generate a polygonal 3D mesh crown model.

6. The method for three-dimensional reconstruction of tooth crowns based on sparse views according to claim 1, characterized in that: Specific implementation of step S4: S41. Optimize the text-to-image diffusion model to a text-to-normal and depth diffusion model; through the rotation matrix of the camera parameters Convert the normal map n′ and depth map s′ from world coordinates to camera coordinates; Set the camera posture It is added as a condition to the view-conditional diffusion model and the model is fine-tuned; the fine-tuning process uses the same optimization objective as the original diffusion model: in is the camera posture, t is the time step; is noise, and is a camera pose adjustment network; and are the parameters of the diffusion scheduler; is the view-conditional diffusion model, is the normal map, is the depth map, is the expected value of the parameter; S42. Model fine-tuning process In the above view condition expansion, the initial estimated camera pose is also refined through the back propagation of the model ; The view-conditioned diffusion model F is initialized with the camera pose and the crown image as the condition to synthesize a new view of the crown, and the probability distribution of the new view I′ of the crown Modeling; According to the Bayesian rule, the view conditional diffusion model F implicitly simulates the camera pose distribution of the input image I and the new view image I′. The goal of optimizing the camera pose is: in is an image The noisy latent code of is the camera pose from image i to image j, is the i-th image; S43. Combine 3D Gaussians with depth and normal maps generated by a view-conditioned diffusion model. When the Gaussian distributions are located on the same local plane in 3D space, the 2D projections are also located in adjacent areas and exhibit similar geometric properties, namely depth and normals. Each view of the input model has a camera external function. , the center of the initialized crown Gaussian point cloud Projected into the camera coordinate system, as The depth value of the 3D Gaussian; in , and is the three-dimensional coordinate value of a point i on the x, y, and z axes, where refers to the Gaussian depth at the current viewing angle, is the camera external function; In Gaussian distribution G, its covariance matrix is ​​expressed as ; Rotation matrix Determine its three orthogonal eigenvectors and scale the matrix Determine the scaling along the eigenvector direction; Covariance matrix of a 3D Gaussian The approximate representation is an ellipsoid shape, where the eigenvectors correspond to the axes of the ellipsoid and the scaling refers to the length of the axes; according to the optimization process of the Gaussian shader, the 3D Gaussian sphere will gradually become flat and approach a plane; for calculating the normal value of the crown Gaussian point cloud, the direction of its shortest axis is approximately the direction of the Gaussian normal , the direction is in , It is the operation of finding the minimum value index; The depth map and normal map generated by the optimized view conditional diffusion model under the current viewpoint are checked by using multi-view geometric consistency, and inaccurate results are filtered out to obtain filtered depth map and normal map; the relative difference values ​​between the filtered depth values ​​and normal values ​​and the depth values ​​and normal values ​​of the crown 3D Gaussian mapping are calculated. For areas where the relative difference values ​​exceed the threshold, the initialized crown Gaussian distribution cannot accurately reconstruct the area in three dimensions; the pixels in the area are projected back to the 3D space to render the depth map and normal map under the current viewpoint, initialize them as 3D Gaussians and add them to the existing Gaussian distribution for iterative optimization.

Images