Automatic repair model generation method, automatic repair method, equipment and storage medium

Abstract

This invention provides an automatic restoration model generation method, an automatic restoration method, an apparatus, and a storage medium. The automatic restoration model generation method includes: determining a first sample defective tooth depth map, a second sample defective tooth depth map, a first sample restored tooth depth map, and a second sample restored tooth depth map based on k first sample defective tooth models, a first sample restored tooth model, a second sample defective tooth model, and a second sample restored tooth model; stitching these k sample stitched depth maps together; performing data augmentation processing on the k sample stitched depth maps to obtain m target training images; and training a neural network algorithm based on the m target training images to obtain a restoration model. This method enables more accurate automatic restoration models, reduces the computational load on the processor, improves computational efficiency, and allows the restoration model to output a corresponding restored tooth depth map based on the depth map of the defective tooth to be restored, eliminating the need for dental technician processing and improving the efficiency of defective tooth restoration.

Description

[Technical Field]

[0001] This invention relates to the field of dental technology, and in particular to an automatic repair model generation method, automatic repair method, device and storage medium. [Background Technology]

[0002] In clinical dentistry, the fabrication of dental inlays often takes into account the location of the defective tooth, the shape of the occlusal surface, and the correct occlusal function. Existing dental inlay fabrication usually relies on dental CAD design, is based on a standard template for defective tooth restoration, and requires a lot of manual adjustment of the occlusal relationship to restore the correct occlusal function.

[0003] The existing fabrication of dental inlays has the following problems: 1. It is semi-automatic, which is limited by the location and size of the patient's tooth defect; 2. It requires manual interaction and a lot of manual occlusal adjustment to restore the correct occlusal function. It requires high design experience from clinical technicians, and the time and economic cost of training experienced technicians is high; 3. The effect cannot be adapted to different patients.

[0004] Therefore, it is necessary to provide an automatic repair model generation method, an automatic repair method, an apparatus, and a storage medium to overcome the above-mentioned deficiencies. [Summary of the Invention]

[0005] The purpose of this invention is to provide an automatic repair model generation method, an automatic repair method, an apparatus, and a storage medium to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, a first aspect of the present invention provides an automatic repair model generation method, comprising the following steps:

[0007] Obtain k first sample defective tooth models and corresponding first sample repair tooth models. The first sample defective tooth model is a three-dimensional model with only defective teeth. Obtain k second sample defective tooth models and corresponding second sample repair tooth models. The second sample defective tooth model is a three-dimensional model with the same defective teeth and adjacent teeth. k is a positive integer.

[0008] A corresponding first sample defective tooth depth map is determined based on the occlusal surface of the first sample defective tooth model for each first sample defective tooth model; a corresponding second sample defective tooth depth map is determined based on the occlusal surface of the second sample defective tooth model for each second sample defective tooth model; a corresponding first sample restored tooth depth map is determined based on the occlusal surface of the first sample restored tooth model for each first sample restored tooth model; and a corresponding second restored tooth depth map is determined based on the occlusal surface of the second sample restored tooth model for each second sample restored tooth model.

[0009] The depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth for the same defective tooth are stitched together to obtain k sample stitched depth maps.

[0010] Data augmentation processing is performed on the k sample stitched depth maps to obtain m target training images. The target training images include a first target defective tooth depth map obtained based on the first sample defective tooth depth map, a second target defective tooth depth map obtained based on the second sample defective tooth depth map, a first target restored tooth depth map obtained based on the first sample restored tooth depth map, and a second target restored tooth depth map obtained based on the second sample restored tooth depth map, where m is a positive integer and greater than k.

[0011] A repair model is obtained by training a neural network algorithm based on m target training images.

[0012] In some embodiments, the step of stitching together the first sample defective tooth depth map, the second sample defective tooth depth map, the first sample restored tooth depth map, and the second sample restored tooth depth map for the same defective tooth to obtain k sample stitched depth maps further includes:

[0013] The open() function of the Image library is used to read the depth map of the first sample defective tooth, the depth map of the second sample defective tooth, the depth map of the first sample restored tooth, and the depth map of the second sample restored tooth, resulting in four corresponding depth map arrays;

[0014] The concatenate function from the NumPy library is used to concatenate the four depth map arrays along axis=1 to obtain a concatenated depth map array;

[0015] The fromarray() function is used to convert the depth map stitching array into an image for export, thus obtaining the sample stitched depth map.

[0016] In some embodiments, the step of performing data augmentation processing on the stitched depth maps of the k samples to obtain m target training images, where m is a positive integer greater than k, further includes:

[0017] The rotate() algorithm from the OpenCV library is used to perform data augmentation on the depth maps stitched together from the k samples, resulting in m target training images.

[0018] In some embodiments, determining a corresponding first sample defective tooth depth map based on the occlusal surface of the first sample defective tooth model for each first sample defective tooth, and determining a corresponding second sample defective tooth depth map based on the occlusal surface of the second sample defective tooth model for each second sample defective tooth; determining a corresponding first sample restored tooth depth map based on the occlusal surface of the first sample restored tooth model for each first sample restored tooth, and determining a corresponding second restored tooth depth map based on the occlusal surface of the second sample restored tooth model for each second sample restored tooth, further includes:

[0019] OBB bounding boxes are constructed for the first sample defective tooth model, the second sample defective tooth model, the first sample restored tooth model, and the second sample restored tooth model, respectively;

[0020] Based on the obb bounding box, establish an interlocking plane with a resolution of 256*256 and a pixel value of 0;

[0021] Using the three-dimensional library Open3D, ray projection and nearest point lookup are performed on the occlusal planes of the first sample defective tooth model, the second sample defective tooth model, the first sample restored tooth model, and the second sample restored tooth model to obtain the ray intersection point and the shortest distance d to the nearest ray intersection point;

[0022] The ray intersection points are mapped into pixels and projected onto a plane based on the shortest distance d, resulting in the first sample defective tooth depth map, the second sample defective tooth depth map, the first sample restored tooth depth map, and the second restored tooth depth map. The mapping method is as follows:

[0023]

[0024] Where i is the grayscale value, and h is the preset value.

[0025] In some embodiments, m ≥ 4k.

[0026] In some embodiments, the step of training a neural network algorithm based on m target training images to obtain a repair model further includes:

[0027] Based on the depth map of the first target defective tooth and the depth map of the first target repaired tooth, a generative adversarial network algorithm is trained to obtain a local generative network model and a local discriminative network model. The local generative network model is used to generate a local repair simulation depth map, and the local discriminative network model is used to identify whether the input image is real.

[0028] Generative Adversarial Network (GAN) algorithms are trained based on the depth maps of the second target defective tooth and the second target repaired tooth to obtain a global generative network model and a global discriminative network model. The global generative network model is used to generate a global repair simulation depth map, and the global discriminative network model is used to discriminate whether the input image is real.

[0029] The local generative network model, the local discriminative network model, the global generative network model, and the global discriminative network model are subjected to error optimization to obtain the repair model.

[0030] In some embodiments, the local restoration simulation depth map includes restoration occlusal features trained based on the depth map of a first target defective tooth, and the global restoration simulation depth map includes global contour features trained based on the depth map of a second target defective tooth. The step of error optimization of the local generation network model, the local discrimination network model, the global generation network model, and the global discrimination network model further includes:

[0031] Error optimization is performed on the local generative network model and the local discriminative network model using a local loss function. for:

[0032]

[0033] in This indicates the difference between the generated repaired occlusal features and the occlusal features in the depth map of the first target repaired tooth. It is the local adversarial loss function, λ i It is a constant, σ is the sigmoid function, and x is a constant. local This is the depth map of the first target defect tooth, y local This is the depth map of the first target tooth to be restored. E is the false local repair simulation depth map information generated by the local generative network model to confuse the local discrimination network model, and E is the cross-entropy loss function;

[0034] Error optimization is performed on the global generator network model and the global discriminator network model using a global loss function. for:

[0035]

[0036] in This indicates the difference between the generated global contour features and the contour features in the depth map of the second target restoration tooth. It is the global adversarial loss function, λ global It is a constant, G globalThe false globally repaired simulated depth map information generated by the global generative network model is used to confuse the adversarial network. global This is the depth map of the second target defect tooth, y global This is the depth map of the second target tooth to be restored;

[0037] Calculate the target loss function to optimize the error. for: Where λ is a constant.

[0038] A second aspect of the present invention provides an automatic repair method, comprising the following steps:

[0039] Obtain a model of the tooth with the defect to be repaired;

[0040] The corresponding depth map of the defective tooth to be repaired is determined based on the occlusal surface of the model of the defective tooth to be repaired;

[0041] Input the depth map of the defective tooth to be repaired into any of the above-mentioned repair models to obtain an automatically repaired tooth depth map;

[0042] The automatic restoration tooth depth map is converted into an automatic restoration tooth model.

[0043] A third aspect of the present invention provides a computing device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the automatic repair model generation method described in any one of the preceding claims.

[0044] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the automatic repair model generation method described in any of the preceding claims.

[0045] The technical effects of this invention are as follows: By simultaneously processing and training the first sample defective tooth model, the second sample defective tooth model, the first sample restored tooth model, and the second sample restored tooth model, the comprehensiveness and accuracy of the tooth data are improved, thereby enabling the subsequently trained automatic restoration model of the defective tooth to more accurately simulate the restoration morphology of the defective tooth. Subsequent training using the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth obtained from the occlusal surface can reflect more three-dimensional details, reduce information loss, and thus make the subsequent training data more accurate. It also reduces the computational load on the processor and improves computational efficiency. By stitching the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth to form a sample stitched depth map, the number of images obtained is reduced. In this case, by performing data augmentation on this sample stitched depth map, multiple target training images are obtained, thereby increasing the amount of training sample data and making the subsequent training results more accurate and comprehensive.

[0046] The restoration model allows for subsequent processing. After inputting a depth map of the tooth with defects to be restored, the model can output a corresponding depth map of the restored tooth. Then, a 3D model of the restored tooth can be reconstructed based on this depth map. This process requires no intervention from dental technicians, simplifying the operation, improving the efficiency of defect tooth restoration, and saving costs. Furthermore, the 3D model of the restored tooth obtained from the restoration model can adapt to a wide range of types of defects, demonstrating good adaptability.

[0047] To make the above-mentioned objects, features and advantages of the invention more apparent and understandable, preferred embodiments of the invention are described below in detail with reference to the accompanying drawings. [Attached Image Description]

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart of an embodiment of the automatic repair model generation method provided by the present invention;

[0050] Figure 2 A flowchart of an embodiment of the automatic repair method provided by the present invention;

[0051] Figure 3A schematic diagram of the automatic repair model generation device provided by the present invention;

[0052] Figure 4 A schematic diagram of the hardware architecture of the computer device provided by the present invention;

[0053] Figure 5 A schematic diagram of the sample stitching depth map provided in an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of the overall structure of the generative adversarial network provided by this invention;

[0055] Figure 7 A schematic diagram of the structural models of the local generative network model and the global generative network model provided by the present invention;

[0056] Figure 8 A schematic diagram of the structural models of the local discrimination network model and the global discrimination network model provided by the present invention;

[0057] Figure 9 The first schematic diagram of the automatic repair tooth model generated from the input depth map of the defective tooth to be repaired, provided by the present invention;

[0058] Figure 10 The second schematic diagram of the repair model provided by the present invention generates an automatic repair tooth model from the input depth map of the defective tooth to be repaired;

[0059] Figure 11 The third schematic diagram shows how the repair model provided by the present invention generates an automatic repair tooth model from the input depth map of the defective tooth to be repaired.

Detailed Implementation Methods

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0061] Please see Figure 1This invention provides an automated restoration model generation method, executed by a terminal, on which a dental technician designs the restoration. The terminal can be a computing device including one or more processors. These processors may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this invention; no limitation is made herein. The computing device includes one or more processors, which can be of the same type, such as one or more CPUs; or of different types, such as one or more CPUs and one or more ASICs; no limitation is made herein.

[0062] Please see Figure 1 This invention provides an automatic repair model generation method, comprising the following steps:

[0063] Step 110: Obtain k first sample defective tooth models and corresponding first sample repair tooth models. The first sample defective tooth model is a three-dimensional model with only defective teeth. Obtain k second sample defective tooth models and corresponding second sample repair tooth models. The second sample defective tooth model is a three-dimensional model with the same defective teeth and adjacent teeth. k is a positive integer.

[0064] Step 120: Determine the corresponding depth map of the first sample defective tooth based on the occlusal surface of the first sample defective tooth model for each first sample defective tooth model; determine the corresponding depth map of the second sample defective tooth based on the occlusal surface of the second sample defective tooth model for each second sample defective tooth model; determine the corresponding depth map of the first sample restored tooth based on the occlusal surface of the first sample restored tooth model for each first sample restored tooth model; determine the corresponding depth map of the second sample restored tooth based on the occlusal surface of the second sample restored tooth model for each second sample restored tooth model.

[0065] Step 130: The depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth are stitched together to obtain k sample stitched depth maps.

[0066] Step 140: Perform data augmentation processing on the k sample stitched depth maps to obtain m target training images. The target training images include a first target defective tooth depth map obtained based on the first sample defective tooth depth map, a second target defective tooth depth map obtained based on the second sample defective tooth depth map, a first target restored tooth depth map obtained based on the first sample restored tooth depth map, and a second target restored tooth depth map obtained based on the second sample restored tooth depth map, where m is a positive integer and greater than k.

[0067] Step 150: Train a neural network algorithm based on m target training images to obtain a repair model.

[0068] In step 110, k first sample defective tooth models, k first sample restored tooth models, k second sample defective tooth models, and k second sample restored tooth models are obtained to acquire sample data for subsequent machine training.

[0069] In this model, both the first and second sample defective tooth models are 3D reconstruction models of actual defective teeth. The first sample restored tooth model is a 3D reconstruction model of the restored tooth obtained by the dental technician from the actual defective tooth corresponding to the first sample defective tooth model. The second sample restored tooth model is a 3D reconstruction model of the restored tooth obtained by the dental technician from the actual defective tooth corresponding to the second sample defective tooth model. This 3D reconstruction method is existing technology and will not be described in detail here.

[0070] The first sample defective tooth model is a 3D model containing only the defective tooth. Accordingly, during subsequent training and learning, interference from other information is reduced, resulting in a relatively larger and more accurate set of features for the defective tooth. This defective tooth can be a single tooth or multiple teeth; no limitation is made here, and it can be set as needed. In this embodiment, the first sample defective tooth model is a 3D model focusing only on a single defective tooth.

[0071] The second sample defective tooth model is a three-dimensional model with the same defective tooth as the first sample defective tooth model, plus adjacent teeth. In this case, during subsequent training and learning, it can more accurately learn the relationship between the defective tooth and adjacent teeth, as well as more comprehensive features. The adjacent teeth are determined based on the location of the defective tooth. For example, in some embodiments, the adjacent teeth are located on both sides of the defective tooth; in other embodiments, the adjacent teeth are located on one side of the defective tooth. There is no limitation here, and it is determined as needed. In this embodiment, the second sample defective tooth model is a single defective tooth corresponding to the first sample defective tooth model, plus the adjacent teeth located on both sides of the single defective tooth. It should be noted that in this invention, the defective tooth is a tooth with a partially defective occlusal surface, and the resulting restoration tooth is an inlay-type restoration tooth.

[0072] The first and second sample defective tooth models are three-dimensional models obtained through oral scanning. For example, scanning images of the defective tooth from different angles are obtained through oral scanning, and then three-dimensional reconstruction is performed based on the scanning images from different angles to obtain the corresponding first and second sample defective tooth models; or the tooth model of the defective tooth is obtained by taking a dental mold, and then the dental mold is photographed to obtain scanning images from different angles, and then three-dimensional reconstruction is performed to obtain the corresponding first and second sample defective tooth models.

[0073] Similarly, the first sample defective tooth model and the second sample defective tooth model are reconstructed into three-dimensional models by oral scanning or dental impression taking, which will not be elaborated here.

[0074] It should be noted that information loss occurs during the 3D reconstruction process. The 3D model data of the defective tooth in the first sample defective tooth model and the 3D model of the defective tooth and adjacent teeth are different. Similarly, the 3D model data of the defective tooth restoration model in the first sample restoration model and the second sample restoration model are also different. By simultaneously processing and training the data of the first sample defective tooth model, the second sample defective tooth model, the first sample restoration model, and the second sample restoration model, the comprehensive accuracy of the tooth data is improved. This allows the automatically restored defective tooth model obtained through subsequent training to more accurately simulate the restoration morphology of the defective tooth.

[0075] To ensure a large number of training samples, k is typically chosen to be relatively large, preferably greater than or equal to 100. A larger k value generally results in a more accurate repair model. To guarantee a sufficient amount of data while minimizing the workload of data collection, the preferred range for k is 400 ≤ k ≤ 800.

[0076] In step 120, directly training based on the 3D model data would result in a huge computational load, affecting the generation efficiency of the restoration model and placing a significant computational burden on the processor. Therefore, to reduce the amount of data, depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth are determined based on the occlusal surfaces of each tooth model. Subsequent training is then conducted using these depth maps, reducing the computational load on the processor and improving computational efficiency. In particular, defective tooth restoration primarily focuses on the occlusal condition of the occlusal surface and its positional relationship with adjacent teeth. In this case, the depth maps obtained based on the occlusal surface can reflect more 3D details, reducing information loss and thus making the subsequent training data more accurate.

[0077] In steps 130 and 140, a sample stitched depth map is formed by stitching together the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth. This reduces the number of images obtained. In this case, data augmentation is performed on this sample stitched depth map to obtain multiple target training images, thus increasing the amount of training data and improving the accuracy and comprehensiveness of subsequent training results. Since the number of sample stitched depth maps is relatively small, the processor can reduce computational load during data augmentation. Data augmentation can be performed directly on a single sample stitched image to obtain the same data augmentation result as the first sample defective tooth depth map, the second sample defective tooth depth map, the first sample restored tooth depth map, and the second sample restored tooth depth map, significantly improving processor efficiency.

[0078] There are multiple ways to stitch together the first sample defective tooth depth map, the second sample defective tooth depth map, the first sample restored tooth depth map, and the second sample restored tooth depth map; this is an existing technology.

[0079] For example, in some embodiments, the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth can be converted into a depth map array. This depth map array is then stitched together to obtain a stitched depth map array, which is then converted into a sample stitched depth map. In some embodiments, the image boundaries of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth can be directly obtained. Then, based on these image boundaries, the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth can be stitched and fused together to obtain a sample stitched depth map.

[0080] There are various methods for data augmentation, such as rotating and / or flipping and / or translating and / or sharpening and / or adding noise to the sample stitched depth map. These operations cause the resulting target training image to change in different ways relative to the sample stitched depth map, thereby increasing the amount of sample data and improving the accuracy and comprehensiveness of the training output. The algorithms related to this data augmentation are existing technologies and will not be described in detail here.

[0081] In this process, a single sample stitched depth map undergoes data augmentation to generate multiple target training images, with the original sample stitched depth map serving as one of these target training images. Accordingly, the content of the data-augmented target training images includes: a data-augmented first target defective tooth depth map based on the first sample defective tooth depth map; a data-augmented second target defective tooth depth map based on the second sample defective tooth depth map; a data-augmented first target restored tooth depth map based on the first sample restored tooth depth map; and a data-augmented second target restored tooth depth map based on the second sample restored tooth depth map.

[0082] The value of m depends on the data augmentation method. If two data augmentation methods are performed, m equals 2k. For example, data augmentation involving rotation and flipping of the sample stitched depth map is considered to be two-method data augmentation. If three data augmentation methods are performed, m equals 3k. For example, data augmentation involving rotation, flipping, and translation of the sample stitched depth map is considered to be three-method data augmentation. Similarly, if four data augmentation methods are performed, m equals 4k, and so on. These values ​​will not be explained in detail here and can be set as needed.

[0083] In step 150, the restoration model obtained by training a neural network algorithm based on m target training images can be used to output a corresponding restoration tooth depth map after inputting a depth map of the tooth with defects to be restored. Then, a 3D model of the restoration tooth is reconstructed based on this depth map. This process requires no intervention from dental technicians, improving the efficiency of tooth restoration.

[0084] There are various types of neural network algorithms, which are existing technologies, such as GAN, CNN, and RNN algorithms, which will not be described in detail here.

[0085] In some embodiments, the processor can directly perform content recognition based on the target training image to obtain the corresponding first target defect tooth depth map, second target defect tooth depth map, first target restored tooth depth map, and second target restored tooth depth map, thereby training the neural network algorithm accordingly to obtain the restoration model.

[0086] In some embodiments, the processor can split the target training image to obtain a first target defective tooth depth map, a second target defective tooth depth map, a first target restored tooth depth map, and a second target restored tooth depth map, thereby training a neural network algorithm accordingly to obtain a restoration model.

[0087] In summary, steps 110 to 150, by simultaneously processing and training the first sample defective tooth model, the second sample defective tooth model, the first sample restored tooth model, and the second sample restored tooth model, improves the comprehensiveness and accuracy of the tooth data. This allows the subsequently trained automatic restoration model of the defective tooth to more accurately simulate the restoration morphology of the defective tooth. Subsequent training using the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth obtained from the occlusal surface reveals more three-dimensional details, reduces information loss, and makes the subsequent training data more accurate. It also reduces the computational load on the processor and improves computational efficiency. By stitching the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth to form a sample stitched depth map, the number of images obtained is reduced. In this case, data augmentation is performed on this sample stitched depth map to obtain multiple target training images, thus increasing the amount of training sample data and contributing to more accurate and comprehensive results in subsequent training.

[0088] The restoration model allows for subsequent processing. After inputting a depth map of the tooth with defects to be restored, the model can output a corresponding depth map of the restored tooth. Then, a 3D model of the restored tooth can be reconstructed based on this depth map. This process requires no intervention from dental technicians, simplifying the operation, improving the efficiency of defect tooth restoration, and saving costs. Furthermore, the 3D model of the restored tooth obtained from the restoration model can adapt to a wide range of types of defects, demonstrating good adaptability.

[0089] In some embodiments, step 130 further includes:

[0090] Step 131: Use the open() function of the Image library to read the depth map of the first sample defective tooth, the depth map of the second sample defective tooth, the depth map of the first sample restored tooth, and the depth map of the second sample restored tooth, to obtain four corresponding depth map arrays;

[0091] Step 132: Use the concatenate function of the NumPy library to concatenate four depth map arrays on axis=1 to obtain a concatenated depth map array;

[0092] Step 133: Use the fromarray() function to convert the depth map stitching array into an image for export, and obtain the sample stitched depth map.

[0093] In steps 131 to 133, the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth are stitched together using an array concatenation method. The operation is relatively simple, and the open() function of the Image library, the concatenate function of the NumPy library, and the fromarray() function are all open-source algorithms, which reduces the development difficulty.

[0094] Where axis=1 indicates that the depth map array is arranged column-wise, with the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth arranged as needed to form the depth map array. For example, in some embodiments, the first sample defective tooth depth map is arranged first, the first sample restored tooth depth map is arranged second, the second sample defective tooth depth map is arranged third, and the second sample restored tooth depth map is arranged fourth; in other embodiments, the first sample defective tooth depth map is arranged first, the second sample defective tooth depth map is arranged second, the first sample restored tooth depth map is arranged third, and the second sample restored tooth depth map is arranged fourth; no limitation is made here, and it can be set as needed. See reference. Figure 5 In this embodiment, the depth map of the second sample restored tooth is arranged as the first, the depth map of the first sample restored tooth is arranged as the second, the depth map of the second sample defective tooth is arranged as the third, and the depth map of the first sample defective tooth is arranged as the fourth.

[0095] In some embodiments, step 140 further includes:

[0096] Step 141: Use the rotate() algorithm in the OpenCV library to perform data augmentation on the k sample stitched depth map to obtain m target training images.

[0097] In step 141, the rotate() algorithm can perform various operations, including rotation and flipping. For example, in this embodiment, the parameters of the rotate() algorithm are set as follows:

[0098] The functions `cv2.ROTATE_90_CLOCKWISE`, `cv2.ROTATE_90_COUNTERCLOCKWISE`, and `flip()` (where `flipcode` takes values ​​of 0, 1, and -1) can be used to rotate and flip images at different angles, increasing the number of target training images.

[0099] In some embodiments, step 120 further includes:

[0100] Step 121: Construct OBB bounding boxes for the first sample defective tooth model, the second sample defective tooth model, the first sample restored tooth model, and the second sample restored tooth model, respectively;

[0101] Step 122: Based on the obb bounding box, establish an interlocking plane with a resolution of 256*256 and a pixel value of 0;

[0102] Step 123: Using the 3D library Open3D, perform ray projection and nearest point lookup on the occlusal planes of the first sample defective tooth model, the second sample defective tooth model, the first sample restored tooth model, and the second sample restored tooth model to obtain the ray intersection point and the shortest distance d to the nearest ray intersection point;

[0103] Step 124: Based on the shortest distance d, map the ray intersection points into pixels and project them onto the plane to obtain the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth. The mapping method is as follows:

[0104]

[0105] Where i is the grayscale value, and h is the preset value.

[0106] In some embodiments, m ≥ 4k to ensure a large number of target training images, thereby making the training results more accurate and obtaining a more accurate repair model. Preferably, 4k ≤ m ≤ 7k to prevent an excessive number of target training images from affecting the processor's computational efficiency.

[0107] See Figures 6 to 8 In some embodiments, step 150 further includes:

[0108] Step 151: Based on the depth map of the first target defective tooth and the depth map of the first target restored tooth, a generative adversarial network algorithm is trained to obtain a local generative network model and a local discriminative network model. The local generative network model is used to generate a local restoration simulation depth map, and the local discriminative network model is used to discriminate whether the input image is real.

[0109] Step 152: Based on the depth map of the second target defective tooth and the depth map of the second target restored tooth, a generative adversarial network algorithm is trained to obtain a global generative network model and a global discriminative network model. The global generative network model is used to generate a global restoration simulation depth map, and the global discriminative network model is used to discriminate whether the input image is real.

[0110] Step 153: Perform error optimization on the local generator network model, local discriminator network model, global generator network model, and global discriminator network model to obtain the repair model.

[0111] In steps 151 to 152, the local generative network model and the local discriminative network model obtained by the generative adversarial network algorithm are subjected to adversarial training, as are the global generative network model and the global discriminative network model. This ensures that the depth map of the restored tooth generated by the local generative network model and the global generative network model meets expectations and yields a more accurate restoration result.

[0112] This generative adversarial network algorithm is usually implemented using the GAN algorithm.

[0113] The local generative network model can learn all features of the depth maps of the first target defective tooth and the first target restored tooth, or it can learn only some features, such as occlusal features including tooth fissures, ridges, and cusps. The local generative network model then performs simulated restoration based on the learned features, resulting in a generated local restoration simulation depth map.

[0114] In this model, the features learned by the local discriminative network model are at least partially the same as those learned by the local generative network model. It learns the corresponding features of the depth maps of the first target defective tooth and the first target restored tooth to form a match. The features learned by the local discriminative network model are considered to be those of the depth map of the partially restored tooth obtained by the dental technician. Then, based on the learned features, the local discriminative network model determines whether the input depth map is real. Specifically, if it determines that the depth map comes from a simulated partially restored tooth, it is a fake depth map; if it determines that the depth map comes from the partially restored tooth obtained by the dental technician, it is a real depth map.

[0115] Similarly, the global generative network model can learn all features of the depth maps of the second target defective tooth and the second target restored tooth, or it can learn only some features, such as the contour features of multiple teeth and / or the distance features between the defective tooth and adjacent teeth, or other features, as needed. Then, the local generative network model performs simulated restoration based on the learned features to obtain the generated global restoration simulation depth map.

[0116] The features learned by the global discriminative network model are at least partially the same as those learned by the global generative network model. It learns corresponding features from the depth maps of the second target defective tooth and the second target restored tooth to form a match. The features learned by the global discriminative network model are considered to be those of the global restored tooth depth map obtained by the dental technician. Then, based on the learned features, the global discriminative network model determines whether the input depth map is real. Specifically, if it determines that the depth map comes from a globally simulated restoration, it is a fake depth map; if it determines that the depth map comes from a globally restored tooth depth map obtained by the dental technician, it is a real depth map.

[0117] In step 153, by performing error optimization, once the adversarial training of the local generator network model and the local discriminator network model, as well as the adversarial training of the global generator network model and the global discriminator network model, meets the optimized error, it can be considered that the repair simulation depth map generated by the local generator network model and the global generator network model meets the expectations.

[0118] In some embodiments, the local restoration simulation depth map includes restoration occlusal features trained based on the depth map of the first target defective tooth, and the global restoration simulation depth map includes global contour features trained based on the depth map of the second target defective tooth. Step 153 further includes:

[0119] Step 1531: Optimize the error of the local generative network model and the local discriminative network model using the local loss function. for:

[0120]

[0121] in This indicates the difference between the generated occlusal features of the restored tooth and the occlusal features in the depth map of the first target restored tooth. It is the local adversarial loss function, λ i It is a constant, σ is the sigmoid function, and x is a constant. local This is the depth map of the first target defective tooth, y local This is the depth map of the first target tooth for restoration. E is the false local repair simulation depth map information generated by the local generative network model to confuse the local discrimination network model, and E is the cross-entropy loss function.

[0122] Step 1532: Optimize the errors of the global generator network model and the global discriminator network model using a global loss function. (Global loss function...) for:

[0123]

[0124] in This indicates the difference between the generated global contour features and the contour features in the depth map of the second target restored tooth. It is the global adversarial loss function, λ global It is a constant, representing the fake global repair simulation depth map information generated by the global generative network model to confuse the adversarial network, x. global This is the depth map of the second target defect tooth, y global This is the depth map of the second target tooth for restoration;

[0125] Step 1533: Calculate the target loss function to optimize the error. for: Where λ is a constant.

[0126] In steps 1531 and 1532, λ i As needed, in this embodiment, λ is set... i The value is 100.

[0127] Among them, the restorative occlusal features represent the features of tooth occlusion such as pits, fissures, ridges, and cusps corresponding to the occlusal site of the restorative tooth, while the global contour features represent the contour features of the restorative tooth and adjacent teeth. Through these restorative occlusal features, the distance relationship between the restorative tooth and adjacent teeth and related adjacency relationships can be represented. In the training and learning process, better restorative tooth shaping data can be obtained based on these restorative occlusal features, which is conducive to more accurate restoration model construction.

[0128] In step 1533, λ is an empirical value, which is set as needed.

[0129] In this embodiment, steps 1531 to 1533 are trained and learned based on the Pix2Pix-CGAN framework.

[0130] Please see Figure 2 The invention also provides an automatic repair method, comprising the following steps:

[0131] Step 210: Obtain a model of the tooth with the defect to be repaired;

[0132] Step 220: Determine the corresponding depth map of the tooth to be repaired based on the occlusal surface of the tooth model;

[0133] Step 230: Input the depth map of the defective tooth to be repaired into the above repair model to obtain the depth map of the automatically repaired tooth;

[0134] Step 240: Convert the automatic restoration tooth depth map into an automatic restoration tooth model.

[0135] In steps 210 to 240, the defective tooth model to be repaired is a three-dimensional reconstruction model of the defective tooth. Then, based on the occlusal surface of the three-dimensional reconstruction model, the corresponding depth map of the defective tooth to be repaired is determined. After the depth map of the defective tooth to be repaired is input into the training-obtained repair model, the repair model generates an automatically repaired tooth depth map accordingly (see...). Figures 9 to 11 The process involves automatically converting the depth map of the restored tooth into an automated restoration model. This process requires no intervention from a dental technician, simplifying the operation, improving the efficiency of restoring defective teeth, and saving costs. Furthermore, the resulting 3D restoration model can adapt to a wide range of types of defective teeth, demonstrating good adaptability.

[0136] Among them, the Pix2mesh algorithm can be used to convert the depth map of the auto-restoring tooth into an auto-restoring tooth model.

[0137] In some embodiments, the automatic restoration tooth model and the defective tooth model to be restored can be compared and trimmed to obtain an inlay model of the defective tooth to be restored.

[0138] like Figure 3 As shown, the automatic repair model generation device 300 includes:

[0139] The acquisition module 310 is used to acquire k first sample defective tooth models and corresponding first sample repair tooth models of the first sample defective tooth models. The first sample defective tooth model is a three-dimensional model with only defective teeth. The module also acquires k second sample defective tooth models and corresponding second sample repair tooth models of the second sample defective tooth models. The second sample defective tooth models are three-dimensional models with the same defective teeth and adjacent teeth. k is a positive integer.

[0140] The determination module 320 is used to determine the corresponding depth map of the first sample defective tooth based on the occlusal surface of the first sample defective tooth model for each first sample defective tooth model, and to determine the corresponding depth map of the second sample defective tooth based on the occlusal surface of the second sample defective tooth model for each second sample defective tooth model. It is also used to determine the corresponding depth map of the first sample restored tooth based on the occlusal surface of the first sample restored tooth model for each first sample restored tooth model, and to determine the corresponding depth map of the second sample restored tooth based on the occlusal surface of the second sample restored tooth model for each second sample restored tooth model.

[0141] The splicing module 330 is used to splice the first sample defective tooth depth map, the second sample defective tooth depth map, the first sample restored tooth depth map, and the second sample restored tooth depth map for the same defective tooth to obtain k sample spliced ​​depth maps.

[0142] The data augmentation module 340 is used to perform data augmentation processing on the stitched depth map of k samples to obtain m target training images, where m is a positive integer and greater than k. The target training images include a first target defective tooth depth map obtained based on the first sample defective tooth depth map, a second target defective tooth depth map obtained based on the second sample defective tooth depth map, a first target restored tooth depth map obtained based on the first sample restored tooth depth map, and a second target restored tooth depth map obtained based on the second sample restored tooth depth map.

[0143] Training module 350 is used to train a neural network algorithm based on m target training images to obtain a repair model.

[0144] Figure 4 This illustration schematically depicts the hardware architecture of a computer device 10000 suitable for implementing an automatic repair model generation method according to an embodiment of the present invention. In this embodiment, the computer device 10000 is a device capable of automatically performing score calculations and / or information processing according to pre-set or stored instructions. For example, it may be a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), gateway, etc. Figure 3 As shown, the computer device 10000 includes, but is not limited to, at least the following: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked to each other via a system bus. Wherein:

[0145] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as the program code for the automatic repair model generation method. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0146] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0147] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0148] It should be pointed out that, Figure 4 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0149] In this embodiment, the automatic repair model generation method stored in memory 10010 can be further divided into one or more program modules and executed by a processor (processor 10020 in this embodiment) to complete the embodiment of the present invention.

[0150] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by at least one processor, implements the steps of the automatic repair model generation method in the embodiments.

[0151] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the automatic repair model generation method in this embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0152] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0153] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An automatic repair model generation method, characterized in that, Includes the following steps: Obtain k first sample defective tooth models and corresponding first sample repair tooth models. The first sample defective tooth model is a three-dimensional model with only defective teeth. Obtain k second sample defective tooth models and corresponding second sample repair tooth models. The second sample defective tooth model is a three-dimensional model with the same defective teeth and adjacent teeth. k is a positive integer. A corresponding first sample defective tooth depth map is determined based on the occlusal surface of the first sample defective tooth model for each first sample defective tooth model; a corresponding second sample defective tooth depth map is determined based on the occlusal surface of the second sample defective tooth model for each second sample defective tooth model; a corresponding first sample restored tooth depth map is determined based on the occlusal surface of the first sample restored tooth model for each first sample restored tooth model; and a corresponding second restored tooth depth map is determined based on the occlusal surface of the second sample restored tooth model for each second sample restored tooth model. The depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth for the same defective tooth are stitched together to obtain k sample stitched depth maps. Data augmentation processing is performed on the k sample stitched depth maps to obtain m target training images. The target training images include a first target defective tooth depth map obtained based on the first sample defective tooth depth map, a second target defective tooth depth map obtained based on the second sample defective tooth depth map, a first target restored tooth depth map obtained based on the first sample restored tooth depth map, and a second target restored tooth depth map obtained based on the second sample restored tooth depth map, where m is a positive integer and greater than k. A repair model is obtained by training a neural network algorithm based on m target training images.

2. The automatic repair model generation method as described in claim 1, characterized in that, The step of stitching together the depth maps of the first sample defective tooth, the second sample defective tooth, the first sample restored tooth, and the second sample restored tooth for the same defective tooth to obtain k sample stitched depth maps further includes: The open() function of the Image library is used to read the depth map of the first sample defective tooth, the depth map of the second sample defective tooth, the depth map of the first sample restored tooth, and the depth map of the second sample restored tooth, resulting in four corresponding depth map arrays; The concatenate function from the NumPy library is used to concatenate the four depth map arrays along axis=1 to obtain a concatenated depth map array; The fromarray() function is used to convert the depth map stitching array into an image for export, thus obtaining the sample stitched depth map.

3. The automatic repair model generation method as described in claim 1 or 2, characterized in that, The step of performing data augmentation processing on the stitched depth maps of the k samples to obtain m target training images further includes: The rotate() algorithm from the OpenCV library is used to perform data augmentation on the depth maps stitched together from the k samples, resulting in m target training images.

4. The automatic repair model generation method as described in claim 1, characterized in that, The first sample defective tooth depth map is determined based on the occlusal surface of the first sample defective tooth model for each first sample defective tooth model, and the second sample defective tooth depth map is determined based on the occlusal surface of the second sample defective tooth model for each second sample defective tooth model. Determining a corresponding first sample restoration tooth depth map based on the occlusal surface of the first sample restoration tooth model for each of the first sample restoration tooth models, and determining a corresponding second sample restoration tooth depth map based on the occlusal surface of the second sample restoration tooth model for each of the second sample restoration tooth models, further includes: OBB bounding boxes are constructed for the first sample defective tooth model, the second sample defective tooth model, the first sample restored tooth model, and the second sample restored tooth model, respectively; Based on the obb bounding box, establish an interlocking plane with a resolution of 256*256 and a pixel value of 0; Using the three-dimensional library Open3D, ray projection and nearest point lookup are performed on the occlusal planes of the first sample defective tooth model, the second sample defective tooth model, the first sample restored tooth model, and the second sample restored tooth model to obtain the ray intersection point and the shortest distance d to the nearest ray intersection point; The ray intersection points are mapped into pixels and projected onto a plane based on the shortest distance d, resulting in the first sample defective tooth depth map, the second sample defective tooth depth map, the first sample restored tooth depth map, and the second restored tooth depth map. The mapping method is as follows: Where i is the grayscale value, and h is the preset value.

5. The automatic repair model generation method as described in claim 1, characterized in that, m≥4k.

6. The automatic repair model generation method as described in claim 1, characterized in that, The step of training a neural network algorithm based on m target training images to obtain a repair model further includes: Based on the depth map of the first target defective tooth and the depth map of the first target repaired tooth, a generative adversarial network algorithm is trained to obtain a local generative network model and a local discriminative network model. The local generative network model is used to generate a local repair simulation depth map, and the local discriminative network model is used to identify whether the input image is real. Generative Adversarial Network (GAN) algorithms are trained based on the depth maps of the second target defective tooth and the second target repaired tooth to obtain a global generative network model and a global discriminative network model. The global generative network model is used to generate a global repair simulation depth map, and the global discriminative network model is used to discriminate whether the input image is real. The local generative network model, the local discriminative network model, the global generative network model, and the global discriminative network model are subjected to error optimization to obtain the repair model.

7. The automatic repair model generation method as described in claim 6, characterized in that, The local restoration simulation depth map includes restoration occlusal features trained based on the depth map of the first target defective tooth, and the global restoration simulation depth map includes global contour features trained based on the depth map of the second target defective tooth. The error optimization of the local generation network model, the local discrimination network model, the global generation network model, and the global discrimination network model further includes: Error optimization is performed on the local generative network model and the local discriminative network model using a local loss function. for: in This indicates the difference between the generated repaired occlusal features and the occlusal features in the depth map of the first target repaired tooth. It is the local adversarial loss function, λ i It is a constant, σ is the sigmoid function, and x is a constant. local This is the depth map of the first target defect tooth, y local This is the depth map of the first target tooth to be restored. E is the false local repair simulation depth map information generated by the local generative network model to confuse the local discrimination network model, and E is the cross-entropy loss function; Error optimization is performed on the global generator network model and the global discriminator network model using a global loss function. for: in This indicates the difference between the generated global contour features and the contour features in the depth map of the second target restoration tooth. It is the global adversarial loss function, λ global It is a constant, G global The false globally repaired simulated depth map information generated by the global generative network model is used to confuse the adversarial network. global This is the depth map of the second target defect tooth, y global This is the depth map of the second target tooth to be restored; Calculate the target loss function to optimize the error. for: Where λ is a constant.

8. An automatic repair method, characterized in that, The following steps are required: Obtain a model of the tooth with the defect to be repaired; The corresponding depth map of the defective tooth to be repaired is determined based on the occlusal surface of the model of the defective tooth to be repaired; Input the depth map of the defective tooth to be repaired into the repair model as described in any one of claims 1-7 to obtain an automatically repaired tooth depth map; The automatic restoration tooth depth map is converted into an automatic restoration tooth model.

9. A computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the automatic repair model generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by at least one processor to perform the steps of the automatic repair model generation method according to any one of claims 1 to 7.

Images