Determination method of tooth defect repair scheme, electronic equipment and storage medium

Abstract

The invention provides a determination method of a tooth defect repair scheme, electronic equipment and a storage medium. The determination method comprises the following steps: acquiring a first image and a second image of a target tooth; inputting the first image and the second image into a preset restoration scheme prediction model to obtain multiple groups of prediction results output by the restoration scheme prediction model; each group of prediction results comprises a plurality of repair schemes and corresponding confidence coefficients, and the confidence coefficients corresponding to the repair schemes are correct probabilities of the repair schemes; determining the average confidence of the same type of repair schemes from the multiple groups of prediction results; and taking the repair scheme with the highest average confidence as a target repair scheme. By adopting the scheme provided by the invention, the accuracy of determining the tooth defect repair scheme can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for determining a tooth defect restoration plan, an electronic device, and a storage medium. Background Art

[0002] Since there are many causes of tooth defects, such as caries, wear, trauma, acid erosion and developmental deformities, and the conditions of the surrounding tissues such as occlusal distance vary, when formulating a tooth defect repair plan, dentists usually need to accurately analyze the state of the tooth defect and the surrounding tissues and make comprehensive decisions in order to formulate a suitable repair plan. However, due to the limitations of the dentist's knowledge reserves and personal clinical experience, when the first doctor's level is poor, it is easy to have low accuracy in the analysis of the tooth defect state and the surrounding tissue conditions and make comprehensive decision errors, resulting in the formulation of a poor repair plan, thereby reducing the accuracy of the repair plan. Summary of the invention

[0003] The embodiment of the present application provides a method for determining a tooth defect repair solution, an electronic device, and a storage medium to solve the technical problems existing in the related art. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for determining a tooth defect restoration plan, comprising: acquiring a first image and a second image of a target tooth; the first image includes information on the occlusal distance distribution between an undefective area of ​​the target tooth and a relative area of ​​its opposing tooth and a two-dimensional defect morphology of a defective area of ​​the target tooth, and the second image includes a three-dimensional defect morphology of a defective area of ​​the target tooth; inputting both the first image and the second image into a preset restoration plan prediction model to obtain multiple groups of prediction results output by the restoration plan prediction model; each group of prediction results includes multiple restoration plans and their corresponding confidence levels; the confidence level corresponding to the restoration plan is the probability that the restoration plan is correct; determining the average confidence level of restoration plans of the same type from multiple groups of prediction results; and taking the restoration plan with the highest average confidence level as the target restoration plan.

[0005] In one embodiment, the repair solution prediction model includes multiple different sub-models, the image samples of the validation sets for training each sub-model are different, and the image samples of the training sets for training each sub-model are partially the same; the first image and the second image are both input into the preset repair solution prediction model to obtain multiple groups of prediction results output by the repair solution prediction model, including: inputting the first image and the second image into each sub-model, so that each sub-model outputs a group of prediction results respectively.

[0006] In one embodiment, each sub-model includes: a feature extraction layer, which is used to extract a first target feature from the first image and a second target feature from the second image; the first target feature includes an occlusal distance feature between an undamaged area of ​​a target tooth and a relative area of ​​its opposing tooth and a two-dimensional defect morphology feature of a defective area of ​​the target tooth, and the second target feature includes a three-dimensional defect morphology feature of a defective area of ​​the target tooth; a feature fusion layer, which is used to cross-attentionally fuse the first target feature and the second target feature to obtain a fused multimodal feature; and a classifier, which is used to identify the prediction results of the corresponding group based on the multimodal features.

[0007] In one embodiment, the feature dimension of the second target feature is different from the feature dimension of the first target feature; before cross-attention fusion of the first target feature and the second target feature, the feature fusion layer is also used to: convert the feature dimension of the second target feature into the same feature dimension as the feature dimension of the first target feature.

[0008] In one embodiment, the feature extraction layer uses a feature extraction module in a ViT model to extract a first target feature from a first image; the ViT model is trained based on multiple first sample images and annotated data of each first sample image, and the first sample image includes occlusal distance distribution information between a non-defective area of ​​the affected tooth and a relative area of ​​its opposing tooth and a two-dimensional defect morphology of the defective area of ​​the affected tooth; the second image is a three-dimensional point cloud image, and the feature extraction layer uses a point cloud feature extraction module of a PointMLP model to extract a second target feature from the second image; the PointMLP model is trained based on multiple second image samples and annotated data of each second image sample, and the second image sample includes a three-dimensional defect morphology of a non-defective area of ​​the affected tooth.

[0009] In one embodiment, the determination method also includes: obtaining m image sample pairs of affected teeth; the image sample pairs include a first image sample and a second image sample, the first image sample includes the occlusal distance distribution information between the non-defective area of ​​the affected tooth and the relative area of ​​its opposing tooth and the two-dimensional defect morphology of the defective area of ​​the affected tooth, and the second image sample includes the three-dimensional defect morphology of the defective area of ​​the affected tooth, and the annotation data of each image sample pair is the restoration plan for the corresponding affected tooth; dividing the m image sample pairs into k data sets; each data set includes multiple image sample pairs and their corresponding annotation data, m and k are both integers greater than 1, and m is an integer multiple of k, and the integer multiple is an integer greater than 1; using the i-th data set as the verification set of the i-th sub-model to be trained, and using the remaining k-1 data sets as the training set of the i-th sub-model to be trained, training and verifying the i-th sub-model to be trained to obtain the i-th sub-model; i is an integer greater than 1 and less than or equal to k.

[0010] In one embodiment, the first target feature and the second target feature are both represented by feature vectors, and the feature fusion layer cross-attentionally fuses the first target feature and the second target feature to obtain a fused multimodal feature, including: respectively calculating the query vector, key vector and value vector of the first target feature in the h-th attention head, and the query vector, key vector and value vector of the second target feature; 1≤h≤H, h and H are both integers, and H is the total number of multiple attention heads; according to the query vector of the first target feature and the key vector of the second target feature in the h-th attention head, calculating the first attention weight of the first target feature to the second target feature, and according to the query vector of the second target feature and the key vector of the first target feature vector, calculate the second attention weight of the second target feature for the first target feature; perform weighted summation of the first attention weight in the h-th attention head and the value vector of the second target feature to obtain the first fused feature in the h-th attention head, and perform weighted summation of the second attention weight in the h-th attention head and the value vector of the first target feature to obtain the second fused feature in the h-th attention head; aggregate the first fused features in multiple attention heads to obtain the first target fused feature, and aggregate the second fused features in multiple attention heads to obtain the second target fused feature; fuse the first target feature, the second target feature, the first target fused feature and the second target fused feature to obtain a multimodal feature.

[0011] In one embodiment, the number of rounds of training for the i-th sub-model is multiple rounds; before each round of training begins, obtaining the first image sample in the image sample pair of m diseased teeth includes: obtaining m original image samples of the diseased teeth; performing at least one of rotation transformation, folding transformation and adding random noise on the m original image samples, respectively, to obtain m first image samples.

[0012] In a second aspect, an embodiment of the present application provides an electronic device, characterized in that it includes: a processor and a memory, the memory stores instructions, and the instructions are loaded and executed by the processor to implement the determination method of any of the above-mentioned implementation methods.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the determination method of any of the above-mentioned embodiments is implemented.

[0014] The advantages or beneficial effects of the above technical solution include at least: by inputting the first image and the second image of the target tooth into a preset restoration scheme prediction model, the restoration scheme prediction model can be used to combine the occlusal distance distribution information between the non-defective area of ​​the target tooth and the relative area of ​​its opposing tooth in the first image, the two-dimensional defect morphology of the defective area of ​​the target tooth, and the three-dimensional defect morphology of the defective area of ​​the target tooth in the second image, to determine multiple groups of prediction results for defect repair of the target tooth, wherein each group of prediction results includes multiple restoration schemes and their corresponding confidence levels, thereby avoiding the problem of low accuracy caused by the limitation of the level of the first-visit doctor in the prior art, resulting in the analysis of the occlusal distance between the target tooth and its opposing tooth and the defect morphology of the target tooth; furthermore, by determining the average confidence of the same type of restoration schemes from multiple groups of prediction results, and taking the restoration scheme with the highest average confidence as the target restoration scheme, it is also possible to avoid the mistakes caused by the comprehensive decision-making of the restoration scheme in the prior art, which is conducive to determining the best restoration scheme and improving the accuracy of the restoration scheme determination.

[0015] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0017] Figure 1 A flow chart showing a method for determining a tooth defect restoration plan according to an embodiment of the present application.

[0018] Figure 2A Shown Figure 1 A processing flow diagram of step S130 and step S140 in FIG.

[0019] Figure 2B Shown Figure 2A Schematic diagram of the processing flow of a single sub-model in .

[0020] Figure 3 Training shown Figure 2A A flow chart of a single sub-model in .

[0021] Figure 4 Shown is a schematic diagram of a process for obtaining a first sample image.

[0022] Figure 5 Shown is a schematic diagram of a process for obtaining a second image.

[0023] Figure 6 Shown is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0025] Figure 1 A flow chart showing a method for determining a tooth defect restoration plan according to an embodiment of the present application.

[0026] like Figure 1 As shown, the method for determining a tooth defect restoration plan includes the following steps S110 to S140.

[0027] Step S110, obtaining a first image and a second image of the target tooth; the first image includes the occlusal distance distribution information between the non-defective area of ​​the target tooth and the relative area of ​​its opposing tooth and the two-dimensional defect morphology of the defective area of ​​the target tooth, and the second image includes the three-dimensional defect morphology of the defective area of ​​the target tooth.

[0028] For example, Figure 2B As shown, in the first image, the occlusal distance between the undefective area 11 of the target tooth 10 and the corresponding area of ​​its opposing tooth can be represented by different colors. For example, red indicates a close occlusal distance, green indicates a long occlusal distance, and yellow indicates a moderate occlusal distance. In this way, the distribution of different colors of the undefective area 11 in the first image can represent the distribution information of the occlusal distance between the target tooth 10 and its opposing tooth. In the first image, the two-dimensional defect morphology of the defective area 12 of the target tooth 10 can be the two-dimensional defect contour of the defective area 12 of the target tooth 10. In the second image, the three-dimensional defect morphology of the defective area 12 of the target tooth 10 can be the three-dimensional defect contour of the defective area 12 of the target tooth 10.

[0029] Preferably, both the first image and the second image are images taken toward the occlusal surface of the target tooth 10, and the shooting angles of the first image and the second image can also be selected and adjusted according to actual needs, which is not limited in this embodiment of the present application.

[0030] Step S120, input both the first image and the second image into a preset restoration scheme prediction model to obtain multiple groups of prediction results output by the restoration scheme prediction model; each group of prediction results includes multiple restoration schemes and their corresponding confidence levels; the confidence level corresponding to the restoration scheme is the correct probability of the restoration scheme.

[0031] For example, Figure 2A As shown, the restoration scheme prediction model can output the first group of prediction results, the second group of prediction results, ..., the kth group of prediction results, where k is an integer greater than 1. Each group of prediction results includes five restoration schemes, namely, resin filling restoration scheme, inlay restoration scheme, high onlay restoration scheme, full crown restoration scheme and post and core restoration scheme, and their corresponding confidence levels.

[0032] Step S130: Determine the average confidence of the same type of repair solutions from multiple groups of prediction results.

[0033] For example, step S130 includes: screening out the same type of repair solutions from multiple groups of prediction results; and calculating the average confidence of the same type of repair solutions.

[0034] Step S140: taking the repair solution with the highest average confidence as the target repair solution.

[0035] Exemplarily, it is taken as an example that each group of prediction results includes five restoration schemes, namely resin filling restoration scheme, inlay restoration scheme, high onlay restoration scheme, full crown restoration scheme and post and core restoration scheme, and the confidence level corresponding to each restoration scheme, and the restoration scheme prediction model can output 10 groups of prediction results.

[0036] In step S120, the first group of prediction results output by the restoration scheme prediction model are that the confidence of the resin filling restoration scheme is a1, the confidence of the inlay restoration scheme is b1, the confidence of the onlay restoration scheme is c1, the confidence of the full crown restoration scheme is d1, and the confidence of the post and core restoration scheme is e1; the second group of prediction results output by the restoration scheme prediction model are that the confidence of the resin filling restoration scheme is a2, the confidence of the inlay restoration scheme is b2, the confidence of the onlay restoration scheme is c2, the confidence of the full crown restoration scheme is d2, and the confidence of the post and core restoration scheme is e2; ..., and so on, the tenth group of prediction results output by the restoration scheme prediction model are that the confidence of the resin filling restoration scheme is a10, the confidence of the inlay restoration scheme is b10, the confidence of the onlay restoration scheme is c10, the confidence of the full crown restoration scheme is d10, and the confidence of the post and core restoration scheme is e10.

[0037] In step S130, it can be determined that the average confidence of the resin filling restoration scheme in multiple groups of prediction results is (a1+a2+…+a10) / 10, the average confidence of the inlay restoration scheme is (b1+b2+…+b10) / 10, the average confidence of the high onlay restoration scheme is (c1+c2+…+c10) / 10, the average confidence of the full crown restoration scheme is (d1+d2+…+d10) / 10, and the average confidence of the post and core restoration scheme is (e1+e2+…+e10) / 10.

[0038] In step S140, the determined repair solution with the highest average confidence is used as the target repair solution.

[0039] The above scheme, by inputting the first image and the second image of the target tooth into a preset restoration scheme prediction model, can use the restoration scheme prediction model to combine the occlusal distance distribution information between the non-defective area of ​​the target tooth and the relative area of ​​its opposing tooth in the first image, the two-dimensional defect morphology of the defective area of ​​the target tooth and the three-dimensional defect morphology of the defective area of ​​the target tooth in the second image, to determine multiple groups of prediction results for defect repair for the target tooth, wherein each group of prediction results includes multiple restoration schemes and their corresponding confidence levels, thereby avoiding the problem of low accuracy caused by the limitation of the level of the first-visit physician in the prior art, resulting in the analysis of the occlusal distance between the target tooth and its opposing tooth and the defect morphology of the target tooth; furthermore, by determining the average confidence of restoration schemes of the same category from multiple groups of prediction results, and taking the restoration scheme with the highest average confidence as the target restoration scheme, it is also possible to avoid the mistakes caused by the prior art in making comprehensive decisions on restoration schemes, which is conducive to determining the best restoration scheme and improving the accuracy of restoration scheme determination. Furthermore, when the level of the first-visit physician is poor, the solution of the present application can assist the first-visit physician in making more accurate treatment decisions, improve clinical prognosis, and enhance medical quality.

[0040] In one embodiment, Figure 2A As shown, the restoration solution prediction model includes a plurality of different sub-models, such as the 1st to the kth sub-models; the image samples of the validation sets for training the various sub-models are different, and the image samples of the training sets for training the various sub-models are partially the same, so that the sub-models finally obtained by training are different. The first image and the second image are both input into the preset restoration solution prediction model to obtain multiple groups of prediction results output by the restoration solution prediction model, including: inputting the first image and the second image into each sub-model, so that each sub-model outputs a group of prediction results respectively. For example, if the first image and the second image are both input into the 1st to the kth sub-models, the 1st sub-model can output the 1st group of prediction results, the 2nd sub-model can output the 2nd group of prediction results, and so on, the kth sub-model outputs the kth group of prediction results.

[0041] The above scheme, by inputting both the first image and the second image into each sub-model, and using each sub-model to combine the first image and the second image to determine a group of prediction results, can enable different sub-models to output their own corresponding groups of prediction results. This is conducive to determining an accurate average confidence when combining multiple groups of prediction results to determine the average confidence of the same type of repair solutions.

[0042] In one embodiment, Figure 2B As shown, each sub-model includes a feature extraction layer, a feature fusion layer and a classifier. The feature extraction layer is used to extract the first target feature from the first image and the second target feature from the second image; the first target feature includes the occlusal distance feature between the undamaged area of ​​the target tooth and the relative area of ​​its opposing tooth and the two-dimensional defect morphology feature of the defective area of ​​the target tooth, and the second target feature includes the three-dimensional defect morphology feature of the defective area of ​​the target tooth. The feature fusion layer is used to cross-attentionally fuse the first target feature and the second target feature to obtain a fused multimodal feature. The classifier is used to identify the prediction results of the corresponding group based on the multimodal features.

[0043] In the related art, the fusion between two image features is usually performed by cumulative fusion or splicing fusion. The inventors also conducted the following comparative experiments in the process of implementing the present application:

[0044] The first target feature with a feature dimension of 1×768 and the second target feature with a feature dimension of 1×768 are fused by cumulative fusion. The fusion result is a multimodal feature with a feature dimension of 1×768. The fusion process will cause a lot of information loss, which is easy to reduce the recognition accuracy of the classifier.

[0045] The first target feature with a feature dimension of 1×768 and the second target feature with a feature dimension of 1×768 are fused by splicing fusion to obtain multimodal features. However, this fusion process lacks the interactive information between the two first target features and the second target features, which makes the multimodal features unable to accurately represent the first target features and the second target features, and it is also easy to reduce the recognition accuracy of the classifier.

[0046] In the above scheme, after the first target feature and the second target feature are extracted by the feature extraction layer, the first target feature and the second target feature are cross-attention fused through the feature fusion layer, so that the fused multimodal features can accurately represent the first target feature, the second target feature and the interaction information between them, and then the classifier can more accurately identify the prediction results of the corresponding group according to the multimodal features.

[0047] In one embodiment, Figure 2BAs shown, the feature dimension of the second target feature is different from the feature dimension of the first target feature; before the first target feature and the second target feature are cross-attention fused, the feature fusion layer is also used to: convert the feature dimension of the second target feature into the same feature dimension as the feature dimension of the first target feature.

[0048] For example, the feature dimension of the first target feature may be 1×1024, and the feature dimension of the second target feature may be 1×768. The feature dimension of the second target may be converted to the same feature dimension as the feature dimension of the first target feature by the following formula (1):

[0049] X g =XW+b formula (1);

[0050] Among them, X g represents the second target feature after conversion, and its feature dimension may be 1×1024; X represents the second target feature before conversion, and its feature dimension may be 1×768; W represents the trainable weight matrix, and its feature dimension may be 768×1024; b represents the bias matrix, which is used to adjust the linear transformation of the output, and its feature dimension may be 1×1024.

[0051] Based on this, the feature dimension of the converted second target feature can be made the same as that of the first target feature, meeting the input requirements for subsequent cross-attention fusion and ensuring the smooth progress of cross-attention fusion.

[0052] In one embodiment, Figure 2B As shown, the feature extraction layer extracts the first target feature from the first image, including: the feature extraction layer uses the feature extraction module of the ViT model to extract the first target feature from the first image; the feature extraction module of the ViT model is trained based on multiple first image samples and annotation data of each first image sample, the first image sample includes the occlusal distance distribution information between the undefective area of ​​the affected tooth and the relative area of ​​its opposing tooth and the two-dimensional defect morphology of the undefective area of ​​the affected tooth, and the annotation data of the first image sample includes the occlusal distance characteristics between the undefective area of ​​the affected tooth and the relative area of ​​the opposing tooth and the two-dimensional defect morphology characteristics of the defective area of ​​the affected tooth.

[0053] The feature extraction layer extracts the second target feature from the second image, including: the feature extraction layer uses the point cloud feature extraction module of the PointMLP model to extract the second target feature from the second image; the PointMLP model is trained based on multiple second image samples and the annotation data of each second image sample, the second image sample includes the three-dimensional defect morphology of the defect area of ​​the diseased tooth, and the annotation data of the second image sample includes the three-dimensional defect morphology characteristics of the non-defective area of ​​the diseased tooth.

[0054] In practical applications, the ViT (Visual Transformer) model is an image classification model pre-trained using a large dataset such as the ImageNet-21k dataset in the prior art, and the PointMLP model is a 3D point cloud processing model pre-trained using a large dataset such as the modelnet40 dataset in the prior art. Since the ImageNet-21k dataset includes 14 million 2D images and the modelnet40 dataset includes 127,915 3D model data, both datasets are large enough, so that in the process of pre-training the ViT model and the PointMLP model, the feature extraction module in the ViT model and the feature extraction module in the PointMLP model can learn general feature extraction capabilities.

[0055] In the present application, a small amount of first image samples and the annotated data of each first image sample are continuously used to train the feature extraction module in the ViT model. For example, 3,000 first sample images and their annotated data are used to train the feature extraction module in the ViT model, so that the feature extraction module in the ViT model can converge quickly and have the ability to extract specific image features, such as the ability to extract the occlusal distance feature between the defect area of ​​the affected tooth and the relative area of ​​the opposing tooth and the two-dimensional defect morphology feature of the defect area of ​​the affected tooth, that is, the ability to extract the first target feature. Similarly, a small amount of second image samples and the annotated data of each second image sample are continuously used to train the feature extraction module in the PointMLP model, so that the feature extraction module in the PointMLP model can converge quickly and have the ability to extract specific three-dimensional point cloud features, such as extracting the three-dimensional defect morphology feature of the defect area of ​​the affected tooth, that is, the ability to extract the second target feature. Furthermore, the feature extraction module in the trained ViT model and the feature extraction module in the trained PointMLP model are migrated to the feature extraction layer, so that the feature extraction layer can extract the first target feature from the first image and the second target feature from the second image.

[0056] In one embodiment, the determination method further includes the following steps S210 to S230.

[0057] Step S210, obtaining image sample pairs of m affected teeth; the image sample pairs include a first image sample and a second image sample, the first image sample includes the occlusal distance distribution information between the non-defective area of ​​the affected tooth and the relative area of ​​its opposing tooth and the two-dimensional defect morphology of the defective area of ​​the affected tooth, the second image sample includes the three-dimensional defect morphology of the defective area of ​​the affected tooth, and the annotated data of each image sample pair is the restoration plan of the corresponding affected tooth.

[0058] Step S220, divide the m image sample pairs into k data sets; each data set includes multiple image sample pairs and their corresponding annotation data; m and k are both integers greater than 1, and m is an integer multiple of k, and the integer multiple is an integer greater than 1. For example, m = 3000, k = 10, then each data set includes 30 image sample pairs and their corresponding annotation data.

[0059] Step S230: use the i-th data set as the validation set of the i-th sub-model to be trained, and use the remaining k-1 data sets as the training sets of the i-th sub-model to be trained, train and verify the i-th sub-model to be trained, and obtain the i-th sub-model; i is an integer greater than 1 and less than or equal to k.

[0060] For example, Figure 3 As shown, the example of 10 sub-models to be trained is used for explanation. After m image sample pairs are divided into 10 data sets, the first data set is used as the validation set of the first model to be trained, and the second to tenth data sets are used as the training sets of the first sub-model to be trained, and the first sub-model to be trained is trained and verified to obtain the first sub-model; the second data set is used as the validation set of the second model to be trained, and the first data set and the third to tenth data sets are used as the training sets of the second sub-model to be trained, and the second sub-model to be trained is trained and verified to obtain the second sub-model; and so on, the tenth data set is used as the validation set of the tenth model to be trained, and the first to ninth data sets are used as the training sets of the tenth sub-model to be trained, and the tenth sub-model to be trained is trained to obtain the tenth sub-model. Based on this, 10 sub-models can be obtained by training using a 10-fold cross-validation strategy.

[0061] The above scheme divides m image sample pairs into k data sets, uses the ith data set as the verification set of the ith sub-model to be trained, and uses the remaining k-1 data sets as the training sets of the ith sub-model to be trained, trains and verifies the ith sub-model to be trained to obtain the ith sub-model, so that in the process of training the ith sub-model to be trained, each image sample pair participates in the training or verification, which can improve the efficiency of using the image sample pairs and ensure that the verification sets in each sub-model are different and the training sets are partially the same, so that each sub-model has differences; furthermore, in the process of training all the sub-models to be trained, each image sample pair has the opportunity to be used as a verification set, and this multiple verification method can help reduce the randomness and generalization error of the performance estimation of the repair solution prediction model integrated for all sub-models, and make its performance estimation more stable and reliable.

[0062] It should be noted that the specific process of training and verifying the i-th sub-model to be trained is a conventional technical means in this field and will not be described in detail here.

[0063] In one embodiment, the first target feature and the second target feature are both represented by feature vectors, and the feature fusion layer cross-attentionally fuses the first target feature and the second target feature to obtain a fused multimodal feature, including the following steps S121 to S125.

[0064] Step S121: Calculate the first target feature X in the hth attention head respectively. v The query vector Key Vector Sum value vector and the second target feature X g The query vector Key Vector Sum value vector 1≤h≤H, h and H are both integers, and H is the total number of attention heads.

[0065] Exemplarily, each vector in step S121 can be calculated using the following formulas (2) to (7):

[0066]

[0067] in, Indicates that the first target feature X in the hth attention head v and the second target feature X g Linear transformation matrices for performing different linear transformations. For example, is the first target feature X v Linear transformation to query vector The linear transformation matrix of is the first target feature X v Linear transformation into key-value vector The linear transformation matrix of is the first target feature X v Linear transformation to value vector The linear transformation matrix of Also the second target feature X g The linear transformation is the linear transformation matrix of the corresponding vector.

[0068] Step S122: According to the first target feature X in the hth attention head v The query vector and the second target feature X g The key vector Calculate the first target feature X v For the second target feature Xg The first attention weight And according to the second target feature X g The query vector and the first target feature X v The key vector Calculate the second target feature X g For the first target feature X v The second attention weight

[0069] Exemplarily, the first attention weight and the second attention weight It can be calculated using the following formula (8) and formula (9):

[0070]

[0071] Among them, d h represents the feature dimension of the h-th attention head. Softmax() is a normalized exponential function, which is used to normalize each row of the matrix so that the sum of each row of the matrix is ​​1.

[0072] Step S123: Set the first attention weight in the hth attention head With the second target feature X g The value vector of Perform weighted summation to obtain the first fusion feature in the hth attention head And the second attention weight in the hth attention head With the first target feature X v The value vector of Perform weighted summation to obtain the second fusion feature in the hth attention head

[0073] For example, the first fused feature in the hth attention head and the second fusion feature in the h-th attention head The following formulas (10) and (11) can be used for calculation:

[0074]

[0075] in, Represents element-wise addition.

[0076] Step S124: Aggregate the first fusion features in multiple attention heads to obtain the first target fusion feature Z v , and aggregate the second fusion features in multiple attention heads to obtain the second target fusion feature Z g .

[0077] For example, the first target fusion feature Z v And the second target fusion feature Z g It can be calculated using the following formula (12) and formula (13):

[0078] Z v =Norm(Z v_fusion W v +Z v_fusion ) formula (12);

[0079] Z g =Norm(Z v_fusion W g +Z v_fusion ) formula (13);

[0080] Among them, Norm() represents the layer normalization operation, W v Z v_fusion The corresponding trainable weight matrix, W g Z v_fusion The corresponding trainable weight matrix.

[0081] Step S125: The first target feature X v , the second target feature X g , the first target fusion feature Z v and the second target fusion feature Z g After fusion, we get the multimodal feature Z.

[0082] For example, the multimodal feature can be calculated using the following formula (14):

[0083] Z = Concatenation(X v ,X g ,Z v ,Z g ) formula (14);

[0084] Among them, Concatenation() represents the matrix concatenation operation.

[0085] Based on this, the first target fusion feature and the second target fusion feature can be determined with the help of the attention mechanism to characterize the interaction information between the first target feature and the second target feature, and the first target feature, the second target feature, the first target fusion feature and the second target fusion feature can be fused into a multimodal feature. The multimodal feature can retain the original modal features composed of the first target feature and the second target feature, and include the interaction information between the first target feature and the second target feature, which is conducive to more accurately characterizing the characteristics of the target tooth through multimodal features, thereby improving the recognition accuracy of subsequent classifiers.

[0086] In one embodiment, the number of rounds of training for the i-th sub-model is multiple rounds; before each round of training begins, obtaining the first image sample of the m pairs of image samples of the affected teeth includes: Figure 4 As shown, m original image samples of the affected teeth are obtained; at least one of rotation transformation, folding transformation and adding random noise is performed on the m original image samples respectively to obtain m first image samples. The original image samples can be two-dimensional images obtained by scanning the affected teeth with an intraoral scanner, for example, Figure 4 Two-dimensional occlusal space color image shown in .

[0087] In the process of transforming the original sample image into the first sample image, the rotation angle, fold line position and type of random noise of the original sample image can be selected and adjusted according to actual needs, and the embodiment of the present application does not limit this. Preferably, the random noise can be Gaussian white noise.

[0088] In the above scheme, after obtaining m original image samples of diseased teeth, the m original images are rotated to increase the perspective diversity of the first image sample, so that the i-th sub-model can recognize the rotated target tooth; or, the m original images are folded to increase the diversity of the first image sample without affecting the semantics of the original image sample to prevent overfitting; or, random noise is added to the m original images to train the i-th sub-model in a noisy environment and improve the performance of the sub-model in different environments. Based on this, the generalization ability of each sub-model can be improved.

[0089] In one implementation, please refer to Figure 5 The second image is a three-dimensional point cloud image, and obtaining the second image of the target tooth includes the following steps S310 to S330.

[0090] Step S310: Acquire a triangular mesh image of the target tooth. For example, the triangular mesh image of the target tooth may be obtained by scanning the target tooth with a three-dimensional laser scanner, and the file format may be 3D STL (Stereolithography).

[0091] Step S320: extracting the vertex coordinates of each triangle from the triangular mesh image to form a three-dimensional point cloud image of the target tooth.

[0092] Step S330: downsample the three-dimensional point cloud image using the farthest point sampling method to obtain a second image. The farthest point sampling method (FPS for short) is a conventional sampling method in the art and will not be described in detail here.

[0093] In the above scheme, since the vertex coordinates of each triangle are extracted from the triangular mesh image to form a three-dimensional point cloud image of the target tooth, the data volume of the three-dimensional point cloud image of the target tooth is large. Therefore, by adopting the farthest point sampling method to continue downsampling the three-dimensional point cloud image, not only can the data volume of the three-dimensional point cloud image of the target tooth be reduced, for example, to 3096 three-dimensional point cloud data to improve the image processing speed, but also it can ensure that the second image obtained is not distorted, which is more conducive to the smooth progress of the subsequent image processing process.

[0094] Figure 6 The figure shows a structural block diagram of an electronic device according to an embodiment of the present application. Figure 6 As shown, the electronic device 30 includes: a memory 31 and a processor 32. The memory 31 stores a computer program that can be run on the processor 32. When the processor 32 executes the computer program, the method for determining the tooth defect restoration plan in the above embodiment is implemented. The number of the memory 31 and the processor 32 can be one or more.

[0095] The electronic device 30 also includes: a communication interface 33, which is used to communicate with external devices and perform data exchange transmission. If the memory 31, the processor 32 and the communication interface 33 are implemented independently, the memory 31, the processor 32 and the communication interface 33 can be interconnected through a bus and complete mutual communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0096] Optionally, in a specific implementation, if the memory 31, the processor 32 and the communication interface 33 are integrated on a chip, the memory 31, the processor 32 and the communication interface 33 can communicate with each other through an internal interface.

[0097] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0098] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0099] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0100] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0102] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0103] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0104] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0105] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0106] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0107] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining a tooth defect repair plan, characterized in that: include: Acquire a first image and a second image of a target tooth; the first image includes occlusal distance distribution information between an undamaged area of ​​the target tooth and a relative area of ​​its opposing tooth and a two-dimensional defect morphology of a defective area of ​​the target tooth, and the second image includes a three-dimensional defect morphology of the defective area of ​​the target tooth; Inputting both the first image and the second image into a preset restoration solution prediction model, obtaining multiple groups of prediction results output by the restoration solution prediction model; each group of prediction results includes multiple restoration solutions and their corresponding confidence levels; the confidence level corresponding to the restoration solution is the correct probability of the restoration solution; Determining an average confidence level of the same type of the repair solutions from multiple groups of the prediction results; The repair solution with the highest average confidence is taken as the target repair solution.

2. The determination method according to claim 1, characterized in that: The restoration solution prediction model includes a plurality of different sub-models, the image samples of the validation sets for training the sub-models are different, and the image samples of the training sets for training the sub-models are partially the same; Inputting both the first image and the second image into a preset restoration solution prediction model to obtain multiple groups of prediction results output by the restoration solution prediction model, including: inputting both the first image and the second image into each of the sub-models, so that each of the sub-models outputs a group of the prediction results respectively.

3. The determination method according to claim 2, characterized in that: Each of the sub-models comprises: A feature extraction layer, used to extract a first target feature from the first image, and to extract a second target feature from the second image; the first target feature includes an occlusal distance feature between an undamaged area of ​​the target tooth and an area opposite to the opposing tooth and a two-dimensional defect morphology feature of a defective area of ​​the target tooth, and the second target feature includes a three-dimensional defect morphology feature of the defective area of ​​the target tooth; A feature fusion layer, used for cross-attentionally fusing the first target feature and the second target feature to obtain a fused multimodal feature; A classifier is used to identify the prediction results of the corresponding group according to the multimodal features.

4. The determination method according to claim 3, characterized in that: The feature dimension of the second target feature is different from the feature dimension of the first target feature; Before cross-attention fusing the first target feature and the second target feature, the feature fusion layer is further used to: convert the feature dimension of the second target feature into the same feature dimension as the feature dimension of the first target feature.

5. The determination method according to claim 3, characterized in that: The feature extraction layer uses a feature extraction module in a ViT model to extract the first target feature from the first image; the ViT model is trained based on a plurality of first sample images and annotated data of each of the first sample images, wherein the first sample images include occlusal distance distribution information between a non-defective area of ​​an affected tooth and a relative area of ​​an opposing tooth and a two-dimensional defect morphology of a defective area of ​​the affected tooth; The second image is a three-dimensional point cloud image, and the feature extraction layer uses the point cloud feature extraction module of the PointMLP model to extract the second target feature from the second image; the PointMLP model is trained based on multiple second image samples and the annotation data of each second image sample, and the second image sample includes the three-dimensional defect morphology of the non-defective area of ​​the diseased tooth.

6. The determination method according to claim 2, characterized in that: Also includes: Acquire m pairs of image samples of the affected teeth; the image sample pairs include a first image sample and a second image sample, the first image sample includes the occlusal distance distribution information between the non-defective area of ​​the affected tooth and the relative area of ​​its opposing tooth and the two-dimensional defect morphology of the defective area of ​​the affected tooth, the second image sample includes the three-dimensional defect morphology of the defective area of ​​the affected tooth, and the annotated data of each of the image sample pairs is a restoration plan for the corresponding affected tooth; Divide the m image sample pairs into k data sets; each data set includes a plurality of image sample pairs and their corresponding annotation data, m and k are both integers greater than 1, and m is an integer multiple of k, and the integer multiple is an integer greater than 1; The i-th data set is used as the validation set of the i-th sub-model to be trained, and the remaining k-1 data sets are used as the training sets of the i-th sub-model to be trained, and the i-th sub-model to be trained is trained and verified to obtain the i-th sub-model; i is an integer greater than 1 and less than or equal to k.

7. The determination method according to claim 3, characterized in that: The first target feature and the second target feature are both represented by feature vectors, and the feature fusion layer cross-attentionally fuses the first target feature and the second target feature to obtain a fused multimodal feature, including: Calculate the query vector, key vector, and value vector of the first target feature, and the query vector, key vector, and value vector of the second target feature in the h-th attention head respectively; 1≤h≤H, h and H are both integers, and H is the total number of multiple attention heads; Calculating a first attention weight of the first target feature for the second target feature according to the query vector of the first target feature and the key vector of the second target feature in the h-th attention head, and calculating a second attention weight of the second target feature for the first target feature according to the query vector of the second target feature and the key vector of the first target feature; Performing a weighted summation on the first attention weight in the h-th attention head and the value vector of the second target feature to obtain a first fused feature in the h-th attention head, and performing a weighted summation on the second attention weight in the h-th attention head and the value vector of the first target feature to obtain a second fused feature in the h-th attention head; Aggregating the first fused features in multiple attention heads to obtain a first target fused feature, and aggregating the second fused features in multiple attention heads to obtain a second target fused feature; The first target feature, the second target feature, the first target fusion feature and the second target fusion feature are fused to obtain the multimodal feature.

8. The determination method according to claim 6, characterized in that: The number of rounds of training for the i-th sub-model is multiple rounds; before each round of training starts, obtaining the first image sample of the m pairs of image samples of the affected tooth includes: Obtaining m original image samples of the affected teeth; Perform at least one of rotation transformation, folding transformation and adding random noise on the m original image samples respectively to obtain m first image samples.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images