Method for determining a tooth defect repair plan, electronic device, and storage medium

Abstract

This application proposes a method, electronic device, and storage medium for determining a restoration plan for a tooth defect. The method comprises: obtaining a first image and a second image of a target tooth; inputting both the first image and the second image into a preset restoration plan prediction model to obtain multiple sets of prediction results output by the restoration plan prediction model; each set of prediction results includes multiple restoration plans and their corresponding confidence levels, where the confidence level of a restoration plan represents the probability that the restoration plan is correct; determining the average confidence level of restoration plans of the same type from the multiple sets of prediction results; and selecting the restoration plan with the highest average confidence level as the target restoration plan. The method of this application can improve the accuracy of determining restoration plans for tooth defects.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for determining a tooth defect restoration plan, an electronic device, and a storage medium. Background Art

[0002] Since there are many causes of tooth defects, such as caries, wear, trauma, acid erosion and developmental deformities, and the conditions of the surrounding tooth tissues, such as occlusal distance, vary, when formulating a tooth defect restoration plan, dentists are usually required to accurately analyze the state of the tooth defect and the surrounding tissues and make comprehensive decisions in order to formulate an appropriate restoration plan. However, due to the limitations of the dentist's knowledge reserves and personal clinical experience, when the first doctor's level is poor, it is easy to make low-accuracy analyses of the tooth defect state and the surrounding tissue conditions and make comprehensive decision-making errors, resulting in the formulation of a poor restoration plan, thereby reducing the accuracy of the restoration plan. Summary of the Invention

[0003] The present invention provides a method for determining a tooth defect restoration solution, an electronic device, and a storage medium to solve the technical problems existing in related technologies. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for determining a tooth defect restoration plan, comprising: obtaining a first image and a second image of a target tooth; the first image includes information on the occlusal distance distribution between the undefected area of the target tooth and the relative area of its opposing tooth and a two-dimensional defect morphology of the defected area of the target tooth, and the second image includes a three-dimensional defect morphology of the defected area of the target tooth; the first image and the second image are both input into a preset restoration plan prediction model to obtain multiple groups of prediction results output by the restoration plan prediction model; each group of prediction results includes multiple restoration plans and their corresponding confidence levels; the confidence level corresponding to the restoration plan is the probability that the restoration plan is correct; the average confidence level of restoration plans of the same type is determined from multiple groups of prediction results; and the restoration plan with the highest average confidence level is used as the target restoration plan.

[0005] In one embodiment, the repair solution prediction model includes multiple different sub-models, the image samples of the validation sets for training each sub-model are different, and the image samples of the training sets for training each sub-model are partially the same; the first image and the second image are both input into the preset repair solution prediction model to obtain multiple sets of prediction results output by the repair solution prediction model, including: inputting the first image and the second image into each sub-model, so that each sub-model outputs a set of prediction results respectively.

[0006] In one embodiment, each sub-model includes: a feature extraction layer for extracting a first target feature from the first image and a second target feature from the second image; the first target feature includes an occlusal distance feature between the undamaged area of the target tooth and the relative area of its opposing tooth and a two-dimensional defect morphology feature of the defected area of the target tooth, and the second target feature includes a three-dimensional defect morphology feature of the defected area of the target tooth; a feature fusion layer for cross-attentionally fusing the first target feature and the second target feature to obtain a fused multimodal feature; a classifier for identifying the prediction results of the corresponding group based on the multimodal feature.

[0007] In one embodiment, the feature dimension of the second target feature is different from the feature dimension of the first target feature; before cross-attention fusing the first target feature and the second target feature, the feature fusion layer is further used to: convert the feature dimension of the second target feature into the same feature dimension as the feature dimension of the first target feature.

[0008] In one embodiment, the feature extraction layer uses the feature extraction module in the ViT model to extract the first target feature from the first image; the ViT model is trained based on multiple first sample images and the annotation data of each first sample image, and the first sample image includes the occlusal distance distribution information between the non-defective area of the affected tooth and the relative area of its opposing tooth and the two-dimensional defect morphology of the defective area of the affected tooth; the second image is a three-dimensional point cloud image, and the feature extraction layer uses the point cloud feature extraction module of the PointMLP model to extract the second target feature from the second image; the PointMLP model is trained based on multiple second image samples and the annotation data of each second image sample, and the second image sample includes the three-dimensional defect morphology of the non-defective area of the affected tooth.

[0009] In one embodiment, the determination method also includes: obtaining m image sample pairs of the affected teeth; the image sample pairs include a first image sample and a second image sample, the first image sample includes the occlusal distance distribution information between the non-defective area of the affected tooth and the relative area of its opposing tooth and the two-dimensional defect morphology of the defective area of the affected tooth, the second image sample includes the three-dimensional defect morphology of the defective area of the affected tooth, and the annotation data of each image sample pair is the restoration plan for the corresponding affected tooth; dividing the m image sample pairs into k data sets; each data set includes multiple image sample pairs and their corresponding annotation data, m and k are both integers greater than 1, and m is an integer multiple of k, and the integer multiple is an integer greater than 1; using the i-th data set as the verification set of the i-th sub-model to be trained, and using the remaining k-1 data sets as the training set of the i-th sub-model to be trained, training and verifying the i-th sub-model to be trained to obtain the i-th sub-model; i is an integer greater than 1 and less than or equal to k.

[0010] In one embodiment, the first target feature and the second target feature are both represented by feature vectors, and the feature fusion layer cross-attentionally fuses the first target feature and the second target feature to obtain a fused multimodal feature, including: respectively calculating the query vector, key vector and value vector of the first target feature in the h-th attention head, and the query vector, key vector and value vector of the second target feature; 1≤h≤H, h and H are both integers, and H is the total number of multiple attention heads; according to the query vector of the first target feature and the key vector of the second target feature in the h-th attention head, calculating the first attention weight of the first target feature to the second target feature, and according to the query vector of the second target feature and the key vector of the first target feature vector, calculate the second attention weight of the second target feature for the first target feature; perform weighted sum of the first attention weight in the hth attention head and the value vector of the second target feature to obtain the first fusion feature in the hth attention head, and perform weighted sum of the second attention weight in the hth attention head and the value vector of the first target feature to obtain the second fusion feature in the hth attention head; aggregate the first fusion features in multiple attention heads to obtain the first target fusion feature, and aggregate the second fusion features in multiple attention heads to obtain the second target fusion feature; fuse the first target feature, the second target feature, the first target fusion feature and the second target fusion feature to obtain a multimodal feature.

[0011] In one embodiment, the number of rounds of training for the i-th sub-model is multiple rounds; before each round of training begins, obtaining the first image sample of the m image sample pairs of the diseased teeth includes: obtaining the original image samples of the m diseased teeth; performing at least one of rotation transformation, folding transformation and adding random noise on the m original image samples respectively to obtain m first image samples.

[0012] In a second aspect, an embodiment of the present application provides an electronic device, characterized in that it includes: a processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the determination method of any of the above-mentioned embodiments.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the determination method of any of the above-mentioned embodiments is implemented.

[0014] The advantages or beneficial effects of the above technical solution include at least: by inputting the first image and the second image of the target tooth into a preset restoration scheme prediction model, the restoration scheme prediction model can be used to combine the occlusal distance distribution information between the non-defective area of the target tooth and the relative area of its opposing tooth in the first image, the two-dimensional defect morphology of the defective area of the target tooth, and the three-dimensional defect morphology of the defective area of the target tooth in the second image to determine multiple groups of prediction results for defect restoration of the target tooth, wherein each group of prediction results includes multiple restoration schemes and their corresponding confidence levels, avoiding the problem of low accuracy caused by the level limitation of the first-visit physician in the existing technology, resulting in the analysis of the occlusal distance between the target tooth and its opposing tooth and the defect morphology of the target tooth; furthermore, by determining the average confidence of the same type of restoration schemes from multiple groups of prediction results, and taking the restoration scheme with the highest average confidence as the target restoration scheme, it can also avoid the mistakes caused by the comprehensive decision-making of restoration schemes in the existing technology, which is conducive to determining the best restoration scheme and improving the accuracy of the restoration scheme determination.

[0015] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0017] Figure 1 A flowchart illustrating a method for determining a tooth defect restoration plan according to an embodiment of the present application is shown.

[0018] Figure 2A Shown Figure 1 A processing flow diagram of step S130 and step S140 in FIG.

[0019] Figure 2B Shown Figure 2A Schematic diagram of the processing flow of a single sub-model in .

[0020] Figure 3 Training shown Figure 2A A flow chart of a single sub-model in .

[0021] Figure 4 Shown is a schematic diagram of a process for obtaining a first sample image.

[0022] Figure 5 Shown is a schematic diagram of a process for obtaining a second image.

[0023] Figure 6 Shown is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0025] Figure 1 A flowchart illustrating a method for determining a tooth defect restoration plan according to an embodiment of the present application is shown.

[0026] like Figure 1 As shown, the method for determining a tooth defect restoration plan includes the following steps S110 to S140.

[0027] Step S110, obtaining a first image and a second image of the target tooth; the first image includes the occlusal distance distribution information between the non-defective area of the target tooth and the relative area of its opposing tooth and the two-dimensional defect morphology of the defective area of the target tooth, and the second image includes the three-dimensional defect morphology of the defective area of the target tooth.

[0028] For example, Figure 2B As shown, in the first image, the occlusal distance between the undefected area 11 of the target tooth 10 and the corresponding area of its opposing tooth can be represented by different colors. For example, red indicates a close occlusal distance, green indicates a far occlusal distance, and yellow indicates a moderate occlusal distance. In this way, the distribution of different colors of the undefected area 11 in the first image can represent the distribution information of the occlusal distance between the target tooth 10 and its opposing tooth. In the first image, the two-dimensional defect morphology of the defect area 12 of the target tooth 10 can be the two-dimensional defect contour of the defect area 12 of the target tooth 10. In the second image, the three-dimensional defect morphology of the defect area 12 of the target tooth 10 can be the three-dimensional defect contour of the defect area 12 of the target tooth 10.

[0029] Preferably, the first image and the second image are both images taken toward the occlusal surface of the target tooth 10. The shooting angles of the first image and the second image can also be selected and adjusted according to actual needs, and the embodiment of the present application does not impose any restrictions on this.

[0030] Step S120: Input the first image and the second image into a preset restoration solution prediction model to obtain multiple groups of prediction results output by the restoration solution prediction model; each group of prediction results includes multiple restoration solutions and their corresponding confidence levels; the confidence level corresponding to the restoration solution is the probability that the restoration solution is correct.

[0031] For example, Figure 2A As shown in FIG, the restoration solution prediction model can output the first group of prediction results, the second group of prediction results, ..., the kth group of prediction results, where k is an integer greater than 1. Each group of prediction results includes five restoration solutions: resin filling restoration solution, inlay restoration solution, onlay restoration solution, full crown restoration solution, and post and core restoration solution, and their corresponding confidence levels.

[0032] Step S130: Determine the average confidence level of the same type of repair solutions from multiple groups of prediction results.

[0033] For example, step S130 includes: screening out the same type of repair solutions from multiple groups of prediction results; and calculating the average confidence of the same type of repair solutions.

[0034] Step S140: The repair solution with the highest average confidence is used as the target repair solution.

[0035] For example, each set of prediction results includes five restoration schemes, namely resin filling restoration scheme, inlay restoration scheme, high onlay restoration scheme, full crown restoration scheme and post and core restoration scheme, and the corresponding confidence level of each restoration scheme, and the restoration scheme prediction model can output 10 sets of prediction results.

[0036] In step S120, the first group of prediction results output by the restoration scheme prediction model are: the confidence level of the resin filling restoration scheme is a1, the confidence level of the inlay restoration scheme is b1, the confidence level of the onlay restoration scheme is c1, the confidence level of the full crown restoration scheme is d1, and the confidence level of the post and core restoration scheme is e1; the second group of prediction results output by the restoration scheme prediction model are: the confidence level of the resin filling restoration scheme is a2, the confidence level of the inlay restoration scheme is b2, the confidence level of the onlay restoration scheme is c2, the confidence level of the full crown restoration scheme is d2, and the confidence level of the post and core restoration scheme is e2; and so on, the tenth group of prediction results output by the restoration scheme prediction model are: the confidence level of the resin filling restoration scheme is a10, the confidence level of the inlay restoration scheme is b10, the confidence level of the onlay restoration scheme is c10, the confidence level of the full crown restoration scheme is d10, and the confidence level of the post and core restoration scheme is e10.

[0037] In step S130, it can be determined that the average confidence of the resin filling restoration scheme in multiple groups of prediction results is (a1+a2+…+a10) / 10, the average confidence of the inlay restoration scheme is (b1+b2+…+b10) / 10, the average confidence of the high onlay restoration scheme is (c1+c2+…+c10) / 10, the average confidence of the full crown restoration scheme is (d1+d2+…+d10) / 10, and the average confidence of the post and core restoration scheme is (e1+e2+…+e10) / 10.

[0038] In step S140 , the determined restoration solution with the highest average confidence is used as the target restoration solution.

[0039] The above scheme, by inputting the first image and the second image of the target tooth into a preset restoration scheme prediction model, can use the restoration scheme prediction model to combine the occlusal distance distribution information between the non-defective area of the target tooth and the relative area of its opposing tooth in the first image, the two-dimensional defect morphology of the defective area of the target tooth and the three-dimensional defect morphology of the defective area of the target tooth in the second image, to determine multiple groups of prediction results for defect restoration of the target tooth, wherein each group of prediction results includes multiple restoration schemes and their corresponding confidence levels, avoiding the problem of low accuracy caused by the level limitation of the first-visit physician in the existing technology, resulting in the analysis of the occlusal distance between the target tooth and its opposing tooth and the defect morphology of the target tooth; furthermore, by determining the average confidence of the restoration schemes of the same category from multiple groups of prediction results, and taking the restoration scheme with the highest average confidence as the target restoration scheme, it can also avoid the mistakes caused by the comprehensive decision-making of the restoration scheme in the existing technology, which is conducive to determining the best restoration scheme and improving the accuracy of the restoration scheme determination. Furthermore, when the first-visit physician's skills are poor, the solution of this application can assist the first-visit physician in making more accurate treatment decisions, improving clinical prognosis and enhancing medical quality.

[0040] In one embodiment, Figure 2A As shown, the repair solution prediction model includes multiple different sub-models, such as the 1st to the kth sub-models; the image samples of the validation sets for training each sub-model are different, and the image samples of the training sets for training each sub-model are partially the same, so that the sub-models finally trained are different. The first image and the second image are both input into the preset repair solution prediction model to obtain multiple groups of prediction results output by the repair solution prediction model, including: inputting the first image and the second image into each sub-model, so that each sub-model outputs a group of prediction results respectively. For example, if the first image and the second image are both input into the 1st to the kth sub-models, the 1st sub-model can output the 1st group of prediction results, the 2nd sub-model can output the 2nd group of prediction results, and so on, the kth sub-model outputs the kth group of prediction results.

[0041] The above scheme, by inputting both the first image and the second image into each sub-model, and using each sub-model to combine the first image and the second image to determine a set of prediction results, can enable different sub-models to output their own corresponding groups of prediction results. This is conducive to determining an accurate average confidence when combining multiple groups of prediction results to determine the average confidence of the same type of repair solutions.

[0042] In one embodiment, Figure 2B As shown, each sub-model includes a feature extraction layer, a feature fusion layer and a classifier. The feature extraction layer is used to extract the first target feature from the first image and the second target feature from the second image; the first target feature includes the occlusal distance feature between the non-defective area of the target tooth and the relative area of its opposing tooth and the two-dimensional defect morphology feature of the defective area of the target tooth, and the second target feature includes the three-dimensional defect morphology feature of the defective area of the target tooth. The feature fusion layer is used to cross-attentionally fuse the first target feature and the second target feature to obtain a fused multimodal feature. The classifier is used to identify the prediction results of the corresponding group based on the multimodal features.

[0043] In related technologies, the fusion of two image features is usually performed using an accumulation fusion method or a splicing fusion method. The inventors also conducted the following comparative experiments in the process of implementing this application:

[0044] The cumulative fusion method is used to fuse the first target feature with a feature dimension of 1×768 and the second target feature with a feature dimension of 1×768. The fusion result is a multimodal feature with a feature dimension of 1×768. The fusion process will cause a lot of information loss, which is likely to reduce the recognition accuracy of the classifier.

[0045] The first target feature with a feature dimension of 1×768 and the second target feature with a feature dimension of 1×768 are fused by splicing fusion to obtain multimodal features. However, this fusion process lacks the interactive information between the two first target features and the second target features, which makes the multimodal features unable to accurately represent the first target features and the second target features, and also easily reduces the recognition accuracy of the classifier.

[0046] In the above scheme, after the first target feature and the second target feature are extracted by the feature extraction layer, the first target feature and the second target feature are cross-attention fused through the feature fusion layer, so that the fused multimodal features can accurately represent the first target feature, the second target feature and the interaction information between them, and thus the classifier can more accurately identify the prediction results of the corresponding group based on the multimodal features.

[0047] In one embodiment, Figure 2BAs shown, the feature dimension of the second target feature is different from the feature dimension of the first target feature; before the first target feature and the second target feature are cross-attention fused, the feature fusion layer is also used to: convert the feature dimension of the second target feature into the same feature dimension as the feature dimension of the first target feature.

[0048] For example, the feature dimension of the first target feature may be 1×1024, and the feature dimension of the second target feature may be 1×768. The feature dimension of the second target may be converted to the same feature dimension as the feature dimension of the first target feature by the following formula (1):

[0049] X g =XW+b formula (1);

[0050] Among them, X g represents the second target feature after conversion, and its feature dimension can be 1×1024; X represents the second target feature before conversion, and its feature dimension can be 1×768; W represents the trainable weight matrix, and its feature dimension can be 768×1024; b represents the bias matrix, which is used to adjust the linear transformation of the output, and its feature dimension can be 1×1024.

[0051] Based on this, the feature dimension of the converted second target feature can be made the same as that of the first target feature, meeting the input requirements of subsequent cross-attention fusion and ensuring the smooth progress of cross-attention fusion.

[0052] In one embodiment, Figure 2B As shown, the feature extraction layer extracts the first target feature from the first image, including: the feature extraction layer uses the feature extraction module of the ViT model to extract the first target feature from the first image; the feature extraction module of the ViT model is trained based on multiple first image samples and the annotation data of each first image sample, the first image sample includes the occlusal distance distribution information between the non-defective area of the affected tooth and the relative area of its opposing tooth and the two-dimensional defect morphology of the non-defective area of the affected tooth, and the annotation data of the first image sample includes the occlusal distance feature between the non-defective area of the affected tooth and the relative area of the opposing tooth and the two-dimensional defect morphology feature of the defective area of the affected tooth.

[0053] The feature extraction layer extracts the second target feature from the second image, including: the feature extraction layer uses the point cloud feature extraction module of the PointMLP model to extract the second target feature from the second image; the PointMLP model is trained based on multiple second image samples and the annotation data of each second image sample, the second image sample includes the three-dimensional defect morphology of the defect area of the diseased tooth, and the annotation data of the second image sample includes the three-dimensional defect morphology feature of the non-defective area of the diseased tooth.

[0054] In practical applications, the ViT (Visual Transformer) model is an image classification model pre-trained using large datasets such as the ImageNet-21k dataset. The PointMLP model is a 3D point cloud processing model pre-trained using large datasets such as the ModelNet40 dataset. Since the ImageNet-21k dataset includes 14 million 2D images and the ModelNet40 dataset includes 127,915 3D model data, both datasets are large enough. This allows the feature extraction modules in the ViT and PointMLP models to learn common feature extraction capabilities during pre-training.

[0055] In the present application, the feature extraction module in the ViT model is continuously trained using a small number of first image samples and the annotated data of each first image sample. For example, 3,000 first sample images and their annotated data are used to train the feature extraction module in the ViT model. This allows the feature extraction module in the ViT model to converge quickly and have the ability to extract specific image features, such as the ability to extract the occlusal distance feature between the defect area of the affected tooth and the relative area of the opposing tooth and the two-dimensional defect morphology feature of the defect area of the affected tooth, that is, the ability to extract the first target feature. Similarly, the feature extraction module in the PointMLP model is continuously trained using a small number of second image samples and the annotated data of each second image sample. This allows the feature extraction module in the PointMLP model to converge quickly and have the ability to extract specific three-dimensional point cloud features, such as the three-dimensional defect morphology feature of the defect area of the affected tooth, that is, the ability to extract the second target feature. Furthermore, the feature extraction module in the trained ViT model and the feature extraction module in the trained PointMLP model are migrated to the feature extraction layer, which allows the feature extraction layer to extract the first target feature from the first image and the second target feature from the second image.

[0056] In one embodiment, the determination method further includes the following steps S210 to S230.

[0057] Step S210, obtaining m pairs of image samples of the affected teeth; the image sample pairs include a first image sample and a second image sample, the first image sample includes the occlusal distance distribution information between the non-defective area of the affected tooth and the relative area of its opposing tooth and the two-dimensional defect morphology of the defective area of the affected tooth, the second image sample includes the three-dimensional defect morphology of the defective area of the affected tooth, and the annotated data of each image sample pair is the restoration plan for the corresponding affected tooth.

[0058] Step S220: Divide the m image sample pairs into k data sets; each data set includes multiple image sample pairs and their corresponding annotated data; m and k are both integers greater than 1, and m is an integer multiple of k, and the integer multiple is an integer greater than 1. For example, if m = 3000 and k = 10, each data set includes 30 image sample pairs and their corresponding annotated data.

[0059] Step S230: Use the i-th data set as the validation set of the i-th sub-model to be trained, and use the remaining k-1 data sets as the training sets of the i-th sub-model to be trained. Train and verify the i-th sub-model to be trained to obtain the i-th sub-model; i is an integer greater than 1 and less than or equal to k.

[0060] For example, Figure 3 As shown, the example of 10 sub-models to be trained is used for explanation. After m image sample pairs are divided into 10 data sets, the first data set is used as the validation set of the first model to be trained, and the second to tenth data sets are used as the training sets of the first sub-model to be trained. The first sub-model to be trained is trained and verified to obtain the first sub-model; the second data set is used as the validation set of the second model to be trained, and the first data set and the third to tenth data sets are used as the training sets of the second sub-model to be trained. The second sub-model to be trained is trained and verified to obtain the second sub-model; and so on, the tenth data set is used as the validation set of the tenth model to be trained, and the first to ninth data sets are used as the training sets of the tenth sub-model to be trained. The tenth sub-model to be trained is trained to obtain the tenth sub-model. Based on this, 10 sub-models can be obtained by training using a 10-fold cross-validation strategy.

[0061] The above scheme divides m image sample pairs into k data sets, uses the i-th data set as the verification set of the i-th sub-model to be trained, and uses the remaining k-1 data sets as the training sets of the i-th sub-model to be trained, and trains and verifies the i-th sub-model to be trained to obtain the i-th sub-model. In this scheme, in the process of training the i-th sub-model to be trained, each image sample pair participates in training or verification, which can improve the efficiency of using image sample pairs and ensure that the verification sets in each sub-model are different and the training sets are partially the same, so that each sub-model has differences; furthermore, in the process of training all sub-models to be trained, each image sample pair has the opportunity to be used as a verification set. This multiple verification method can help reduce the randomness and generalization error of the performance estimation of the repair solution prediction model integrated for all sub-models, and make its performance estimation more stable and reliable.

[0062] It should be noted that the specific process of training and verifying the i-th sub-model to be trained is a conventional technical means in this field and will not be described in detail here.

[0063] In one embodiment, the first target feature and the second target feature are both represented by feature vectors, and the feature fusion layer cross-attentionally fuses the first target feature and the second target feature to obtain a fused multimodal feature, including the following steps S121 to S125.

[0064] Step S121: Calculate the first target feature X in the h-th attention head respectively. v The query vector Key Vector Sum value vector and the second target feature X g The query vector Key Vector Sum vector 1≤h≤H, h and H are both integers, and H is the total number of attention heads.

[0065] For example, each vector in step S121 can be calculated using the following formulas (2) to (7):

[0066]

[0067] in, Indicates the first target feature X in the h-th attention head v and the second target feature X g Linear transformation matrices for different linear transformations. For example, is the first target feature X v Linear transformation to query vector The linear transformation matrix of is the first target feature X v Linear transformation into key-value vector The linear transformation matrix of is the first target feature X v Linear transformation to value vector The linear transformation matrix of ; similarly, Also the second target feature X g The linear transformation is the linear transformation matrix of the corresponding vector.

[0068] Step S122: Based on the first target feature X in the hth attention head v The query vector and the second target feature X g The key vector Calculate the first target feature X v For the second target feature Xg The first attention weight And according to the second target feature X g The query vector and the first target feature X v The key vector Calculate the second target feature X g For the first target feature X v The second attention weight

[0069] For example, the first attention weight and the second attention weight It can be calculated using the following formulas (8) and (9):

[0070]

[0071] Among them, d h represents the feature dimension of the h-th attention head, and softmax() is a normalized exponential function used to normalize each row of the matrix so that the sum of each row of the matrix is 1.

[0072] Step S123: Set the first attention weight in the hth attention head With the second target feature X g The value vector of Perform weighted summation to obtain the first fusion feature in the h-th attention head And the second attention weight in the hth attention head With the first target feature X v The value vector of Perform weighted summation to obtain the second fusion feature in the h-th attention head

[0073] For example, the first fused feature in the hth attention head and the second fusion feature in the h-th attention head The following formulas (10) and (11) can be used for calculation:

[0074]

[0075] in, Represents element-wise addition.

[0076] Step S124: Aggregate the first fusion features in multiple attention heads to obtain the first target fusion feature Z v , and aggregate the second fusion features in multiple attention heads to obtain the second target fusion feature Z g .

[0077] For example, the first target fusion feature Z v and the second target fusion feature Z g It can be calculated using the following formula (12) and formula (13):

[0078] Z v =Norm(Z v_fusion W v +Z v_fusion ) formula (12);

[0079] Z g =Norm(Z v_fusion W g +Z v_fusion ) formula (13);

[0080] Among them, Norm() represents the layer normalization operation, W v Z v_fusion The corresponding trainable weight matrix, W g Z v_fusion The corresponding trainable weight matrix.

[0081] Step S125: The first target feature X v , the second target feature X g , the first target fusion feature Z v and the second target fusion feature Z g The multimodal feature Z is obtained by fusion.

[0082] For example, the multimodal features can be calculated using the following formula (14):

[0083] Z=Concatenation(X v ,X g ,Z v ,Z g ) formula (14);

[0084] Among them, Concatenation() represents the matrix concatenation operation.

[0085] Based on this, the first target fusion feature and the second target fusion feature can be determined with the help of the attention mechanism to characterize the interaction information between the first target feature and the second target feature, and the first target feature, the second target feature, the first target fusion feature and the second target fusion feature can be fused into a multimodal feature. The multimodal feature can retain the original modal features composed of the first target feature and the second target feature, and contain the interaction information between the first target feature and the second target feature, which is conducive to more accurately characterizing the characteristics of the target tooth through multimodal features, thereby improving the recognition accuracy of subsequent classifiers.

[0086] In one embodiment, the number of rounds of training for the i-th sub-model is multiple rounds; before each round of training begins, obtaining the first image sample of the m pairs of image samples of the affected tooth includes: Figure 4 As shown, m original image samples of the affected tooth are obtained; at least one of rotation transformation, folding transformation and random noise addition is performed on the m original image samples to obtain m first image samples. The original image samples can be two-dimensional images obtained by scanning the affected tooth with an intraoral scanner, for example, Figure 4 2D occlusion space color image shown in .

[0087] In the process of transforming the original sample image into the first sample image, the rotation angle, the position of the fold line, and the type of random noise of the original sample image can be selected and adjusted according to actual needs, and the embodiment of the present application does not limit this. Preferably, the random noise can be Gaussian white noise.

[0088] In the above scheme, after acquiring m original image samples of a diseased tooth, the m original images can be rotated to increase the perspective diversity of the first image sample, enabling the i-th sub-model to recognize the rotated target tooth. Alternatively, the m original images can be flipped to increase the diversity of the first image sample without affecting the semantics of the original image samples, preventing overfitting. Alternatively, random noise can be added to the m original images to train the i-th sub-model in a noisy environment, improving its performance in different environments. Based on this, the generalization ability of each sub-model can be improved.

[0089] In one embodiment, please refer to Figure 5 The second image is a three-dimensional point cloud image, and obtaining the second image of the target tooth includes the following steps S310 to S330.

[0090] Step S310: Acquire a triangular mesh image of the target tooth. For example, the triangular mesh image of the target tooth can be obtained by scanning the target tooth with a three-dimensional laser scanner, and the file format can be 3D STL (Stereolithography)

[0091] Step S320: extracting the vertex coordinates of each triangle from the triangular mesh image to form a three-dimensional point cloud image of the target tooth.

[0092] Step S330: Downsample the 3D point cloud image using the farthest point sampling method to obtain a second image. The farthest point sampling method (FPS) is a conventional sampling method in the art and will not be described in detail here.

[0093] In the above scheme, since the vertex coordinates of each triangle are extracted from the triangular mesh image to form a three-dimensional point cloud image of the target tooth, the data volume of the three-dimensional point cloud image of the target tooth is large. Therefore, by adopting the farthest point sampling method to continue downsampling the three-dimensional point cloud image, not only can the data volume of the three-dimensional point cloud image of the target tooth be reduced, for example, to 3096 three-dimensional point cloud data to improve the image processing speed, but also it can ensure that the obtained second image is not distorted, which is more conducive to the smooth progress of the subsequent image processing process.

[0094] Figure 6 The figure shows a structural block diagram of an electronic device according to an embodiment of the present application. Figure 6 As shown, the electronic device 30 includes a memory 31 and a processor 32. The memory 31 stores a computer program that can be executed by the processor 32. When the processor 32 executes the computer program, it implements the method for determining a tooth defect restoration plan in the above-mentioned embodiment. The number of the memory 31 and the processor 32 can be one or more.

[0095] The electronic device 30 also includes: a communication interface 33 for communicating with external devices and performing data exchange transmission. If the memory 31, the processor 32 and the communication interface 33 are implemented independently, the memory 31, the processor 32 and the communication interface 33 can be interconnected through a bus and complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0096] Optionally, in a specific implementation, if the memory 31 , the processor 32 and the communication interface 33 are integrated on a chip, the memory 31 , the processor 32 and the communication interface 33 can communicate with each other through an internal interface.

[0097] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the embodiment of the present application.

[0098] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0099] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.

[0100] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0101] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0103] Any process or method description in a flow chart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes additional implementations in which the functions may be performed in a different order than shown or discussed, including in a substantially simultaneous manner or in a reverse order depending on the functions involved.

[0104] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0105] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0106] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0107] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for determining a tooth defect restoration plan, characterized in that: include: Acquire a first image and a second image of a target tooth; the first image includes information on the distribution of the occlusal distance between an undamaged area of the target tooth and an opposing area of the tooth, and a two-dimensional defect morphology of a defected area of the target tooth; and the second image includes information on the three-dimensional defect morphology of the defected area of the target tooth; Inputting both the first image and the second image into a preset restoration solution prediction model to obtain multiple sets of prediction results output by the restoration solution prediction model; each set of prediction results includes multiple restoration solutions and their corresponding confidence levels; the confidence level corresponding to the restoration solution is the probability that the restoration solution is correct; Determining an average confidence level of the same type of the repair solutions from a plurality of groups of the prediction results; The repair solution with the highest average confidence is taken as the target repair solution.

2. The determination method according to claim 1, characterized in that The repair solution prediction model includes a plurality of different sub-models, the image samples of the validation set for training the sub-models are different, and the image samples of the training set for training the sub-models are partially the same; Inputting both the first image and the second image into a preset repair solution prediction model to obtain multiple groups of prediction results output by the repair solution prediction model, including: inputting both the first image and the second image into each of the sub-models, so that each of the sub-models outputs a group of prediction results respectively.

3. The determination method according to claim 2, characterized in that: Each of the sub-models includes: a feature extraction layer for extracting a first target feature from the first image and a second target feature from the second image; the first target feature includes an occlusal distance feature between an undamaged area of the target tooth and an opposing area of the opposing tooth and a two-dimensional defect morphology feature of a defected area of the target tooth; and the second target feature includes a three-dimensional defect morphology feature of the defected area of the target tooth; A feature fusion layer, configured to perform cross-attention fusion on the first target feature and the second target feature to obtain a fused multimodal feature; A classifier is used to identify the prediction results of the corresponding group based on the multimodal features.

4. The determination method according to claim 3, characterized in that: The characteristic dimension of the second target feature is different from the characteristic dimension of the first target feature; Before cross-attention fusing the first target feature and the second target feature, the feature fusion layer is further used to: convert the feature dimension of the second target feature into the same feature dimension as the feature dimension of the first target feature.

5. The determination method according to claim 3, characterized in that: The feature extraction layer extracts the first target feature from the first image using a feature extraction module in a ViT model; the ViT model is trained based on a plurality of first sample images and labeled data for each of the first sample images, wherein the first sample images include information on the distribution of the occlusal distance between the undamaged area of the affected tooth and the opposing area of its opposing tooth, and a two-dimensional defect morphology of the damaged area of the affected tooth; The second image is a three-dimensional point cloud image, and the feature extraction layer uses the point cloud feature extraction module of the PointMLP model to extract the second target feature from the second image; the PointMLP model is trained based on multiple second image samples and the labeled data of each second image sample, and the second image sample includes the three-dimensional defect morphology of the non-defective area of the affected tooth.

6. The determination method according to claim 2, characterized in that: Also includes: Obtain m pairs of image samples of the affected tooth; the image sample pairs include a first image sample and a second image sample, the first image sample including occlusal distance distribution information between an undamaged area of the affected tooth and an opposing area of its opposing tooth and a two-dimensional defect morphology of a damaged area of the affected tooth, and the second image sample including a three-dimensional defect morphology of the damaged area of the affected tooth, and the annotated data of each image sample pair is a restoration plan for the corresponding affected tooth; Divide the m image sample pairs into k data sets; each data set includes a plurality of image sample pairs and their corresponding labeled data, m and k are both integers greater than 1, and m is an integer multiple of k, and the integer multiple is an integer greater than 1; The i-th data set is used as the validation set of the i-th sub-model to be trained, and the remaining k-1 data sets are used as the training sets of the i-th sub-model to be trained. The i-th sub-model to be trained is trained and verified to obtain the i-th sub-model; i is an integer greater than 1 and less than or equal to k.

7. The determination method according to claim 3, characterized in that: The first target feature and the second target feature are both represented by feature vectors. The feature fusion layer performs cross-attention fusion on the first target feature and the second target feature to obtain a fused multimodal feature, including: Calculate the query vector, key vector, and value vector of the first target feature, and the query vector, key vector, and value vector of the second target feature in the h-th attention head respectively; 1≤h≤H, h and H are integers, and H is the total number of multiple attention heads; Calculating a first attention weight of the first target feature for the second target feature based on the query vector of the first target feature and the key vector of the second target feature in the h-th attention head, and calculating a second attention weight of the second target feature for the first target feature based on the query vector of the second target feature and the key vector of the first target feature; Performing a weighted summation on the first attention weight in the h-th attention head and the value vector of the second target feature to obtain a first fused feature in the h-th attention head, and performing a weighted summation on the second attention weight in the h-th attention head and the value vector of the first target feature to obtain a second fused feature in the h-th attention head; Aggregating the first fused features from multiple attention heads to obtain a first target fused feature, and aggregating the second fused features from multiple attention heads to obtain a second target fused feature; The first target feature, the second target feature, the first target fusion feature and the second target fusion feature are fused to obtain the multimodal feature.

8. The determination method according to claim 6, characterized in that: The number of rounds of training for the i-th sub-model is multiple rounds; before each round of training begins, obtaining the first image sample of the m pairs of image samples of the affected tooth includes: Obtaining m original image samples of the affected teeth; Perform at least one of rotation transformation, folding transformation and random noise addition on the m original image samples respectively to obtain m first image samples.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 8. 10 . A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to claim 1 is implemented.

Images