Denture three-dimensional modeling based on oral image and model training method and system thereof

Abstract

This application discloses a method and system for three-dimensional modeling of dentures based on oral images and its training. The method includes: acquiring a first oral sample image, a positive sample three-dimensional denture model, and a negative sample three-dimensional denture model, and a model to be trained including a first sub-model and a second sub-model; reconstructing a second oral sample image using the first sub-model based on the first oral sample image; determining the first sub-model loss based on the difference between the first oral sample image and the second oral sample image; generating an initial three-dimensional denture model using the second sub-model based on the second oral sample image; determining the second sub-model loss based on the differences between the initial three-dimensional denture model and the positive sample three-dimensional denture model, and the differences between the initial three-dimensional denture model and the negative sample three-dimensional denture model; determining a comprehensive loss based on the first sub-model loss and the second sub-model loss; and iteratively training the model to be trained based on the comprehensive loss, which can improve the efficiency and accuracy of three-dimensional denture modeling.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for three-dimensional modeling of dentures based on oral images and training the model. Background Technology

[0002] 3D printing is widely used in the field of denture manufacturing. To manufacture dentures that are suitable for the patient's actual situation, it is usually necessary to first use 3D modeling simulation software to create a 3D model of the denture suitable for the patient based on the patient's oral data. Then, based on the 3D model obtained from the modeling, 3D printing is used to manufacture dentures that match the patient.

[0003] However, this type of 3D modeling method has the problem of low modeling efficiency and may even have the problem of insufficient modeling accuracy, which in turn affects the efficiency and accuracy of denture 3D printing. Summary of the Invention

[0004] To address the aforementioned technical issues, this application proposes a method and system for three-dimensional modeling of dentures based on oral images, which can improve the efficiency and accuracy of three-dimensional modeling of dentures.

[0005] In a first aspect, embodiments of this application provide a model training method for three-dimensional modeling of dentures, including:

[0006] Acquire a first oral cavity sample image and its corresponding positive sample denture 3D model and negative sample denture 3D model, as well as a training model including a first sub-model and a second sub-model;

[0007] Using the first sub-model, a second oral cavity sample image is obtained by reconstructing the first oral cavity sample image.

[0008] The first sub-model loss is determined based on the difference between the first oral cavity sample image and the second oral cavity sample image;

[0009] Based on the second oral cavity sample image, an initial three-dimensional model of the denture is generated using the second sub-model;

[0010] The second sub-model loss is determined based on the differences between the initial three-dimensional denture model and the positive sample three-dimensional denture model, and the differences between the initial three-dimensional denture model and the negative sample three-dimensional denture model.

[0011] The comprehensive loss is determined based on the loss of the first sub-model and the loss of the second sub-model. The model to be trained is iteratively trained according to the comprehensive loss. The three-dimensional denture generation model obtained after the iterative training is used for three-dimensional denture modeling.

[0012] Optionally, the step of reconstructing a second oral cavity sample image using the first sub-model based on the first oral cavity sample image includes:

[0013] Using the first sub-model, the first image features of the first oral sample image are extracted, and the first image features are reconstructed to obtain the second image features. Furthermore, a second oral sample image is generated based on the second image features, wherein the second image features are the image features of the second oral sample image.

[0014] Optionally, generating an initial three-dimensional model of the denture using the second sub-model based on the second oral cavity sample image includes:

[0015] Using the second sub-model, the key points of the denture structure are analyzed based on the second oral cavity sample image, and an initial three-dimensional model of the denture is generated based on the analysis results.

[0016] Optionally, the step of using the second sub-model to perform key point analysis of the denture structure based on the second oral cavity sample image, and generating an initial three-dimensional model of the denture based on the key point analysis results, includes:

[0017] Using the second sub-model, feature information related to key points of the denture structure in the second oral sample image is identified to obtain the analysis results of key points of the denture structure. Furthermore, using the analysis results of key points of the denture structure as constraints, an initial three-dimensional model of the denture is generated based on the second oral sample image.

[0018] Optionally, determining the second sub-model loss based on the differences between the initial three-dimensional denture model and the positive sample three-dimensional denture model, and the differences between the initial three-dimensional denture model and the negative sample three-dimensional denture model, includes:

[0019] Determine the key structural differences of the first denture between the initial three-dimensional model and the positive sample three-dimensional model, and determine the key structural differences of the second denture between the initial three-dimensional model and the negative sample three-dimensional model;

[0020] The loss of the second sub-model is determined based on the differences in key points of the first and second denture structures.

[0021] Secondly, embodiments of this application provide a method for three-dimensional modeling of dentures based on oral images, including:

[0022] Acquire target oral cavity images of the target patient;

[0023] A pre-trained three-dimensional denture generation model is invoked to generate a three-dimensional denture model corresponding to the target patient based on the target oral image, wherein the three-dimensional denture generation model is trained according to any one of the methods described in the first aspect above.

[0024] Optionally, acquiring the target oral cavity image of the target patient includes:

[0025] Acquire multiple oral images of the target patient;

[0026] According to a preset division method, the multiple oral cavity images are divided into a first oral cavity image and a second oral cavity image to be enhanced;

[0027] Based on the first oral cavity image and the second oral cavity image, generate the fusion feature corresponding to the second oral cavity image;

[0028] Based on the fusion features corresponding to the second oral image, image enhancement processing is performed on the second oral image to obtain a third oral image, and the first oral image and the third oral image are combined to form the target oral image of the target patient.

[0029] Optionally, the fusion feature corresponding to the second oral cavity image includes at least one fusion domain feature corresponding to the second oral cavity image;

[0030] The step of generating fusion features corresponding to the second oral cavity image based on the first oral cavity image and the second oral cavity image includes:

[0031] At least one type of domain information is determined for each of the plurality of oral cavity images, wherein the at least one type of domain information corresponds one-to-one with the at least one fusion domain feature;

[0032] For each type of domain information in the at least one type of domain information, the corresponding domain features are extracted from the respective domain information of the first oral cavity image and the second oral cavity image, and the corresponding domain features of the first oral cavity image and the second oral cavity image are fused based on an attention mechanism to obtain the fused domain features of the second oral cavity image under the domain information.

[0033] Optionally, the step of performing image enhancement processing on the second oral image based on the fusion features corresponding to the second oral image to obtain a third oral image includes:

[0034] The fusion features corresponding to the second oral cavity image are transformed into the representation domain of the large model to obtain the feature representation information corresponding to the second oral cavity image;

[0035] Based on the second oral cavity image and the feature representation information, model prompt information is generated, wherein the model prompt information is used to instruct the large model to perform image enhancement processing on the second oral cavity image under the guidance of the feature representation information;

[0036] The model prompt information is processed by the large model to obtain a third oral cavity image.

[0037] Thirdly, embodiments of this application provide a three-dimensional modeling system for dentures based on oral images, comprising:

[0038] The image acquisition module is used to acquire target oral cavity images of the target patient;

[0039] The model generation module is used to call a pre-trained three-dimensional denture generation model to generate a three-dimensional denture model corresponding to the target patient based on the target oral image, wherein the three-dimensional denture generation model is trained according to the method described in any one of the first aspects above.

[0040] In summary, the embodiments of this application have at least the following beneficial effects:

[0041] In this embodiment, a first oral cavity sample image and its corresponding positive and negative sample 3D denture models are acquired, along with a training model including a first sub-model and a second sub-model. Using the first sub-model, a second oral cavity sample image is reconstructed from the first oral cavity sample image. Based on the difference between the first and second oral cavity sample images, a first sub-model loss is determined. Based on the second oral cavity sample image, an initial 3D denture model is generated using the second sub-model. Based on the differences between the initial 3D denture model and the positive sample 3D denture model, and between the initial 3D denture model and the negative sample 3D denture model, a second sub-model loss is determined. A comprehensive loss is determined based on the first and second sub-model losses, and the training model is iteratively trained based on the comprehensive loss. The resulting 3D denture generation model is used for 3D denture modeling, thereby improving the efficiency and accuracy of the 3D denture generation model for 3D denture modeling of input images. Attached Figure Description

[0042] Figure 1 This is a schematic flowchart of the model training method for three-dimensional modeling of dentures provided in the embodiments of this application;

[0043] Figure 2 This is a schematic diagram of the structure of the model to be trained provided in an embodiment of this application;

[0044] Figure 3This is a flowchart illustrating the three-dimensional modeling method for dentures based on oral images provided in an embodiment of this application.

[0045] Figure 4 This is a schematic diagram of the structure of the three-dimensional modeling system for dentures based on oral images provided in the embodiments of this application;

[0046] Figure 5 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments / examples are only a part of the embodiments / examples of this application, and not all of the embodiments / examples. Based on the embodiments / examples in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0048] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. In the description of this application, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment / example" means "at least one embodiment / example"; the term "another embodiment / example" means "at least one additional embodiment / example"; the term "some embodiments / examples" means "at least some embodiments / examples."

[0049] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0050] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0051] Firstly, see [the following] Figure 1 The diagram shows a flowchart of a model training method for three-dimensional modeling of dentures provided in an embodiment of this application. The method includes steps S101-S106, as detailed below.

[0052] S101, acquire the first oral cavity sample image and its corresponding positive sample denture 3D model and negative sample denture 3D model, as well as the training model including the first sub-model and the second sub-model.

[0053] In some examples, the first oral sample image described above may include the area in the oral cavity of the corresponding patient that requires implantation of a prosthesis.

[0054] In some examples, the aforementioned first oral sample image may be an image obtained by scanning the patient's oral cavity.

[0055] In some examples, the aforementioned first oral sample image may include at least one of the following: three-dimensional oral sample image, two-dimensional oral sample image, oral sample X-ray image, and oral sample CT (Computed Tomography) image.

[0056] S102, using the first sub-model, reconstruct the first oral cavity sample image to obtain the second oral cavity sample image.

[0057] In some examples, the aforementioned first sub-model may include: CNN (Convolutional Neural Network), ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks), and CycleGAN (Cycle Generative Adversarial Networks). Among them, CNN is more suitable for repairing specific regions in an image (such as image denoising, blur removal, and / or repairing tooth defects), ESRGAN is more suitable for improving image resolution, and CycleGAN is more suitable for transforming images under different conditions (such as simulating images under different lighting conditions).

[0058] S103, Based on the difference between the first oral cavity sample image and the second oral cavity sample image, determine the first sub-model loss.

[0059] In some examples, the feature similarity between the first image features of the first oral cavity sample image and the second image features of the second oral cavity sample image can be calculated, and the first sub-model loss can be calculated using a first preset loss function based on the feature similarity. In any embodiment of this application, the feature similarity can be calculated using methods such as cosine similarity.

[0060] S104, Based on the second oral cavity sample image, generate an initial three-dimensional model of the denture using the second sub-model.

[0061] In some examples, the aforementioned second sub-model can be built based on CNN and / or Transformer architectures.

[0062] In some examples, the second oral sample image may include a two-dimensional image and / or a three-dimensional image. In this way, the second sub-model can directly perform three-dimensional modeling for the denture to be implanted based on the second oral sample image to generate an initial three-dimensional model of the denture.

[0063] S105, based on the differences between the initial three-dimensional denture model and the positive sample three-dimensional denture model, and the differences between the initial three-dimensional denture model and the negative sample three-dimensional denture model, determine the second sub-model loss.

[0064] In some examples, the aforementioned positive sample denture 3D model refers to the desired 3D model of the denture corresponding to the first oral sample image, while the aforementioned negative sample denture 3D model refers to the unwanted 3D model of the denture corresponding to the first oral sample image (for example, the model may contain denture structural key points that clearly do not match the first oral sample image). Thus, by comparing positive and negative samples, the second sub-model can learn to distinguish between reasonable and unreasonable designs, thereby improving the clinical usability of the generated results.

[0065] In some examples, the second sub-model loss can be calculated using a second preset loss function based on the feature similarity between the model feature vectors of the initial denture 3D model and the positive sample denture 3D model, and the feature similarity between the model feature vectors of the initial denture 3D model and the negative sample denture 3D model.

[0066] Thus, the second sub-model loss comprehensively considers the differences between the generated initial denture 3D model and the positive sample denture 3D model and the negative sample denture 3D model, respectively. Compared with a single type of sample, it can improve the number of iterations required for training and the output accuracy of the model after training.

[0067] S106, determine the comprehensive loss based on the first sub-model loss and the second sub-model loss, and iteratively train the model to be trained according to the comprehensive loss, wherein the three-dimensional denture generation model obtained after the iterative training is used for three-dimensional denture modeling.

[0068] In this embodiment, since the input first oral sample image may contain noise, blur, occlusion, uneven lighting, and large differences in shooting parameters between different images, the first sub-model can solve this problem through reconstruction. This improves the image quality of the input image to the second sub-model used to generate the 3D denture model and ensures the stability of the input image parameters. This improves the accuracy of the generated 3D denture model and reduces the generalization requirements of the second sub-model for images of different quality, simplifying the complexity of the second sub-model in terms of model structure, model parameters, and / or training methods. Furthermore, by combining the losses of the first and second sub-models, they can share gradients during training, achieving joint training. Thus, the performance of the second sub-model depends on the output quality of the first sub-model through joint training, while the first sub-model can automatically learn the most useful image features for denture generation through joint training. This allows the first sub-model to achieve task-driven (i.e., based on the output feedback of the second sub-model) image reconstruction rather than general denoising. In summary, this improves the efficiency and accuracy of the 3D denture generation model for input images.

[0069] For example, see Figure 2 The first sub-model and the second sub-model in the model to be trained can be cascaded, that is, the model to be trained includes a first sub-model and a second sub-model that are electrically connected in sequence.

[0070] In some examples, the losses of the first and second sub-models can be weighted and summed to obtain a combined loss. In this way, the weighted summation assigns weights to the first and second sub-model losses in the current training iteration.

[0071] Furthermore, the comprehensive loss can be calculated by weighted summation of the first sub-model loss and the second sub-model loss based on the first rate of change of the first sub-model loss and the second rate of change of the second sub-model loss in the current iteration of training. Here, the first rate of change of loss refers to the rate of change of the first sub-model loss in the current iteration of training compared to the first sub-model loss in the previous iteration, and the second rate of change of loss refers to the rate of change of the second sub-model loss in the current iteration of training compared to the second sub-model loss in the previous iteration. Generally, if the difference between the rates of change of the two sub-model losses is less than or equal to a preset rate of change threshold, a smaller rate of change means a slower rate of loss decrease, and thus a higher weight can be assigned. And / or, if the difference between the rates of change of the two sub-model losses is greater than the preset rate of change threshold, it indicates that the learning speed of the sub-model corresponding to the larger rate of change loss is much greater than that of the other sub-model. In this case, the weight of the sub-model corresponding to the larger rate of change loss can be reduced to prevent the sub-model corresponding to the larger rate of change loss from dominating the iteration training.

[0072] In one optional implementation, the step of reconstructing a second oral cavity sample image using the first sub-model based on the first oral cavity sample image includes:

[0073] Using the first sub-model, the first image features of the first oral sample image are extracted, and the first image features are reconstructed to obtain the second image features. Furthermore, a second oral sample image is generated based on the second image features, wherein the second image features are the image features of the second oral sample image.

[0074] In some examples, the first sub-model may include an encoder network and a decoder network, wherein the encoder network can be used to encode a first oral cavity sample image to extract a first image feature from the first oral cavity sample image, and the decoder network can be used to decode a second image feature to generate a second oral cavity sample image.

[0075] In some examples, the first sub-model may include a DAE (Denoising Autoencoder), which, in cases where the first image features include features of noisy and / or missing image data, can be used to recover a clean feature representation from the features of the noisy and / or missing image data to at least partially achieve reconstruction, thereby generating a second image feature.

[0076] In some examples, the first sub-model may include a sparse autoencoder, which can be used to remove irrelevant information noise from the first image features to at least partially achieve reconstruction.

[0077] In some examples, the first sub-model may include a generative adversarial network (GAN) that is at least partially reconstructed through iterative adversarial training, the GAN comprising a generative network and a discriminative network.

[0078] In one optional implementation, generating an initial three-dimensional model of the denture using the second sub-model based on the second oral cavity sample image includes:

[0079] Using the second sub-model, the key points of the denture structure are analyzed based on the second oral cavity sample image, and an initial three-dimensional model of the denture is generated based on the analysis results.

[0080] In some examples, key structural points of the denture may include at least one of the following: the location of existing teeth, the shape outline of existing teeth, the location of the gingival line, and the boundaries and features of the old denture corresponding to the denture to be implanted.

[0081] In some examples, the second sub-model can be configured to identify and analyze key points of the denture structure in the second oral sample image through object detection, and directly determine the key points of the denture to be implanted based on the analysis results of the key points of the denture structure. Then, based on the key points of the denture to be implanted, a three-dimensional model of the denture to be implanted is performed to generate an initial three-dimensional model of the denture.

[0082] In one optional implementation, the step of using the second sub-model to perform key point analysis of the denture structure based on the second oral cavity sample image, and generating an initial three-dimensional model of the denture based on the key point analysis results, includes:

[0083] Using the second sub-model, feature information related to key points of the denture structure in the second oral sample image is identified to obtain the analysis results of key points of the denture structure. Furthermore, using the analysis results of key points of the denture structure as constraints, an initial three-dimensional model of the denture is generated based on the second oral sample image.

[0084] In some examples, the second sub-model may include an object detection network that can be configured to identify key points of the denture structure in a second oral sample image through object detection, and then generate feature information (e.g., key point feature vectors of the denture structure) related to the key points of the denture structure as the result of the key point analysis of the denture structure.

[0085] In some examples, the second sub-model may include a 3D modeling network built on a CNN and / or Transformer architecture. This 3D modeling network may include an image encoder built on a CNN or Transformer architecture and a 3D decoder built on a Transformer architecture (e.g., a voxel decoder, a point cloud decoder, and / or an implicit field decoder). Thus, the image encoder can be used to encode the second oral sample image to output corresponding image features. The 3D decoder is configured to use the feature vector of the denture structure key points corresponding to the analysis results of the denture structure key points as a conditional vector and the image features output by the image encoder as a basis to decode and generate an initial denture 3D model.

[0086] In some examples, this conditional vector can be used to concatenate with the image features output by the image encoder and input into the 3D decoder for decoding.

[0087] In some examples, an attention mechanism can also be configured in the 3D decoder, allowing it to dynamically weight and fuse the feature information carried by the conditional vector during the decoding process of the image features output by the image encoder. This attention mechanism can include channel attention and / or cross-attention. The channel attention mechanism can be used to automatically generate channel weights based on the conditional vector, which can then be used to automatically modulate the image features output by the image encoder to enhance the response in areas containing key points of the denture structure. The cross-attention mechanism can instruct the 3D decoder to query relevant information in the conditional vector at each step of the decoding process, thereby dynamically guiding the decoding process (i.e., the 3D modeling process) and improving decoding (modeling) accuracy.

[0088] In one optional implementation, determining the second sub-model loss based on the differences between the initial three-dimensional denture model and the positive sample three-dimensional denture model, and the differences between the initial three-dimensional denture model and the negative sample three-dimensional denture model, includes:

[0089] Determine the key structural differences of the first denture between the initial three-dimensional model and the positive sample three-dimensional model, and determine the key structural differences of the second denture between the initial three-dimensional model and the negative sample three-dimensional model;

[0090] The loss of the second sub-model is determined based on the differences in key points of the first and second denture structures.

[0091] In some examples, the differences in the key points of the denture structure (i.e., the differences in the key points of the first denture structure and the differences in the key points of the second denture structure) can be characterized by calculating the spatial coordinate distance between the key points of the denture structure of the two models (i.e., the initial three-dimensional denture model and the positive sample three-dimensional denture model, and the initial three-dimensional denture model and the negative sample three-dimensional denture model).

[0092] In some examples, a third pre-defined loss function can be used to determine the second sub-model loss based on the differences in key points of the first and second denture structures. This second sub-model loss indicates a reduction in the differences in key points of the first denture structure and an increase in the differences in key points of the second denture structure. This aims to make the initial 3D denture model closer to the positive sample 3D denture model (i.e., smaller differences in key points of the first denture structure) and to make the difference between the initial 3D denture model and the negative sample 3D denture model more pronounced (i.e., larger differences in key points of the second denture structure). For example, an exemplary mathematical expression can be shown below.

[0093]

[0094] in, This represents the loss of the second sub-model. This indicates the key differences in the structure of the first denture. This indicates the key differences in the structure of the second denture. Indicates the preset marginal value. Indicates the lower bound of the loss value (e.g., 0). This represents the first control factor. , All parameters are greater than 0 and can be manually preset to control the specific parameters of the second sub-model loss. For example, this... These are control parameters used to control the weights of differences in key points of the first denture structure; in some examples, this... The value of can also be 1, meaning it does not pass. To set weights related to the differences in key points of the first denture structure.

[0095] Secondly, see Figure 3 The diagram shows a flowchart of a three-dimensional modeling method for dentures based on oral images provided in an embodiment of this application. The method includes steps S301-S302, as detailed below.

[0096] S301, acquire the target oral cavity image of the target patient.

[0097] S302, invoke the pre-trained three-dimensional denture generation model, and generate a three-dimensional denture model corresponding to the target patient based on the target oral image, wherein the three-dimensional denture generation model is trained according to any one of the methods in the first aspect above.

[0098] In some examples, in step S302, the first sub-model in the three-dimensional denture generation model can be called first to reconstruct the target oral cavity image based on the target oral cavity image to obtain the reconstructed oral cavity image; then, the second sub-model in the three-dimensional denture generation model can be called to generate the target denture three-dimensional model based on the reconstructed oral cavity image.

[0099] In one optional implementation, acquiring the target oral cavity image of the target patient includes:

[0100] Acquire multiple oral images of the target patient;

[0101] According to a preset division method, the multiple oral cavity images are divided into a first oral cavity image and a second oral cavity image to be enhanced;

[0102] Based on the first oral cavity image and the second oral cavity image, generate the fusion feature corresponding to the second oral cavity image;

[0103] Based on the fusion features corresponding to the second oral image, image enhancement processing is performed on the second oral image to obtain a third oral image, and the first oral image and the third oral image are combined to form the target oral image of the target patient.

[0104] In some cases, the aforementioned 3D denture generation model can be configured to generate a 3D model of the target denture based on multiple target oral cavity images. In this case, it is easy to understand that it is simply a matter of setting the first oral cavity sample images used in each iteration of the relevant embodiments of the above training method to multiple images, which will not be elaborated further here. However, since some of the acquired oral cavity images may not provide sufficient information, thus affecting the accuracy of the 3D modeling, this embodiment can perform image enhancement processing on the portion of the second oral cavity image to be enhanced, in order to improve the accuracy of subsequent 3D modeling.

[0105] In some examples, the preset segmentation method may include a first segmentation method based on sharpness and / or a second segmentation method based on image modality.

[0106] When the preset segmentation method includes a first segmentation method based on sharpness, the sharpness of each of the multiple oral cavity images can be determined first. Then, the first N oral cavity images with the highest sharpness (e.g., oral cavity images in the same image modality), and / or oral cavity images with a sharpness higher than a preset sharpness threshold (e.g., oral cavity images in the same image modality) are determined as the first oral cavity images, and the remaining oral cavity images (e.g., oral cavity images in the same image modality) are determined as the second oral cavity images.

[0107] When the preset segmentation method includes a second segmentation method based on image modality, the first oral cavity image can be a three-dimensional scan image, and the second oral cavity image can include a two-dimensional oral cavity image and / or an oral X-ray image and / or an oral CT image.

[0108] In some examples, based on the first oral cavity image and each second oral cavity image, the fusion feature corresponding to each second oral cavity image can be generated. Specifically, the fusion feature corresponding to each second oral cavity image can be obtained directly by fusing the image features of the first oral cavity image with the image features of each second oral cavity image.

[0109] In some examples, image enhancement processing can be performed on each second oral cavity image based on the fusion features corresponding to each second oral cavity image to obtain a third oral cavity image corresponding to each second oral cavity image, and the first oral cavity image and each third oral cavity image can be combined to form the target oral cavity image of the target patient.

[0110] In one optional implementation, the fusion feature corresponding to the second oral cavity image includes at least one fusion domain feature corresponding to the second oral cavity image;

[0111] The step of generating fusion features corresponding to the second oral cavity image based on the first oral cavity image and the second oral cavity image includes:

[0112] At least one type of domain information is determined for each of the plurality of oral cavity images, wherein the at least one type of domain information corresponds one-to-one with the at least one fusion domain feature;

[0113] For each type of domain information in the at least one type of domain information, the corresponding domain features are extracted from the respective domain information of the first oral cavity image and the second oral cavity image, and the corresponding domain features of the first oral cavity image and the second oral cavity image are fused based on an attention mechanism to obtain the fused domain features of the second oral cavity image under the domain information.

[0114] As can be seen, for each type of domain information, the domain features corresponding to the first oral cavity image and the domain features corresponding to each second oral cavity image can be extracted separately. The domain features corresponding to the first oral cavity image and the domain features corresponding to each second oral cavity image are then fused based on an attention mechanism to obtain the fused domain features corresponding to each second oral cavity image under that type of domain information.

[0115] In some examples, the aforementioned domain information may include at least one of the following: illumination domain information (e.g., used to indicate the brightness, contrast, and / or shadow distribution of an image), texture domain information (used to indicate the surface details and / or gingival texture of oral teeth, especially the teeth on both sides of the denture to be implanted), edge / structure domain information (used to indicate the contours, occlusal lines, and / or gaps of oral teeth, especially the teeth on both sides of the denture to be implanted), color domain information (used to indicate the color gradation of oral teeth, especially the teeth on both sides of the denture to be implanted, and / or the redness of the gingiva), and sharpness domain information (used to indicate the sharpness and blurriness of the image). In this embodiment, multi-dimensional feature decoupling of the image is achieved. The image is no longer treated as a whole, but rather decomposed into multiple independent domains (such as illumination domain, texture domain, etc.), allowing subsequent feature fusion to be performed on demand and accurately, thereby improving the accuracy of the generated fused domain features.

[0116] In some scenarios, image enhancement processing can include frequency-domain based image enhancement or spatial-domain based image enhancement. However, directly performing frequency-domain based image enhancement on the second oral cavity image can easily lead to significant changes in the spatial domain features of the image during the enhancement process (e.g., the coordinates of key points of the denture to be implanted and / or the teeth on both sides of the denture), resulting in visual inaccuracies in the enhanced image. On the other hand, while directly performing spatial-domain based image enhancement on the second oral cavity image can improve the visual effect, it may lead to significant and unreasonable changes in the overall shape and contour of the denture to be implanted and / or the teeth on both sides of the denture in the enhanced image due to the lack of constraints on physical realism.

[0117] In some examples, for the above-mentioned scenarios, the at least one type of domain information may also include frequency domain information and spatial domain information. Thus, by converting each oral cavity image to the frequency and spatial domains respectively, the resulting frequency domain information can be used to provide frequency distribution, and the spatial domain information can be used to provide rich details. Under the dynamic feature fusion of the attention mechanism, these two types of domain information can provide richer and more accurate information for the fused domain features, thereby improving the ability to capture details of oral cavity images during image enhancement processing, resulting in more precise image enhancement. Furthermore, this embodiment considers both the frequency and spatial domains simultaneously, so spatial details can be enhanced under the constraints of the physical properties represented by the frequency domain information during image enhancement, thus solving the problems existing in the above-mentioned scenarios.

[0118] In one optional implementation, the step of performing image enhancement processing on the second oral image based on the fusion features corresponding to the second oral image to obtain a third oral image includes:

[0119] The fusion features corresponding to the second oral cavity image are transformed into the representation domain of the large model to obtain the feature representation information corresponding to the second oral cavity image;

[0120] Based on the second oral cavity image and the feature representation information, model prompt information is generated, wherein the model prompt information is used to instruct the large model to perform image enhancement processing on the second oral cavity image under the guidance of the feature representation information;

[0121] The model prompt information is processed by the large model to obtain a third oral cavity image.

[0122] In some exemplary scenarios, large models in related technologies may suffer from the problem of "large model illusion." Large model illusion typically refers to a phenomenon that occurs when using large-scale pre-trained language models, where the content generated by the model may not conform to real-world facts, or it may automatically generate seemingly reasonable but actually erroneous or fictitious information without clear evidence. Specifically, in this embodiment, if a large model is directly used to perform general image enhancement processing on the second oral cavity image, it may result in the large model adding erroneous or fictitious information to the enhanced image, thereby affecting the reliability and accuracy of the generated target denture 3D model.

[0123] In some examples, for the scenarios described above, this embodiment adds feature representation information transformed from fusion features to assist / guide the large model in performing image enhancement processing on the second oral cavity image, thereby reducing the probability of the aforementioned large model illusion. Furthermore, the fusion features here are intentionally transformed into the representation domain of the large model, allowing the large model to better understand the information of the fusion features and reducing the probability of model illusions related to the fusion features occurring during the understanding process.

[0124] In some examples, the model cue information may be obtained by fusing (e.g., stitching) the feature representation information and the image representation information obtained by converting the second oral image into the representation domain of the larger model.

[0125] In some examples, the model prompt information can also be generated by filling a preset prompt template based on the second oral cavity image and the feature representation information. Specifically, the preset prompt template may include image feature filling positions and guidance information filling positions. In this case, the image features of the second oral cavity image can be filled into the image feature filling positions, and the feature representation information can be filled into the guidance information filling positions to generate the model prompt information.

[0126] In some examples, the large model can also be configured to perform image enhancement processing on the second oral image related to key points of the denture structure, guided by the feature representation information, so as to focus image enhancement on the relevant areas of key points of the denture structure.

[0127] Thirdly, correspondingly, the embodiments of this application also provide a three-dimensional modeling system for dentures based on oral images, which can realize all the processes of the three-dimensional modeling method for dentures based on oral images provided in the above embodiments.

[0128] See Figure 4 The diagram illustrates the structure of a three-dimensional denture modeling system based on oral cavity images provided in this application embodiment. This three-dimensional denture modeling system based on oral cavity images includes:

[0129] Image acquisition module 401 is used to acquire target oral cavity images of the target patient;

[0130] The model generation module 402 is used to call a pre-trained three-dimensional denture generation model to generate a three-dimensional denture model corresponding to the target patient based on the target oral image, wherein the three-dimensional denture generation model is trained according to the method described in any one of the first aspects above.

[0131] In one optional implementation, acquiring the target oral cavity image of the target patient includes:

[0132] Acquire multiple oral images of the target patient;

[0133] According to a preset division method, the multiple oral cavity images are divided into a first oral cavity image and a second oral cavity image to be enhanced;

[0134] Based on the first oral cavity image and the second oral cavity image, generate the fusion feature corresponding to the second oral cavity image;

[0135] Based on the fusion features corresponding to the second oral image, image enhancement processing is performed on the second oral image to obtain a third oral image, and the first oral image and the third oral image are combined to form the target oral image of the target patient.

[0136] In one optional implementation, the fusion feature corresponding to the second oral cavity image includes at least one fusion domain feature corresponding to the second oral cavity image;

[0137] The step of generating fusion features corresponding to the second oral cavity image based on the first oral cavity image and the second oral cavity image includes:

[0138] At least one type of domain information is determined for each of the plurality of oral cavity images, wherein the at least one type of domain information corresponds one-to-one with the at least one fusion domain feature;

[0139] For each type of domain information in the at least one type of domain information, the corresponding domain features are extracted from the respective domain information of the first oral cavity image and the second oral cavity image, and the corresponding domain features of the first oral cavity image and the second oral cavity image are fused based on an attention mechanism to obtain the fused domain features of the second oral cavity image under the domain information.

[0140] In one optional implementation, the step of performing image enhancement processing on the second oral image based on the fusion features corresponding to the second oral image to obtain a third oral image includes:

[0141] The fusion features corresponding to the second oral cavity image are transformed into the representation domain of the large model to obtain the feature representation information corresponding to the second oral cavity image;

[0142] Based on the second oral cavity image and the feature representation information, model prompt information is generated, wherein the model prompt information is used to instruct the large model to perform image enhancement processing on the second oral cavity image under the guidance of the feature representation information;

[0143] The model prompt information is processed by the large model to obtain a third oral cavity image.

[0144] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0145] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in any of the preceding claims.

[0146] Sixthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0147] See Figure 5 The computer device in this embodiment includes a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501, such as a model training program for 3D denture modeling and / or a 3D denture modeling program based on oral images. When the processor 501 executes the computer program, it implements the steps in the above-described embodiments of the various 3D denture modeling methods for model training and / or 3D denture modeling based on oral images.

[0148] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 502 and executed by the processor 501 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0149] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0150] The processor 501 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 501 can be any conventional processor. The processor 501 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0151] The memory 502 can be used to store the computer programs and / or modules. The processor 501 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 502 and calling the data stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0152] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a non-transitory computer-readable storage medium. When the computer program is executed by the processor 501, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0153] In summary, the embodiments of this application have at least the following beneficial effects:

[0154] In this embodiment, a first oral cavity sample image and its corresponding positive and negative sample 3D denture models are acquired, along with a training model including a first sub-model and a second sub-model. Using the first sub-model, a second oral cavity sample image is reconstructed from the first oral cavity sample image. Based on the difference between the first and second oral cavity sample images, a first sub-model loss is determined. Based on the second oral cavity sample image, an initial 3D denture model is generated using the second sub-model. Based on the differences between the initial 3D denture model and the positive sample 3D denture model, and between the initial 3D denture model and the negative sample 3D denture model, a second sub-model loss is determined. A comprehensive loss is determined based on the first and second sub-model losses, and the training model is iteratively trained based on the comprehensive loss. The resulting 3D denture generation model is used for 3D denture modeling, thereby improving the efficiency and accuracy of the 3D denture generation model for 3D denture modeling of input images.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware platforms, or it can be implemented entirely by hardware. Based on this understanding, all or part of the technical solutions of this application that contribute to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0156] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A model training method for three-dimensional modeling of dentures, characterized in that, include: Acquire a first oral sample image and its corresponding positive sample denture 3D model and negative sample denture 3D model, and a training model including a first sub-model and a second sub-model electrically connected in sequence, wherein the first oral sample image contains the area in the oral cavity of the corresponding patient that needs to be implanted with denture, and the first sub-model includes a convolutional neural network (CNN) for repairing a specific area in the image, wherein the repair of the specific area includes the repair of tooth defects. Using the first sub-model, the second oral cavity sample image is obtained by reconstructing the first oral cavity sample image. The first sub-model loss is determined based on the difference between the first oral cavity sample image and the second oral cavity sample image; Based on the second oral cavity sample image, the second sub-model is used to perform three-dimensional modeling for the denture to be implanted, so as to generate an initial three-dimensional model of the denture. The second sub-model loss is determined based on the differences between the initial three-dimensional denture model and the positive sample three-dimensional denture model, and the differences between the initial three-dimensional denture model and the negative sample three-dimensional denture model. A comprehensive loss is determined based on the loss of the first sub-model and the loss of the second sub-model. The model to be trained is iteratively trained according to the comprehensive loss. The three-dimensional denture generation model obtained after the iterative training is used for three-dimensional denture modeling. The step of using the first sub-model to reconstruct the first oral cavity sample image to obtain the second oral cavity sample image includes: using the first sub-model to extract the first image features of the first oral cavity sample image, reconstructing the first image features to obtain the second image features, and generating the second oral cavity sample image based on the second image features, wherein the second image features are the image features of the second oral cavity sample image. The step of using the second sub-model to perform three-dimensional modeling of the denture to be implanted based on the second oral sample image to generate an initial three-dimensional denture model includes: using the second sub-model to identify feature information related to key points of the denture structure in the second oral sample image, obtaining the analysis results of key points of the denture structure, and using the analysis results of key points of the denture structure as constraints to generate an initial three-dimensional denture model based on the second oral sample image. The second sub-model includes a three-dimensional modeling network, which includes an image encoder and a 3D decoder. The image encoder is used to encode the second oral sample image to output corresponding image features. The 3D decoder is configured to use the feature vector of the key points of the denture structure corresponding to the analysis result of the key points of the denture structure as a condition vector and the image features output by the image encoder as a basis to decode and generate the initial three-dimensional model of the denture. The 3D decoder is equipped with an attention mechanism, which enables the 3D decoder to dynamically weight and fuse the feature information carried by the conditional vector during the decoding process of the image features output by the image encoder.

2. The method of claim 1, wherein, The determination of the second sub-model loss based on the differences between the initial 3D denture model and the positive sample 3D denture model, and the differences between the initial 3D denture model and the negative sample 3D denture model, includes: Determine the key structural differences of the first denture between the initial three-dimensional model and the positive sample three-dimensional model, and determine the key structural differences of the second denture between the initial three-dimensional model and the negative sample three-dimensional model; The loss of the second sub-model is determined based on the differences in key points of the first and second denture structures.

3. A method of three-dimensional modeling of a denture based on an oral image, characterized by, include: Acquire target oral cavity images of the target patient; A pre-trained three-dimensional denture generation model is invoked to generate a three-dimensional denture model corresponding to the target patient based on the target oral image, wherein the three-dimensional denture generation model is trained according to the method described in any one of claims 1-2; The acquisition of the target oral cavity image of the target patient includes: Acquire multiple oral images of the target patient; According to a preset division method, the multiple oral cavity images are divided into a first oral cavity image and a second oral cavity image to be enhanced; Based on the first oral cavity image and the second oral cavity image, generate the fusion feature corresponding to the second oral cavity image; Based on the fusion features corresponding to the second oral image, image enhancement processing is performed on the second oral image to obtain a third oral image, and the first oral image and the third oral image are combined to form the target oral image of the target patient. The step of performing image enhancement processing on the second oral image based on the fusion features corresponding to the second oral image to obtain a third oral image includes: The fusion features corresponding to the second oral cavity image are transformed into the representation domain of the large model to obtain the feature representation information corresponding to the second oral cavity image; The feature representation information and the image representation information obtained by converting the second oral image into the representation domain of the large model are spliced ​​together to obtain model prompt information, wherein the model prompt information is used to instruct the large model to perform image enhancement processing on the second oral image under the guidance of the feature representation information; The large model processes the model prompt information to obtain a third oral image, wherein the large model is further configured to perform image enhancement processing on the second oral image related to key points of the denture structure, guided by the feature representation information in the model prompt information.

4. The method of claim 3, wherein, The fusion features corresponding to the second oral cavity image include at least one fusion domain feature corresponding to the second oral cavity image; The step of generating fusion features corresponding to the second oral cavity image based on the first oral cavity image and the second oral cavity image includes: At least one type of domain information is determined for each of the plurality of oral cavity images, wherein the at least one type of domain information corresponds one-to-one with the at least one fusion domain feature; For each type of domain information in the at least one type of domain information, the corresponding domain features are extracted from the respective domain information of the first oral cavity image and the second oral cavity image, and the corresponding domain features of the first oral cavity image and the second oral cavity image are fused based on an attention mechanism to obtain the fused domain features of the second oral cavity image under the domain information.

5. A denture three-dimensional modeling system based on oral images, characterized by, include: The image acquisition module is used to acquire target oral cavity images of the target patient; The model generation module is used to call a pre-trained three-dimensional denture generation model to generate a three-dimensional denture model corresponding to the target patient based on the target oral image, wherein the three-dimensional denture generation model is trained according to the method described in any one of claims 1-2; The acquisition of the target oral cavity image of the target patient includes: Acquire multiple oral images of the target patient; According to a preset division method, the multiple oral cavity images are divided into a first oral cavity image and a second oral cavity image to be enhanced; Based on the first oral cavity image and the second oral cavity image, generate the fusion feature corresponding to the second oral cavity image; Based on the fusion features corresponding to the second oral image, image enhancement processing is performed on the second oral image to obtain a third oral image, and the first oral image and the third oral image are combined to form the target oral image of the target patient. The step of performing image enhancement processing on the second oral image based on the fusion features corresponding to the second oral image to obtain a third oral image includes: The fusion features corresponding to the second oral cavity image are transformed into the representation domain of the large model to obtain the feature representation information corresponding to the second oral cavity image; The feature representation information and the image representation information obtained by converting the second oral image into the representation domain of the large model are spliced ​​together to obtain model prompt information, wherein the model prompt information is used to instruct the large model to perform image enhancement processing on the second oral image under the guidance of the feature representation information; The large model processes the model prompt information to obtain a third oral image, wherein the large model is further configured to perform image enhancement processing on the second oral image related to key points of the denture structure, guided by the feature representation information in the model prompt information.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-4.

7. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-4.

8. A computer device, comprising: The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-4.

Images