Method for generating simulated two-dimensional image of tooth wearing shell-shaped dental instrument
The patient's tooth-exposed photos are processed by computer, the tooth area is extracted and the edges of the shell-shaped dental instrument are simulated, and two-dimensional images of teeth that simulate wearing shell-shaped dental instruments are generated, which solves the problem of patients' lack of confidence in treatment and promotes communication between doctors and patients.
Patent Information
- Application Number
- CN202311386076.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-24
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art is difficult to show patients that wearing shell-like dental devices before treatment will not adversely affect their image, affecting the patient's confidence in treatment and communication between the orthodontist and the patient.
Through a computer-executed method, a photo of the patient's tooth-exposed oral part is obtained, and the tooth area is extracted using edge detection algorithm and deep neural network, brightening processing is performed and the edges of the shell-shaped dental instrument is simulated. It is superimposed on the photo to generate a simulated two-dimensional image of the teeth wearing shell-shaped dental instruments.
It realizes the image of the teeth that simulates wearing shell-like dental devices to patients before treatment, helps patients build confidence in treatment and promotes communication between doctors and patients.
Smart Images
Figure CN119887534A_ABST
Abstract
Description
Technical Field
[0001] The present application generally relates to methods of generating simulated two-dimensional images of teeth wearing shell-shaped dental appliances. Background Art
[0002] Nowadays, more and more people are beginning to understand that orthodontic treatment is not only good for health, but also can improve personal image. Due to its advantages in appearance, convenience and easy cleaning, orthodontic treatment using shell-shaped dental appliances is being accepted by more and more people.
[0003] For patients who are unfamiliar with orthodontic treatment, being able to demonstrate to them before treatment that wearing shell-shaped dental appliances will not adversely affect their image can help them build confidence in the treatment and promote communication between orthodontists and patients.
[0004] Therefore, it is necessary to provide a method for generating a simulated two-dimensional image of teeth wearing a shell-shaped dental appliance. Summary of the invention
[0005] One aspect of the present application provides a computer-implemented method for generating a simulated two-dimensional image of teeth wearing a shell-shaped dental appliance, comprising: obtaining a photograph including a patient's toothy mouth area; processing the mouth area image of the photograph using an edge detection algorithm to obtain gradient information of the mouth area image; extracting a mask of the tooth area from the mouth area image using a trained first deep neural network; extracting the gradient information of the tooth area image from the gradient information of the mouth area image using the mask, selecting an area to be brightened in the tooth area image based on the gradient information of the tooth area image, and brightening the selected area in the tooth area image; generating an edge of a simulated shell-shaped dental appliance toward the opposite jaw at an edge of a first tooth row in the mouth area image close to the opposite jaw side, the first tooth row being the one located in front of the maxillary tooth row and the mandibular tooth row; and superimposing the image of the brightened area and the edge of the simulated shell-shaped dental appliance on the photograph to obtain a simulated two-dimensional image of teeth wearing a shell-shaped dental appliance.
[0006] In some embodiments, the first deep neural network is an instance segmentation network or a semantic segmentation network.
[0007] In some embodiments, the method of generating a simulated two-dimensional image of teeth wearing a shell-shaped dental appliance further comprises: using a trained second deep neural network to capture the mouth area image from the photograph, wherein the second deep neural network is a target detection network.
[0008] In some embodiments, the method of generating a simulated two-dimensional image of teeth wearing a shell-shaped dental appliance further comprises: using a trained third deep neural network to determine which of the maxillary dentition and the mandibular dentition is located in front based on the oral area image.
[0009] In some embodiments, the method of generating a simulated two-dimensional image of teeth wearing a shell-shaped dental appliance further comprises: before the brightening operation, using the mask to determine the tooth area in the mouth area image, and performing white balance processing on the tooth area image to whiten it.
[0010] In some embodiments, the method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance further comprises: filtering out pixels whose gradient values are within a predetermined range from the gradient information of the tooth region image, and selecting the filtered out pixels and the pixels surrounding them as the area to be brightened.
[0011] In some embodiments, the edge of the simulated shell-shaped dental appliance is a region extending a predetermined number of pixels from an edge of the first dentition close to the opposing jaw toward the opposing jaw.
[0012] In some embodiments, the color of the edge of the simulated shell-shaped dental appliance gradually changes from gray to white from one side of the first dentition to the opposing jaw.
[0013] In some embodiments, the grayscale of the edge of the simulated shell-shaped dental appliance is positively correlated with the gradient of the corresponding portion of the edge of the first dentition on the opposing jaw side.
[0014] In some embodiments, the photograph is a frontal open-face photograph of the patient.
[0015] Another aspect of the present application provides a computer system for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance, which includes a processor and a storage device, wherein the storage device stores a computer program, and when the computer program is run, the processor will execute the method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The above and other features of the present disclosure will be more fully understood through the following description and the attached claims in conjunction with the accompanying drawings. It should be understood that these drawings only depict several embodiments of the present disclosure and should not be considered as limiting the scope of the present disclosure. By using the drawings, the present disclosure will be more clearly and detailed.
[0017] Figure 1A schematic flow chart of a method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance in one embodiment of the present application;
[0018] Figure 2 A frontal smiling photo of a patient in an example;
[0019] Figure 3 Demonstrated the use of the first deep neural network trained Figure 2 The mouth area detected in the photo shown;
[0020] Figure 4 Demonstrated the use of a second deep neural network trained Figure 3 The mask of teeth obtained by extracting the mouth area image shown;
[0021] Figure 5 Shows the selected area that needs to be brightened;
[0022] Figure 6 showing the edge of a simulated shell-shaped dental appliance; and
[0023] Figure 7 Demonstrated the use of the method of the present application based on Figure 2 The photograph shown is a simulated oral region of a two-dimensional image of teeth wearing a shell-shaped dental appliance. DETAILED DESCRIPTION
[0024] In the following detailed description, reference is made to the accompanying drawings which form a part hereof. In the drawings, similar symbols generally indicate similar components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be employed, and other changes may be made, without departing from the spirit or scope of the subject matter described herein. It should be readily understood that various configurations, substitutions, combinations, designs of various aspects of the disclosure generally described herein and illustrated in the accompanying drawings may be made, all of which are expressly contemplated and form a part of the disclosure.
[0025] One aspect of the present application provides a method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance. The method of the present application mainly generates a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance through two operations, one of which is to brighten a local area of the tooth surface in a patient's tooth photo to simulate the reflection effect of light on the shell-shaped dental appliance, and the other is to generate a simulated edge of the shell-shaped dental appliance toward the opposite jaw at the edge of the front of the maxillary dentition and the mandibular dentition in the photo, and its texture is similar to the edge of the real shell-shaped dental appliance.
[0026] The shell-shaped dental appliance is made of a transparent material and is a one-piece shell that forms a cavity to house a plurality of teeth.
[0027] Please refer to Figure 1 , which is a schematic flow chart of a method 100 for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance in one embodiment of the present application.
[0028] Another aspect of the present application provides a computer system for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance, which includes a processor and a storage device, wherein the storage device stores a computer program for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance. When the computer program is run, the processor will execute the method 100 for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance.
[0029] In 101 , a dental photograph of a patient is obtained.
[0030] Since people usually care more about their image when they smile with their teeth showing, in one embodiment, the photo can be a full frontal photo of the patient's face when he smiles with his teeth showing, and accordingly, the generated photo will be a simulated full frontal photo of the patient's face with his teeth showing and wearing a shell-shaped dental appliance. In light of this application, it can be understood that the photo can also be a photo of a part of the face, and the angle of the photo can also be other angles besides the front, and accordingly, the range and angle of the generated simulated two-dimensional image will also change accordingly.
[0031] Please refer to Figure 2 , is a toothy face photo of a patient in an example. To protect the patient's privacy, the patient's eyes are covered in the photo.
[0032] In 103, an image of the mouth area is captured from the photograph.
[0033] Compared with complete facial photos, mouth area images have fewer features, and subsequent processing based only on mouth area images can simplify calculations.
[0034] In one embodiment, the mouth area image may be captured from the photo using a trained first deep neural network.
[0035] In one embodiment, the first deep neural network can be a target detection network, which can adopt any applicable network, for example, YOLOv5, YOLOv4, YOLOv3, SSD (Single Shot MultiBox Detector), Faster R-CNN, DETR (Detection Transformer), etc.
[0036] In one embodiment, the object detection network selects the mouth area with a box.
[0037] Please refer to Figure 3 , schematically showing the target detection network Figure 2 The mouth area (the area within the box) is detected from the frontal face photo of the patient showing teeth.
[0038] In 105, a mask of the teeth region is extracted from the mouth region image.
[0039] In one embodiment, a mask of the tooth region may be extracted from the mouth region image using a trained second deep neural network.
[0040] In one embodiment, the second deep neural network can be an instance segmentation network, which can extract a mask of each tooth from the mouth region image. The instance segmentation network can adopt any applicable network, for example, YOLOv5 and Mask R-CNN.
[0041] In another embodiment, the second deep neural network may be a semantic segmentation network, which can extract the mask of the entire maxillary teeth and the mask of the entire mandibular teeth from the mouth region image. The instance segmentation network may adopt any applicable network, for example, U-Net and FCN (Fully Convolutional Networks).
[0042] Please refer to Figure 4 , schematically showing the semantic segmentation network Figure 3 The mask of the tooth area is obtained by extracting the mouth area image. Figure 2 Only the maxillary teeth are shown in the photo. Figure 4 The tooth region mask shown includes only a mask of the entirety of the exposed maxillary teeth.
[0043] In 107, white balance processing is performed on the teeth area in the mouth area image.
[0044] The inventor of the present application has found through experiments that if the tooth area is subjected to white balance processing to make it whiter, the final generated two-dimensional image of the simulated teeth wearing the shell-shaped dental appliance will be more realistic. This operation is optional.
[0045] First, the tooth region image is cut out from the mouth region image using the mask of the tooth region, and then white balance processing is performed on the tooth region image.
[0046] In one embodiment, Gray World Assumption / Algorithm can be used to balance the values of the red, blue and green channels of the tooth region to make them close to white. In the light of this application, it can be understood that in addition to the above algorithm, any other applicable white balance processing algorithm can be used to process the tooth region image.
[0047] In 109, the mouth area image is processed using an edge detection algorithm.
[0048] In one embodiment, the mouth region image may be processed using a Sobel operator to calculate a gradient value for each pixel of the mouth region image, and their collection is referred to as the gradient information of the mouth region image.
[0049] Since only the gradient information of the tooth region is needed, the gradient information of the non-tooth region in the mouth region image can be filtered out using the mask of the tooth region, leaving only the gradient information of the tooth region.
[0050] In light of the present application, it can be understood that, in addition to the Sobel operator, any other applicable edge detection algorithm can be used to process the mouth region image, for example, the Canny operator.
[0051] In 111, based on the gradient information, an area to be brightened is selected in the tooth region, and a brightening operation is performed on the selected area in the tooth region image.
[0052] According to experience, shell-shaped dental appliances are prone to reflect light on the uneven surfaces of teeth. Brightening these parts of the tooth area image can simulate the reflective effect of light on the shell-shaped dental appliance, making the final simulated two-dimensional image of the teeth wearing the shell-shaped dental appliance more realistic.
[0053] The gradient of the uneven tooth surface is larger than that of the flat surface, but smaller than that of the tooth edge. Since it is not desired to brighten the flat surface and the tooth edge, pixels with excessively large and small gradients in the tooth area can be filtered out. For the filtered pixels, the pixels within a certain range centered on them are brightened.
[0054] For a screened pixel, pixels to be brightened may be selected from its periphery based on a predetermined rule, or pixels to be brightened may be selected randomly from its periphery.
[0055] For images encoded in YUV, the brightness of a pixel can be increased by increasing the value of the Luma channel of the pixel.
[0056] In one embodiment, pixels outside a predetermined gradient range may be filtered out, and pixels within the predetermined gradient range may be retained. For example, the predetermined gradient range may be 3-15.
[0057] Please refer to Figure 5 , schematically showing the selected area to be brightened.
[0058] In one embodiment, the degree of brightening may be positively correlated with the gradient, that is, the higher the gradient, the higher the degree of brightening.
[0059] In 113, at an edge of the front side of the maxillary dentition and the mandibular dentition close to the opposite jaw in the oral region image, an edge of a simulated shell-shaped dental appliance is generated toward the opposite jaw.
[0060] In one embodiment, the trained third deep neural network can be used to determine whether the patient's teeth are overbite or underbite based on the oral region image, and based on the result of the determination, it is determined whether to generate the edge of the simulated shell-shaped dental appliance at the lower edge of the maxillary dentition or the upper edge of the mandibular dentition. In other words, the selected dentition is the one located in front of the maxillary dentition and the mandibular dentition, or in other words, the dentition that partially covers the other of the maxillary dentition and the mandibular dentition.
[0061] by Figure 3 Taking the mouth area image shown as an example, the patient's teeth are overhanging. Therefore, the lower edge of the maxillary dentition can be extracted from the tooth mask, and the range of a predetermined number of pixels from the lower edge downward is defined as the edge area of the simulated shell-shaped dental appliance, and the color of the edge area is gradually changed from gray to white from top to bottom to simulate the material characteristics of the edge of the shell-shaped dental appliance.
[0062] In one embodiment, the grayscale of the edge area can be changed according to the gradient at the edge of the tooth. The greater the gradient, the higher the grayscale.
[0063] Please refer to Figure 6 , schematically showing the Figure 3 The image of the oral region shown produces the edge of a simulated shell-shaped dental appliance.
[0064] In one embodiment, a denoising algorithm may be used to remove redundant noise points generated by the brightening operation and smooth the simulated edge. The denoising algorithm may be any applicable denoising algorithm, such as anisotropic diffusion fusion.
[0065] In 115 , the image of the tooth region subjected to the brightening operation and the edge of the generated simulated shell-shaped dental appliance are superimposed on the photo to obtain an image of a simulated tooth wearing the shell-shaped dental appliance.
[0066] In one embodiment, only the image of the brightened area can be superimposed on the photo. In another embodiment, the image of the entire tooth area that has been brightened can also be superimposed on the photo.
[0067] Please refer to Figure 7 , showing that the method of this application is based on Figure 2 The photograph shown is a simulated oral region of a two-dimensional image of teeth wearing a shell-shaped dental appliance.
[0068] Although various aspects and embodiments of the present application are disclosed herein, other aspects and embodiments of the present application will be apparent to those skilled in the art in light of the present application. The various aspects and embodiments disclosed herein are for illustrative purposes only and not for limiting purposes. The scope and subject matter of the present application are determined solely by the appended claims.
[0069] Likewise, various diagrams may illustrate exemplary architectures or other configurations of the disclosed methods and systems that aid in understanding the features and functions that may be included in the disclosed methods and systems. The claimed content is not limited to the exemplary architectures or configurations shown, and the desired features may be implemented with various alternative architectures and configurations. In addition, for flow charts, functional descriptions, and method claims, the order of blocks presented herein should not be limited to various embodiments that are implemented in the same order to perform the described functions, unless otherwise clearly indicated in the context.
[0070] Unless otherwise expressly noted, the terms and phrases used herein and their variations should be interpreted as open ended, not restrictive. In some instances, the appearance of broad words and phrases such as "one or more", "at least", "but not limited to", or other similar terms should not be understood as an intent or need to indicate a narrowing of the example where such broad terms may not be present.
Claims
1. A computer-implemented method for generating a two-dimensional image of a simulated tooth wearing a shell-shaped dental appliance, comprising: Obtain photographs that include the patient's toothy oral area; Processing the mouth region image of the photo using an edge detection algorithm to obtain gradient information of the mouth region image; Extracting a mask of a tooth region from the mouth region image using a trained first deep neural network; extracting gradient information of the tooth region image from the gradient information of the mouth region image using the mask, selecting a region to be brightened in the tooth region image based on the gradient information of the tooth region image, and performing a brightening operation on the selected region in the tooth region image; At an edge of a first dentition near an opposing jaw in the oral region image, an edge of a simulated shell-shaped dental appliance is generated toward the opposing jaw, wherein the first dentition is the front one of the maxillary dentition and the mandibular dentition; and The image of the brightened area and the edge of the simulated shell-shaped dental appliance are superimposed on the photograph to obtain a simulated two-dimensional image of a tooth wearing the shell-shaped dental appliance.
2. The method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance according to claim 1, characterized in that: The first deep neural network is an instance segmentation network or a semantic segmentation network.
3. The method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance according to claim 1, characterized in that: It also includes: using a trained second deep neural network to capture the mouth area image from the photo, wherein the second deep neural network is a target detection network.
4. The method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance according to claim 1, characterized in that: It also includes: using a trained third deep neural network to determine which of the maxillary dentition and the mandibular dentition is located in front based on the oral area image.
5. The method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance according to claim 1, characterized in that: It also includes: before the brightening operation, using the mask to determine the tooth area in the mouth area image, and performing white balance processing on the tooth area image to whiten it.
6. The method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance according to claim 1, characterized in that: It also includes: selecting pixels whose gradient values are within a predetermined range from the gradient information of the tooth region image, and selecting the selected pixels and the pixels around them as the brightened region.
7. The method of generating a two-dimensional image of a simulated tooth wearing a shell-shaped dental appliance according to claim 1, characterized in that: The edge of the simulated shell-shaped dental appliance is a region extending a predetermined number of pixels from the edge of the first tooth row close to the opposing jaw toward the opposing jaw.
8. The method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance according to claim 7, characterized in that: The color of the edge of the simulated shell-shaped dental appliance changes gradually from gray to white from one side of the first tooth row to the opposite jaw.
9. The method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance according to claim 8, characterized in that: The grayscale of the edge of the simulated shell-shaped dental appliance is positively correlated with the gradient of the corresponding portion of the edge of the first dentition on the opposing jaw side.
10. The method of generating a two-dimensional image of a simulated tooth wearing a shell-shaped dental appliance according to claim 1, characterized in that: The photograph is a frontal open-face photograph of the patient.
11. A computer system for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance, comprising a processor and a storage device, wherein the storage device stores a computer program, and when the computer program is run, the processor will execute the method for generating a simulated two-dimensional image of a tooth wearing a shell-shaped dental appliance as described in claim 1.