Tooth beautifying method and device based on deep learning

Abstract

The invention discloses a tooth beautifying method and device based on deep learning. The method comprises the following steps: acquiring image data of teeth needing to be beautified; inputting the image data into a pre-trained tooth beautifying model; taking the image data as input of a tooth beautifying model, and generating a beautifying result graph corresponding to the image data by the tooth beautifying model based on the input image data; wherein the tooth beautifying model comprises a backbone network, a plurality of types of generator modules and a discriminator module; according to the invention, a multi-task learning framework of parallel branches is integrated in a neural network, so that the teeth are beautified in all directions.

Description

Technical Field

[0001] The present invention relates to the technical field of tooth beautification, and in particular to a tooth beautification method and device based on deep learning. Background Art

[0002] Existing methods for tooth beautification mainly rely on manual editing or simple automated tools, which have problems such as long time consumption, high technical barriers, and unnatural effects. With the development of deep learning technology, it has become possible to use convolutional neural networks (CNN) and generative adversarial networks (GANs) for image processing. Among them, although the existing tooth restoration algorithms on the market have good restoration effects, this method regenerates the entire mouth of teeth and cannot retain the texture of the original teeth. There are problems such as misalignment between the restored tooth data and the original teeth, and the color and gloss of the teeth do not correspond to the original teeth. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a teeth beautification method and device based on deep learning, which realizes all-round beautification of teeth by integrating a multi-task learning framework with parallel branches in a neural network.

[0004] According to one aspect of the present invention, a method for tooth beautification based on deep learning is provided, the steps comprising: Acquire image data of teeth that need to be beautify; The image data is input into a pre-trained tooth beautification model; the image data is used as input of the tooth beautification model, and the tooth beautification model generates a beautification result image corresponding to the image data based on the input image data; wherein, The teeth beautification model includes: a backbone network, several types of generator modules, and a discriminator module; A backbone network configured to: extract a first image feature from the image data of beautified teeth; A generator module for several defect categories, which is configured to: extract a second image feature corresponding to the defect category based on the image feature, and generate a defect-removed image based on the second image feature; The discriminator module is configured to respectively receive the de-defected images generated by the generator modules of several defect categories and respectively evaluate the authenticity of the images.

[0005] In the above technical solution, the present invention aims to simultaneously complete the task of restoring and beautifying different dental defects in a generative adversarial network. The network is a multi-task learning framework containing multiple parallel branches. Each branch handles a different task. To achieve this goal, the generator and discriminator of the generative adversarial network share the underlying features, and then are divided into several task branches based on the high-level features.

[0006] In some embodiments, the generator modules for the plurality of defect categories include: a defect removal generator module, a padding generator module, and an alignment generator module.

[0007] In the above technical solution, the present invention designs the generator into a shared backbone network, and divides it into three parallel branches at the end, respectively processing the tasks of removing defects, filling and aligning teeth. Each branch outputs different image transformation results. First, it is necessary to obtain the tooth image data that needs to be beautified. These data will be provided as input to the tooth beautification model. Then, the acquired tooth image data is input into a pre-trained tooth beautification model. This model has been trained and can handle the related tasks of tooth beautification. Finally, the beautification result is generated: the tooth beautification model will generate the corresponding beautification result image based on the input image data. The tooth beautification model includes three main parts: a backbone network, several categories of generator modules and a discriminator module. Backbone network: The backbone network is responsible for extracting the first image features from the input beautified tooth image data. These features are the basis for subsequent processing and can be information such as the shape, color and texture of the teeth. Generator module: There are multiple generator modules for different categories of tooth defects. Each generator module is responsible for processing a specific type of defect, such as removing stains on teeth, filling gaps or adjusting tooth arrangement. These generator modules further extract second image features corresponding to the category of defects based on the first image features extracted by the backbone network, and use these features to generate defect-free images. The discriminator module is responsible for evaluating the authenticity of the defect-free images generated by the generator modules. It receives defect-free images from different generator modules and evaluates the realism of these images respectively. The purpose of this process is to generate tooth images that are both realistic and aesthetically pleasing while maintaining consistency with other features in the original image.

[0008] In some embodiments, the training data of the defect removal generator module is obtained by the following steps: Collect a large number of high-resolution flawless tooth images, use the tooth segmentation algorithm to extract the tooth area in the flawless tooth image, obtain the tooth mask, and perform the following optional operations on the image: a. Simulate tooth decay: by adding black and / or dark spots of any shape and size at specified positions and / or random positions on the tooth mask; b. Simulate yellow teeth: Convert the flawless tooth image to HSV color space, increase the saturation of the tooth area and reduce the brightness, and then convert it back to BGR color space; c. Simulate cracks: Draw lines of random length and width at specified locations and / or random locations on the tooth mask.

[0009] The original flawless tooth image and the flawed tooth image generated after the above operation are combined into a data pair; each pair of data includes an original flawless tooth image and a flawed tooth image generated by the above method; The training data of the defect removal generator module is constructed based on the above method. In the above technical solution, the purpose of making tooth defect removal data pairs is to generate paired data sets for training machine learning models, where each pair of data contains an original flawless tooth image and an artificially flawed tooth image. Collect a large number of high-resolution flawless tooth images. These images will serve as the basis for generating defect images, so it is necessary to ensure that their quality is high enough for subsequent image processing. The process of generating tooth defect images includes three sub-steps, each of which is used to simulate different types of tooth defects.

[0010] a. Simulate tooth decay Add black or dark spots to the tooth image using image processing software or programming libraries such as OpenCV, PIL, etc. Determine the locations in the tooth image where you want to simulate cavities and draw circular or irregular shaped spots at those locations.

[0011] b. Simulate yellow teeth Convert the tooth image from BGR color space to HSV (hue, saturation, brightness) color space. Adjust the saturation and brightness values ​​in the HSV color space to simulate the effect of yellowing teeth. Typically, this involves increasing the saturation and decreasing the brightness appropriately. Convert the adjusted image back to BGR color space.

[0012] c. Simulate cracks Draw thin black or dark lines on the tooth image to simulate cracks on the tooth. Determine the location and direction of the cracks and use image processing tools to draw lines at these locations.

[0013] A tooth mask is used to ensure that the above defect effects are only applied to the tooth area. The tooth mask can be a binary image of the same size as the tooth image, where the tooth area is white (or value 1) and the background is black (or value 0). The mask is applied to the defect image to ensure that the defect does not spread to areas outside the teeth. The generated data pair includes an original tooth image without defect and an image of the tooth with the defect added.

[0014] The generated data pairs are organized into a dataset, where each pair of images contains a folder or record containing the original image and the corresponding defect image. This dataset can be used to train machine learning models, such as generative adversarial networks (GANs), to learn how to automatically remove defects from dental images. Through this process, a high-quality dental defect removal dataset can be created for the development and training of image processing models that can automatically repair dental defects.

[0015] In some embodiments, the training data of the completion generator module is obtained by the following steps: Collect a large number of high-resolution images of teeth without gaps and perform the following operations on the images: a. Use the tooth segmentation algorithm to extract the tooth area in the tooth image without gaps and obtain the tooth mask; b. Using edge extraction algorithm and coordinate sorting on the tooth mask, obtain the lowest point of a random upper tooth or the highest point of a random lower tooth; c. Draw a random irregular polygon on the flawless tooth image with the obtained point as the center; The original tooth image without gaps and the tooth image with gaps generated after the above operation are combined into data pairs; each pair of data includes an original tooth image without gaps and a tooth image with gaps generated by the above method; The training data of the generator module is constructed based on the above method.

[0016] In the above technical solution, the purpose of making teeth filling data pairs is to generate paired data sets for training machine learning models, where each pair of data contains an original image of neat teeth and an image of teeth with artificial gaps. Collect a large number of high-resolution images of teeth without gaps. These images will serve as the basis for generating gap images, so it is necessary to ensure that their quality is high enough to facilitate subsequent image processing.

[0017] Generating tooth gap images includes three sub-steps for simulating gaps in tooth images.

[0018] a. Obtain tooth mask using tooth segmentation Use a tooth segmentation algorithm (which can be a traditional image processing method or a deep learning method) to obtain a tooth mask. This mask can be a binary image in which the tooth area is white (or the value is 1) and the background is black (or the value is 0). Use edge extraction algorithms and coordinate sorting on the tooth mask to obtain the lowest point of a random upper tooth or the highest point of a random lower tooth. Apply an edge detection algorithm (such as Canny edge detection) to the tooth mask to identify the contour of the tooth. Sort the coordinates of the detected edges to determine the lowest point (for upper teeth) or the highest point (for lower teeth) of the tooth. Select a random point as the center point of the gap, and draw an irregular polygon on the original tooth image based on the selected center point to simulate the shape of the gap. The shape, size, and position of the polygon should be randomly generated to simulate different types and degrees of tooth gaps.

[0019] For each original unchipped tooth image, generate a corresponding tooth image with a chip. Make sure that the chip only appears in the tooth area and does not affect other areas. Each pair of data includes an original unchipped tooth image and an added tooth image. Organize the generated data pairs into a dataset, and each pair of images is contained in a folder or record, which contains the original image and the corresponding chipped image. This dataset can be used to train machine learning models, such as generative adversarial networks (GANs), to learn how to automatically fill in gaps in tooth images. Through this process, a high-quality tooth filling dataset can be created for developing and training image processing models that can automatically repair tooth gaps.

[0020] In some embodiments, the training data of the alignment generator module is obtained by the following steps: Collect a large number of high-resolution images of neat teeth, use the tooth segmentation algorithm to extract the tooth area in the neat tooth image, obtain the tooth mask, and perform the following operations on the image: a. Divide the neat teeth image into several regular rectangular grids, and each grid vertex serves as the control point of the image; b. Using the tooth mask to extract the edge of any tooth, in the tooth edge area, select a control point to move a preset distance in a specific direction, and / or, move a preset distance in a random direction; c. According to the deformed mesh vertex positions, the pixel deformation caused by the movement of the control points is smoothed by bilinear interpolation to generate a natural deformed image; The original teeth image without gaps and the teeth image with gaps generated after the above operation are combined into data pairs, each pair of data includes an original uniform teeth image and an irregular teeth image generated by the above method; The training data of the generator module is constructed based on the above method.

[0021] In the above technical solution, the purpose of making teeth alignment data pairs is to generate paired data sets for training machine learning models, where each pair of data contains an original image of aligned teeth and an image of teeth with artificially added misaligned teeth.

[0022] Collect a large number of high-resolution images of straight teeth. These images will serve as the basis for generating images of irregular teeth, so it is necessary to ensure that their quality is high enough for subsequent image processing.

[0023] The process of generating crooked teeth images consists of three sub-steps to simulate malalignment in the teeth images.

[0024] a. Divide the image into grids. Divide the tooth image into several regular rectangular grids. The vertices of each grid are called control points, which determine the shape and position of the local image. The control points should be dense enough to simulate the slight movement and deformation of the teeth.

[0025] b. Define control point deformation. Use the tooth mask to extract the edges of specific teeth and determine the position and outline of the teeth. Within the tooth edge area, select specific mesh control points to move to simulate the misalignment of teeth. Control points can be moved in a specific direction or their positions can be randomly adjusted to simulate rotation, tilt or misalignment of teeth. The movement of control points should be small to keep the deformed image looking natural and realistic.

[0026] c. Image interpolation. The image is interpolated based on the position of the deformed mesh vertices. Common interpolation methods include bilinear interpolation or bicubic interpolation, which can smooth the pixel deformation caused by the movement of the control points and generate a natural deformed image. The interpolation process ensures the continuity and smoothness of the image, making the movement of the teeth look more realistic.

[0027] For each original aligned teeth image, generate a corresponding misaligned teeth image. Make sure that the deformation of the teeth only appears in the tooth area and does not affect other areas such as gums and lips. Each pair of data includes an original aligned teeth image and an image of the teeth with the misalignment added. Organize the generated data pairs into a dataset, and each pair of images is contained in a folder or record, which contains the original image and the corresponding misaligned image. This dataset can be used to train machine learning models, such as generative adversarial networks (GANs) or image inpainting models, to learn how to automatically align teeth in dental images.

[0028] According to another aspect of the present invention, a method for training a teeth beautification model is provided, based on the above-mentioned teeth beautification method based on deep learning; the teeth beautification model comprises: a backbone network, several types of generator modules and a discriminator module; the method comprises: Constructing training data sets of several defect categories, each data set contains several data pairs; each pair of data includes an original non-defective tooth image and a defective tooth image generated based on the non-defective image; Input defective tooth images in several categories of training data sets into the backbone network and extract first image features respectively; According to the defect category, the corresponding first image feature is input into the corresponding pre-built generator module, the generator module extracts the second image feature of the defect of the corresponding category based on the image feature, and generates a defect-removed image based on the second image feature; Inputting the de-defected images of several defect categories into the discriminator module respectively, and outputting the corresponding first loss values; comparing the de-defected images of several defect categories with the corresponding original defect-free images, and calculating the second loss value; A teeth beautification model is trained based on the first loss value and the second loss value.

[0029] In the above technical solution, a generative adversarial network (GAN) structure is used to process the beautification of tooth images, including the removal of tooth defects. First, a training data set containing multiple categories of defects needs to be constructed. This data set consists of multiple pairs of images, each pair of images includes an original defect-free tooth image and a corresponding defective tooth image. The defective tooth image is the result of artificially adding defects (such as cavities, yellow teeth, gaps, etc.) to the original image. The backbone network is used to extract features from the input defective tooth image. The backbone network is usually a pre-trained deep network that can extract rich feature representations from the image. According to the category of the defect, the extracted features are input into the corresponding generator module. Each generator module can further extract features related to the defect of a specific defect category and use these features to generate a tooth image with the defect removed. The generated defect-free image is input into the discriminator module to calculate the first loss value. This loss value usually reflects the difference between the generated image and the real defect-free image. The discriminator aims to distinguish the real image from the generated image. At the same time, the generated defect-free image is compared with the corresponding original defect-free image to calculate the second loss value. This loss value usually reflects the difference between the generated image and the target image and is used to optimize the performance of the generator. The first loss value and the second loss value are combined to train and optimize the teeth beautification model. This training method can improve the ability of the generator to generate high-quality defect-free images and improve the discriminant ability of the discriminator. Multi-task processing: By designing multiple generator modules, the model can handle multiple types of dental defects, improving the applicability and flexibility of the model. The shared feature extraction of the backbone network allows the model to share knowledge between different tasks, reducing the number of parameters and improving training efficiency. Through adversarial training of the generator and the discriminator, the model can generate more realistic and high-quality defect-free images.

[0030] According to another aspect of the present invention, a teeth beautification device based on deep learning is provided, based on the above-mentioned teeth beautification method based on deep learning; comprising the following modules connected in sequence: Acquisition module: used to acquire image data of teeth that need to beautify; A generation module is used to input the image data into a pre-trained tooth beautification model; the image data is used as the input of the tooth beautification model, and the tooth beautification model generates a beautification result image corresponding to the image data based on the input image data; wherein, The teeth beautification model includes: a backbone network, several types of generator modules, and a discriminator module; A backbone network configured to: extract a first image feature from the image data of beautified teeth; A generator module for several defect categories, which is configured to: extract a second image feature corresponding to the defect category based on the image feature, and generate a defect-removed image based on the second image feature; The discriminator module is configured to respectively receive the de-defected images generated by the generator modules of several defect categories and respectively evaluate the authenticity of the images.

[0031] In the above technical solution, in order to better use the above method, the present application proposes a teeth beautification device based on deep learning, and each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0032] According to another aspect of the present invention, a teeth beautification device based on deep learning is provided, comprising: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0033] In the above technical solution, in order to better run and process the method, the above method is stored in a memory, and the stored method is executed by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0034] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0035] In the above technical solution, in order to better run and use the method, the above method is stored in a computer-readable storage medium, and the above method is implemented by a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a flowchart of an embodiment of a method for teeth beautification based on deep learning according to the present invention; Figure 2 It is a flow chart of an embodiment of a training method for a teeth beautification model of the present invention; Figure 3 It is a schematic diagram of a training framework of an embodiment of a training method for a teeth beautification model of the present invention; Figure 4 It is a structural schematic diagram of an embodiment of a teeth beautification device based on deep learning of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0039] The present invention provides a teeth beautification method and device based on deep learning, which realizes all-round beautification of teeth by integrating a multi-task learning framework with parallel branches in a neural network.

[0040] Embodiment 1 See also Figure 1 , a method for tooth beautification based on deep learning, the steps comprising: A. Obtain image data of teeth that need to beautify; B. Inputting the image data into a pre-trained tooth beautification model; using the image data as input of the tooth beautification model, the tooth beautification model generates a beautification result image corresponding to the image data based on the input image data; wherein, The tooth beautification model includes: a backbone network, a defect removal generator module, a filling generator module, an alignment generator module, and a discriminator module; the present invention designs the generator into a shared backbone network, and divides it into three parallel branches at the end, respectively processing the defect removal, filling, and alignment tasks of teeth. Each branch outputs a different image transformation result.

[0041] A backbone network configured to: extract a first image feature from the image data of beautified teeth; A generator module for several defect categories, which is configured to: extract a second image feature corresponding to the defect category based on the image feature, and generate a defect-removed image based on the second image feature; The discriminator module is configured to respectively receive the de-defected images generated by the generator modules of several defect categories and respectively evaluate the authenticity of the images.

[0042] The generator modules of the several defect categories include: a defect removal generator module, a filling generator module and an alignment generator module.

[0043] It should be noted that in order to ensure the effectiveness and generalization ability of the model, the pre-trained teeth beautification model needs to be trained on a diverse dataset that includes various cases of tooth defects. To this end, the following three training datasets for three category modules are designed in this case: (1) The training data of the defect removal generator module is obtained by the following steps: Collect a large number of high-resolution flawless tooth images, use the tooth segmentation algorithm to extract the tooth area in the flawless tooth image, obtain the tooth mask, and perform the following optional operations on the image: a. Simulate tooth decay: by adding black and / or dark spots of any shape and size at specified positions and / or random positions on the tooth mask; b. Simulate yellow teeth: Convert the flawless tooth image to HSV color space, increase the saturation of the tooth area and reduce the brightness, and then convert it back to BGR color space; c. Simulate cracks: Draw lines of random length and width at specified locations and / or random locations on the tooth mask.

[0044] The original flawless tooth image and the flawed tooth image generated after the above operation are combined into a data pair; each pair of data includes an original flawless tooth image and a flawed tooth image generated by the above method; The training data of the defect removal generator module is constructed based on the above method. In this embodiment, the purpose of making tooth defect removal data pairs is to generate paired data sets for training machine learning models, where each pair of data contains an original flawless tooth image and an artificially added flawed tooth image. Collect a large number of high-resolution flawless tooth images. These images will serve as the basis for generating defect images, so it is necessary to ensure that their quality is high enough to facilitate subsequent image processing. The process of generating tooth defect images includes three sub-steps, which are used to simulate different types of tooth defects.

[0045] a. Simulate tooth decay Add black or dark spots to the tooth image using image processing software or programming libraries such as OpenCV, PIL, etc. Determine the locations in the tooth image where you want to simulate cavities and draw circular or irregular shaped spots at those locations.

[0046] b. Simulate yellow teeth Convert the tooth image from BGR color space to HSV (hue, saturation, brightness) color space. Adjust the saturation and brightness values ​​in the HSV color space to simulate the effect of yellowing teeth. Typically, this involves increasing the saturation and decreasing the brightness appropriately. Convert the adjusted image back to BGR color space.

[0047] c. Simulate cracks Draw thin black or dark lines on the tooth image to simulate cracks on the tooth. Determine the location and direction of the cracks and use image processing tools to draw lines at these locations.

[0048] A tooth mask is used to ensure that the above defect effects are only applied to the tooth area. The tooth mask can be a binary image of the same size as the tooth image, where the tooth area is white (or value 1) and the background is black (or value 0). The mask is applied to the defect image to ensure that the defect does not spread to areas outside the teeth. The generated data pair includes an original tooth image without defect and an image of the tooth with the defect added.

[0049] The generated data pairs are organized into a dataset, where each pair of images contains a folder or record containing the original image and the corresponding defect image. This dataset can be used to train machine learning models, such as generative adversarial networks (GANs), to learn how to automatically remove defects from dental images. Through this process, a high-quality dental defect removal dataset can be created for the development and training of image processing models that can automatically repair dental defects.

[0050] (2) The training data of the completion generator module is obtained by the following steps: Collect a large number of high-resolution images of teeth without gaps and perform the following operations on the images: a. Use the tooth segmentation algorithm to extract the tooth area in the tooth image without gaps and obtain the tooth mask; b. Using edge extraction algorithm and coordinate sorting on the tooth mask, obtain the lowest point of a random upper tooth or the highest point of a random lower tooth; c. Draw a random irregular polygon on the flawless tooth image with the obtained point as the center; The original tooth image without gaps and the tooth image with gaps generated after the above operation are combined into data pairs; each pair of data includes an original tooth image without gaps and a tooth image with gaps generated by the above method; The training data of the generator module is constructed based on the above method.

[0051] In this embodiment, the purpose of making teeth filling data pairs is to generate paired data sets for training machine learning models, where each pair of data contains an original neat teeth image and an artificially added gap teeth image. Collect a large number of high-resolution gap-free teeth images. These images will serve as the basis for generating gap images, so it is necessary to ensure that their quality is high enough to facilitate subsequent image processing.

[0052] Generating tooth gap images includes three sub-steps for simulating gaps in tooth images.

[0053] a. Obtain tooth mask using tooth segmentation Use a tooth segmentation algorithm (which can be a traditional image processing method or a deep learning method) to obtain a tooth mask. This mask can be a binary image in which the tooth area is white (or the value is 1) and the background is black (or the value is 0). Use edge extraction algorithms and coordinate sorting on the tooth mask to obtain the lowest point of a random upper tooth or the highest point of a random lower tooth. Apply an edge detection algorithm (such as Canny edge detection) to the tooth mask to identify the contour of the tooth. Sort the coordinates of the detected edges to determine the lowest point (for upper teeth) or the highest point (for lower teeth) of the tooth. Select a random point as the center point of the gap, and draw an irregular polygon on the original tooth image based on the selected center point to simulate the shape of the gap. The shape, size, and position of the polygon should be randomly generated to simulate different types and degrees of tooth gaps.

[0054] For each original unchipped tooth image, generate a corresponding tooth image with a chip. Make sure that the chip only appears in the tooth area and does not affect other areas. Each pair of data includes an original unchipped tooth image and an added tooth image. Organize the generated data pairs into a dataset, and each pair of images is contained in a folder or record, which contains the original image and the corresponding chipped image. This dataset can be used to train machine learning models, such as generative adversarial networks (GANs), to learn how to automatically fill in gaps in tooth images. Through this process, a high-quality tooth filling dataset can be created for developing and training image processing models that can automatically repair tooth gaps.

[0055] (3) The training data of the alignment generator module is obtained by the following steps: Collect a large number of high-resolution images of neat teeth, use the tooth segmentation algorithm to extract the tooth area in the neat tooth image, obtain the tooth mask, and perform the following operations on the image: a. Divide the neat teeth image into several regular rectangular grids, and each grid vertex serves as the control point of the image; b. Using the tooth mask to extract the edge of any tooth, in the tooth edge area, select a control point to move a preset distance in a specific direction, and / or, move a preset distance in a random direction; c. According to the deformed mesh vertex positions, the pixel deformation caused by the movement of the control points is smoothed by bilinear interpolation to generate a natural deformed image; The original teeth image without gaps and the teeth image with gaps generated after the above operation are combined into data pairs, each pair of data includes an original uniform teeth image and an irregular teeth image generated by the above method; The training data of the generator module is constructed based on the above method.

[0056] In this embodiment, the purpose of making teeth alignment data pairs is to generate paired data sets for training machine learning models, where each pair of data includes an original image of aligned teeth and an image of teeth with artificially added misaligned teeth.

[0057] Collect a large number of high-resolution images of straight teeth. These images will serve as the basis for generating images of irregular teeth, so it is necessary to ensure that their quality is high enough for subsequent image processing.

[0058] The process of generating crooked teeth images consists of three sub-steps to simulate malalignment in the teeth images.

[0059] a. Divide the image into grids. Divide the tooth image into several regular rectangular grids. The vertices of each grid are called control points, which determine the shape and position of the local image. The control points should be dense enough to simulate the slight movement and deformation of the teeth.

[0060] b. Define control point deformation. Use the tooth mask to extract the edges of specific teeth and determine the position and outline of the teeth. Within the tooth edge area, select specific mesh control points to move to simulate the misalignment of teeth. Control points can be moved in a specific direction or their positions can be randomly adjusted to simulate rotation, tilt or misalignment of teeth. The movement of control points should be small to keep the deformed image looking natural and realistic.

[0061] c. Image interpolation. The image is interpolated based on the position of the deformed mesh vertices. Common interpolation methods include bilinear interpolation or bicubic interpolation, which can smooth the pixel deformation caused by the movement of the control points and generate a natural deformed image. The interpolation process ensures the continuity and smoothness of the image, making the movement of the teeth look more realistic.

[0062] For each original aligned teeth image, generate a corresponding misaligned teeth image. Make sure that the deformation of the teeth only appears in the tooth area and does not affect other areas such as gums and lips. Each pair of data includes an original aligned teeth image and an image of the teeth with the misalignment added. Organize the generated data pairs into a dataset, and each pair of images is contained in a folder or record, which contains the original image and the corresponding misaligned image. This dataset can be used to train machine learning models, such as generative adversarial networks (GANs) or image inpainting models, to learn how to automatically align teeth in dental images.

[0063] In this embodiment, the present invention designs the generator as a shared backbone network, and divides it into three parallel branches at the end, respectively processing the tasks of removing defects, filling and aligning teeth. Each branch outputs different image transformation results. First, it is necessary to obtain the tooth image data that needs to be beautified. These data will be provided as input to the tooth beautification model. Then, the acquired tooth image data is input into a pre-trained tooth beautification model. This model has been trained and can handle the related tasks of tooth beautification. Finally, the beautification result is generated: the tooth beautification model will generate the corresponding beautification result image based on the input image data. The tooth beautification model includes three main parts: a backbone network, several categories of generator modules and a discriminator module. Backbone network: The backbone network is responsible for extracting the first image features from the input beautified tooth image data. These features are the basis for subsequent processing and can be information such as the shape, color and texture of the teeth. Generator module: There are multiple generator modules for different categories of tooth defects. Each generator module is responsible for processing a specific type of defect, such as removing stains on teeth, filling gaps or adjusting tooth alignment. These generator modules further extract second image features of the corresponding category defects based on the first image features extracted by the backbone network, and use these features to generate defect-free images. The discriminator module is responsible for evaluating the authenticity of the defect-free images generated by the generator module. It receives defect-free images from different generator modules and evaluates the realism of these images respectively. The purpose of this process is to generate dental images that are both realistic and beautiful while maintaining consistency with other features in the original image. It should be noted that during reasoning, if all three effects are needed, they will pass through the three branches separately. For example, first go through the defect branch to get the defect removal result, then go through the filling branch to get the filling result, and finally go through the alignment branch to get the alignment result, and then finally output the result image.

[0064] Embodiment 2 See also Figure 2 A training method for a teeth beautification model is based on a teeth beautification method based on deep learning described in one of the embodiments; the teeth beautification model includes: a backbone network, several types of generator modules and a discriminator module; the method includes: C. Construct training data sets of several defect categories, each of which contains several data pairs; each pair of data includes an original defect-free tooth image and a defective tooth image generated based on the defect-free image; D. Input defective tooth images in several categories of training data sets into the backbone network and extract the first image features respectively; E. According to the defect category, the corresponding first image feature is input into the corresponding pre-built generator module, and the generator module extracts the second image feature of the corresponding category defect based on the image feature, and generates a defect-removed image based on the second image feature; F. Inputting the de-defected images of several defect categories into the discriminator module respectively, and outputting the corresponding first loss values; and calculating the second loss values ​​based on comparing the de-defected images of several defect categories with the corresponding original defect-free images; G. Based on the first loss value and the second loss value, train a teeth beautification model.

[0065] In this embodiment, the present invention aims to simultaneously complete the tasks of tooth defect removal, filling and alignment in a generative adversarial network, which is a multi-task learning framework containing three parallel branches. Each branch handles different tasks - defect removal, filling and alignment. To achieve this goal, the generator and discriminator of the generative adversarial network share the underlying features, and then are divided into three task branches on the high-level features. The present invention uses a generative adversarial network (GAN) structure to process the beautification of tooth images, including the removal of tooth defects. Generator of Generative Adversarial Network (GAN): The present invention designs the generator as a shared backbone network and divides it into three parallel branches at the end to handle the tasks of tooth defect removal, filling and alignment respectively. Each branch outputs a different image transformation result. Discriminator of Generative Adversarial Network (GAN): The present invention designs the discriminator as a multi-task structure to determine whether the input image is a real image, and at the same time judge the effects of defect removal, filling and alignment.

[0066] The training method first needs to construct a training dataset containing multiple categories of defects. This dataset consists of multiple pairs of images, each pair of images includes an original defect-free tooth image and a corresponding defective tooth image. The defective tooth image is the result of artificially adding defects (such as cavities, yellow teeth, gaps, etc.) to the original image. Use the backbone network to extract features from the input defective tooth image. The backbone network is usually a pre-trained deep network that can extract rich feature representations from the image. According to the category of the defect, the extracted features are input into the corresponding generator module. Each generator module can further extract features related to the defect of a specific defect category and use these features to generate a tooth image with the defect removed. The generated defect-free image is input into the discriminator module to calculate the first loss value. This loss value usually reflects the difference between the generated image and the real defect-free image. The goal of the discriminator is to distinguish between the real image and the generated image. At the same time, the generated defect-free image is compared with the corresponding original defect-free image to calculate the second loss value. This loss value usually reflects the difference between the generated image and the target image and is used to optimize the performance of the generator. The first loss value and the second loss value are combined to train and optimize the tooth beautification model. This training method can improve the generator's ability to generate high-quality defect-free images and improve the discriminator's ability to discriminate. Multi-task processing: By designing multiple generator modules, the model can handle multiple types of tooth defects, improving the applicability and flexibility of the model. The shared feature extraction of the backbone network allows the model to share knowledge between different tasks, reducing the number of parameters and improving training efficiency. Through adversarial training of the generator and the discriminator, the model can generate more realistic and high-quality defect-free images.

[0067] See also Figure 3 According to the above description, the training flow chart of the present invention is shown in the figure below, which shows a multi-task generative adversarial network for tooth defect removal, filling and alignment. The generator includes a backbone network and three branches, which handle different tasks respectively. The original image is processed from the generator through each branch to the discriminator for discrimination. The loss calculation includes two parts: GAN Loss is fed back from the discriminator to the generator to improve the authenticity of the generated image; VGG Loss is directly fed back from the generated image to optimize the perceived similarity. The loss feedback represented by the dotted line is distinguished from the forward network transmission represented by the solid line to clarify the action paths of the two.

[0068] The present invention uses VGG Loss and GAN Loss to evaluate the generated tooth images, and calculates the difference between the image pixels with the images before blemish removal, before filling, and before alignment, so as to evaluate the performance of the network.

[0069] Among them, the GAN Loss formula is:

[0070] in, represents the loss function of GAN, Represents the real data distribution To have expectations, represents the real data sample, represents the output of the discriminator D for the real data sample x, which represents the probability that the sample x is judged as the real data, Represents the noise distribution To have expectations, represents the noise vector sampled from the noise distribution, represents the output of the generator G to the noise z, represents the generated fake sample, Denotes the discriminator D to generate samples The output evaluation of The probability of being classified as false data.

[0071] The VGG Loss formula is:

[0072] in, represents the VGG perceptual loss function, represents the number of samples, Represents the feature extraction function of the VGG network, usually referring to the output of a specific layer, Represents the generator G for the i-th noise The output of represents the generated fake sample, represents the i-th true sample, represents the L2 norm, which is used to calculate the Euclidean distance between two vectors. Represents the VGG network to generate samples The extracted features, Represents the VGG network for real samples Extracted features.

[0073] Embodiment 3 See also Figure 4 , a teeth beautification device based on deep learning, based on the above-mentioned teeth beautification method based on deep learning; comprising the following modules connected in sequence: Acquisition module: used to acquire image data of teeth that need to beautify; A generation module is used to input the image data into a pre-trained tooth beautification model; the image data is used as the input of the tooth beautification model, and the tooth beautification model generates a beautification result image corresponding to the image data based on the input image data; wherein, The teeth beautification model includes: a backbone network, several types of generator modules, and a discriminator module; A backbone network configured to: extract a first image feature from the image data of beautified teeth; A generator module for several defect categories, which is configured to: extract a second image feature corresponding to the defect category based on the image feature, and generate a defect-removed image based on the second image feature; The discriminator module is configured to respectively receive the de-defected images generated by the generator modules of several defect categories and respectively evaluate the authenticity of the images.

[0074] In this embodiment, in order to better use the method described in one of the embodiments, the present application proposes a teeth beautification device based on deep learning, and each module corresponds to each step of the above method. The specific principles have been described above and will not be repeated here.

[0075] Embodiment 4 A teeth beautification device based on deep learning, comprising: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in one of the embodiments.

[0076] In this embodiment, in order to better run and process the method described in one of the embodiments, the above method is stored in a memory, and a processor is used to execute the stored method. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0077] Embodiment 5 A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the method described in one of the embodiments is implemented.

[0078] In this embodiment, in order to better run and use the method, the above method is stored in a computer-readable storage medium, and the above method is implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be described in detail here.

[0079] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for tooth beautification based on deep learning, characterized in that: The steps include: Acquire image data of teeth that need to be beautify; The image data is input into a pre-trained tooth beautification model; the image data is used as input of the tooth beautification model, and the tooth beautification model generates a beautification result image corresponding to the image data based on the input image data; wherein, The teeth beautification model includes: a backbone network, several types of generator modules, and a discriminator module; A backbone network configured to: extract a first image feature from the image data of beautified teeth; A generator module for several defect categories, which is configured to: extract a second image feature corresponding to the defect category based on the image feature, and generate a defect-removed image based on the second image feature; The discriminator module is configured to respectively receive the de-defected images generated by the generator modules of several defect categories and respectively evaluate the authenticity of the images.

2. A method for tooth beautification based on deep learning as claimed in claim 1, characterized in that: The generator modules of the several defect categories include: a defect removal generator module, a filling generator module and an alignment generator module.

3. A method for tooth beautification based on deep learning as claimed in claim 2, characterized in that: The training data of the defect removal generator module is obtained by the following steps: Collect a large number of high-resolution flawless tooth images, use the tooth segmentation algorithm to extract the tooth area in the flawless tooth image, obtain the tooth mask, and perform the following optional operations on the image: a. Simulate tooth decay: by adding black and / or dark spots of any shape and size at specified positions and / or random positions on the tooth mask; b. Simulate yellow teeth: Convert the flawless tooth image to HSV color space, increase the saturation of the tooth area and reduce the brightness, and then convert it back to BGR color space; c. Simulate cracks: draw lines of random length and width at specified locations and / or random locations on the tooth mask; The original flawless tooth image and the flawed tooth image generated after the above operation are combined into a data pair; each pair of data includes an original flawless tooth image and a flawed tooth image generated by the above method; The training data of the defect removal generator module is constructed based on the above method.

4. A method for tooth beautification based on deep learning as claimed in claim 2, characterized in that: The training data of the completion generator module is obtained by the following steps: Collect a large number of high-resolution images of teeth without gaps and perform the following operations on the images: a. Use the tooth segmentation algorithm to extract the tooth area in the tooth image without gaps and obtain the tooth mask; b. Using edge extraction algorithm and coordinate sorting on the tooth mask, obtain the lowest point of a random upper tooth or the highest point of a random lower tooth; c. Draw a random irregular polygon on the flawless tooth image with the obtained point as the center; The original tooth image without gaps and the tooth image with gaps generated after the above operation are combined into data pairs; each pair of data includes an original tooth image without gaps and a tooth image with gaps generated by the above method; The training data of the generator module is constructed based on the above method.

5. The method for tooth beautification based on deep learning as claimed in claim 2, characterized in that: The training data of the alignment generator module is obtained by the following steps: Collect a large number of high-resolution images of neat teeth, use the tooth segmentation algorithm to extract the tooth area in the neat tooth image, obtain the tooth mask, and perform the following operations on the image: a. Divide the neat teeth image into several regular rectangular grids, and each grid vertex serves as the control point of the image; b. Extract the edge of any tooth using the tooth mask, and select a control point in the tooth edge area to move a preset distance in a specific direction, and / or, move a preset distance in a random direction; c. According to the position of the deformed mesh vertices, the pixel deformation caused by the movement of the control points is smoothed by bilinear interpolation to generate a natural deformed image; The original teeth image without gaps and the teeth image with gaps generated after the above operation are combined into data pairs, each pair of data includes an original uniform teeth image and an irregular teeth image generated by the above method; The training data of the generator module is constructed based on the above method.

6. A method for training a tooth beautification model, characterized in that: A method for tooth beautification based on deep learning according to any one of claims 1 to 5; The tooth beautification model includes: a backbone network, several types of generator modules and a discriminator module; the method includes: Constructing training data sets of several defect categories, each data set contains several data pairs; each pair of data includes an original non-defective tooth image and a defective tooth image generated based on the non-defective image; Input defective tooth images in several categories of training data sets into the backbone network and extract first image features respectively; According to the defect category, the corresponding first image feature is input into the corresponding pre-built generator module, the generator module extracts the second image feature of the defect of the corresponding category based on the image feature, and generates a defect-removed image based on the second image feature; Inputting the de-defected images of several defect categories into the discriminator module respectively, and outputting the corresponding first loss values; comparing the de-defected images of several defect categories with the corresponding original defect-free images, and calculating the second loss value; A teeth beautification model is trained based on the first loss value and the second loss value.

7. A tooth beautification device based on deep learning, characterized in that: A method for tooth beautification based on deep learning according to any one of claims 1 to 5; comprising the following modules connected in sequence: Acquisition module: used to acquire image data of teeth that need to beautify; A generation module is used to input the image data into a pre-trained tooth beautification model; the image data is used as the input of the tooth beautification model, and the tooth beautification model generates a beautification result image corresponding to the image data based on the input image data; wherein, The teeth beautification model includes: a backbone network, several types of generator modules, and a discriminator module; A backbone network configured to: extract a first image feature from the image data of beautified teeth; A generator module for several defect categories, which is configured to: extract a second image feature corresponding to the defect category based on the image feature, and generate a defect-removed image based on the second image feature; The discriminator module is configured to respectively receive the de-defected images generated by the generator modules of several defect categories and respectively evaluate the authenticity of the images.

8. A tooth beautification device based on deep learning, characterized in that: include: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images