Method for detecting dental disease based on dental tissue image

Abstract

One aspect of the invention provides a computer-implemented method for detecting dental diseases based on a dental tissue image. The method comprises the following steps: acquiring a dental tissue image of a patient; and detecting a dental disease based on the dental tissue image by using a trained first deep neural network, the first deep neural network comprising a feature extraction module and a dental disease detection module, the training of the first deep neural network comprising a first stage of training and a second stage of training, the first-stage training is to train the feature extraction module for pre-tasks by using a first training set, at least one of the pre-tasks is trained in a self-supervised training mode in the first-stage training, and the second-stage training is to train the whole deep neural network by using a second training set, and at least one of the pre-tasks is trained in a self-supervised training mode in the second-stage training. At least part of the training data in the first training set is unlabeled training data, and the training data in the second training set is labeled training data of dental diseases.

Description

Technical Field

[0001] The present application generally relates to a method for detecting dental diseases based on dental tissue images, and more particularly to a method for detecting dental diseases based on dental tissue images using a feature extractor trained by self-supervision. Background Art

[0002] With the development of computer vision, deep learning, and medical image analysis technologies, the automated detection of oral diseases based on intraoral tissue photographs has been significantly improved in accuracy and efficiency.

[0003] Deep learning, especially convolutional neural network (CNN), has performed well in image classification, object detection and semantic segmentation.

[0004] However, on the one hand, the quality of intraoral tissue photos taken with different equipment and conditions varies greatly, which may affect the generalization ability of the model. On the other hand, accurate labeling of dental tissue lesions requires professional knowledge and is difficult, especially for complex cases. It is difficult and costly to obtain a large amount of training data for training deep learning models. On the other hand, tooth morphology and lesion characteristics vary from individual to individual, and deep learning models need to be robust enough to cope with such changes.

[0005] In view of the above difficulties in using deep learning to detect oral diseases based on intraoral tissue photos, it is necessary to provide a new method for detecting dental diseases based on dental tissue photos. Summary of the invention

[0006] One aspect of the present application provides a computer-implemented method for detecting dental diseases based on dental tissue images, comprising: acquiring a dental tissue image of a patient; and detecting dental diseases based on the dental tissue image using a trained first deep neural network, wherein the first deep neural network comprises a feature extraction module and a dental disease detection module, and the training of the first deep neural network comprises first and second stage training, wherein the first stage training is to train the feature extraction module on a pre-task using a first training set, and in the first stage training, at least one of the pre-tasks is trained in a self-supervised training manner, and the second stage training is to train the entire deep neural network using a second training set, wherein at least part of the training data in the first training set is unlabeled training data, and the training data in the second training set is labeled training data for dental diseases.

[0007] In some embodiments, the pre-task includes contrastive learning and mask image modeling.

[0008] In some embodiments, the computer-implemented method for detecting dental diseases based on dental tissue images also includes: detecting the position of each tooth based on the dental tissue image; cropping the features extracted by the feature extraction module based on the tooth position information; and inputting the cropped features into the dental disease detection network respectively to obtain dental disease detection results for the areas corresponding to each tooth in the dental tissue image.

[0009] In some embodiments, the computer-implemented method for detecting dental diseases based on dental tissue images also includes: detecting the tooth number of each tooth therein based on the dental tissue image; and assigning a corresponding tooth number to the area corresponding to each tooth in the dental tissue image.

[0010] In some embodiments, the tooth position information is obtained using a trained second deep neural network.

[0011] In some embodiments, the tooth number is obtained using a trained third deep neural network.

[0012] In some implementations, the second and third deep neural networks are the same neural network.

[0013] In some embodiments, the feature extraction module is a Vision Transformer.

[0014] In some embodiments, the dental disease detection module is a Transformer.

[0015] Another aspect of the present application provides a computer execution system for detecting dental diseases based on dental tissue images, which includes: a processor; and a storage device, wherein the storage device stores a computer program for detecting dental diseases based on dental tissue images, and when it is run, the processor will execute the method for detecting dental diseases based on dental tissue images as described in claim 1. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features of the present application will be further described below in conjunction with the accompanying drawings and detailed descriptions thereof. It should be understood that these drawings only illustrate several exemplary embodiments according to the present application and should not be considered as limiting the scope of protection of the present application. Unless otherwise specified, the drawings are not necessarily to scale and similar reference numerals represent similar components.

[0017] Figure 1 A schematic flow chart of a method for detecting dental diseases based on dental tissue images executed by a computer in one embodiment of the present application; Figures 2A-2CThese are intraoral photos of a patient taken from three angles: left, middle, and right; Figure 3 A schematic module diagram of a deep neural network in one embodiment of the present application; Figures 4A-4D Respectively Figure 2B The partial images corresponding to four different teeth are cropped out; Figure 5A is a dental tissue image to be detected in an example; and Figure 5B In order to utilize the deep neural network based Figure 5A The dental tissue image shown is used to detect dental diseases. DETAILED DESCRIPTION

[0018] The following detailed description refers to the drawings that form part of this specification. The illustrative embodiments mentioned in the specification and drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present application. Under the guidance of the present application, those skilled in the art will understand that many other embodiments can be adopted and various changes can be made to the described embodiments without departing from the subject matter and scope of protection of the present application. It should be understood that the various aspects of the present application described and illustrated herein can be arranged, replaced, combined, separated and designed according to many different configurations, and these different configurations are within the scope of protection of the present application.

[0019] One aspect of the present application provides a computer-implemented method for detecting dental diseases based on dental tissue images, which uses a trained deep neural network to detect dental diseases in dental tissue images. The deep neural network includes a feature extraction module and a dental disease detection module, and the training of the deep neural network includes two stages of training, the first stage of training is to use a first training set to train only the feature extraction module in a self-supervised training manner, and the second stage of training is to use a second training set to train the entire deep neural network, wherein at least part of the training data of the first training set is unlabeled training data, and the training data of the second training set is labeled training data.

[0020] Another aspect of the present application provides a computer system for detecting dental diseases based on dental tissue images, which includes a processor and a storage device, wherein the storage device stores a computer program for detecting dental diseases based on dental tissue images, and when it is run, the processor will execute the method for detecting dental diseases based on dental tissue images.

[0021] There are many types of oral diseases, including caries, periodontal disease, wedge defects, etc. Early detection of these diseases can effectively avoid more serious oral health problems.

[0022] With the development of medical imaging technology, there are more and more ways and types to obtain dental tissue images, such as traditional photos (such as non-radiographic black and white or color photos), X-rays, and cone beam computed tomography (CBCT) images. Modern imaging equipment can generate high-resolution images, making it possible to accurately detect diseases using computers.

[0023] The traditional method of using deep neural networks to detect diseases based on dental tissue images requires training deep neural networks with large-scale labeled data. However, it is difficult and expensive to obtain a large amount of labeled data. In addition, the traditional method also faces the following challenges: the differences between dental tissue images taken with different equipment and conditions may be large, which is likely to affect the generalization ability of the model; tooth morphology and lesion characteristics vary from individual to individual, and the model needs to be robust enough to cope with such changes.

[0024] In the process of training the deep neural network of the present application, the feature extraction module is first pre-trained with a large amount of unlabeled training data, wherein at least one pre-task in the pre-training is trained in a self-supervised training manner, and then the entire network is fine-tuned with a small amount of labeled data. Since self-supervised training does not rely on large-scale labeled data, a large amount of unlabeled data can be used to train the feature extraction module, which greatly reduces the dependence of the deep neural network training of the present application on labeled data. In addition, since a large number of unlabeled dental tissue images taken under different equipment and conditions and from different patients are relatively easy to obtain, training the feature extraction module in this way can enable it to effectively learn the features of these images, thereby improving the generalization ability and robustness of the network.

[0025] If the deep neural network of the present application is trained with multiple types of dental tissue images, such as traditional dental tissue photos, dental tissue X-rays, and dental tissue CBCT images, then it can detect dental diseases based on any of these types of dental tissue images. For ease of explanation, the method of the present application is described in detail below using intraoral photos as an example.

[0026] Please refer to Figure 1 , which is a schematic flowchart of a method 100 for detecting dental diseases based on dental tissue images and executed by a computer in one embodiment of the present application.

[0027] In 101 , a dental tissue image is acquired.

[0028] In one embodiment, the dental tissue image may be an intraoral photograph. An intraoral photograph is a commonly used image for dental diagnosis, which can be obtained by taking a photo with a camera after the patient's lips are stretched open with a lip stretcher. Each intraoral photograph includes multiple teeth and the soft tissues near them. For the same patient, multiple intraoral photographs may be taken from different angles.

[0029] Please refer to Figures 2A-2C , which are intraoral photos of a patient taken from three angles: left, middle, and right.

[0030] In 103 , dental diseases are detected based on the dental tissue image using the trained deep neural network.

[0031] The dental tissue image is input into the deep neural network, which detects dental diseases based on the former and outputs the detection result.

[0032] Please refer to Figure 3 , which is a schematic module diagram of a deep neural network 200 in one embodiment of the present application.

[0033] The deep neural network 200 includes a feature extraction module 201 , a tooth position information detection network 203 and a dental disease detection module 205 .

[0034] In order to reduce the dependence of the training of the deep neural network 200 on the labeled data, the training is divided into two stages. The first stage of training is to use a large number of first training data sets including unlabeled data to train the feature extraction module 201, which can also be called pre-training.

[0035] In one embodiment, the following pre-tasks (PreText Task) may be designed for the pre-training: contrastive learning, masked image modeling, tooth number prediction, and multi-view tooth consistency judgment.

[0036] In one embodiment, the contrastive learning may adopt the SimCLR (Simple Contrastive Learning) method. It is understood that, in addition to the SimCLR method, the contrastive learning may also adopt any other applicable method, for example, MoCo (Momentum Contrast), SwAV (Swap Assignment Views), SimSiam (Simple Siamese), or DINO (Deep In-Depth Navigation) and the like.

[0037] In one embodiment, the mask image modeling may adopt the MAE (Masked Autoencoder) method. It is understood that in addition to the MAE method, the mask image modeling may also adopt any other applicable method, such as BEIT (Bidirectional Encoder representation from Image Transformers) or VAE (Variational AutoEncoder) and other methods.

[0038] Tooth number prediction is to classify the pixels belonging to a tooth in the dental tissue image as the tooth number corresponding to the tooth. The first training data set includes dental tissue images with labeled tooth numbers, which are used for training the tooth number prediction task.

[0039] Multi-view tooth consistency judgment is to identify teeth with the same tooth number in different dental tissue images. The training of this pre-task is helpful for correctly identifying the tooth number in dental tissue images that have been rotated or taken at abnormal angles.

[0040] In addition to the above labeled data, the first training data set may also include a large amount of unlabeled data for self-supervised training of the feature extraction module 201 for some pre-tasks. In the above embodiment, the two pre-tasks of contrastive learning and mask image modeling are trained in a self-supervised training manner.

[0041] In one embodiment, in order to enable the feature extraction module 201 to learn features at different levels, a trained tooth position information detection network can be used to crop at least a portion of the dental tissue images in the first training data set into corresponding local images according to the teeth, and then these local images are mixed with the original images to obtain the first training data set.

[0042] Please refer to Figures 4A-4D , respectively Figure 2B Partial images corresponding to four different teeth cropped from .

[0043] In one embodiment, the feature extraction module 201 may use a network based on Vision Transformer (ViT for short). It is understandable that in addition to ViT, the feature extraction module 201 may also use other applicable network structures, such as ResNet (Residual Network) or EfficientNet.

[0044] The second stage of training is to use the second training data set with labeled dental diseases to train the deep neural network 200 as a whole. This stage of training can also be called fine-tuning.

[0045] In one embodiment, the deep neural network 200 can be trained to detect corresponding dental diseases as required, as long as the second training data set includes training data labeled with corresponding dental diseases. For example, the deep neural network 200 can be trained to detect the following dental diseases: caries, soft plaque, demineralization, tartar, pigmentation, periodontal disease, residual root, residual crown, pulp lesions, and gingival abnormalities.

[0046] In the second stage of training, the dental tissue image used for training is input into the feature extraction module 201 and the tooth position information detection network 203, wherein the latter is a trained deep neural network that can detect the position information of each tooth based on the dental tissue image. In one embodiment, the tooth position information detection network 203 can adopt a network based on SOLO (Segmenting Objects by Locations). It can be understood that in addition to the SOLO network, the tooth position information detection network 203 can also adopt other applicable networks, such as Faster R-CNN (Region with Convolutional) or YOLO (You Only Look Once) network.

[0047] In one embodiment, the features extracted by the feature extraction module 201 can be clipped according to the tooth position information to obtain the features of the regions where each tooth is located, and the features are respectively input into the dental disease detection module 205 to detect whether there is a dental disease in these tooth regions and the degree of the disease. At the same time, the trained pre-task can also predict the corresponding tooth number for these tooth regions.

[0048] In one embodiment, the dental disease detection module 205 may use a Transformer-based network. It is understood that in addition to the Transformer, the dental disease detection module 205 may also use any other applicable network, such as MLP (Multilayer Perceptron). Dental disease detection is equivalent to the downstream task of the pre-training.

[0049] In order to intuitively display the dental disease detection results, in one embodiment, the dental disease detection results can be displayed in a combination of text and images, that is, the detected dental disease area is marked in the dental tissue image, and the type and degree of the dental disease are marked with text. It can be understood that in addition to the above forms of displaying the dental disease detection results, any other forms of displaying the dental disease detection results can be used according to needs.

[0050] Please refer to Figure 5A , is an example of a dental tissue image to be detected. Figure 5B To use deep neural network 200 based Figure 5AThe dental tissue image shown is used to detect dental diseases.

[0051] In yet another embodiment, only the presence or absence of dental disease and the type of dental disease present may be detected without detecting the extent of the dental disease.

[0052] In another embodiment, the dental disease detection results may not be divided into regions according to teeth and tooth numbers may not be assigned. Accordingly, the pre-training may not include the two pre-tasks of tooth number prediction and multi-view tooth consistency judgment.

[0053] In another embodiment, the tooth position information detection network 203 may be replaced by a deep neural network trained to detect tooth number and tooth position. In this embodiment, the pre-training may not include the two pre-tasks of tooth number prediction and multi-view tooth consistency judgment.

[0054] In this embodiment, the dental tissue image is respectively input into the feature extraction module 201 and the tooth number and tooth position detection network. The features extracted by the feature extraction module 201 are cropped and assigned tooth numbers based on the tooth position information and tooth numbers detected by the tooth number and tooth position detection network. The cropped features are then respectively input into the dental disease detection module 205 to obtain the dental disease detection results of the areas of the dental tissue image corresponding to each tooth.

[0055] Although various aspects and embodiments of the present application are disclosed herein, other aspects and embodiments of the present application will be apparent to those skilled in the art in light of the present application. The various aspects and embodiments disclosed herein are for illustrative purposes only and not for limiting purposes. The scope and subject matter of the present application are determined solely by the appended claims.

[0056] Likewise, various diagrams may illustrate exemplary architectures or other configurations of the disclosed methods and systems that aid in understanding the features and functions that may be included in the disclosed methods and systems. The claimed content is not limited to the exemplary architectures or configurations shown, and the desired features may be implemented with various alternative architectures and configurations. In addition, for flow charts, functional descriptions, and method claims, the order of blocks presented herein should not be limited to various embodiments that are implemented in the same order to perform the described functions, unless otherwise clearly indicated in the context.

[0057] Unless otherwise expressly noted, the terms and phrases used herein and their variations should be interpreted as open ended, not restrictive. In some instances, the appearance of broad words and phrases such as "one or more", "at least", "but not limited to", or other similar terms should not be understood as an intent or need to indicate a narrowing of the example where such broad terms may not be present.

Claims

1. A computer-implemented method for detecting dental diseases based on dental tissue images, comprising: obtaining an image of dental tissue of the patient; as well as Dental diseases are detected based on the dental tissue image using a trained first deep neural network, wherein the first deep neural network includes a feature extraction module and a dental disease detection module, and the training of the first deep neural network includes first and second stage training, the first stage training is to train the feature extraction module on a pre-task using a first training set, in the first stage training, at least one of the pre-tasks is trained in a self-supervised training manner, the second stage training is to train the entire deep neural network using a second training set, at least part of the training data in the first training set is unlabeled training data, and the training data in the second training set is training data labeled with dental diseases.

2. The computer-implemented method for detecting dental diseases based on dental tissue images as claimed in claim 1, characterized in that: The pre-tasks include contrastive learning and mask image modeling.

3. The computer-implemented method for detecting dental diseases based on dental tissue images as claimed in claim 1, characterized in that: It also includes: Detecting the position of each tooth therein based on the dental tissue image; Cutting the features extracted by the feature extraction module based on the tooth position information; and The clipped features are respectively input into the dental disease detection network to obtain dental disease detection results of the areas corresponding to the teeth in the dental tissue image.

4. The computer-implemented method for detecting dental diseases based on dental tissue images as claimed in claim 3, characterized in that: It also includes: Detecting the tooth number of each tooth therein based on the dental tissue image; and A corresponding tooth number is assigned to the area corresponding to each tooth in the dental tissue image.

5. The computer-implemented method for detecting dental diseases based on dental tissue images as claimed in claim 3, characterized in that: The tooth position information is obtained using a trained second deep neural network.

6. The computer-implemented method for detecting dental diseases based on dental tissue images as claimed in claim 4, characterized in that: The tooth number is obtained using a trained third deep neural network.

7. The computer-implemented method for detecting dental diseases based on dental tissue images as claimed in claim 6, characterized in that: The second and third deep neural networks are the same neural network.

8. The computer-implemented method for detecting dental diseases based on dental tissue images as claimed in claim 1, characterized in that: The feature extraction module is Vision Transformer.

9. The computer-implemented method for detecting dental diseases based on dental tissue images as claimed in claim 1, characterized in that: The dental disease detection module is a Transformer.

10. A computer-implemented system for detecting dental diseases based on dental tissue images, comprising: processor; as well as A storage device storing a computer program for detecting dental diseases based on dental tissue images. When the computer program is run, the processor will execute the method for detecting dental diseases based on dental tissue images as claimed in claim 1.

Images