Method for enriching a learning base
The method enhances the learning base by transforming original dental scene photos into hyperrealistic images to simulate dental events, addressing the scarcity of images for rare pathologies and enabling effective neural network analysis.
Patent Information
- Application Number
- EP2019736742
- Authority / Receiving Office
- EP · EP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-12-21
- Filing Date
- 2019-07-10
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2039-07-10
AI Technical Summary
The creation of a large learning base for neural networks to analyze dental arches is hindered by the limited number of photos available, particularly in rare dental pathologies, due to confidentiality and scarcity of images.
A method that enriches a historical learning base by transforming original dental scene photos into hyperrealistic images simulating specific dental events, allowing the creation of a rich dataset from limited initial images, which is then used to train a second neural network for analysis.
Enables the analysis of dental scenes with rare pathologies using a neural network, even with a limited initial dataset, by generating hyperrealistic images that simulate dental events, thereby expanding the learning base and improving analysis accuracy.
Smart Images

Figure IMGF0001 
Figure IMGF0002
Abstract
Description
Technical field
[0001] The present invention relates to the field of analysis of dental arch photos. It relates in particular to a method for creating a learning base intended for training a neural network intended to carry out such an analysis. Prior art
[0002] The most recent techniques use neural networks to assess dental situations from images, typically X-rays, particularly for identification post mortem.
[0003] A “neural network” or “artificial neural network” is a set of algorithms well known to those skilled in the art. The neural network may in particular be chosen from: specialized networks for image classification, called "CNN" ("Convolutional neural network"), for example AlexNet (2012) ZF Net (2013) VGG Net (2014) GoogleNet (2015) Microsoft ResNet (2015) Caffe: BAIR Reference CaffeNet, BAIR AlexNet Torch:VGG_CNN_S,VGG_CNN_M,VGG_CNN_M_2048,VGG_CNN_M_10 24,VGG_CNN_M_128,VGG_CNN_F,VGG ILSVRC-2014 16-layer,VGG ILSVRC-2014 19-layer,Network-in-Network (Imagenet & CIFAR-10) Google: Inception (V3, V4). networks specialized in localization and detection of objects in an image, Object Detection Network, for example: R-CNN (2013) SSD (Single Shot MultiBox Detector: Object Detection network), Faster R-CNN (Faster Region-based Convolutional Network method: Object Detection network) Faster R-CNN (2015) SSD (2015).
[0004] The above list is not exhaustive.
[0005] To be operational, a neural network must be trained by a learning process called " deep learning”,from a learning base consisting of a set of recordings, each containing an image and a description of the image. By presenting the recordings as input to the neural network, the latter gradually learns how to generate a description for an image presented to it.
[0006] For example, each record in the learning database may include an image of a dental arch and a description identifying, in this image, the representations of the teeth, or "tooth zones", and the corresponding tooth numbers. Conventionally, these descriptions are generated by an operator who delimits, using a computer, the tooth zones and who, after having identified the corresponding tooth, for example "upper right canine", assigns them a number accordingly. This operation is called "labeling". After being trained, the neural network will thus be able to identify, on an image presented to it, the representations of the teeth and the corresponding tooth numbers.
[0007] The quality of the analysis performed by the neural network directly depends on the number of records in the training database. Typically, the training database must contain more than 10,000 records. There is a need for the rapid creation of such a training database.
[0008] The publication titled "Image Captioning and Visual Question Answering Based on Attributes and Their Related External Knowledge" by QI WU ET AL in arXiv.org 2016-03-09 Cornell University Library 201 Olin Library Cornell University Ithaca, NY 14853 , discloses a method for generating captions and attributes related to an image fed into a convolutional neural network, previously trained for this purpose.
[0009] The paper "Deep visual-semantic alignments for generating image descriptions" by KARPATHY ANDREJ ET AL, 2015 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, June 7, 2015, pages 3128-3137, discloses a model that generates a natural language description of an image and its regions.
[0010] The publication entitled "Deep Learning for Image-to-Text Generation: A Technical Overview", by Xiaodong He ET AL: "IEEE Signal Processing Magazine, vol. 34, no. 6, November 1, 2017, pages 109-116, is a state-of-the-art on methods for automatically generating a description of an image in natural language.
[0011] In some dental contexts, particularly in the context of rare pathologies, the creation of a large number of records is made difficult by the limited number of photos produced, particularly by orthodontists and dentists, and by the generally confidential nature of these photos. There is therefore a need for a method allowing an analysis of a dental context, by means of a neural network, from a limited learning base.
[0012] One aim of the invention is to meet these needs. Statement of the invention Summary of the invention
[0013] The invention proposes a method for analyzing a photo to be analyzed representing a dental scene to be analyzed, said method comprising the following steps: A) creating a historical learning base comprising more than 1,000 historical records, by implementing a method for enriching a historical learning base, said enrichment method comprising the following steps: 1) training a first neural network to transform first photos of first dental scenes into first hyperrealistic images to simulate the effect of a dental event on said first dental scenes; 2) submitting, to the first trained neural network, an original photo representing an original dental scene in a dental context, so as to obtain a final image representing, in a hyperrealistic manner, the original dental scene after simulation of the dental event; 3) creating a description of said final image, or “final description”;4) creation of a historical record consisting of the final image and the final description, and addition of the historical record to the historical training base; B) training of a second neural network, using the historical training base; C) submission to the second trained neural network, of the photo to be analyzed, so as to obtain a description of the photo to be analyzed.
[0014] As will be seen in more detail in the rest of the description, the enrichment process makes it possible to obtain a historical learning base comprising numerous images representing, with the quality of photos, dental scenes in dental contexts for which few photos are available. The quality of these images advantageously allows them to be used to train the second neural network so that it can search, in photos to be analyzed, for patterns specific to the dental context.
[0015] The enrichment process may further include one or more of the following optional features: the dental event is chosen from an application of an orthodontic appliance, a dental treatment, the occurrence of a pathology, a modification of a shape, a color or a position of a tooth or several teeth and / or of the tongue and / or of the gum and / or of an arch and / or of the temporomandibular joint and / or of the shape of the face, and / or of a relationship between the two arches, and a modification of the observation conditions of a first dental scene; more than 10,000 first photos exist and, preferably, are public; the dental context is that of a dental pathology which affects less than 10% of the population and / or for which less than 10,000 photos representing a dental arch victim of said dental pathology exist and, preferably, are public; the original photo does not show symptoms of said pathology;the first neural network is trained to, in step 2), only add one or more elements to the original photo; the first neural network is trained to, in step 2), only add an orthodontic appliance to the original photo; in step 1), several first neural networks are trained, preferably more than 2, more than 3, more than 5 or more than 10 first neural networks, to simulate the effect of different dental events, then, in step 2), the original photo is submitted to said trained first neural networks so as to generate several final images; the first photos and / or the original photo are extra-oral views taken by means of a dental retractor. ;
[0016] When the dental context is, for example, the occurrence of a rare pathology, the second neural network thus advantageously makes it possible to evaluate whether the dental scene in the photo to be analyzed corresponds to this pathology.
[0017] Preferably, before step C), the photo to be analyzed is taken by the patient with his mobile phone, step C) is implemented by a computer, integrated into the mobile phone or with which the mobile phone can communicate, after step C), the computer informs the patient, preferably via the mobile phone, of the result of the analysis.
[0018] The methods are carried out by computer. In particular, the first and second neural networks are created, trained and implemented by computer. Submitting a photo to a neural network, creating a description and adding a record are also operations carried out by means of a computer, possibly with the assistance of an operator, preferably without the assistance of an operator except to launch a computer program.
[0019] The invention therefore also relates to: a computer program comprising program code instructions for executing one or more steps of any method according to the invention, when said program is executed by a computer, a computer medium on which such a program is recorded, for example a memory or a CD-ROM. Definitions
[0020] A “patient” is a person for whom a method according to the invention is implemented, regardless of whether this person is undergoing orthodontic treatment or not.
[0021] A "dental situation" defines a set of characteristics relating to a dental arch, for example the position of the teeth, their shape, the position of an orthodontic appliance, etc. A "dental context" characterizes a set of dental situations, for example characterizes a pathology. A dental context can be, for example, that of a malocclusion.
[0022] An "image" is a two-dimensional, pixel-based representation of a dental scene. A "photo" is therefore a specific image, typically in color, taken with a camera. A "camera" refers to any device capable of taking a photo, including a video camera, a mobile phone, a tablet, or a computer.
[0023] A "hyperrealistic image" is an image that is not a photo but appears to be a photo.
[0024] A "scene" is a set of elements that can be observed simultaneously. A "dental scene" is a scene containing at least part of a dental arch.
[0025] By "image of an arch", "photo of an arch", "representation of an arch", etc. is meant an image, a photo, a representation, etc. of all or part of said dental arch.
[0026] A retractor is a device that has an upper rim and a lower rim extending around a retractor opening. In the operating position, the patient's upper and lower lips rest on the upper and lower rims, respectively. The retractor is configured to elastically separate the upper and lower lips so as to expose the teeth visible through the opening. A retractor thus allows the teeth to be observed without being obstructed by the lips. The teeth, however, do not rest on the retractor, so that the patient can, by turning their head relative to the retractor, change which teeth are visible through the retractor opening. They can also change the spacing between the arches. Preferably, the retractor has cheek retracting ears, which allow the buccal surfaces of teeth at the back of the mouth, such as molars, to be observed.
[0027] A "learning database" is a database of computer records suitable for training a neural network. Each record contains an object, for example an image, and information about this object, or "description." A description contains values for attributes of the object. For example, an attribute of an image of a dental scene can be used to identify the numbers of the teeth represented. The attribute is then "tooth number," and for each tooth, the value of this attribute is the number of that tooth. An attribute of an image of a dental scene can also define a pathology from which the patient suffers, an occlusion class, a position of the mandible relative to the maxilla ("overbite" or "overjet"), an overall hygiene index, or a crowding index, for example.
[0028] In this description, the terms "historical", "first", "second", "current", "rare", "original", "final", and "analysis" are used for clarity.
[0029] "Comprising" or "comprising" or "presenting" should be interpreted in a non-restrictive manner, unless otherwise indicated. Brief description of the drawings
[0030] Other characteristics and advantages of the invention will become apparent upon reading the detailed description which follows and upon examining the attached drawing in which: [ Fig 1 ] schematically represents the different stages of an enrichment process; [ Fig 2 ] schematically represents the different stages of an analysis method according to the invention; [ Fig 3 ] represents an original photo; [ Fig 4 ] represents a final image obtained by means of a process from the original photo of the figure 3 . Detailed description
[0031] The following detailed description is of preferred embodiments, but is not limiting. Enrichment process
[0032] As illustrated in the figure 1 , a process for enriching a historical learning base includes steps 1) to 3).
[0033] In step 1), we can train a first neural network with a first learning base made up of a set of first recordings, each first recording comprising: a first photo representing a first dental scene; and a first description specifying a value for at least one attribute of the first photo relating to the occurrence of a dental event for the first dental scene; at least some of the earliest photographs depicting early dental scenes after they were submitted to the dental event.
[0034] The first neural network can be chosen in particular from the list provided in the preamble to this description.
[0035] The first photos are preferably extra-oral views, for example photos taken facing the patient, preferably with a dental retractor.
[0036] The dental event could be, for example, the fitting of orthodontic appliances. The value of this attribute then indicates whether an orthodontic appliance is shown in the first photo. The attribute could be, for example, "presence of orthodontic appliances" and take the values "yes" or "no".
[0037] The dental event is preferably “common,” that is, widely spread.
[0038] Preferably, more than 500, more than 1,000, more than 5,000, preferably more than 10,000 photos depicting dental scenes where a dental arch has been subjected to the dental event, for example the fitting of an orthodontic appliance, exist, preferably are public.
[0039] The dental event may in particular be chosen from an application of an orthodontic appliance, a dental treatment, in particular orthodontic, therapeutic or not, the occurrence of a pathology, and a modification of a shape, color or position of a tooth or teeth and / or of the tongue and / or of the gum and / or of an arch and / or of the temporomandibular joint (TMJ) and / or of the shape of the face, and / or of a relationship between the two arches (for example modification of the occlusal plane or of the occlusal class).
[0040] The first training base preferably comprises more than 1,000, more than 5,000, preferably more than 10,000, preferably more than 30,000, preferably more than 50,000, preferably more than 100,000 first records. The higher the number of records, the better the ability of the first neural network to modify a photo in a hyperrealistic way.
[0041] Training the first neural network using the first learning base is well known to those skilled in the art.
[0042] Through this training, the first neural network learns how to modify, in a hyper-realistic way, a photo representing a dental scene to simulate the dental event, for example, to add the representation of an orthodontic appliance. It therefore makes it possible to virtually subject the dental event to the dental scene and to visualize the result.
[0043] In particular, training can be performed by following the lessons of the article by Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks" (Open access Computer Vision Foundation).
[0044] In step 2), We submit an original photo (Fig. 3A) to the first neural network so that it transforms it into a final hyperrealistic image (Fig. 3B), that is to say one which appears to be a photo, the transformation simulating the dental event. For example, it adds to the original photo the representation of an orthodontic appliance.
[0045] The original photo is preferably an extra-oral view, for example a photo taken facing the patient, preferably with a dental retractor.
[0046] The original photo represents an original dental scene in a dental context, for example, represents the dental arch of a patient suffering from a pathology "P". Preferably, however, it does not represent a dental scene containing an arch that has been subjected to the dental event. In the example considered, it does not represent a dental scene in which the arch wears an orthodontic appliance.
[0047] In a preferred embodiment, the dental context is rare.
[0048] The rare dental context is preferably that of a dental pathology, preferably a pathology which affects less than 10%, less than 5%, less than 2% or less than 1% of the population.
[0049] The dental context may be a context for which fewer than 10,000, fewer than 5,000, or fewer than 1,000 photos exist or are public, i.e., accessible to anyone, possibly for a fee. Difficulty in access may result, in particular, from regulatory constraints or practical constraints, for example, in taking photos illustrating the dental context.
[0050] Preferably, photos illustrating the dental context show the symptoms of a dental pathology, for example a rare pathology, for example one that affects less than 10%, less than 5%, less than 2% or less than 1% of the population.
[0051] When the dental context relates to the occurrence of a pathology, the original photo may show symptoms of this pathology identifiable by a dental professional, in particular an orthodontist, or not.
[0052] If it does not represent such symptoms, it does represent the original dental scene in this dental context because the pathology was diagnosed with other means, for example from tomographic images, preferably obtained by cone beam computed tomography, in English CBCT for "cone beam computed tomography". For example, the photos show the arches of patients who have undergone rhizalysis.
[0053] Since there are few photos illustrating the dental context, it is difficult to train a neural network with only these photos. Step 2) advantageously transforms the original photo into a hyper-realistic final image that represents the result of the dental event if it were applied to the original dental scene.
[0054] By training several initial neural networks to simulate different dental events, and then submitting the original photo to these networks, it is possible to multiply the final images.
[0055] Tests (Fig. 3A and 3B) have shown that the final images are of very good quality, so that it is very difficult to detect that they are not photos.
[0056] In step 3), a "final" description is created for the final image resulting from step 2). It provides a value for at least one attribute of the final image relating to the occurrence of the dental event, and a value for at least one attribute of the final image relating to the dental event.
[0057] For example, it specifies that the attribute "presence of an orthodontic appliance" has the value "yes" and that the attribute "presence of pathology P" has the value "yes".
[0058] The description of the final image can be done by hand, at least partially. Preferably, it is done, at least partially, by a computer program.
[0059] At step 4), A historical record consisting of the final image and the final description is created and added to the historical training base. The final image and the final description are then referred to as the historical image and the historical description, respectively.
[0060] The historical learning base may consist only of historical records generated using an enrichment process. Alternatively, the historical learning base may include historical records generated using an enrichment process and other historical records, for example created using conventional processes, including by labeling photos, and including by labeling the original photos.
[0061] Preferably, the cycle of steps 2) to 4) is repeated, preferably until the historical learning base comprises more than 5,000, preferably more than 10,000, preferably more than 30,000, preferably more than 50,000, preferably more than 100,000 historical records. Analysis of an analysis photo
[0062] As illustrated in the figure 2 , to analyze a photo to be analyzed, we proceed according to steps A) to C).
[0063] The photo to be analyzed is preferably in color, preferably in real color.
[0064] At step A), a historical learning base is created, comprising historical records obtained through an enrichment process.
[0065] In step B), a second neural network is trained using the historical learning base. Such training is well known to those skilled in the art.
[0066] The second neural network can be chosen in particular from the list provided in the preamble to this description.
[0067] Through this training, the second neural network learns to determine, for the photos to be analyzed that are presented to it, values for the attributes evaluated in the historical descriptions. For example, each historical description can specify a value ("yes" or "no") for the attribute "presence of pathology P". The second neural network then learns to determine whether a photo to be analyzed that is submitted to it presents patterns indicating the presence of pathology P.
[0068] At step C), the photo to be analyzed is presented to the second neural network, and an evaluation is thus obtained for the different attributes, for example "yes", with a probability of more than 99%, for the presence of pathology P.
[0069] The analysis process can be used for therapeutic or non-therapeutic purposes, for example for research or purely aesthetic purposes.
[0070] It can be used, for example, to assess a patient's dental situation during orthodontic treatment or teeth whitening treatment. It can be used to monitor tooth movement or the development of a dental pathology.
[0071] In one embodiment, the patient takes the photo to be analyzed, for example with his mobile phone, and a computer, integrated into the mobile phone or with which the mobile phone can communicate, implements the process. The patient can thus very easily request an analysis of his dental situation, without even having to move, by simply transmitting one or preferably several photos of his teeth.
[0072] Analyzing a photo to be analyzed is particularly useful for detecting a rare disease. As is now clear, the first neural network can transform original photos representing an original dental scene in a dental context, into final images representing the original dental scene differently, the difference between the final images and the original photos being the result of the simulation of the dental event. It is thus possible to create a rich historical learning base, from a few original photos, by adding these final images.
[0073] The addition of hyper-realistic final images thus makes it possible to very quickly enrich the historical learning base.
[0074] Of course, the invention is not limited to the embodiments described above and shown. Rebalancing a learning base
[0075] In particular, steps 1) to 4) can be implemented to enrich a historical learning base in order to “rebalance” it.
[0076] For example, in step 1), the first neural network can be trained to provide first hyperrealistic images of a dental arch representing this arch in so-called “simulated” observation conditions, in particular following a so-called “simulated” observation direction, from first photos of the dental arch taken in first observation conditions different from the simulated observation conditions, in particular following a first observation direction different from the simulated observation direction.
[0077] In particular, the first neural network can be trained to transform first photos of first dental scenes representing certain teeth, for example incisors, into first hyperrealistic images representing other teeth, for example molars.
[0078] The dental event is then a modification of the observation conditions of the dental arch, from the first observation conditions implemented to acquire the first photos to the simulated observation conditions. The first neural network is thus trained to simulate the effect of this modification on the first dental scenes.
[0079] In step 2), we submit to the first trained neural network an original photo representing an original dental scene in a dental context in which the original dental scene is observed under the first observation conditions, for example an original photo representing incisors, without the molars being visible.
[0080] The first neural network then generates a final image representing, in a hyper-realistic manner, the original dental scene after simulation of the dental event. In other words, the final image represents, in a hyper-realistic manner, the original dental scene observed under the simulated observation conditions. For example, it represents molars in the dental arch.
[0081] Steps 3) and 4) create a description of the final image and associate it with the final image to create a historical record. The historical record is then added to the historical learning database.
[0082] While the historical training base initially included an excessive proportion of records relating to images representing incisors, this addition helps to increase the number of records relating to images representing molars. The training base is therefore advantageously better balanced.
[0083] The patient is not limited to a human being. A method according to the invention can be used for another animal.
[0084] A learning base is not necessarily made up of “pair” recordings.
Claims
1. Method for analyzing a photograph to be analyzed representing a dental scene to be analyzed, said method comprising the following steps: A) creating a historical learning base comprising more than 1,000 historical records, by implementing an enrichment method for a historical learning base, said enrichment method comprising the following steps: 1) training a first neural network to convert first photographs of first dental scenes into first hyperrealistic images to simulate the effect of a dental event on said first dental scenes; 2) submitting, to the first trained neural network, an original photograph representing an original dental scene in a dental context, so as to obtain a final image representing, in a hyperrealistic manner, the original dental scene after simulation of the dental event; 3) creating a description of said final image, or "final description"; 4) creating a historical record consisting of the final image and the final description, and adding the historical record to the historical learning database; B) training a second neural network, using the historical learning base; C) submitting the photograph to be analyzed to the second trained neural network, so as to obtain a description of the photograph to be analyzed.
2. Method according to the preceding claim, wherein the dental event is selected from an application of an orthodontic appliance, a dental treatment, the occurrence of a pathology, a change in shape, color or position of one or more teeth and / or the tongue and / or the gum and / or an arch and / or the temporomandibular joint and / or the face shape, and / or a relationship between the two arches, and a change in the observation conditions of a first dental scene.
3. Method according to either of the preceding claims, wherein more than 10,000 first photographs are public.
4. Method according to any of the preceding claims, wherein the dental context is that of a dental pathology affecting less than 10% of the population and / or for which less than 10,000 photographs representing a dental arch victim of said dental pathology are public.
5. Method according to the preceding claim, wherein the public photographs do not show the symptoms of said dental pathology.
6. Method according to any of the preceding claims, wherein the first neural network is trained to, in step 2), only add one or more elements to the original photograph.
7. Method according to the preceding claim, wherein the first neural network is trained to, in step 2), only add an orthodontic appliance to the original photograph.
8. Method according to any of the preceding claims, wherein, in step 1), multiple first neural networks are trained to simulate the effect of different dental events, then, in step 2), the original photograph is subjected to said trained first neural networks so as to generate multiple final images.
9. Method according to any of the preceding claims, wherein the first photographs and / or the original photograph are extra-oral views taken by means of a mouthpiece.
10. Method according to any of the preceding claims, wherein: - before step C), the photograph to be analyzed is taken by the patient using his or her mobile phone, - step C) is implemented by a computer, which is integrated into the mobile phone or with which the mobile phone can communicate, - after step C), the computer informs the patient, preferably via the mobile phone, of the result of the analysis.
11. Method according to any of the preceding claims, with the learning base consisting of "pair" records.
Citation Information
Patent Citations
Augmented reality enhancements for dental practitioners
US20180168780A1
