Method for generating a dental image

Abstract

A method for generating an image of a dental arch of a beneficiary, called "modified image", is disclosed, said method comprising the following steps: a) at an acquisition time, acquiring a photograph, called "raw image", depicting said dental arch; b) processing said raw image so that it depicts a dental arch outline; c) submitting said raw image to a neural network, called "simulation neural network", trained to simulate the effect of a dental event on said raw image, thereby obtaining said modified image; d) preferably, processing said modified image to make it hyper-realistic; e) preferably, - presenting said modified image; and / or - selecting an orthodontic appliance based on said modified image.

Description

TECHNICAL FIELD

[0001] The present invention relates to generating an image of a dental arch. It particularly relates to a method for realistically visualizing in an image the impact of a dental event on a dental arch liable to affect a beneficiary, for example a patient undergoing orthodontic treatment. BACKGROUND

[0002] PCT / EP2019 / 068558 describes a method in which a digital three-dimensional model of a dental arch, in particular a scan made by a dental professional, is deformed to simulate, that is to say to adapt to artificially reproduce, a dental situation expected at a past simulation time or predicted at a future simulation time. The deformed model is used to create a hyper-realistic view equivalent to a photograph.

[0003] Thus, one has to go to an orthodontist to create their dental arch model. The field of application of this method is therefore limited.

[0004] There is therefore a need for a method for generating an image of a dental arch simulating the impact of a dental event and not requiring the generation of a model.

[0005] One object of the present invention is to meet this need. SUMMARY

[0006] According to a first main aspect, the invention relates to a method for generating an image of a dental arch of a beneficiary, called "modified image", said method comprising the following successive steps:

[0007] a) at an acquisition time, acquiring a photograph, called "original image", depicting said dental arch;

[0008] b) processing said original image so that it depicts discriminant information, preferably the contours of the dental arch;

[0009] c) submitting said original image to the input of a neural network, called "simulation neural network", trained to simulate on said original image the impact of a dental event, thereby obtaining said modified image;

[0010] d) preferably, processing said modified image to make it hyper-realistic;

[0011] e) preferably,

[0012] - presenting said modified image, preferably at least to said beneficiary and / or to a dental professional; and / or

[0013] - selecting an orthodontic appliance, performed by a computer and / or by the beneficiary and / or by a dental professional based on the modified image, then preferably manufacturing said orthodontic appliance.

[0014] As will be seen in more detail in the rest of the description, the generation method according to the application does not require generating a model of the beneficiary's dental arch. Notably, anyone, whether or not subject to treatment, is able to simulate the impact of a dental event on their teeth without having to go to an orthodontist.

[0015] In particular, the modified image depicts the dental arch simulated after application of the dental event. The beneficiary can thus benefit from the simulation, allowing them to accurately gauge the visual impact of the dental event.

[0016] The generation method according to the application can also comprise one or more of the following optional features:

[0017] - the dental event is chosen from the group consisting of time elapsing in orthodontic or non-orthodontic treatment situations, time elapsing in pathologies or bruxism situations, fitting of dental components on the dental arch, time elapsing without treatment, and combinations of these dental events;

[0018] - the discriminating information is chosen from the group consisting of: contour information, color information, density information, distance information, luminosity information, saturation information, information about reflections, and combinations of these information;

[0019] - the simulation neural network is trained by the training method according to the application, as described below;

[0020] - a loop of steps 1) to 4) is repeated, modifying the first observation conditions at the end of each step 4);

[0021] - in step a), the photos are taken extra-orally, preferably using a mobile phone, preferably with the beneficiary wearing dental expanders;

[0022] - the method comprises a step d) of processing the modified image to make it hyper-realistic, step d) comprising the following steps:

[0023] d0) for each photo called "texture photo", depicting the dental arch from a set comprising more than 1000 texture photos,

[0024] processing the texture photos, preferably as in step b), to obtain an image called "texture image" depicting the contours;

[0025] d1) creating a so-called "texture" learning library consisting of records called "texture records", each texture record comprising a texture photo and a texture image obtained by processing the texture photo in step d0);

[0026] d2) training a neural network called "texture neural network" by the texture learning library;

[0027] d3) submitting the modified image to a trained texture neural network to obtain a hyper-realistic modified image;

[0028] - the dental event is the time lapse from the acquisition time to a simulated time more than 1 day before or after the acquisition time and in step e) the modified image is presented to the beneficiary in order to show the beneficiary the simulated dental condition determined at said simulated time;

[0029] - before step c) the dental event is determined by specifying a simulated time and / or treatment parameters applied to the beneficiary and / or parameters of the orthodontic appliance worn by the beneficiary and / or functional parameters of the beneficiary and / or anatomical parameters of the beneficiary other than the positioning parameters of their teeth and / or the age or age bracket of the beneficiary and / or the gender;

[0030] - steps a) to e) are repeated in a loop with successive acquisition of original images, each loop lasting less than 5 seconds, preferably less than 1 second.

[0031] According to a second main aspect, the application relates to a method for training a neural network, for each of a plurality of digital three-dimensional models of a dental arch called "historical models", said method comprising the following successive steps 1) to 3):

[0032] 1) acquiring a first view of the historical model under first observation conditions, if said first view does not depict a discriminating information of the historical dental arch called "first discriminating information", preferably does not depict a contour of the historical dental arch called "first contour", processing said first view so that it depicts said first discriminating information, preferably the first contour;

[0033] 2) modifying said historical model, for example by deforming and / or adding dental pieces, to reproduce the effect of a dental event on said historical dental arch;

[0034] 3) acquiring a second view of the historical model under second observation conditions identical to the first observation conditions, if said second view does not depict a discriminating information of said historical dental arch called "second discriminating information", in particular if said second view does not depict a contour of the historical dental arch called "second contour", processing the second view so that it depicts said second discriminating information, preferably the second contour, and creating a historical record using the first view and the second view;

[0035] Then, all historical records are created for all historical models:

[0036] 4) introducing said first view and said second view respectively at the input and at the output of the neural network to train said neural network to convert an input view depicting an analyzed dental arch into an output view depicting the analyzed dental arch after application of said dental event.

[0037] The observation of the historical model under the same observation conditions makes it possible to easily obtain perfectly aligned first and second views. No cropping is necessary.

[0038] In one embodiment,

[0039] - in step 1), more than 10, more than 100, more than 1000, more than 10000 first views are acquired under each different first observation condition, then,

[0040] - in step 3), for each first view acquired under a first observation condition, a second view is acquired under a second observation condition identical to said first observation condition, and a history is created using said first and second views.

[0041] Preferably, the simulated neural network used in step c) is trained with the training method according to the application.

[0042] The generation method and / or the training method according to the application are at least partially implemented by a computer. The application therefore also relates to:

[0043] - a computer program comprising program code instructions for executing steps b) to e) of the simulation method according to the application and / or steps 1), 3) and 4) of the training method according to the application, and preferably 2), when said program is executed by a computer,

[0044] - a computer medium, such as a memory or a CD-ROM, on which such a program is recorded.

[0045] Any computer can be envisaged, in particular a telephone, a PC, a server or a tablet.

[0046] Definitions

[0047] "Orthodontic treatment" is the whole or part of the treatment aimed at modifying the configuration of the dental arch.

[0048] "Dental member" is understood to mean any device intended to be worn by the dental arch, in particular an orthodontic appliance, a crown, an implant, a bridge or a veneer.

[0049] "Dental condition" defines a set of characteristics related to the dental arch of a patient at a time, for example the position of the teeth at this time, their shape, their color, the position of the orthodontic devices, etc.

[0050] "Beneficiary" is the person for whom the method according to the application is implemented, whether or not this person is undergoing orthodontic treatment.

[0051] "Dental professional" is understood to mean any person qualified to provide dental care, in particular including orthodontists and dentists.

[0052] A "computer" is understood to mean any electronic device having data processing capabilities.

[0053] A "retractor" or "dental retractor" is a device intended to push the lips backwards. It has an upper edge and a lower edge extending around the retractor opening. In the working position, the patient's upper lip and lower lip rest on the upper edge and the lower edge, respectively. The retractor is configured to elastically space the upper lip and the lower lip from each other so that the teeth are visible through the opening. Thus, the retractor makes it possible to observe the teeth without obstruction by the lips. However, the teeth do not rest on the retractor, which means that the patient is able to modify the teeth visible through the opening of the retractor by turning their head with respect to the retractor. The retractor also makes it possible to modify the spacing between the dental arches. In particular, the retractor does not press on the teeth to space the two jaws from each other. Preferably, the retractor comprises ear parts for expanding the cheeks, thus making it possible to take a photo of the vestibular surface of the posterior teeth (e.g. molars) of the oral cavity through the retractor opening.

[0054] A "model" is understood to mean a digital three-dimensional model. A model consists of a set of voxels.

[0055] For the sake of clarity, a distinction is generally made between "segmenting" a dental arch model into "elementary models" and "segmenting" an image into "elementary regions". An elementary model and an elementary region are respectively a 3D or 2D depiction of a real scene element (e.g. a tooth).

[0056] An "observation condition" of a model specifies a position in space, an orientation in space, and a calibration, e.g. a value of the aperture of the virtual image acquisition device associated with this model and / or the exposure time and / or the focal length and / or the sensitivity.

[0057] A "calibration" of an acquisition device consists of a set of calibration parameter values. A calibration parameter is a parameter inherent to the acquisition device (different from its position and orientation) whose value influences the acquired image. For example, the aperture is a calibration parameter that modifies the depth of field. The exposure time is a calibration parameter that modifies the brightness (or "exposure") of the image. The focal length is a calibration parameter that modifies the angle of view, i.e. the degree of "zoom". The "sensitivity" is a calibration parameter that modifies the reaction of the sensor of the digital acquisition device to the incident light.

[0058] A model observed under a determined observation condition is called a "view".

[0059] An "image" is a two-dimensional depiction of a dental arch formed of pixels. Thus, a "photo" is a particular image taken with a camera, usually in color. A "camera" is understood to mean any device for taking photos, including a dedicated camera, a cell phone, a tablet or a computer. A view is another example of an image.

[0060] "A dental arch photograph", "dental arch depiction", "dental arch scan", "dental arch model", "dental arch image", "dental arch view" or "dental arch outline" are understood to mean a photograph, depiction, scan, model, image, view or outline of all or a part of the dental arch, preferably of at least 2, preferably of at least 3, preferably of at least 4 teeth.

[0061] "Dental event" is understood to mean an event liable to change the dental arch, such as the wearing of an orthodontic appliance or simply the passage of time.

[0062] "Distinguishing information" is characteristic information ("image characteristics") that can be extracted from an image, generally by computer processing of this image.

[0063] Distinguishing information can exhibit a variable number of values. For example, contour information can represent the probability that a pixel belongs to a contour, and for example take values from 0 to 255, 0 representing a very low probability that a pixel belongs to a contour and 255 a very high probability. In one embodiment, the distinguishing information is thresholded, that is to say only the distinguishing information that exceeds a predetermined threshold is depicted. For example, in the example of contour information described above, only the values greater than 200 are depicted. Intensity information can take many values. Image processing makes it possible to extract and quantify the distinguishing information.

[0064] Distinguishing information can be depicted in an image, sometimes called a "map". The map is thus the result of processing an image to reveal the distinguishing information. For example, the contours of the teeth and gums can be a depiction of the contour information from the original image. The depiction of the contours is thus a depiction of the contour information in image form.

[0065] "A contour" is a line or set of lines that delimits an object, preferably delimits the elements making up this object. For example, the contour of a dentition can be a line that defines the outer limit of this dentition. Preferably, the "contour" also includes the lines that define the boundaries between adjacent teeth. The contour of a dentition thus preferably consists of all the contours of the teeth forming this dentition.

[0066] The contours depicted in an image can be complete and thus closed in on themselves, or incomplete.

[0067] The contour of a dental arch preferably depicts the contour of the dentition of this dental arch, and preferably, the contour of each tooth depicted, called "elementary contour".

[0068] In the method according to the application, the processing operations applied to the image are preferably configured to reveal, preferably isolate, the same contours, preferably the contours comprising all the elementary contours of the teeth depicted, preferably consisting essentially of all the elementary contours of the teeth depicted, as Figure 7 Figure 7 as ​

[0069] A "neural network" or "artificial neural network" is a set of algorithms well known by the person skilled in the art. To be run, a neural network must be trained through a learning process called "deep learning" from a learning library.

[0070] A "learning library" is a computer library suitable for training a neural network. The quality of the analysis performed by the neural network depends directly on the number of records in the learning library. Typically, the learning library contains more than 10 000 records.

[0071] The training of a neural network is adapted to the desired goal and does not pose any particular difficulty to the person skilled in the art.

[0072] Training a neural network consists in confronting it with a learning library containing information about two types of objects that the neural network must learn to "match", i.e. to correlate.

[0073] The training can be performed from a "paired" learning library or "learning library with pairs", consisting of "pairs" of records, that is to say, each record comprises a first object of a first type used as input for the neural network, and a second corresponding object of a second type used as output for the neural network. It can also be said that the input and output of the neural network are "paired". Training the neural network with all these pairs teaches the neural network to provide a corresponding object of a second type from any object of a first type.

[0074] For example, each record in the learning library can comprise a first view of a dental arch model and a second view of this model after a dental event has occurred. After training using this learning library, the neural network will be able to convert a view of a dental arch model into a modified view of this model to simulate the impact of said dental event.

[0075] The article "Image-to-Image Translation with Conditional Adversarial Networks" by Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, BAIR Laboratory, University of California, Berkeley, shows the use of a paired learning library.

[0076] In this specification, the qualifiers "historical", "original", "texture", "simulation" and "analysis" are used for clarity.

[0077] Unless otherwise stated, "comprise" or "comprising" or "exhibit" should be interpreted as non-restrictively. BRIEF DESCRIPTION OF DRAWINGS

[0078] Other features and advantages of the invention will become more apparent from the following detailed description and study of the accompanying drawings, wherein:

[0079] -[ Figure 1 ] Figure 1 The steps of a preferred embodiment of the image generation method according to the present invention are illustrated schematically;

[0080] -[ Figure 2 ] Figure 2 The steps of a preferred embodiment of the training method according to the present invention are illustrated schematically;

[0081] -[ Figure 3 ] Figure 3 An example of the photograph acquired in step a) is shown, as well as an example of a modified image obtained by the method according to the invention, which uses methods such as Figure 7 Training is performed using the records in the database;

[0082] -[ Figure 4 ] Figure 4 An example of a dental arch model is shown;

[0083] -[ Figure 5 ] Figure 5 A view of a dental arch model is shown;

[0084] -[ Figure 6 ] Figure 6 An example of a puller is shown;

[0085] -[ Figure 7 ] Figure 7 An example of a record used to train a simulated neural network is shown to simulate the event “orthodontic treatment using an orthodontic appliance with archwire and brackets”, with the left and right images introduced at the input and output of the simulated neural network, respectively.

[0086] -[ Figure 8 ] Figure 8 An example of a model divided into multiple tooth models is shown; other elements of the dental arch are not shown. Detailed Implementation

[0087] The following detailed description is a preferred embodiment, but is not limiting.

[0088] In particular, the following detailed description describes the use of discriminative information, i.e., contour information. However, discriminative information can be of another type.

[0089] Training a neural network

[0090] The method for training a neural network according to the present invention preferably includes steps 1) to 4).Figure 1 ).

[0091] Steps 1) to 3) of the method advantageously make it possible to increase the number of records used in step 4).

[0092] Before these steps, one, preferably a plurality, preferably more than 100, preferably more than 1000, preferably more than 10000 historical models are generated.

[0093] Each historical model depicts the dental arch of a so-called “historical” individual.

[0094] The historical models can be prepared from measurements made on the teeth of historical individuals or on casts of their teeth, for example plaster casts.

[0095] The historical models are preferably obtained from real conditions, preferably created using a 3D scanner. Such a model, called a “3D” model, can be observed from any angle Figure 4 ).

[0096] Preferably, the historical models are segmented. In particular, preferably, for each tooth, a model of said tooth or “tooth model” is defined based on the historical model.

[0097] Segmenting a historical model of a dental arch into a plurality of tooth models is a routine operation by which a dental arch model is segmented so as to delimit the depiction of one or more teeth in the model Figure 8 ).

[0098] The historical models can be segmented manually by an operator using a computer or automatically by a computer, preferably by implementing a deep learning device, preferably a neural network. In particular, the tooth models can be defined as described for example in international application PCT / EP2015 / 074896.

[0099] In the historical models, the tooth models are preferably delimited by the gingival margin, which can be decomposed into an internal gingival margin (relative to the side of the tooth inside the mouth), an external gingival margin (relative to the side of the tooth facing outside the mouth) and two lateral gingival margins.

[0100] Similarly, based on the historical models, models of the basic elements other than the tooth models can be defined, in particular of the tongue and / or mouth and / or lips and / or jaws and / or gums and / or dental elements, in particular orthodontic appliances.

[0101] In one embodiment, the historical model is theoretical, that is to say does not correspond to a real situation. In particular, the historical model can be created by assembling a set of dental models chosen from a digital library. The arrangement of the dental models is determined so that the historical model is realistic, that is to say corresponds to a situation that the individual can encounter. In particular, the dental models are arranged in an arch according to their nature and are realistically oriented. The use of a theoretical historical model can advantageously simulate a dental arch exhibiting rare characteristics.

[0102] The historical model preferably provides information on the positioning of the teeth with an error of less than 5 / 10 mm, preferably less than 3 / 10 mm, preferably less than 1 / 10 mm.

[0103] The historical model is for example of.stl or.Obj,.DXF 3D, IGES, STEP, VDA or point cloud type. Advantageously, the model, called "3D" model, can be observed from any angle.

[0104] Steps 1) to 4) are carried out for each historical model.

[0105] In step 1), a first view of the historical model is acquired under first observation conditions, that is to say by virtually placing a virtual image acquisition device under these first observation conditions, then using this device configured in this way to acquire the first view.

[0106] The first view is preferably an extraoral view, for example a view corresponding to a photograph that can have been taken facing the patient, preferably using a retractor.

[0107] Figure 6 One example of a retractor is shown.

[0108] If the first view does not depict contours, or if the contours cannot be identified, a processing operation is applied to isolate the contours, preferably the contours of the teeth.

[0109] Figure 5 One example of a first view before processing to isolate the contours is shown.

[0110] Figure 7 The left part of Figure 1 shows one example of a first view after processing to isolate the contours of the teeth.

[0111] In step 2), the historical model is modified to simulate the effects of the dental event.

[0112] Modifying the historical model can in particular comprise displacing, deforming or removing one or more teeth ("tooth model") and / or gingiva, and / or one or both jaws, and / or orthodontic appliances in the base model.

[0113] The modifications can be performed manually by an operator, preferably by a dental professional, more preferably by an orthodontist, preferably using a computer allowing them to view the model that is currently being modified.

[0114] Step 2) leads to a historical model depicting a theoretical dental situation.

[0115] This model can advantageously simulate unmeasurable dental situations. In particular, historical models corresponding to different stages of rare pathologies can be created.

[0116] In step 3), a view of the historical model modified in step 2), called "second view", is acquired under observation conditions identical to those enabling the acquisition of the first view. In other words, in the first and second views, the depiction of the teeth that has not been moved between steps 1) and 3) can perfectly overlap, i.e. "align".

[0117] If the second view does not depict contours, or contours cannot be identified, a processing operation is applied to isolate the contours, preferably of the teeth.

[0118] Figure 7 The right part of Fig. 2 shows one example of a second view after processing to isolate the contours of the teeth. Comparing the left part of Fig. 2 and the right part of Fig. 2, the impact of the dental event, in this case orthodontic treatment by an appliance with archwires and brackets, can be seen. Figure 7

[0119] A pair or "history record" is thus generated, comprising a first view and an associated second view representing the application of a dental event to the dental arch depicted in the first image. The history record is added to a historical learning library.

[0120] Figure 2 Fig. 3 shows one example of a history record.

[0121] In one embodiment, in step 1), more than 10, more than 100, more than 1000, more than 10000 first views are acquired under each different first observation condition, then, in step 3), for each first view acquired under a first observation condition, a second view is acquired under a second observation condition identical to said first observation condition, and a history record is created with said first and second views.

[0122] ​This process is equivalent to performing steps 1) and 3) without step 2) for more than 10, more than 100, more than 1000, more than 10000 cycles after step 3) by modifying the first observation condition in each cycle. In other words, the first view and the second view are acquired by moving around, in particular by rotating around, the historical model and / or by approaching or moving away and / or by modifying the calibration of the virtual acquisition device used to acquire the first view and the second view.

[0123] The invention thus advantageously makes it possible to increase the number of historical records with the same historical model.

[0124] Next, the historical model is changed and the process returns to step 1).

[0125] A historical learning library is thus formed, comprising preferably more than 5000, preferably more than 10000, preferably more than 30000, preferably more than 50000, preferably more than 100000 historical records.

[0126] In step 4), the historical learning library is then used to train a neural network. Such training is well known to the person skilled in the art.

[0127] Such training generally comprises providing all the first views at the input of the neural network and all the second views at the output of the neural network by establishing a bijection (that is to say belonging to the same record) between each first view and the corresponding second view.

[0128] Through such training, the neural network learns to convert an input view (for example a first view) depicting the profile of an analyzed dental arch into an output view depicting the same dental arch but after a dental event has occurred.

[0129] The neural network can in particular be chosen from networks dedicated to image generation, for example:

[0130] - Cycle-consistent adversarial networks (2017)

[0131] - Enhanced CycleGAN (2018)

[0132] - Deep photo style transfer (2017)

[0133] - Fast photo style (2018)

[0134] - pix2pix (2017)

[0135] - Style-based GAN generator architecture (2018)

[0136] - SRGAN (2018).

[0137] The above list is not limiting.

[0138] The neural network trained in this way can be used to simulate the effect of the dental event depicted in the simple photograph on the dental arch according to the following steps a) to e). This neural network is then called "conversion neural network".

[0139] Simulating dental events

[0140] In step a), the original image is created by taking a photograph using a camera, preferably chosen from a mobile phone comprising a photograph taking system, a so-called "connected" camera, a so-called "smart watch", a tablet or a fixed or portable personal computer. Preferably, the camera is a mobile phone.

[0141] Preferably, when taking the photograph, the camera is at a distance greater than 5 cm, greater than 8 cm or even greater than 10 cm from the dental arch, so as to avoid condensation of water vapour on the camera optics and to facilitate focusing. Furthermore, preferably, the camera, in particular a mobile phone, is not equipped with any specific optics for taking the photograph, which is possible in particular because of the distance from the dental arch during taking.

[0142] In order to facilitate the taking of the photograph, the retractor and the camera are preferably fixed on the same support, so as to fix their relative position. Preferably, the support is portable and should be held by the beneficiary when taking the photograph.

[0143] Preferably, the photograph is in colour, preferably in true colour.

[0144] Preferably, the photograph is taken by the beneficiary, preferably without using a support for fixing the camera, in particular without using a tripod.

[0145] The photograph is preferably an extra-oral view, for example corresponding to the view of a photograph that can have been taken facing the patient, preferably using a retractor.

[0146] The retractor can have the characteristics of a conventional retractor.

[0147] In step b), the processing of the original image aims to highlight, or even isolate, the discriminant information contained in the original image. The use of the discriminant information significantly improves the efficiency of the neural network implemented in step c).

[0148] Preferably, the original image is processed to reveal and preferably isolate the contours.

[0149] Preferably, the original image is processed so as to depict substantially nothing but the contours. Preferably, the contours comprise, or even consist of, the basic contours of each tooth depicted.

[0150] The contours can also comprise, or even consist of, the contours of all the teeth depicted. However, this embodiment is not preferred.

[0151] The person skilled in the art knows how to process a photograph or view so as to isolate the contours. Such processing comprises, for example, the application of well-known masks or filters that come with image processing software. Such processing operations make it possible, for example, to detect high-contrast areas.

[0152] These processing operations comprise, in particular, one or more of the following known and preferred methods:

[0153] - application of a Canny filter, in particular using the Canny algorithm to search for contours;

[0154] - application of a Sobel filter, in particular to calculate derivatives by means of an extended Sobel operator;

[0155] - application of a Laplace filter, in order to calculate the Laplacian of the image;

[0156] - detection of blobs in the image (“Blob detector”);

[0157] - application of a “threshold” to apply a fixed threshold to each element of the vector;

[0158] - use of a relationship between regions of pixels (“Resize (Area)”), or bicubic interpolation resizing in the environment of a pixel;

[0159] - erosion of the image by a particular structuring element;

[0160] - dilation of the image by a particular structuring element;

[0161] - image correction, in particular using the areas in the vicinity of the recovered areas;

[0162] - application of a bilateral filter;

[0163] - application of a Gaussian blur;

[0164] - application of an Otsu filter to search for a threshold that minimizes the variance between classes;

[0165] - application of an A* filter to search for a path between points;

[0166] - applying a "adaptive threshold" to apply an adaptive threshold to a vector;

[0167] - applying in particular a histogram equalization filter to a grayscale image;

[0168] - blur detection ("BlurDetection") to compute the entropy of an image using its Laplacian;

[0169] - detecting the contour of a binary image ("FindContour");

[0170] - color filling ("FloodFill"), in particular to fill the associated elements with a determined color.

[0171] The following non-limiting methods, although not preferred, can also be implemented:

[0172] - applying a "MeanShift" filter to find an object in the projection of an image;

[0173] - applying a "CLAHE" filter, CLAHE standing for "Contrast Limited Adaptive Histogram Equalization";

[0174] - applying a "Kmeans" filter to determine the center of clusters and the center of groups of samples around clusters;

[0175] - applying a DFT filter to perform a forward or inverse discrete Fourier transform on a vector;

[0176] - computing the moment of force;

[0177] - applying a "HuMoments" filter to compute the invariants Hu invariants;

[0178] - computing the integral of an image;

[0179] - applying a Scharr filter so that the derivative of an image can be computed by implementing the Scharr operator;

[0180] - searching the convex hull of a point ("ConvexHull");

[0181] - searching the convexity defects of a contour ("ConvexityDefects");

[0182] - comparing shapes ("MatchShapes");

[0183] - checking if a point is on a contour ("PointPolygonTest");

[0184] - detecting Harris corner points ("CornerHarris");

[0185] - searching for the smallest eigenvalue of the gradient matrix to detect a corner ("CornerMinEigenVal");

[0186] - applying a Hough transform to find circles in the grayscale image ("HoughCircles");

[0187] - "active contour modeling" (drawing the contour of an object from a potentially "noisy" 2D image);

[0188] - computing a force field in a part of the image, called GVF ("Gradient Vector Flow");

[0189] - Cascade Classification.

[0190] This processing can also be performed by a neural network trained for this purpose. This neural network is preferably chosen from among networks dedicated to the localization and detection of objects in images, such as the following "object detection networks":

[0191] - R-CNN (2013)

[0192] - SSD (Single Shot MultiBox Detector: object detection network), Faster (faster region-based convolutional network method: object detection network)

[0193] - Faster R-CNN (2015)

[0194] - SSD (2015).

[0195] The determination of the tooth profile can be optimized by the teachings of PCT / EP2015 / 074900 or FR1901755.

[0196] Step b) advantageously makes it possible to obtain an image that can be processed very easily by a neural network.

[0197] In step c), the original image from step b) is presented at the input of the simulation neural network. The simulation neural network then modifies the original image to simulate the effect of a dental event on the dental arch depicted therein.

[0198] Advantageously, only the original image needs to be presented at the input of the simulation neural network. There is no need to present three-dimensional information, such as a three-dimensional digital model, to the simulation neural network. Furthermore, the entire original image can be submitted to the simulation neural network. There is no need to isolate elements of this image. The simulation neural network generates a new image from the original image. In other words, no part of the original image is simply copied. Any part of the original image is susceptible to being modified when the new image is generated.

[0199] Preferably, the analog neural network has been previously trained by providing it with:

[0200] - at input, a set of "input" images each depicting a respective "input" real image depicting a respective dental arch, for example a set of photographs of the dental arch or hyper-realistic views of a three-dimensional model of the dental arch, and

[0201] - at output, a set of "output" images each associated with an input image and depicting a respective "output" realistic image depicting the dental arch depicted in the associated "input" hyper-realistic image but after the dental event has occurred, the output realistic image being for example a photograph of the dental arch or a hyper-realistic view of a three-dimensional model of the dental arch.

[0202] By "realistic" is meant that the depicted dental arch resembles the dental arch that a person can observe with the naked eye in reality.

[0203] By this training, the analog neural network learns to transform the input images into the output images, thereby learning to simulate the dental event.

[0204] Preferably, the distinctive information depicted in the input images and in the output images is the contour.

[0205] The analog neural network used can in particular be trained according to steps 1) to 4).

[0206] In step d) (optional but preferred), the modified image obtained at the end of step c) is modified so that it is hyper-realistic, that is to say so that it looks like a photograph.

[0207] All methods of making the modified image hyper-realistic are possible.

[0208] Some image texture techniques are described in the article "Unpaired image-to-image translation using cycle-consistent adversarial networks" by Zhu, Jun-Yan et al.

[0209] Preferably, a so-called "texture" neural network is used, the "texture" neural network being trained to make images depicting a contour hyper-realistic, for example the modified image, and preferably comprising the following steps d0) to d3).

[0210] For the neural network implemented in the training method according to the application, the texture neural network can be chosen from the list of neural networks presented above. The texture neural network can in particular be chosen from the networks dedicated to image generation, for example:

[0211] Cycle-consistent adversarial networks (2017)

[0212] Enhanced CycleGAN (2018)

[0213] Deep photo style transfer (2017)

[0214] Fast photo style (2018)

[0215] pix2pix (2017)

[0216] Style-based GAN generator architecture (2018)

[0217] SRGAN (2018).

[0218] However, the texture neural network is not limited to the above list.

[0219] In step dO), for each photo called "texture photo", a dental arch is depicted from a set comprising more than 1000 texture photos,

[0220] The texture photos are processed, preferably as in step b), to obtain an image depicting the contours called "texture image";

[0221] In step d1), a so-called "texture" learning library consisting of records called "texture records" is created, each texture record comprising:

[0222] - a texture photo depicting a dental arch, and

[0223] - a texture image obtained by processing said texture photos of step dO) in a step dO) preceding step d1);

[0224] In step d2), a texture neural network is trained by the texture learning library. Such training is well known to the person skilled in the art.

[0225] Such training generally comprises providing all said texture images at the input of the texture neural network and all said texture photos at the output of the texture neural network, while informing the texture neural network of the texture photo corresponding to each texture image.

[0226] By such training, the texture neural network learns to convert an image depicting the contours of a dental arch, such as a modified image, into a hyper-realistic image.

[0227] In step d3), the modified image is submitted to the trained texture neural network. The texture neural network converts the modified image into a hyper-realistic image.

[0228] Figure 3 One example of a photo taken in step a) (left) and the modified image obtained at the end of step d) (right) is shown. The effect of the dental event on certain tooth positions can be observed.

[0229] In step e), the modified image, which is preferably hyper-realistic, can in particular be presented to the beneficiary, preferably on a screen, preferably on the screen of a smartphone, tablet, portable computer or virtual reality headset. The screen can also be the glass of a mirror.

[0230] The generation method and the training method are implemented by computer. In general, the computer comprises in particular a processor, a memory, a human-machine interface, generally comprising a screen, a module for communicating by internet, Wi-Fi, or telephone network. The software configured to implement the method of the application under consideration is loaded into the memory of the computer.

[0231] The computer can also be connected to a printer.

[0232] In one embodiment, the human-machine interface makes it possible to communicate with the computer:

[0233] - the dental event comprising the time elapsed between the time of acquisition and the simulated time, and / or

[0234] - the treatment parameters applied to the beneficiary; and / or

[0235] - the parameters of the orthodontic appliance worn by the beneficiary, for example related to the category and / or the configuration of the orthodontic appliance; and / or

[0236] - the functional parameters of the beneficiary, in particular the neurological functional parameters, for example the ease of breathing, swallowing or mouth closure; and / or

[0237] - the anatomical parameters of the beneficiary other than the positioning parameters of their teeth, for example the arrangement and / or the structure of the bone tissue (in particular the jaws) and / or the alveolar dental tissue and / or the soft tissue (in particular the gums and / or the frenum and / or the tongue and / or the cheeks); and / or

[0238] - the age or age group and / or the sex of the beneficiary.

[0239] The computer can thus select a simulated neural network trained accordingly.

[0240] For example, for a man aged 30 to 40 with "normal" bone tissue, a simulated neural network trained to simulate the movement of the teeth with an orthodontic appliance with archwire and brackets can be selected.

[0241] This method makes it possible in particular to simulate the effects of various orthodontic treatments, thereby facilitating the choice of the treatment best suited to the needs or wishes of the beneficiary.

[0242] Preferably, the human-machine interface comprises a screen with a field for inputting the simulated time.

[0243] In one embodiment, the human-machine interface makes it possible to display or not display on the screen the orthodontic appliance worn by the beneficiary.

[0244] Examples

[0245] Simulation of past or future dental conditions

[0246] In one embodiment, the beneficiary takes the original image, for example with their smartphone (step a)), and the steps b) to e) are performed by a computer integrated into the smartphone or with which the smartphone is able to communicate. The modified image is preferably presented on the screen of the smartphone.

[0247] Preferably, the computer is integrated into the smartphone, thereby allowing the beneficiary to implement the generation method according to the application completely autonomously.

[0248] The beneficiary can thus very easily request a simulation of the dental situation on the basis of one or preferably several photographs of their teeth, even without having to move.

[0249] In particular, it is possible to simulate the dental situation at a past or future simulated time. The simulated time can for example be more than 1 day, more than 10 days or more than 100 days before or after the time of acquisition of the original image.

[0250] In one particular case, the dental event is the passage of time in the case of a treatment, for example orthodontic treatment during which the beneficiary wears an orthodontic appliance.

[0251] Preferably, the method comprises step d). The modified image presented is then displayed as a photograph that would have been taken at the simulated time. The modified image can be presented to the beneficiary in order to show them their future or past dental situation, thereby motivating them to follow the treatment.

[0252] In one particular case, the dental event is the passage of time in the case of orthodontic treatment during which the beneficiary does not respect the medical recommendations, for example does not wear their orthodontic appliance correctly. The presentation of the realistic modified image thus makes it possible to visualize the effects of incorrect respect.

[0253] The generation method according to the application can in particular be used to simulate:

[0254] - the impact of one or more orthodontic appliances on the teeth of the beneficiary, in particular in order to select the orthodontic appliance that is most suitable for the beneficiary;

[0255] - the impact of temporarily or finally stopping a treatment that is in progress;

[0256] - the impact of the application of instructions;

[0257] - the impact of a therapeutic or non-therapeutic medical treatment.

[0258] In particular, the method can be used, in particular for educational purposes, to visualize the impact of changing the brushing frequency and / or the brushing duration and / or the brushing technique, or the impact of delaying the replacement of orthodontic brackets and / or the impact of delaying an appointment with a dental professional.

[0259] Dynamic simulation

[0260] In one embodiment, steps a) to e) are repeated in a loop, wherein the original image is continuously acquired. Preferably, each loop lasts less than 5 seconds, preferably less than 2 seconds, preferably less than 1 second. In step a), a camera is preferably used.

[0261] For example, the beneficiary can see the modified image on a mirror equipped with a camera in which they look at themselves. Preferably, the modified image is presented in such a way as to be aligned with the image reflected by the mirror, that is to say that the beneficiary sees the modified image as if the image was obtained by reflection. Thus, the beneficiary has the impression of observing themselves in a simulated time.

[0262] Preferably, between two loops from steps a) to e), the simulated time can be modified, for example by modifying the position of a cursor shown on the screen. Preferably, the screen is a touch screen and the simulated time is modified by the interaction of a finger on said screen, preferably by a swipe.

[0263] Events with transient effects

[0264] In a particular case, the dental event has an immediate effect that it is desirable to visualize.

[0265] For example, the dental event is the fitting of an orthodontic appliance. Thus, the method can integrate a depiction of the orthodontic appliance into the original image, or modify the orthodontic appliance depicted in the original image, or remove the orthodontic appliance depicted in the original image, without modifying the position of the teeth.

[0266] The simulated neural network is trained to create a modified image from the original image provided to it. This method is thus quite different from a method that for example adds an element, e.g. a depiction of an existing orthodontic appliance, to an image. In fact, in order to integrate a depiction of an orthodontic appliance into an original image, the simulated neural network creates that depiction. The depiction is thus not a reproduction of a real orthodontic appliance or a 3D model of a real orthodontic appliance, but is artificially generated by the simulated neural network at the same time as the rest of the image.

[0267] Surprisingly, the depiction of the orthodontic appliance is highly realistic and allows a good simulation of the beneficiary. In particular, training the simulated neural network teaches it to depict an orthodontic appliance in the context of an original image and with the corresponding contrast, sharpness, shading and reflections. The simulation is thus much more realistic than simply adding a pre-existing depiction of an orthodontic appliance to an image depicting a dental arch.

[0268] The modification of the original image by the neural network can result in modifications to areas of the original image other than the area in which the orthodontic appliance is depicted. These differences can be detrimental if the modified image is used to intervene on the teeth, for example to guide a dentist during a drilling operation, but are not detrimental when the modified image is intended to be presented to the beneficiary. The performance of the neural network can even be such that it is almost impossible to detect the differences outside the area in which the orthodontic appliance is depicted.

[0269] As will now be clear, the invention allows beneficiaries to simulate the impact of a dental event on the teeth of their dental arch without them having to scan that dental arch. In a preferred embodiment, anyone equipped with a smartphone can advantageously perform such a simulation.

[0270] Of course, the invention is not limited to the embodiments described and illustrated above.

[0271] In particular, the beneficiaries are not limited to humans. The method according to the invention can be used for other animals.

[0272] Furthermore, the training method does not necessarily comprise steps 1) to 4), but steps 1) to 4) are preferred.

[0273] The neural network, in particular the simulated neural network, can for example be trained by implementing a method comprising the following steps for a set of individuals preferably comprising more than 100, preferably more than 1000, preferably more than 10000 individuals:

[0274] 1') obtaining a photograph of the dental arch of an individual and processing said photograph to obtain a first image depicting the contours of the dental arch depicted in the photograph;

[0275] 2') generating a digital three-dimensional model depicting said dental arch after the dental event has taken place;

[0276] 3') a view of the model outlining the photograph is taken, and if the view does not depict the profile, the view is processed so that it depicts the profile of the depicted dental arch; then, all the first images and views are taken for all the individuals:

[0277] 4') the photograph and the view are introduced at the input and output of a neural network, respectively, to train the neural network to convert an input view describing an analyzed dental arch into an output view describing the analyzed dental arch after application of the dental event.

[0278] In step 2'), the generation of the model depicting the dental arch after the dental event has occurred can be the result of:

[0279] - the generation of the model before the dental event has occurred, for example at substantially the same time as the photograph is taken (step 1'), then

[0280] - the simulation of the dental event on the model as described in step 2).

[0281] In step 3'), the view should outline the photograph. Preferably, the observation conditions of the model that correspond best ("best fit") to the photograph taking conditions are sought. In other words, what is sought is the position, orientation and calibration of the virtual taking device that allows the model to be observed with a view in which the depiction of the teeth that did not move during the dental event can overlap in a manner aligned with the depiction of said teeth in the photograph.

Claims

1. A method for generating an image of a dental arch of a beneficiary, called "modified image", said method comprising the following successive steps: a) at an acquisition time, acquiring a photograph, called "original image", depicting said dental arch; b) processing said original image so that said original image depicts discriminant information; c) submitting said original image to an input of a neural network, called "simulation neural network", trained on the basis of said original image to simulate the effect of a dental event on said original image, thereby obtaining said modified image, wherein said image being a two-dimensional image, wherein said simulation neural network is trained by a training method comprising, for each of a plurality of digital three-dimensional models, called "historical models", of "historical" dental arches, the following successive steps: 1) acquiring a first view of said historical model under a first observation condition, processing said first view so that it depicts discriminant information, called "first discriminant information", of said historical dental arch, if said first view does not depict said first discriminant information; 2) modifying said historical model to reproduce the effect of said dental event on said historical dental arch; 3) acquiring a second view of said historical model under a second observation condition identical to said first observation condition, processing said second view so that it depicts discriminant information, called "second discriminant information", of said historical dental arch, if said second view does not depict said second discriminant information; then, acquiring all first and second views for all historical models: 4) introducing said first and second views at said input and output of said simulation neural network, respectively, to train said simulation neural network to convert an input view depicting an analyzed dental arch into an output view depicting said analyzed dental arch after application of said dental event.

2. The method of claim 1, further comprising: after step c), a step d) of processing said modified image so that said modified image is photorealistic, and / or a step e) as follows: - presenting said modified image; and / or - selecting an orthodontic appliance on the basis of said modified image.

3. The method of claim 1, wherein, said dental event is selected from the group consisting of time elapsing in orthodontic or non-orthodontic treatment, time elapsing in pathology or bruxism, fitting of a dental component on said dental arch, time elapsing without treatment, and combinations of these dental events.

4. The method of claim 1, wherein, said discriminant information is selected from the group consisting of contour information, color information, density information, distance information, information about reflections, and combinations of these information.

5. The method of claim 4, wherein, the depiction of said discriminant information is a contour of said dental arch.

6. The method of claim 1, wherein, said discriminant information is selected from the group consisting of luminance information and saturation information.

7. The method of claim 1, wherein, said simulation neural network is trained by a training method comprising, for each of a plurality of digital three-dimensional models, called "historical models", of "historical" dental arches, the following successive steps: 1) acquiring a first view of said historical model under a first observation condition, processing said first view so that it depicts a contour, called "first contour", of said historical dental arch, if said first view does not depict said first contour; 2) modifying said historical model to reproduce the effect of said dental event on said historical dental arch; 3) acquiring a second view of said historical model under a second observation condition identical to said first observation condition, processing said second view so that it depicts a contour, called "second contour", of said historical dental arch, if said second view does not depict said second contour; then, acquiring all first and second views for all historical models: 4) introducing said first and second views at said input and output of said simulation neural network, respectively, to train said simulation neural network to convert an input view depicting an analyzed dental arch into an output view depicting said analyzed dental arch after application of said dental event. 2) modifying said historical model to reproduce the effect of said dental event on said historical dental arch; 3) acquiring a second view of said historical model under a second observation condition identical to said first observation condition, and if said second view does not depict a contour of said historical dental arch called "second contour", processing said second view so that it depicts said second contour; Then, all first views and second views are acquired for all historical models: 4) introducing said first views and said second views at said inputs and said outputs of said simulated neural network, respectively, to train said simulated neural network to convert an input view depicting an analyzed dental arch into an output view of said analyzed dental arch after application of said dental event.

8. The method of claim 1, wherein, - in step 1), more than 10 first views are acquired under each different first observation condition, and then, - in step 3), for each first view acquired under a first observation condition, a second view is acquired under a second observation condition identical to said first observation condition, and a historical record is created using said first view and said second view.

9. The method of claim 1, wherein, In step a), the photos are acquired by extra-oral way using a mobile phone while the beneficiary wears a dental retractor.

10. The method of claim 1, comprising a step d) of processing said modified image so that it is hyper-realistic, step d) comprising the following steps: dO) for each photo called "texture photo", depicting a dental arch from a set comprising more than 1000 texture photos, processing said texture photo to obtain an image called "texture image" depicting a contour; dl) creating a so-called "texture" learning base consisting of records called "texture records", each texture record comprising a texture photo and a texture image obtained by processing said texture photo in step dO); d2) training a neural network called "texture neural network" by said texture learning base; d3) submitting said modified image to the trained texture neural network to obtain a hyper-realistic modified image.

11. The method of claim 10, wherein, In step dO), said texture photo is processed in step b).

12. The method of claim 2, wherein, said dental event is a time lapse from said acquisition time to a simulated time more than 1 day before or after said acquisition time, and wherein in step e), said modified image is presented to said beneficiary in order to show said beneficiary a determined dental condition at said simulated time.

13. The method of claim 1, wherein, Before step c), said dental event is determined by specifying a simulated time and / or treatment parameters applied to said beneficiary and / or parameters of an orthodontic appliance worn by said beneficiary and / or functional parameters of said beneficiary and / or anatomical parameters of said beneficiary other than positioning parameters of the teeth of said beneficiary and / or age or age range and / or gender of said beneficiary.

14. The method of claim 2, wherein, Steps a) to e) are repeated in a loop, with successive acquisition of original images, each loop lasting less than 5 seconds.

15. The method of claim 1, wherein, In step c), only said original image is submitted to the input of said simulated neural network.

Images